rpmguard -- what's next?

[James Laska]

Greetings folks,

Please don't take this post to mean, "drop what you are doing and let's
focus on rpmguard".  I believe our current focus is correctly centered
around automating the package update acceptance test plan [1].  Kamil
and I have discussed some long-term plans for rpmguard, but I also
wanted to bring the discussion to the list for group consumption/review.

Quick background, rpmguard does a fair amount of comparative tests
against old and new set of packages.  Heck, the tests are event
documented [2] (nicely done Kamil!).  Today, Seth Vidal directed me to
bug#564018 and asked whether rpmguard had support for testing whether a
package has a *large* change in it's %provides.  In the bug, the erlang
package went from 50 provides to 22157 provides.  This package did go
through rpmguard and warnings were generated (for a good laugh, see
[3]).  So kudos are in order for capturing the change in an automated
manner.

What interested me about this exercise was how a new test might be added
to rpmguard.  The power of rpmguard is in abstracting the details of
locating the previous packages and providing a common framework for
comparisons against the two package sets.  Presently, the tests exist in
the main driver.  

How do we want to extend this in the future make it easier/faster to
support new comparison tests?  Should tests exist in stand-alone python
scripts, where they all accept a common set of arguments?  It seems like
rpmlint is structured this way [4], is this good/bad ... does this make
user-contributed comparative tests easier?

Something Kamil proposed a while back, should rpmguard be a stand-alone
tool (much like rpmlint)?  Instead of being bundled inside autoqa,
anyone could do: `yum install rpmguard`

Apologies for the ranty nature of the email, I'm still thinking this
through.  My main objective is to get a sense where rpmguard should go.
Hopefully these thoughts can lead to a wiki or TRAC roadmap, and
something we could implement once our immediate objects are behind us.

Thanks,
James

[1]
https://fedoraproject.org/wiki/QA:Package_Update_Acceptance_Test_Plan
[2] https://fedorahosted.org/autoqa/wiki/RpmguardChecks
[3]
https://fedorahosted.org/pipermail/autoqa-results/2010-April/017795.html
[4] http://rpmlint.zarb.org/cgi-bin/trac.cgi/browser/trunk

[Seth Vidal]

On Tue, 11 May 2010, James Laska wrote:

>  What interested me about this exercise was how a new test might be
> added
>  to rpmguard.  The power of rpmguard is in abstracting the details of
>  locating the previous packages and providing a common framework for
>  comparisons against the two package sets.  Presently, the tests exist in
>  the main driver.
>
>  How do we want to extend this in the future make it easier/faster to
>  support new comparison tests?  Should tests exist in stand-alone python
>  scripts, where they all accept a common set of arguments?  It seems like
>  rpmlint is structured this way [4], is this good/bad ... does this make
>  user-contributed comparative tests easier?
>
>  Something Kamil proposed a while back, should rpmguard be a stand-alone
>  tool (much like rpmlint)?  Instead of being bundled inside autoqa,
>  anyone could do: `yum install rpmguard`
>
>  Apologies for the ranty nature of the email, I'm still thinking this
>  through.  My main objective is to get a sense where rpmguard should go.
>  Hopefully these thoughts can lead to a wiki or TRAC roadmap, and
>  something we could implement once our immediate objects are behind us.
>  

okay - so this might be my own imagination but I thought there was a goal 
of autoqa to do the following:

1. show issues with pkgs
2. show signficant/dangerous CHANGES to pkgs
3. provide a way for a packager/the-powers-that-be to stop a package if it 
doesn't get at least N score for a pkg


That last one might be my own wishful-thinking.

But I was sorta thinking that rpmguard (or something else above it) could 
have a dir of python scripts - like yum plugins.

Each one of the [enabled] scripts would be passed a 'conduit' that maybe 
looks like:

conduit.get_old_package_file()
conduit.get_new_package_file()
conduit.get_old_package_hdr()
conduit.get_new_package_hdr()
#and this is just my own wishfulness)
conduit.get_old_package_object()#using yum's package objects
conduit.get_new_package_object()#using yum's package objects


Then the scripts could do whatever they need to do and maybe feedback to 
the code a result object of some kind:

for example:
test_result.code = RPMGUARD_PASS
test_result.output = "a lot of strings here"
test_result.score = 23



We then collect and compile those to pitch back to the 
user/builder/resultsdb.

Does that make sense?

-sv

[James Laska]

On Tue, 2010-05-11 at 17:16 -0400, Seth Vidal wrote:
>  
>  On Tue, 11 May 2010, James Laska wrote:
>  
>  > What interested me about this exercise was how a new test might be added
>  > to rpmguard.  The power of rpmguard is in abstracting the details of
>  > locating the previous packages and providing a common framework for
>  > comparisons against the two package sets.  Presently, the tests exist in
>  > the main driver.
>  >
>  > How do we want to extend this in the future make it easier/faster to
>  > support new comparison tests?  Should tests exist in stand-alone python
>  > scripts, where they all accept a common set of arguments?  It seems like
>  > rpmlint is structured this way [4], is this good/bad ... does this make
>  > user-contributed comparative tests easier?
>  >
>  > Something Kamil proposed a while back, should rpmguard be a stand-alone
>  > tool (much like rpmlint)?  Instead of being bundled inside autoqa,
>  > anyone could do: `yum install rpmguard`
>  >
>  > Apologies for the ranty nature of the email, I'm still thinking this
>  > through.  My main objective is to get a sense where rpmguard should go.
>  > Hopefully these thoughts can lead to a wiki or TRAC roadmap, and
>  > something we could implement once our immediate objects are behind us.
>  >
>  
>  
>  okay - so this might be my own imagination but I thought there was a goal 
>  of autoqa to do the following:
>  
>  1. show issues with pkgs
>  2. show signficant/dangerous CHANGES to pkgs
>  3. provide a way for a packager/the-powers-that-be to stop a package if it 
>  doesn't get at least N score for a pkg 
Yeah, in a nutshell.  AutoQA allows for the above to occur.  Perhaps the
steps above might be more the focus for specific tests like rpmlint,
package_sanity, depcheck and rpmguard.  Same result in the end.

>  That last one might be my own wishful-thinking.
>  
>  But I was sorta thinking that rpmguard (or something else above it) could 
>  have a dir of python scripts - like yum plugins. 
Love the plugin idea!  It's always fun to think of code in this regard,
but I always forget that yum plugins didn't just appear out of thin air.
It took time for the need to develop and the API to mature.

>  Each one of the [enabled] scripts would be passed a 'conduit'
> that maybe 
>  looks like: 
What's a conduit?  Is that a way for each test to gain access to the old
and new packages etc..., or something else?

>  conduit.get_old_package_file()
>  conduit.get_new_package_file()
>  conduit.get_old_package_hdr()
>  conduit.get_new_package_hdr()
>  #and this is just my own wishfulness)
>  conduit.get_old_package_object()#using yum's package objects
>  conduit.get_new_package_object()#using yum's package objects
>  
>  
>  Then the scripts could do whatever they need to do and maybe feedback to 
>  the code a result object of some kind:
>  
>  for example:
>  test_result.code = RPMGUARD_PASS
>  test_result.output = "a lot of strings here"
>  test_result.score = 23 
How were you thinking score would be used here?

>  We then collect and compile those to pitch back to the 
>  user/builder/resultsdb.
>  
>  Does that make sense? 
Yeah, certainly does.  At this high level, it seems like it would
provide a nice structure to allow for future test additions and
dynamically enabling or disabling any subset of tests.

Thanks,
James

[Seth Vidal]

On Wed, 12 May 2010, James Laska wrote:
> > okay - so this might be my own imagination but I thought there
> was a goal
> > of autoqa to do the following:
> >
> > 1. show issues with pkgs
> > 2. show signficant/dangerous CHANGES to pkgs
> > 3. provide a way for a packager/the-powers-that-be to stop a package if it
> > doesn't get at least N score for a pkg
>
>  Yeah, in a nutshell.  AutoQA allows for the above to occur.  Perhaps the
>  steps above might be more the focus for specific tests like rpmlint,
>  package_sanity, depcheck and rpmguard.  Same result in the end. 
right - a series of tests.

So maybe the solution is to make it simpler to create more package tests?

I'm not sure where the abstraction needs to be really. If we do it at 
rpmguard then you get the results back from rpmguard. If we do it at the 
rpmlint layer then it comes back from those tests. It doesn't REALLY 
matter where it happens as long as it happens.


>  Love the plugin idea!  It's always fun to think of code in this
> regard,
>  but I always forget that yum plugins didn't just appear out of thin air.
>  It took time for the need to develop and the API to mature. 
sorta - Menno Smits did some great work with that infrastructure.

>  What's a conduit?  Is that a way for each test to gain access to the old
>  and new packages etc..., or something else? 
It's the thing that gives a plugin its access to the info in the main 
code.


> > conduit.get_old_package_file()
> > conduit.get_new_package_file()
> > conduit.get_old_package_hdr()
> > conduit.get_new_package_hdr()
> > #and this is just my own wishfulness)
> > conduit.get_old_package_object()#using yum's package objects
> > conduit.get_new_package_object()#using yum's package objects
> >
> >
> > Then the scripts could do whatever they need to do and maybe feedback to
> > the code a result object of some kind:
> >
> > for example:
> > test_result.code = RPMGUARD_PASS
> > test_result.output = "a lot of strings here"
> > test_result.score = 23
>
>  How were you thinking score would be used here? 
Generally I was thinking it could be:

  you get positive points for each thing you pass and zero or negative 
points for things you fail. Then we could set an arbitrary number that 
says if you don't get at least this many points then you don't pass over 
all.

Or maybe just  as a simple - the last pkg got a score of 4000, this one 
has a score of 400. We ran the same number of tests, what the hell 
happened?


>  Yeah, certainly does.  At this high level, it seems like it would
>  provide a nice structure to allow for future test additions and
>  dynamically enabling or disabling any subset of tests. 
right - so doing it like the yum or mock plugins is relatively easy 
enough to implement in rpmguard. It takes only a little bit of 
infrastructure. I figure just a directory of these files, each one 
referenced by name in some config for rpmguard to enable/disable them?

Since the test is just a one shot we don't even really need multiple hooks 
at this point - just the one call out and expect back data in a standard 
way.

anyone object to mandating that all tests be in python?

-sv

[James Laska]

On Wed, 2010-05-12 at 15:24 -0400, Seth Vidal wrote:
>  
>  On Wed, 12 May 2010, James Laska wrote:
>  >> okay - so this might be my own imagination but I thought there was a goal
>  >> of autoqa to do the following:
>  >>
>  >> 1. show issues with pkgs
>  >> 2. show signficant/dangerous CHANGES to pkgs
>  >> 3. provide a way for a packager/the-powers-that-be to stop a package if it
>  >> doesn't get at least N score for a pkg
>  >
>  > Yeah, in a nutshell.  AutoQA allows for the above to occur.  Perhaps the
>  > steps above might be more the focus for specific tests like rpmlint,
>  > package_sanity, depcheck and rpmguard.  Same result in the end.
>  
>  right - a series of tests.
>  
>  So maybe the solution is to make it simpler to create more package tests?
>  
>  I'm not sure where the abstraction needs to be really. If we do it at 
>  rpmguard then you get the results back from rpmguard. If we do it at the 
>  rpmlint layer then it comes back from those tests. It doesn't REALLY 
>  matter where it happens as long as it happens. 
You're right, the specific short-term goal is to implement the package
update acceptance criteria.  My initial thoughts here were around what
might be needed to adjust rpmguard to allow for easier comparative test
contributions in the future.

>  > Love the plugin idea!  It's always fun to think of code in
> this regard,
>  > but I always forget that yum plugins didn't just appear out of thin air.
>  > It took time for the need to develop and the API to mature.
>  
>  sorta - Menno Smits did some great work with that infrastructure.
>  
>  >
>  > What's a conduit?  Is that a way for each test to gain access to the old
>  > and new packages etc..., or something else?
>  
>  It's the thing that gives a plugin its access to the info in the main 
>  code. 
Gotcha, thx.

>  >> conduit.get_old_package_file()
>  >> conduit.get_new_package_file()
>  >> conduit.get_old_package_hdr()
>  >> conduit.get_new_package_hdr()
>  >> #and this is just my own wishfulness)
>  >> conduit.get_old_package_object()#using yum's package objects
>  >> conduit.get_new_package_object()#using yum's package objects
>  >>
>  >>
>  >> Then the scripts could do whatever they need to do and maybe feedback to
>  >> the code a result object of some kind:
>  >>
>  >> for example:
>  >> test_result.code = RPMGUARD_PASS
>  >> test_result.output = "a lot of strings here"
>  >> test_result.score = 23
>  >
>  > How were you thinking score would be used here?
>  
>  Generally I was thinking it could be:
>  
>    you get positive points for each thing you pass and zero or negative 
>  points for things you fail. Then we could set an arbitrary number that 
>  says if you don't get at least this many points then you don't pass over 
>  all.
>  
>  Or maybe just  as a simple - the last pkg got a score of 4000, this one 
>  has a score of 400. We ran the same number of tests, what the hell 
>  happened? 
Interesting concept, I can't say that's good or bad.  I like the
approach we take now in the package update acceptance test plan in
stating things as fairly binary (pass/fail).  Mainly because it is
easier to explain, understand and code to.  But I'm open to the concept
of [some] tests determining pass/fail by ensuring a score is within a
specified range.

Is it crazy to have something like the existing "provides-added" test
Kamil wrote [1] to continue providing an ADVISORY result of any %
provides added.  The results of that test would be available for other
tests to potentially review (through the conduit).  For example, we
could potentially have future tests that say ...
      * the kernel package should never %provide kernel-drm (just making
        this up) -- so the new "kernel-no-provides-drm" test would
        examine the provides previously generated and fail if
        "kernel-drm" is now provided
      * or, another test could do some score checking based on results
        of other tests -- so a new test "provides-score" would provide
        an INTROSPECTION result if there is a 30% increase/decrease

Perhaps a future refinement.

[1] https://fedorahosted.org/autoqa/wiki/RpmguardChecks#provision-added

>  > Yeah, certainly does.  At this high level, it seems like it
> would
>  > provide a nice structure to allow for future test additions and
>  > dynamically enabling or disabling any subset of tests.
>  
>  right - so doing it like the yum or mock plugins is relatively easy 
>  enough to implement in rpmguard. It takes only a little bit of 
>  infrastructure. I figure just a directory of these files, each one 
>  referenced by name in some config for rpmguard to enable/disable them?
>  
>  Since the test is just a one shot we don't even really need multiple hooks 
>  at this point - just the one call out and expect back data in a standard 
>  way.
>  
>  anyone object to mandating that all tests be in python? 
That seems appropriate.  The existing rpmguard tests are all in python.
Kamil is the rpmguard author, so I value his opinion too.

Thanks,
James