Updates Criteria Summary/Brainstorming

[Toshio Kuratomi]

On Sat, Nov 20, 2010 at 03:35:43PM -0700, Kevin Fenzi wrote:
> ok, I dug through the devel list for the last month or two and wrote
> down all the various ideas folks have come up with to change/improve
> things. 
> 
> Here (in no particular order) are the ideas and some notes from me on
> how we could enable them. Please feel free to add new (actual/concrete
> ideas or notes): 
> 
> * Just drop all the requirements/go back to before we had any updates
>   criteria. 
> 
> * Change FN-1 to just security and major bugfix
> 
> This may be hard to enforce or figure out if something is a major bugfix. 
> 
> * allow packages with a %check section to go direct to stable
> 
> Bodhi would have to have a list or some way to note these packages,
> it would also need to change as they were added/removed. Perhaps it could
> just be an AutoQA +1 for having a check section? 
> On the con side, some checks may be simple and may not note things 
> that are fedora only issues. 
> 
> * setup a remote test env that people could use to test things.
> 
> This would be lovely with some kind of cloud setup. Have a set of base kickstart 
> files and a package/tester could request an instance, install the update, test, and then
> the instance would be removed. We would need a timelimit of some kind, 
> and not sure there is any HW in infrastructure for this, but it would sure be cool. ;) 
> Perhaps working with the cloud sig we could use EC2 instances for this? 
> 
> * require testing only for packages where people have signed up to be testers
> 
> Packages without 'official' testers could bypass testing or have some lower karma
> requirement. We would need for this a list of packages that have had people sign 
> up to test. 
> 
> * Ask maintainers to provide test cases / test cases in wiki for each package?
> 
> Test cases are not easy to make, many maintainers won't can't do so, but 
> it would be lovely to have even a base checklist of things that should work 
> in the package everytime. 
> 
> * have a way to get interested testers notified on bodhi updates for packages
> they care about.
> 
> We would need to add some kind of tester list to pkgdb, and bodhi would need to be
> able to get this to mail them when a update changed state. 
> We may not get many people signing up for some packages, but this might be a 
> good way to know what packages we have testers for and get them more involved
> in testing. Ideally it could mail them on update submission at least. 
> 
> * reduced karma requirement on other releases when one has gone stable
> 
> Bodhi would need to note when a update went to stable if the exact same
> version (with dist tag differences) was in testing for other releases.
> It could then allow less karma to go stable, or add +1 from the other
> update going stable. 
> 
> Other concrete ideas?
> 

Security update ideas:

* Security updates may push directly to stable -- this would depend on
  our weighing of costs:  a security update may be broken.  OTOH, if we
  don't get security updates out fast, is that a worse danger to our users?
  I think I'd like to try one of the other options below first but those
  require a bit more work/coordination so we need to think of the cost to
  implement as well.

* Ask the QA team to commit to testing security updates.  The idea is that
  updates flagged security have a dedicated pool of testers that will check
  them and get them out ASAP.

* Security updates have a small timeout period -- there's two questions
  here: Is a week (as is currently the case for non-critpath) too long?
  Can wehave a timeout even for critpath packages so a security update
  doesn't get held up forever?

Critpath ideas:

* critpath package timeout -- let's have a timeout period even for critpath
  packages.  Perhaps this could be longer than for non-critpath.  The big
  issue that people have observed with depending on timeouts, though, is
  that pushing new updates in the meantime resets the time that a package
  needs to wait to get to stable.  Could bodhi be changed to let multiple
  packages be in the testing repository at one time and only obsolete them
  when a newer package enters the stable repo?  That would allow longer
  timeout periods to make more sense.

Lack of manpower ideas:

* Allow anonymous karma to count.  Anonymous karma would allow more people
  who report bugs in bugzilla to add karma in bodhi without having to get
  a second account in the Fedora Account System.  For critpath packages,
  we're already mandating that a proventester must give karma so we're
  making sure that someone with an account is giving feedback there.

* Do we believe that there's value in silent testing (knowing that people
  have a package installed from testing but have not submitted karma either
  way?)  If so, create a tool that ties into yumdb and if the user opts in,
  submits that data to us.  Then add karma based upon that data.

Variation on an option you recorded:
* Drop testing requirement until AutoQA can give karma.  Rework based on
  what tests auto qacan perform.

-Toshio
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 198 bytes
Desc: not available
Url : http://lists.fedoraproject.org/pipermail/devel/attachments/20101121/5628e92f/attachment.bin