updates improvements/changes ideas

[James Laska]

On Mon, 2010-11-29 at 09:56 -0700, Kevin Fenzi wrote:
> Greetings. 
> 
> On the devel list recently there was a long thread discussing how we
> could improve the current https://fedoraproject.org/wiki/Updates_Policy
> 
> I gathered up ALL the ideas people put forth that were concrete in a
> list for fesco to look into. I'd love feedback from QA/Testers as well. 
> 
> If you have additional (concrete) ideas to add, please feel free. 
> 
> Note that these are not my ideas. ;) 

Thanks for sending these along Kevin.

> The list I have so far: 
> 
> General: 
> 
> * Just drop all the requirements/go back to before we had any updates
>   criteria. 

Hmm, certainly an idea.  I feel like this is definitely a step backward,
not forward.  Has the initial motivation for an updates policy gone away
or changed?  Have we encountered problems that didn't yet exist, or
weren't as painful, when the policy was first enabled?  Are there other
problems we need to focus on resolving (I suspect this is the case)?

> * back off current setup until autoqa is ready, see what we want to do
> after that lands. 

Much the same as above.  In addition, automation doesn't magically solve
problems, it only automates a repeatable process that already exists.
If we haven't documented the steps manually, I wouldn't know where to
get started.  I'd like to see us work out the manual steps required to
test stuff ... then we can incorporate automation where sensible.

> * Change FN-1 to just security and major bugfix. Nothing else allowed. 

I don't have objections, and with repos.fedoraproject.org we have an
avenue for maintainers to provide more disruptive packages to existing
releases.  From a QA standpoint, this would certainly lower the volume
of updates that need QA attention.  But this definitely seems like a
general Fedora strategy/audience topic, than a QA specific one.

> * allow packages with a %check section to go direct to stable.

Interesting, I like the spirit of this idea, but would like to see if we
can incorporate this with autoqa karma plans.  So, perhaps packages with
%check get automated karma?  Just the same as with packages that pass
automated tests ... they'll eventually get positive karma of some form.

> * setup a remote test env that people could use to test things.

I could use more details on this point.  Is this talking about setting
up QA systems hosted in Fedora infrastructure that any tester could
login and use to test updates?

> * require testing only for packages where people have signed up to be
> testers.

Hmm, I like this idea in part as it allows an opt-in for testing for
maintainers.  I'd stand behind this for packages outside critpath, but
for critpath, we need to test them.  We've moved forward with the policy
without a implementation from QA for hosting/contributing test
documentation.  I suspect that may be the cause of the policy push back,
that there isn't enough test feedback.  I believe that's in part because
we haven't provided clear test instructions for people to follow.  It's
a *significant* time investment to test an update that has no documented
test instructions and that you aren't familiar with.

QA plans to address this starting in Fedora 15 by laying some wiki
ground work for documenting test procedures for specific updates.

> * Ask maintainers to provide test cases / test cases in wiki for each package

This is always the case.  Proper testing comes from all levels,
including development.  However, as noted above, Fedora QA needs to
provide some area/place for test documentation to develop/mature.
Currently, we can discuss test guidance and instructions on the test@
mailing list.  This practice works well for tests related to release
verification.  This is a great public forum to knock out the details.
Moving forward, we need to capture those instructions on the wiki ...
and ideally integrate it with the bodhi update request (e.g. "Click here
for test instructions").

> * have a way to get interested testers notified on bodhi updates for packages
> they care about.

Interesting, almost like a watch-list for bodhi updates.  Seems like a
feature worth exploring in more detail.  I use a similar feature with
koji to watch for new builds.

> * reduced karma requirement on other releases when one has gone stable
> * aggregated karma across the releases for the same package version.

I don't have data to indicate how many updates have been released, and
then reverted/obsoleted on only a subset of releases.

> * PK updates-testing integration of some kind. 

Open to any ideas here ... are you thinking about some PK
updates-testing feedback workflow.  Like integrating fedora-easy-karma?
Something else?

> * allow anon karma to count. 

Or maybe it counts, but counts less (.5 karma or something).

> * setup fedora-qa package or group to more easily bring up more testers. 

Certainly not opposed to it, we've been focused on building a larger
community of testers (proventesters).  Over time, I could see that group
breaking out into more specific interest groups.  I'm inclined to listen
to that request more if it consistently came from proventesters noting
it as a obstacle to participation.  Meaning, I'm not aware that the lack
of component-specific test groups is preventing tester participation.

That said, I like the idea of micro-communities developing to focus on
specific use cases or components.

> * Testing is only required for certain packages.  Those packages are the
>   packages where problems have occurred before so fesco or other maintainers
>   affected by the changes deem it necessary to supplement the maintainer's
>   testing with outside help.
> 
>   - Option: supplement this list with critpath packages where the
>     maintainers desire extra testing.  This means that we would no longer be
>     dragging in dependencies immediately... only if updates by the
>     dependency's maintaner to that package are breaking things.
> 
> * updates that only modify the spec could have a lower requirement.
> (ie, to fix a packaging issue, no changes in the upstream software).  

All %obsoletes, %requires, %provides, %files and %patch statements are
only recorded in the .spec file.  Just because they are in the .spec
file doesn't mean they are any less disruptive.  

> Security updates: 
> 
> * allow security updates to go direct to stable

Risky

> * ask QA to commit to testing security updates

We can't commit to testing without guidance or instructions.  Let's
commit to documenting repeatable procedures that testers can follow and
expand upon.

For some security updates, they may have already been functionally
tested upstream.  I think it's reasonable to provide proxy karma linking
to upstream functional tests.  Though, I don't think upstream functional
tests alone can bless a security update.

> * allow timeout for security updates before going to stable. 
> 
> Critpath updates: 
> 
> * allow critpath timeout for going to stable. 

I think this will come back to hurt us.  These things landed in critpath
for a reason.  I'd like to work on increasing tester engagement, before
we loosen the process around critpath packages.

> Non critpath/security: 
> 
> * reduce timeout for non critpath from 7 to 3 days. 
> 
> * change default autokarma to 2 or 1. 

No immediate thoughts on these points.

Thanks,
James

-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 198 bytes
Desc: This is a digitally signed message part
Url : http://lists.fedoraproject.org/pipermail/test/attachments/20101129/49addb65/attachment.bin