updates improvements/changes ideas

[Adam Williamson]

On Mon, 2010-11-29 at 12:40 -0500, James Laska wrote:

> > * Just drop all the requirements/go back to before we had any updates
> >   criteria. 
> 
> Hmm, certainly an idea.  I feel like this is definitely a step backward,
> not forward.  Has the initial motivation for an updates policy gone away
> or changed?  Have we encountered problems that didn't yet exist, or
> weren't as painful, when the policy was first enabled?  Are there other
> problems we need to focus on resolving (I suspect this is the case)?

As I see it, the thing that everyone agrees is problematic is critpath
updates for old releases not getting pushed or taking a very long time
to push. It's also generally agreed that the quality of critpath testing
can be improved by taking some steps we're already looking at
(package-specific test cases).

Things that some people see as problematic are:

* Having to wait a week to push an update if you can't find testing
* Testing being required for packages with automated test suites
* The delay to security updates which is introduced by the testing
requirements

> > * allow packages with a %check section to go direct to stable.
> 
> Interesting, I like the spirit of this idea, but would like to see if we
> can incorporate this with autoqa karma plans.  So, perhaps packages with
> %check get automated karma?  Just the same as with packages that pass
> automated tests ... they'll eventually get positive karma of some form.

Yes, I was going to suggest the same thing. I'd suggest packages with a
%check section should get +1 proventester karma. Of course, that relies
on the automated test suite actually testing the things proventester
testing is meant to cover; do we want to audit the test suites in
question?

> > * setup a remote test env that people could use to test things.
> 
> I could use more details on this point.  Is this talking about setting
> up QA systems hosted in Fedora infrastructure that any tester could
> login and use to test updates?

Yes - it's an idea to make it easier to test older releases, or packages
you don't want to / can't install (or configure) on your active systems.

> > * reduced karma requirement on other releases when one has gone stable
> > * aggregated karma across the releases for the same package version.
> 
> I don't have data to indicate how many updates have been released, and
> then reverted/obsoleted on only a subset of releases.

Yeah, I'd really like to see some data here. I did ask Luke on the
-devel thread, but so far no response. Everyone more or less agrees on
the factors here (on the one hand it's hard to get testing for old
releases, on the other hand it *is* possible for the 'same' update to
work fine on one distro but not another), it's hard to balance these
without hard numbers. I think you can possibly draw a distinction
between different types of updates here too: as I wrote on the -devel
list, I can see the argument for a single leaf package update, but
pushing an update to, say, an entire desktop environment and relying on
testing from another release seems scary.

> > * allow anon karma to count. 
> 
> Or maybe it counts, but counts less (.5 karma or something).

Something else to consider here is to make more people login; I suspect
relatively few people are actually doing testing who don't have a FAS
account, but I think we could make the login link more prominent, and
try harder to get people to log in (have a big scare-step when posting
anonymous feedback which says 'your feedback will not count unless you
log in!' and requires you to re-confirm to submit the feedback
anonymously; a nag screen, basically.

> > Security updates: 
> > 
> > * allow security updates to go direct to stable
> 
> Risky

Right. As was pointed out on -devel, the update which caused us to start
thinking about an update testing process in the first place - the
infamous udev update - was a security update.

> > * ask QA to commit to testing security updates
> 
> We can't commit to testing without guidance or instructions.  Let's
> commit to documenting repeatable procedures that testers can follow and
> expand upon.
> 
> For some security updates, they may have already been functionally
> tested upstream.  I think it's reasonable to provide proxy karma linking
> to upstream functional tests.  Though, I don't think upstream functional
> tests alone can bless a security update.

That seems like it'd be tricky to automate.

> > Non critpath/security: 
> > 
> > * reduce timeout for non critpath from 7 to 3 days. 
> > 
> > * change default autokarma to 2 or 1. 
> 
> No immediate thoughts on these points.

I suggested the default auto-push karma change, though really what
should change is the linking of auto-push and approval. Right now,
whatever you set as the threshold for auto-push is *also* the threshold
for approval, which is more of a hack/unintended consequence than
intentional design.
-- 
Adam Williamson
Fedora QA Community Monkey
IRC: adamw | Fedora Talk: adamwill AT fedoraproject DOT org
http://www.happyassassin.net