old_testing_critpath notifications

[Adam Williamson]

On Thu, 2010-12-02 at 15:43 -0500, Clyde E. Kunkel wrote:

> >> That being the case, I test the package fairly rigorously myself.  But
> >> this process doesn't take that into account.  I test far more things
> >> than you get with a few people just doing smoke tests, but the smoke
> >> tests are actually the gating factor.  If you changed the process so a
> >> maintainer can indicate they've done their own fairly extensive testing,
> >> that would satisfy me.  But that also opens the door for abuse, so you
> >> would have to watch maintainers once you enabled this ability.
> >
> > I've posted in the thread earlier that I'd actually like to do this,
> > others seem opposed however.
> 
> What about putting the maintainer's tests somewhere a tester can get 
> them and use on their own system(s)? Maybe even require critical path 
> packages to actually provide test cases.  This is not to say we don't 
> believe the maintainer, it is just a validation that the package passes 
> reasonable testing of the actual changes.

We're working on this. It won't always be practical, however; in the
current case, for example, you need specific hardware to test mdadm.

> I continue to be frustrated by not being able to always test the actual 
> change.  In most cases all I can do is make sure there are no 
> regressions and then that means all I have done is use the system with 
> the changed package installed and see if anything at all fails.

ensuring there's no regressions is actually more important than testing
whether the fix works. it's the main point of proven tester testing. if
we ship an update which doesn't fix the bug it tries to fix but doesn't
break anything else, we haven't *lost* anything. if we ship an update
that fixes the bug it's trying to fix but also breaks boot for half of
all users, we've lost a lot. :)
-- 
Adam Williamson
Fedora QA Community Monkey
IRC: adamw | Fedora Talk: adamwill AT fedoraproject DOT org
http://www.happyassassin.net