measuring success [was Re: Bodhi 0.7.5 release]

[Adam Williamson]

On Fri, 2010-07-02 at 18:24 +0200, Ralf Corsepius wrote:
> On 07/02/2010 06:20 PM, Will Woods wrote:
> 
> 
> > The main reasons we want to perform testing are things like: to avoid
> > pushing updates with broken dependencies, or updates that cause serious
> > regressions requiring manual intervention / emergency update
> > replacements. That sort of thing.
> >
> Should be done scripted as part of the "package push process".
> No need for karmas, no need for "provenpackager"

That only handles a subset of the 'broken dependencies' problem. We've
already had an example this year of a dependency issue the proposed
autoqa depcheck test wouldn't catch, and Michael's script didn't - the
nss-softokn update (as the broken dep is only apparent if you take into
consideration multilib issues and which repositories will have which
updated packages available).

Given Murphy's Law, I am willing to bet that, even when we have an
automated dependency checker in place, someone will manage to push an
update which causes a dependency problem which the automated test
doesn't catch. Manual testing will help us catch that.

And, again, though Will mentioned two types of broken update, you only
considered one in your reply.

> Doesn't Michael Schwendt have a script for this?

Yes. Is a script all you need to implement an automated step in the
updaate push process? No.
-- 
Adam Williamson
Fedora QA Community Monkey
IRC: adamw | Fedora Talk: adamwill AT fedoraproject DOT org
http://www.happyassassin.net