The new Update Acceptance Criteria are broken (was: Re: Heads Up - New Firefox update)

[Adam Williamson]

On Mon, 2010-11-01 at 18:29 +0100, Miloslav Trmač wrote:

> > On the other hand, other scenarios were also brought up, which have not
> > come to pass - for instance, the same thing happening to Fedora 13 or
> > Fedora 14. If we had simply accepted the predictions of doom and not
> > implemented the policy at all, we would be without its benefits for the
> > development of F13 and F14.

> A problem with this line of argument is that the benefits are not quite
> apparent to me.

The policies prevented us from shipping a number of completely broken
updates, which is exactly what they were intended to do. I don't have a
command handy to do a search for rejected proposed critpath updates for
F14, but if you figure it out, you can see the precise results of the
policy there.

> > > In addition to being constructive about remedying the situation,
> > > shouldn't we try to be more constructive about _not introducing such
> > > situations_ in the first place?
> 
> > Saying 'oh dear, this might not work, we'd better not try' is rarely a
> > good approach, IMHO.
> That is a cost-benefit comparison.  "New" does not imply "improved".

We had an extensive discussion about the benefits of testing important
updates at the time the policy went into effect. I don't think it's
really necessary to re-hash the entire thing. For the record, I did not
say nor do I believe that "new" inevitably implies "improved".

> > It's better to try things, with the proviso that
> > you accept when they aren't working and withdraw or modify them.
> It's even better not to dismiss known problems with the policy, and to
> make sure the policy can handle them properly from the start.  This was
> not a surprise, this was an "unforced error".

Sorry, but characterizing it as a 'known problem' is misleading. It's
easy to forecast failure, and you'll likely always be correct in *some*
cases if you forecast enough failures. Only if you precisely forecast
only the failures that actually happen, and do not forecast any failures
that don't happen, can your forecast be considered truly reliable. If
this had truly been a 'known problem' then those who predicted it would
also have correctly chosen *not* to predict failure in the case of
Fedora 13 and Fedora 14. The fact is that they did predict a failure
which has not, in fact, come to pass (neither F13 nor F14 have long
queues of old critpath updates).
-- 
Adam Williamson
Fedora QA Community Monkey
IRC: adamw | Fedora Talk: adamwill AT fedoraproject DOT org
http://www.happyassassin.net