The new Update Acceptance Criteria are broken

[Adam Williamson]

On Sat, 2010-11-20 at 14:49 -0700, Kevin Fenzi wrote:
> On Fri, 19 Nov 2010 22:04:24 -0800
> Adam Williamson <awilliam at redhat.com> wrote:
> 
> ...snip...
> 
> > > https://fedorahosted.org/bodhi/ticket/277
> > 
> > hum, that wasn't well publicised, and I wasn't aware of it. (I should
> > probably show up to more FESCo meetings...picture FESCo members going
> > 'no, no, really, it's fine!') I'd disagree, for the reasons above.
> 
> Well, my thought on it is: 
> 
> * As maintainer, shouldn't you be testing your updates already? Granted
>   there's often no way you could test everything, but at least
>   installing it and confirming the bug(s) you claim are fixed are
>   fixed? 

I do. I don't believe all maintainers do. It's pretty hard to explain
why updates that completely prevent the app in question from working, or
even prevent the system from booting, got pushed in the past, if all
maintainers actually test their updates.

The advantage of doing it my way (allowing maintainers to test their own
updates and file bodhi feedback, but requiring bodhi feedback) is that
it leaves an audit trail: it requires the maintainer to effectively make
an explicit public declaration that they tested the update and it
worked, rather than just relying on the implied 'oh of course they must
have tested it'. What this means is that if we come across cases where a
maintainer builds an update, submits it, files bodhi feedback saying
they've tested it, and it turns out to be completely broken in a way
they should have caught if they tested it, we now have all the necessary
evidence to take some kind of sanctions against that packager.

Of course, the idea would be that we'd never have to do that, because
the fact that the above is the case would be sufficient motivation to
ensure that packagers really *do* test their updates properly.

> * Having just one pair of eyes (even if they are exceptionally talented
>   ones) can lead to things slipping through. I know personally I have
>   messed up and tested something, confirmed the bug was fixed, then due
>   to one of: trying to clean up the commit before building without
>   another test cycle, testing on a machine with different package
>   versions, etc the update I push out is not the one that really fixes
>   the issues or introduces some other issue. Unless you have a really
>   really good test workflow this kind of thing can happen. 
> 
> We can surely revisit this if folks like... but if we do, we might
> consider for non critpath packages just re-enabling a 'push to stable',
> as thats what it would allow with less clicking. ;) 
> 

See above for what I consider the advantage of preserving the karma
requirement but allowing the maintainer to provide it. This is for
non-critpath packages, of course. I don't think this should be allowed
for critpath.
-- 
Adam Williamson
Fedora QA Community Monkey
IRC: adamw | Fedora Talk: adamwill AT fedoraproject DOT org
http://www.happyassassin.net