The new Update Acceptance Criteria are broken

[David Nalley]

On Sat, Nov 20, 2010 at 5:09 PM, Adam Williamson <awilliam at redhat.com> wrote:
> On Sat, 2010-11-20 at 14:49 -0700, Kevin Fenzi wrote:
>> On Fri, 19 Nov 2010 22:04:24 -0800
>> Adam Williamson <awilliam at redhat.com> wrote:
>>
>> ...snip...
>>
>> > > https://fedorahosted.org/bodhi/ticket/277
>> >
>> > hum, that wasn't well publicised, and I wasn't aware of it. (I should
>> > probably show up to more FESCo meetings...picture FESCo members going
>> > 'no, no, really, it's fine!') I'd disagree, for the reasons above.
>>
>> Well, my thought on it is:
>>
>> * As maintainer, shouldn't you be testing your updates already? Granted
>>   there's often no way you could test everything, but at least
>>   installing it and confirming the bug(s) you claim are fixed are
>>   fixed?
>
> I do. I don't believe all maintainers do. It's pretty hard to explain
> why updates that completely prevent the app in question from working, or
> even prevent the system from booting, got pushed in the past, if all
> maintainers actually test their updates.
>
> The advantage of doing it my way (allowing maintainers to test their own
> updates and file bodhi feedback, but requiring bodhi feedback) is that
> it leaves an audit trail: it requires the maintainer to effectively make
> an explicit public declaration that they tested the update and it
> worked, rather than just relying on the implied 'oh of course they must
> have tested it'. What this means is that if we come across cases where a
> maintainer builds an update, submits it, files bodhi feedback saying
> they've tested it, and it turns out to be completely broken in a way
> they should have caught if they tested it, we now have all the necessary
> evidence to take some kind of sanctions against that packager.
>
> Of course, the idea would be that we'd never have to do that, because
> the fact that the above is the case would be sufficient motivation to
> ensure that packagers really *do* test their updates properly.
>

I think this is an interesting idea, but I'll also say I think it can
be made simpler. Why not just hold package maintainers accountable
period. Make them accountable to FESCo (which in theory they are to
begin with) If I, as a package maintainer continuously want to 'push
directly to stable' and continuously screw it up, I'd hope FESCo and
my original sponsor would at least tell me I am doing it wrong.
Having a +1 button click recorded in Bodhi strikes me as no more
damning evidence than the fact that I committed the update and asked
for it to be pushed to stable. (whether I wait 7 days, or push it
immediately).

I am curious to know a few things?

How many updates submitted to bodhi since the policy has been in place?
How many updates received any feedback?
How many updates received only neutral or negative feedback?
How many updates had an overall negative score. (assuming this is the
number of 'problems' we can confidently confirm we caught - though
more possibly exist)
How many updates received no feedback - and of that group - how long
were they queued up for in updates-testing?