Rethinking proventester and critpath

[Matthew Garrett]

It's a common complaint that it's too difficult to get updates to 
critpath packages through the update system at the moment. We've been 
looking into trying to make that easier without just dropping the 
critpath requirements, and one thing we looked at was whether the 
requirement for positive karma from proventesters was a net benefit.

Thankfully this is the kind of thing that we can actually generate 
numbers for. Luke pulled some statistics which are available at 
http://lists.fedoraproject.org/pipermail/test/2011-October/104084.html . 
The relevant section here is the set of packages that have (a) 
sufficient positive karma to be pushed, but (b) negative proventester 
karma - that is, the packages where negative proventester karma 
prevented a push.

Straight off, we can see that these amount to 1-2% of all critpath 
updates. It's simply not common for proventester to make a difference to 
the outcome. If we look at the individual packages, things get even more 
interesting. Many of the updates receive a mixture of proventester 
karma, so even with the negative the push would still go ahead. As far 
as I can tell:

https://admin.fedoraproject.org/updates/xorg-x11-drv-geode-2.11.9-1.fc14
https://admin.fedoraproject.org/updates/system-setup-keyboard-0.8.6-2.fc14

are the only two updates where the proventester karma requirement would 
have made a difference, out of 1942 critpath updates that made it to 
stable. That doesn't seem like a great hit rate.

So, assuming I'm not grossly misanalysing the data, it seems that we 
could drop the proventester requirement from critical path updates with 
a negligable change in the quality of the updates. Thoughts?

-- 
Matthew Garrett | mjg59 at srcf.ucam.org