Rethinking proventester and critpath

[Adam Williamson]

On Tue, 2011-11-01 at 13:59 +0000, Matthew Garrett wrote:
> It's a common complaint that it's too difficult to get updates to 
> critpath packages through the update system at the moment. We've been 
> looking into trying to make that easier without just dropping the 
> critpath requirements, and one thing we looked at was whether the 
> requirement for positive karma from proventesters was a net benefit.
> 
> Thankfully this is the kind of thing that we can actually generate 
> numbers for. Luke pulled some statistics which are available at 
> http://lists.fedoraproject.org/pipermail/test/2011-October/104084.html . 
> The relevant section here is the set of packages that have (a) 
> sufficient positive karma to be pushed, but (b) negative proventester 
> karma - that is, the packages where negative proventester karma 
> prevented a push.
> 
> Straight off, we can see that these amount to 1-2% of all critpath 
> updates. It's simply not common for proventester to make a difference to 
> the outcome. If we look at the individual packages, things get even more 
> interesting. Many of the updates receive a mixture of proventester 
> karma, so even with the negative the push would still go ahead. As far 
> as I can tell:
> 
> https://admin.fedoraproject.org/updates/xorg-x11-drv-geode-2.11.9-1.fc14
> https://admin.fedoraproject.org/updates/system-setup-keyboard-0.8.6-2.fc14
> 
> are the only two updates where the proventester karma requirement would 
> have made a difference, out of 1942 critpath updates that made it to 
> stable. That doesn't seem like a great hit rate.
> 
> So, assuming I'm not grossly misanalysing the data, it seems that we 
> could drop the proventester requirement from critical path updates with 
> a negligable change in the quality of the updates. Thoughts?

So, we discussed this at the QA meeting a couple of weeks back.
Initially the plan was for anyone who had concerns to reply to this
mail, but that doesn't seem to have happened, so I will try to summarize
the various responses from that meeting. You can check the full summary
and log of the meeting:

https://fedoraproject.org/wiki/QA/Meetings/20111107

if you want to make sure I'm not misrepresenting anything.

adamw: 1-2% of all critpath updates (where proventester process is seen
to have 'had an impact') is a small number, but it is not 0. It only
takes *one* really bad update to cause big problems.

adamw: it's possible we could use proven tester feedback more
effectively: there have been cases where bad updates went out despite
negative feedback, because we currently don't give negative feedback -
even from proventesters - much significance at all. So there may have
been more cases where proventester input *may have been significant*
with a system where negative karma is more strongly considered. e.g.
updates could be blocked from being auto-pushed if there is any negative
feedback from a proven tester

red_alert (sandro mathys): critpath packages should have detailed test
plans, as currently proventesters often do not know exactly what they
need to verify with some critpath packages: "the process as its done
right now doesn't work (i.e. is no improvement to not having pts) but
having pts with a better process would surely add to the overall
quality"

tflink: we have to keep in mind the reason for the proposal -
maintainers frustrated that packages get stuck in updates-testing for
weeks - is a valid reason and a significant problem. "if we object to
getting rid of the proventester process, do we have any solutions to the
root of their complaints?"

brunowolff: how about keeping the PT karma requirement but adopting the
plan to allow critpath updates to go through without 'required' karma
after a two week wait

adamw: "i assume the stats made the assumption that the proventester
feedback on any update would still have been present but treated it no
different from regular feedback. so, that's not necessarily a safe
assumption: proventesters may feel a stronger 'responsibility to test',
and if you cancel the process, they might stop doing so. but that's hard
to gauge."

red_alert: proposes a meeting during FUDCon NA to try and come up with a
better pt process

That's about all the concrete thoughts / suggestions I can filter out of
the log.
-- 
Adam Williamson
Fedora QA Community Monkey
IRC: adamw | Twitter: AdamW_Fedora | identi.ca: adamwfedora
http://www.happyassassin.net