Some numbers on blocker/NTH nominations (regarding process revision)

[Adam Williamson]

Hey, folks. So the question of whether and how to streamline the
blocker/NTH review process is one that keeps coming up lately. Here's
some numbers I thought may be of interest in regards to that:

		f14		f15		f16		f17		f18
alpha		39		39		46		35		91
alpha nth	-		18		13		14		42
beta		44		78		81		99		178
beta nth	11		20		32		35		86
final		85		96		119		113		146
final nth	62		58		65		45		99

total blocker	168		213		246		247		415
total nth	73		96		110		94		227

total		241		309		356		341		642
%age nth	30.29		31.07		30.89		27.57		35.36

Those are the numbers of bugs ever nominated as blocker or NTH for each
release point for each release going back to F14, which was the first
release where we had NTH (starting at Beta). Methodology is simple. Just
fire off this BZ search:

https://bugzilla.redhat.com/buglist.cgi?f1=blocked&list_id=957029&o1=changedto&query_format=advanced&v1=635218

changing the BZID on the very end (635218 in the example) to the
appropriate tracker bug, and write down the bug count. That search looks
for all bugs which were ever changed to block the bug in question. It
will have captured a few cases where a nomination was made and then
withdrawn before any discussion/voting took place, but I don't think
that has a significant effect.

I think it's quite interesting, as it shows there hasn't really been a
gradual upward trend in the amount of nominations (and thus the amount
of review work), but a sudden massive spike in F18, to nearly double the
previous normal count (and F18 isn't even done yet, so those numbers
will climb a bit more, probably). F14 is a bit lower but that was the
first time we had NTH, so people were getting used to the process. The
F15, F16 and F17 numbers are pretty consistent.

To me this supports the theory that F18 is an outlier (the result of the
newUI work causing a much greater than normal bugload) and it may not be
the best idea to adjust our process based on the assumption that the
amount of work we had to do for F18 is the 'new normal'. Obviously we
would need numbers from the next release or two to confirm this, but
right now, that's the suggestion I get from this exercise. 

It also shows we aren't getting a trend of increasing numbers of NTH
bugs: I wondered if that might be the case, that we were spending more
and more effort on NTH discussion, but it doesn't seem to be true. NTH
as a percentage of the total seems to hold pretty steady across all the
releases.
-- 
Adam Williamson
Fedora QA Community Monkey
IRC: adamw | Twitter: AdamW_Fedora | identi.ca: adamwfedora
http://www.happyassassin.net