Fedora.next QA planning: generic / Product-specific release criteria and blocker review issues

[Adam Williamson]

Hi, folks. So continuing with the theme of setting up the Fedora.next QA
process, I thought before going too far with draft release criteria etc,
we could discuss a couple of important points that have come up since I
sent out the draft test plan.

There are two kind of similar issues in particular I'm thinking of. 

First, at the QA meeting this week, tflink pointed out that we would
need to decide whether we will have sort of 'parallel' blocker/freeze
exception bug processes for each product. That is, do we have:

F21FinalBlocker
F21ServerFinalBlocker
F21WorkstationFinalBlocker

etc etc, and separate lists of blockers on
http://qa.fedoraproject.org/blockerbugs/current per product (and
non-Product-specific), and separate meetings? Or do we keep the
'unified' blocker nomination / review process, and just handle blocker
bugs for all Products within it?

At first blush I'd incline to keeping a single unified process, because
splitting them up seems like an awful lot of work and I'm not sure it
comes with a clear benefit. It also relies either on reporters correctly
determining what product their bug is relevant to, or on a triage step
for blocker nominations.

However, it's worth noting that if we allow the release schedules for
the Products to diverge in future, it would probably then be inevitable
that we'd need split blocker review (and release validation, for that
matter).

Second, there's a similar issue of organization with regard to the
release criteria. I haven't explicitly written this down anywhere, but
I've been sort of unconsciously expecting that we'd keep the existing
'generic' release criteria pages for criteria that are not
Product-specific, and then we'd have Product-specific release criteria
pages which would basically be *additive*: they'd add additional
criteria applicable only to that Product. They could also perhaps
'patch' the generic criteria to a limited degree, though this would get
messy if the delta got too great.

However, Mike (roshi) has produced draft release criteria for the Cloud
product as part of his work on the 'test outline' for that product -
https://fedoraproject.org/wiki/User:Roshi/QA/Cloud_Docs/Cloud_Alpha_Release_Criteria , https://fedoraproject.org/wiki/User:Roshi/QA/Cloud_Docs/Cloud_Beta_Release_Criteria , https://fedoraproject.org/wiki/User:Roshi/QA/Cloud_Docs/Cloud_Final_Release_Criteria - and used another possible approach; the criteria he has drawn up are clearly intended to be *comprehensive* with regard to the Cloud product. They would not need to be read in conjunction with the 'generic' criteria.

I'm not as sure which approach I prefer here, and to a degree the
difference between them isn't as clear cut as it appears at first
glance; however the criteria are ultimately presented, we'll likely have
several that are applicable to multiple Products. Even if these are
displayed multiple times on 'comprehensive' pages for each Product, they
might be shared at the 'source' level using mediawiki transclusion, for
instance (to prevent them diverging unintentionally). And of course we
aren't necessarily tied to mediawiki for presenting the criteria, which
is another factor to bear in mind (I know one of tflink's Grand Plans
involves a different way of maintaining and presenting the release
criteria).

Thoughts on the best approach for each of these issues would certainly
be appreciated! I thought it'd be best to take some time to think them
over before moving ahead with drafts and so on.
-- 
Adam Williamson
Fedora QA Community Monkey
IRC: adamw | Twitter: AdamW_Fedora | XMPP: adamw AT happyassassin . net
http://www.happyassassin.net