Fedora.next QA planning: generic / Product-specific release criteria and blocker review issues

[Mike Ruckman]

On Tue, 27 May 2014 18:22:21 -0700
Adam Williamson <awilliam at redhat.com> wrote:

> Hi, folks. So continuing with the theme of setting up the Fedora.next
> QA process, I thought before going too far with draft release
> criteria etc, we could discuss a couple of important points that have
> come up since I sent out the draft test plan.
> 
> There are two kind of similar issues in particular I'm thinking of. 
> 
> First, at the QA meeting this week, tflink pointed out that we would
> need to decide whether we will have sort of 'parallel' blocker/freeze
> exception bug processes for each product. That is, do we have:
> 
> F21FinalBlocker
> F21ServerFinalBlocker
> F21WorkstationFinalBlocker
> 
> etc etc, and separate lists of blockers on
> http://qa.fedoraproject.org/blockerbugs/current per product (and
> non-Product-specific), and separate meetings? Or do we keep the
> 'unified' blocker nomination / review process, and just handle blocker
> bugs for all Products within it?
> 
> At first blush I'd incline to keeping a single unified process,
> because splitting them up seems like an awful lot of work and I'm not
> sure it comes with a clear benefit. It also relies either on
> reporters correctly determining what product their bug is relevant
> to, or on a triage step for blocker nominations.
> 
> However, it's worth noting that if we allow the release schedules for
> the Products to diverge in future, it would probably then be
> inevitable that we'd need split blocker review (and release
> validation, for that matter).

I think we might be able to strike a balance here between total
separation and a unified blocker process (though it incurs a bit of
overhead). Just as we currently mark bugs as blockers, we can continue
to do that - but *also* mark them as F21<foo>FinalBlocker. This would
allow for tracking at the product level but also keep some sanity with
blocker meetings. Then, later down the road if the blocker process
diverges with each product, they can simply drop the generic blocker
tracking bug.

I don't immediately see much of a downside to this - but then again,
that doesn't mean there isn't :) At the very least it might lessen the
learning curve as we carve a new path with Fedora.next.

> Second, there's a similar issue of organization with regard to the
> release criteria. I haven't explicitly written this down anywhere, but
> I've been sort of unconsciously expecting that we'd keep the existing
> 'generic' release criteria pages for criteria that are not
> Product-specific, and then we'd have Product-specific release criteria
> pages which would basically be *additive*: they'd add additional
> criteria applicable only to that Product. They could also perhaps
> 'patch' the generic criteria to a limited degree, though this would
> get messy if the delta got too great.
> 
> However, Mike (roshi) has produced draft release criteria for the
> Cloud product as part of his work on the 'test outline' for that
> product -
> https://fedoraproject.org/wiki/User:Roshi/QA/Cloud_Docs/Cloud_Alpha_Release_Criteria ,
> https://fedoraproject.org/wiki/User:Roshi/QA/Cloud_Docs/Cloud_Beta_Release_Criteria ,
> https://fedoraproject.org/wiki/User:Roshi/QA/Cloud_Docs/Cloud_Final_Release_Criteria
> - and used another possible approach; the criteria he has drawn up
> are clearly intended to be *comprehensive* with regard to the Cloud
> product. They would not need to be read in conjunction with the
> 'generic' criteria.

Just so we're clear here: I split out the cloud specific (from what I
could tell) criteria out so that it was easier for me to keep track of
in my head [0]. I wasn't intending it to be an end solution (even though
I suppose it could be, at some point).
 
> I'm not as sure which approach I prefer here, and to a degree the
> difference between them isn't as clear cut as it appears at first
> glance; however the criteria are ultimately presented, we'll likely
> have several that are applicable to multiple Products. Even if these
> are displayed multiple times on 'comprehensive' pages for each
> Product, they might be shared at the 'source' level using mediawiki
> transclusion, for instance (to prevent them diverging
> unintentionally). And of course we aren't necessarily tied to
> mediawiki for presenting the criteria, which is another factor to
> bear in mind (I know one of tflink's Grand Plans involves a different
> way of maintaining and presenting the release criteria).

I too share this Grand Plan, but it's not something I feel needs to be
pushed at this point with all the things on our plates. I imagine the
importance of this will be determined as we move forward and figure
out what works best. 

// Mike

[0] From https://fedoraproject.org/wiki/User:Roshi/QA/Cloud_Docs :
"This is not meant to replace the existing RC pages. It was just easier
for me to parse after taking out all the parts that don't directly deal
with Cloud images."
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 490 bytes
Desc: not available
URL: <http://lists.fedoraproject.org/pipermail/test/attachments/20140527/6a547a21/attachment.sig>