Fedora.next QA planning: generic / Product-specific release criteria and blocker review issues

[Adam Williamson]

Hi, folks. So continuing with the theme of setting up the Fedora.next QA
process, I thought before going too far with draft release criteria etc,
we could discuss a couple of important points that have come up since I
sent out the draft test plan.

There are two kind of similar issues in particular I'm thinking of. 

First, at the QA meeting this week, tflink pointed out that we would
need to decide whether we will have sort of 'parallel' blocker/freeze
exception bug processes for each product. That is, do we have:

F21FinalBlocker
F21ServerFinalBlocker
F21WorkstationFinalBlocker

etc etc, and separate lists of blockers on
http://qa.fedoraproject.org/blockerbugs/current per product (and
non-Product-specific), and separate meetings? Or do we keep the
'unified' blocker nomination / review process, and just handle blocker
bugs for all Products within it?

At first blush I'd incline to keeping a single unified process, because
splitting them up seems like an awful lot of work and I'm not sure it
comes with a clear benefit. It also relies either on reporters correctly
determining what product their bug is relevant to, or on a triage step
for blocker nominations.

However, it's worth noting that if we allow the release schedules for
the Products to diverge in future, it would probably then be inevitable
that we'd need split blocker review (and release validation, for that
matter).

Second, there's a similar issue of organization with regard to the
release criteria. I haven't explicitly written this down anywhere, but
I've been sort of unconsciously expecting that we'd keep the existing
'generic' release criteria pages for criteria that are not
Product-specific, and then we'd have Product-specific release criteria
pages which would basically be *additive*: they'd add additional
criteria applicable only to that Product. They could also perhaps
'patch' the generic criteria to a limited degree, though this would get
messy if the delta got too great.

However, Mike (roshi) has produced draft release criteria for the Cloud
product as part of his work on the 'test outline' for that product -
https://fedoraproject.org/wiki/User:Roshi/QA/Cloud_Docs/Cloud_Alpha_Relea... ,
https://fedoraproject.org/wiki/User:Roshi/QA/Cloud_Docs/Cloud_Beta_Releas... ,
https://fedoraproject.org/wiki/User:Roshi/QA/Cloud_Docs/Cloud_Final_Relea... - and
used another possible approach; the criteria he has drawn up are clearly intended to be
*comprehensive* with regard to the Cloud product. They would not need to be read in
conjunction with the 'generic' criteria.

I'm not as sure which approach I prefer here, and to a degree the
difference between them isn't as clear cut as it appears at first
glance; however the criteria are ultimately presented, we'll likely have
several that are applicable to multiple Products. Even if these are
displayed multiple times on 'comprehensive' pages for each Product, they
might be shared at the 'source' level using mediawiki transclusion, for
instance (to prevent them diverging unintentionally). And of course we
aren't necessarily tied to mediawiki for presenting the criteria, which
is another factor to bear in mind (I know one of tflink's Grand Plans
involves a different way of maintaining and presenting the release
criteria).

Thoughts on the best approach for each of these issues would certainly
be appreciated! I thought it'd be best to take some time to think them
over before moving ahead with drafts and so on.
-- 
Adam Williamson
Fedora QA Community Monkey
IRC: adamw | Twitter: AdamW_Fedora | XMPP: adamw AT happyassassin . net
http://www.happyassassin.net

[Stephen Gallagher]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 05/27/2014 09:22 PM, Adam Williamson wrote:
>  Hi, folks. So continuing with the theme of setting up the
>  Fedora.next QA process, I thought before going too far with draft
>  release criteria etc, we could discuss a couple of important points
>  that have come up since I sent out the draft test plan.
>  
>  There are two kind of similar issues in particular I'm thinking of.
>  
>  
>  First, at the QA meeting this week, tflink pointed out that we
>  would need to decide whether we will have sort of 'parallel'
>  blocker/freeze exception bug processes for each product. That is,
>  do we have:
>  
>  F21FinalBlocker F21ServerFinalBlocker F21WorkstationFinalBlocker
>  
>  etc etc, and separate lists of blockers on 
>  http://qa.fedoraproject.org/blockerbugs/current per product (and 
>  non-Product-specific), and separate meetings? Or do we keep the 
>  'unified' blocker nomination / review process, and just handle
>  blocker bugs for all Products within it?
>  
>  At first blush I'd incline to keeping a single unified process,
>  because splitting them up seems like an awful lot of work and I'm
>  not sure it comes with a clear benefit. It also relies either on
>  reporters correctly determining what product their bug is relevant
>  to, or on a triage step for blocker nominations.
>  
>  However, it's worth noting that if we allow the release schedules
>  for the Products to diverge in future, it would probably then be
>  inevitable that we'd need split blocker review (and release
>  validation, for that matter).
>   
I'm of the opinion that we should keep the blocker process unified
until and unless that becomes impossible. The only benefit I can think
of to separating them would be metrics, and I cannot measure the
degree to which I don't care about metrics :)


>  Second, there's a similar issue of organization with regard to
> the 
>  release criteria. I haven't explicitly written this down anywhere,
>  but I've been sort of unconsciously expecting that we'd keep the
>  existing 'generic' release criteria pages for criteria that are
>  not Product-specific, and then we'd have Product-specific release
>  criteria pages which would basically be *additive*: they'd add
>  additional criteria applicable only to that Product. They could
>  also perhaps 'patch' the generic criteria to a limited degree,
>  though this would get messy if the delta got too great.
>  
>  However, Mike (roshi) has produced draft release criteria for the
>  Cloud product as part of his work on the 'test outline' for that
>  product -
>   
https://fedoraproject.org/wiki/User:Roshi/QA/Cloud_Docs/Cloud_Alpha_Relea...
,
https://fedoraproject.org/wiki/User:Roshi/QA/Cloud_Docs/Cloud_Beta_Releas...
,
https://fedoraproject.org/wiki/User:Roshi/QA/Cloud_Docs/Cloud_Final_Relea...
- - and used another possible approach; the criteria he has drawn up are
clearly intended to be *comprehensive* with regard to the Cloud product.
They would not need to be read in conjunction with the 'generic' criteria.
>  
>  I'm not as sure which approach I prefer here, and to a degree the 
>  difference between them isn't as clear cut as it appears at first 
>  glance; however the criteria are ultimately presented, we'll likely
>  have several that are applicable to multiple Products. Even if
>  these are displayed multiple times on 'comprehensive' pages for
>  each Product, they might be shared at the 'source' level using
>  mediawiki transclusion, for instance (to prevent them diverging
>  unintentionally). And of course we aren't necessarily tied to
>  mediawiki for presenting the criteria, which is another factor to
>  bear in mind (I know one of tflink's Grand Plans involves a
>  different way of maintaining and presenting the release criteria).
>  
>  Thoughts on the best approach for each of these issues would
>  certainly be appreciated! I thought it'd be best to take some time
>  to think them over before moving ahead with drafts and so on.
>   

I like the transclusion idea, personally. We can have individual pages
for each Product and follow them straight through (this also reduces
the amount of paging about and information to keep track of).
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v1
> Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/
>
> iEYEARECAAYFAlOF0mQACgkQeiVVYja6o6NzCwCcDHsuB2+EEHqrFrVcv8E7t3CQ
> ec4AoJha8SuPxOde0SKpLBH8rUMIKPve
> =B81N
> -----END PGP SIGNATURE-----

[Bruno Wolff III]

On Wed, May 28, 2014 at 08:11:16 -0400,
  Stephen Gallagher <sgallagh(a)redhat.com&gt; wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
>
> I'm of the opinion that we should keep the blocker process unified
> until and unless that becomes impossible. The only benefit I can think
> of to separating them would be metrics, and I cannot measure the
> degree to which I don't care about metrics :) 
If the release dates of the products diverge, we might want to track 
which specific products are being blocked. That isn't happening now, 
but has been mentioned as a future possibility.

[Matthew Miller]

On Wed, May 28, 2014 at 08:11:16AM -0400, Stephen Gallagher wrote:
>  I'm of the opinion that we should keep the blocker process
> unified
>  until and unless that becomes impossible. The only benefit I can think
>  of to separating them would be metrics, and I cannot measure the
>  degree to which I don't care about metrics :) 
I think we should care about metrics, and I think this would be very useful
to track, even though I agree it'd be better to combine into one actual
process at this point.


-- 
Matthew Miller    --   Fedora Project    --    <mattdm(a)fedoraproject.org&gt;
                                  "Tepid change for the somewhat better!"

[Stephen Gallagher]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 05/28/2014 10:11 AM, Matthew Miller wrote:
>  On Wed, May 28, 2014 at 08:11:16AM -0400, Stephen Gallagher wrote:
> > I'm of the opinion that we should keep the blocker process
> > unified until and unless that becomes impossible. The only
> > benefit I can think of to separating them would be metrics, and I
> > cannot measure the degree to which I don't care about metrics :)
>  
>  I think we should care about metrics, and I think this would be
>  very useful to track, even though I agree it'd be better to combine
>  into one actual process at this point.
>  
>   
I specifically meant "metrics about whether issues are specific to one
Fedora release or another". I should probably start drinking coffee.
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v1
> Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/
>
> iEYEARECAAYFAlOF+dMACgkQeiVVYja6o6PNfACeJ3aXX+JLc9AC9zisDeaU5KnH
> EAYAn3nRkFpJnGyLpMoySeh5Yyxmz5Lk
> =aucv
> -----END PGP SIGNATURE-----

[Matthew Miller]

On Wed, May 28, 2014 at 10:59:31AM -0400, Stephen Gallagher wrote:
>  >> I'm of the opinion that we should keep the blocker
> process
>  >> unified until and unless that becomes impossible. The only
>  >> benefit I can think of to separating them would be metrics, and I
>  >> cannot measure the degree to which I don't care about metrics :)
>  > I think we should care about metrics, and I think this would be
>  > very useful to track, even though I agree it'd be better to combine
>  > into one actual process at this point.
>  I specifically meant "metrics about whether issues are specific to one
>  Fedora release or another". I should probably start drinking coffee. 
Coffee is delicious. But even in a highly caffeinated state, I'm curious why
you don't think this specific thing is interesting. Unless you are saying
that it's immeasurable because you *care so much*. :)

-- 
Matthew Miller    --   Fedora Project    --    <mattdm(a)fedoraproject.org&gt;
                                  "Tepid change for the somewhat better!"

[Stephen Gallagher]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On Wed 28 May 2014 11:04:39 AM EDT, Matthew Miller wrote:
>  On Wed, May 28, 2014 at 10:59:31AM -0400, Stephen Gallagher wrote:
> >>> I'm of the opinion that we should keep the blocker process 
> >>> unified until and unless that becomes impossible. The only 
> >>> benefit I can think of to separating them would be metrics,
> >>> and I cannot measure the degree to which I don't care about
> >>> metrics :)
> >> I think we should care about metrics, and I think this would
> >> be very useful to track, even though I agree it'd be better to
> >> combine into one actual process at this point.
> > I specifically meant "metrics about whether issues are specific
> > to one Fedora release or another". I should probably start
> > drinking coffee.
>  
>  Coffee is delicious. But even in a highly caffeinated state, I'm
>  curious why you don't think this specific thing is interesting.
>  Unless you are saying that it's immeasurable because you *care so
>  much*. :)
>   
Mostly just that it seems like knowing which blockers came from which
product is data for data's sake, rather than necessarily something
from which we can take action.

I *guess* we could use it to make resourcing decisions, if we actually
had resources to distribute.
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v1
> Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/
>
> iEYEARECAAYFAlOF/fUACgkQeiVVYja6o6PoEwCdHts6HN8ee89BhZgWJ1uVf+yD
> I54An35Ap0io1+ughZPZrESa5Rz2HqHT
> =tzG5
> -----END PGP SIGNATURE-----

[Adam Williamson]

On Tue, 2014-05-27 at 18:22 -0700, Adam Williamson wrote:

>  Second, there's a similar issue of organization with regard to
> the
>  release criteria. I haven't explicitly written this down anywhere, but
>  I've been sort of unconsciously expecting that we'd keep the existing
>  'generic' release criteria pages for criteria that are not
>  Product-specific, and then we'd have Product-specific release criteria
>  pages which would basically be *additive*: they'd add additional
>  criteria applicable only to that Product. They could also perhaps
>  'patch' the generic criteria to a limited degree, though this would get
>  messy if the delta got too great.
>  
>  However, Mike (roshi) has produced draft release criteria for the Cloud
>  product as part of his work on the 'test outline' for that product -
>  https://fedoraproject.org/wiki/User:Roshi/QA/Cloud_Docs/Cloud_Alpha_Relea... ,
> https://fedoraproject.org/wiki/User:Roshi/QA/Cloud_Docs/Cloud_Beta_Releas... ,
> https://fedoraproject.org/wiki/User:Roshi/QA/Cloud_Docs/Cloud_Final_Relea... - and
> used another possible approach; the criteria he has drawn up are clearly intended to be
> *comprehensive* with regard to the Cloud product. They would not need to be read in
> conjunction with the 'generic' criteria.
>  
>  I'm not as sure which approach I prefer here, and to a degree the
>  difference between them isn't as clear cut as it appears at first
>  glance; however the criteria are ultimately presented, we'll likely have
>  several that are applicable to multiple Products. Even if these are
>  displayed multiple times on 'comprehensive' pages for each Product, they
>  might be shared at the 'source' level using mediawiki transclusion, for
>  instance (to prevent them diverging unintentionally). And of course we
>  aren't necessarily tied to mediawiki for presenting the criteria, which
>  is another factor to bear in mind (I know one of tflink's Grand Plans
>  involves a different way of maintaining and presenting the release
>  criteria). 
So, just an update on this part: I've spent today poking at various
approaches to this. The bad news is that handling it at all cleanly with
the approach we generally seemed to like - having separate pages for
each product, but using transclusion to share 'common' criteria - is
going to be difficult and would wind up looking pretty ugly on the 'back
end', due to the limitations of mediawiki transclusion.

Basically I think we'd unavoidably wind up with a huge pile of template
pages containing very small numbers of release criteria - or even just
one criterion, or even a *subsection* of one criterion. The resultant
criteria pages would look nice and clean when you just viewed them, but
it'd be a bit of a mess behind the scenes, and it might be hard to
figure out how the heck to edit anything properly. There's a few reasons
for this:

* Criteria don't just map nicely as 'universal' or 'specific to one
product'. Some are specific to a given type of install medium, for
instance - which would make them apply to, say, two products but not the
third. We couldn't just have template pages for 'universal' and each
product, but we'd also need template pages for each install medium type,
and the criteria pages for each product would have to transclude the
correct template pages for the install medium types that product
actually ships. There may be even more factors like this that I haven't
noticed yet: each one would add another degree of complexity.

* Transclusion is a kind of 'all or nothing' thing. You can't break up
transcluded blocks. This is a problem, because we organize the criteria
into sub-sections. So we wind up needing a set of template pages *for
each criteria sub-section*.

* There are many criteria which cover multiple circumstances, only some
of which apply to each Product (or media type). The criterion
"Release-blocking images must boot" is universal, for instance, but its
notes are not: some of its notes wouldn't apply to some products. So do
we split a single release criterion up and have template pages for each
line of its notes? That seems a bit nutty. But it also reads awkwardly
if we have a "Fedora 21 Workstation Alpha release criteria" page which
includes notes like "Supported cloud environments: Release-blocking
cloud images must boot in the Fedora OpenStack Cloud and in Amazon EC2."
Do we just give up on 'dynamically' sharing that criterion between
pages, and write static copies of it into each product's page? Then
maybe it unintentionally diverges between the three some time.

So...I'm not optimistic we'll be able to handle it particularly cleanly
by transclusion. For now what I'm planning to do is mock up completely
static versions of the criteria pages as they apply to each Product, and
then just look at them to see just how bad the problem is, just how
tricky it would be to reproduce that same result using shared templates.
If it's really as bad as it seems at first glance, I suspect the only
viable approaches will be:

1) have separate criteria pages for each product with maybe just basic
transclusion for only those criteria which can be shared perfectly
cleanly between all three products

2) keep one combined set of release criteria, and just add new sections
to it which are specified only to apply to particular products

3) design a much better system for storing and displaying the release
criteria than Mediawiki

3) would be nice, of course, but takes resources. tflink would like to
work on it, but we can hardly call it a high priority right now with
taskotron work still needing to be done. 2) is a bit ugly, but it's the
closest to our current approach, and probably the easiest to do. 1)
gives us a nice 'clean looking' result but is somewhat more work to
implement and carries a risk of unintentional divergence of the criteria
in future, when changes made to the criteria for one product aren't
properly applied to the other products when they should be.

Anyway, that's where I'm at with this right now...when I have anything
concrete I'll throw it up for people to look at. If anyone has any
brilliant ideas or a really elegant implementation of transclusion, or
something, that'd really help.
-- 
Adam Williamson
Fedora QA Community Monkey
IRC: adamw | Twitter: AdamW_Fedora | XMPP: adamw AT happyassassin . net
http://www.happyassassin.net