Blocker bug process proposal: waiving late-discovered blockers to next release

[Adam Williamson]

Hi, folks!

So, during the blocker review process in the last few cycles, we have
several times come up against the unfortunate situation that a bug that
in the usual course of events would block a release is discovered
extremely late - say the day before the go/no-go meeting - and at least
some folks have argued that it's sometimes appropriate to not block the
release in this case.

This position has gained quite a bit of acceptance, and we agreed at
the F26 Final go/no-go meeting to draft up some formal policy for this
so we can make such decisions consistently and not in an ad hoc way
that might lead to it becoming a loophole that gets abused.

So, here's my proposal for that. The actual guts of the 'review
blockers' process are kind of split between
https://fedoraproject.org/wiki/QA:SOP_blocker_bug_process and 
https://fedoraproject.org/wiki/QA:SOP_Blocker_Bug_Meeting , so we could
kinda document this in either, but my preference is for the former. I
propose we add a new section to
https://fedoraproject.org/wiki/QA:SOP_blocker_bug_process , called
'Exceptional cases' or similar, as a sub-section of the 'Reviewing
blocker bugs' section. This is my proposed text. Note that it also
covers another type of case we have occasionally come across in the
past, but similarly never specifically formulated.

##################

=== Exceptional cases ===

Generally speaking, any bug that is agreed to be a violation of the
[[Fedora Release Criteria|release criteria]] should be accepted as a
blocker bug for the next relevant milestone release.

However, as explained in the [[Fedora_Release_Life_Cycle|Fedora life
cycle page]] and the
[[Fedora_Release_Criteria#Release_Constraints|release criteria]], we
consider Fedora's release process not to be strictly based on time
''or'' strictly based on quality, but to take both into consideration.
This can mean that, in some exceptional circumstances, we may agree
that a bug constitutes a sufficient violation of the release criteria
that it would ordinarily be accepted as a blocker bug for the next
milestone release, but in fact accept it as a blocker bug for a later
milestone release.

There are currently two established circumstances in which this may
occur.

Firstly, it may occur if it is agreed to be very unlikely that the bug
can possibly be fixed within a reasonable time frame for the release to
be made. For instance, fixing the bug may be a task of such technical
complexity that it cannot possibly be achieved for several weeks or
months, and it may be held that such a delay would be too disruptive to
Fedora's development to be justified.

Secondly, it may occur if the bug is discovered very late in the
release validation process. Sometimes, a relatively less important
blocker bug (such as a non-vital default installed application on a
release-blocking medium failing to run, for instance) may only be found
very near the end of the release validation process, too late for it to
be reasonably possible to fix it without delaying the release. Again,
we may make the determination that in such a case it is preferable to
go ahead with the release rather than delay it to fix such a late-
discovered bug.

All such cases must be evaluated and discussed by the usual parties
(usually at a blocker bug review meeting) and all relevant factors must
be taken into account, much like the discussion of a bug that is a
'conditional' violation of the release criteria. At least the following
will almost always be relevant:

* The severity and likely prevalence of the bug
* Whether the bug could, or should, have been discovered earlier
* How long the release in question has already been delayed
* Whether delaying the release may give us an opportunity to carry out
other desirable work
* The possible effects of the expected delay (to Fedora itself, and
also to other things influenced by Fedora's schedules, including
downstream projects)

It is expected that in almost 'exceptional' cases, the bug will be
accepted as a blocker either for the very next milestone release, or
for the equivalent milestone for the next release (e.g. if this
'exceptional' provision is agreed to apply to a bug that otherwise
would have blocked {{FedoraVersion|long|next}} Final, it should be
accepted as a blocker either for {{FedoraVersion|long|next2}} Alpha or
{{FedoraVersion|long|next2}} Final).

#################

That's a bit wordy (suggestions for cutting it down are appreciated!),
but I think it covers all the salient points. Thoughts? Concerns?
Suggestions? Thanks!
-- 
Adam Williamson
Fedora QA Community Monkey
IRC: adamw | Twitter: AdamW_Fedora | XMPP: adamw AT happyassassin . net
http://www.happyassassin.net

[Kevin Kofler]

Adam Williamson wrote:
>  So, during the blocker review process in the last few cycles, we
> have
>  several times come up against the unfortunate situation that a bug that
>  in the usual course of events would block a release is discovered
>  extremely late - say the day before the go/no-go meeting - and at least
>  some folks have argued that it's sometimes appropriate to not block the
>  release in this case.
>  
>  This position has gained quite a bit of acceptance, and we agreed at
>  the F26 Final go/no-go meeting to draft up some formal policy for this
>  so we can make such decisions consistently and not in an ad hoc way
>  that might lead to it becoming a loophole that gets abused. 
IMHO, this is a slippery slope eroding the quality of Fedora just because 
some people are not willing to wait a week longer for their "fix". The 
argument that this steals time from the next cycle is also invalid, because 
the obvious answer there is: "Don't Do That Then" – the schedule for the 
next release needs to start from the ACTUAL release date of the current 
release, NOT the planned release.

There is at least one user a day asking on #fedora-kde about the Discover 
issue that you decided to ignore in violation of the process, and no doubt 
there are many more who don't bother asking and just wipe Fedora and install 
Kubuntu or Neon instead. The bug would have been trivial to fix.

A blocker ought to be a blocker, no matter when it is discovered and how 
long it takes to fix.

        Kevin Kofler

[Adam Williamson]

On Tue, 2017-07-18 at 03:37 +0200, Kevin Kofler wrote:
>  
>  IMHO, this is a slippery slope eroding the quality of Fedora just because 
>  some people are not willing to wait a week longer for their "fix". The 
>  argument that this steals time from the next cycle is also invalid, because 
>  the obvious answer there is: "Don't Do That Then" – the schedule for the 
>  next release needs to start from the ACTUAL release date of the current 
>  release, NOT the planned release.
>  
>  There is at least one user a day asking on #fedora-kde about the Discover 
>  issue that you decided to ignore in violation of the process, 
This kind of language just isn't helpful. No-one chose to 'ignore'
anything 'in violation' of anything. It clearly wasn't 'ignored',
since
we had about an hour of debate over it. And as ultimately the decision
as to whether a bug is a blocker is the preserve of the relevant groups
- as the process states - nothing is 'violated' by that decision.

>   and no doubt 
>  there are many more who don't bother asking and just wipe Fedora and install 
>  Kubuntu or Neon instead. The bug would have been trivial to fix.
>  
>  A blocker ought to be a blocker, no matter when it is discovered and how 
>  long it takes to fix. 
I don't think that re-arguing the concept is a useful thing to do in
this thread. All the relevant groups were represented at the meeting
and agreed that this *should* be covered in the process documentation.
The point of this thread is to agree on the details of the explanation,
not to argue over whether this should even be a thing at all. Of course
if it somehow becomes evident there's a large number of relevant people
who think this was a terrible idea, we can re-consider, but that
doesn't seem terribly likely. As the text notes, we have for a *long
time* stated that Fedora's release process is not entirely time-based
or entirely quality-based, so it's not feasible to hold that we
absolutely must hold the release for any criteria violation, no matter
how long it might take to fix. That would be too close to an entirely
quality-based process.

As a side point, I'll note that it is *also* a settled point that
desktop teams have a responsibility to help test their own stuff. We
produce KDE lives nightly throughout the release cycle. We (QA) create
validation events, with all the announcement emails and result tables
and associated paraphernalia, at least every two weeks during the
release cycle. AFAICS, no-one from the KDE team actually contributed
any tests of KDE in the F26 cycle *at all*. As seems to be the trend
lately, we have to point out once more that Fedora is supposed to be a
*collaborative* project. Declaring that X Must Be Done but not actually
being willing to contribute to doing X at all isn't very sustainable
for a project like Fedora. We really need the groups responsible for
release-blocking parts of the distribution to *help test them*. If KDE
is going to continue shipping dozens and dozens and dozens of things by
default (including, last I checked, three different things that deal
somehow with software installation), it would be nice for the KDE team
to help check they all actually *work*.
-- 
Adam Williamson
Fedora QA Community Monkey
IRC: adamw | Twitter: AdamW_Fedora | XMPP: adamw AT happyassassin . net
http://www.happyassassin.net

[Michael Catanzaro]

On Mon, Jul 17, 2017 at 8:37 PM, Kevin Kofler <kevin.kofler(a)chello.at&gt; 
wrote:
>  A blocker ought to be a blocker, no matter when it is discovered and
>
>  how
>  long it takes to fix. 
https://bugzilla.redhat.com/show_bug.cgi?id=1405539

It violates a release criterion (IMO), but it's been broken forever and 
fixing it would be a complex engineering project, far beyond the scope 
of our usual blocker issues. Do we really want to block Fedora for 
months on such an issue? F26 would probably be delayed until next year 
if that was our policy.

Michael

[Kevin Kofler]

Michael Catanzaro wrote:
>  https://bugzilla.redhat.com/show_bug.cgi?id=1405539 
IMHO, this isn't even a bug. Everything is working as designed.

And won't editing the kernel command line (which can be done interactively 
in GRUB) to add vconsole.keymap= the desired keymap solve the problem? So 
where is the data loss?

        Kevin Kofler

[Zbigniew Jędrzejewski-Szmek]

On Mon, Jul 17, 2017 at 05:48:09PM -0700, Adam Williamson wrote:
>  === Exceptional cases ===
>  
>  Generally speaking, any bug that is agreed to be a violation of the
>  [[Fedora Release Criteria|release criteria]] should be accepted as a
>  blocker bug for the next relevant milestone release.
>  
>  However, as explained in the [[Fedora_Release_Life_Cycle|Fedora life
>  cycle page]] and the
>  [[Fedora_Release_Criteria#Release_Constraints|release criteria]], we
>  consider Fedora's release process not to be strictly based on time
>  ''or'' strictly based on quality, but to take both into consideration.
>  This can mean that, in some exceptional circumstances, we may agree
>  that a bug constitutes a sufficient violation of the release criteria
>  that it would ordinarily be accepted as a blocker bug for the next
>  milestone release, but in fact accept it as a blocker bug for a later
>  milestone release. 
+1

>  There are currently two established circumstances in which this may
>  occur.
>  
>  Firstly, it may occur if it is agreed to be very unlikely that the bug
>  can possibly be fixed within a reasonable time frame for the release to
>  be made. For instance, fixing the bug may be a task of such technical
>  complexity that it cannot possibly be achieved for several weeks or
>  months, and it may be held that such a delay would be too disruptive to
>  Fedora's development to be justified. 
"cannot possibly" — that's pretty strong words. I sure almost anything
could be achieved in several months, if enough people banded up to do it.
So I'd just keep the first sentence, without "possibly", and drop the
rest of the paragraph. That'd cut down on the wordiness too.

>  Secondly, it may occur if the bug is discovered very late in the
>  release validation process. Sometimes, a relatively less important
>  blocker bug (such as a non-vital default installed application on a
>  release-blocking medium failing to run, for instance) may only be found
>  very near the end of the release validation process, too late for it to
>  be reasonably possible to fix it without delaying the release. Again,
>  we may make the determination that in such a case it is preferable to
>  go ahead with the release rather than delay it to fix such a late-
>  discovered bug. 
>  All such cases must be evaluated and discussed by the usual parties
>  (usually at a blocker bug review meeting) and all relevant factors must
>  be taken into account, much like the discussion of a bug that is a
>  'conditional' violation of the release criteria. At least the following
>  will almost always be relevant:
>  
>  * The severity and likely prevalence of the bug
>  * Whether the bug could, or should, have been discovered earlier
>  * How long the release in question has already been delayed
>  * Whether delaying the release may give us an opportunity to carry out
>  other desirable work
>  * The possible effects of the expected delay (to Fedora itself, and
>  also to other things influenced by Fedora's schedules, including
>  downstream projects)
>  
>  It is expected that in almost 'exceptional' cases, the bug will be
>  accepted as a blocker either for the very next milestone release, or
>  for the equivalent milestone for the next release (e.g. if this
>  'exceptional' provision is agreed to apply to a bug that otherwise
>  would have blocked {{FedoraVersion|long|next}} Final, it should be
>  accepted as a blocker either for {{FedoraVersion|long|next2}} Alpha or
>  {{FedoraVersion|long|next2}} Final). 
"almost" seems misplaced, or maybe you meant "almost all".

Zbyszek

[Adam Williamson]

On Tue, 2017-07-18 at 02:01 +0000, Zbigniew Jędrzejewski-Szmek wrote:
>  
>  > Firstly, it may occur if it is agreed to be very unlikely that the bug
>  > can possibly be fixed within a reasonable time frame for the release to
>  > be made. For instance, fixing the bug may be a task of such technical
>  > complexity that it cannot possibly be achieved for several weeks or
>  > months, and it may be held that such a delay would be too disruptive to
>  > Fedora's development to be justified.
>  
>  "cannot possibly" — that's pretty strong words. I sure almost anything
>  could be achieved in several months, if enough people banded up to do it.
>  So I'd just keep the first sentence, without "possibly", and drop the
>  rest of the paragraph. That'd cut down on the wordiness too. 
Good point, thanks.

>  > It is expected that in almost 'exceptional' cases, the
> bug will be
>  > accepted as a blocker either for the very next milestone release, or
>  > for the equivalent milestone for the next release (e.g. if this
>  > 'exceptional' provision is agreed to apply to a bug that otherwise
>  > would have blocked {{FedoraVersion|long|next}} Final, it should be
>  > accepted as a blocker either for {{FedoraVersion|long|next2}} Alpha or
>  > {{FedoraVersion|long|next2}} Final).
>  
>  "almost" seems misplaced, or maybe you meant "almost all". 
Indeed, you're exactly correct. Thank you.
-- 
Adam Williamson
Fedora QA Community Monkey
IRC: adamw | Twitter: AdamW_Fedora | XMPP: adamw AT happyassassin . net
http://www.happyassassin.net

[Matthew Miller]

On Mon, Jul 17, 2017 at 05:48:09PM -0700, Adam Williamson wrote:
>  All such cases must be evaluated and discussed by the usual parties
>  (usually at a blocker bug review meeting) and all relevant factors must
>  be taken into account, much like the discussion of a bug that is a
>  'conditional' violation of the release criteria. At least the following
>  will almost always be relevant:
>  
>  * The severity and likely prevalence of the bug
>  * Whether the bug could, or should, have been discovered earlier
>  * How long the release in question has already been delayed
>  * Whether delaying the release may give us an opportunity to carry out
>  other desirable work
>  * The possible effects of the expected delay (to Fedora itself, and
>  also to other things influenced by Fedora's schedules, including
>  downstream projects) 

For "could, or should, have been discovered earlier", there's also
"raised as a blocker earlier". There were a couple this time around
that actually had bugs filed but we didn't prioritize them until the
last minute.

Another consideration that might be relevant: is this a *new* issue or
something that also affects the current release (either as released or
with updates)? If something is a clear-cut blocker criterion violation
but isn't a regression *and* we're running late, using further release
delay as a forcing function feels like cutting off our nose to spite
our face. 


-- 
Matthew Miller
<mattdm(a)fedoraproject.org&gt;
Fedora Project Leader

[Adam Williamson]

On Tue, 2017-07-18 at 09:48 -0400, Matthew Miller wrote:
>  On Mon, Jul 17, 2017 at 05:48:09PM -0700, Adam Williamson wrote:
>  > All such cases must be evaluated and discussed by the usual parties
>  > (usually at a blocker bug review meeting) and all relevant factors must
>  > be taken into account, much like the discussion of a bug that is a
>  > 'conditional' violation of the release criteria. At least the following
>  > will almost always be relevant:
>  > 
>  > * The severity and likely prevalence of the bug
>  > * Whether the bug could, or should, have been discovered earlier
>  > * How long the release in question has already been delayed
>  > * Whether delaying the release may give us an opportunity to carry out
>  > other desirable work
>  > * The possible effects of the expected delay (to Fedora itself, and
>  > also to other things influenced by Fedora's schedules, including
>  > downstream projects)
>  
>  
>  For "could, or should, have been discovered earlier", there's also
>  "raised as a blocker earlier". There were a couple this time around
>  that actually had bugs filed but we didn't prioritize them until the
>  last minute.
>  
>  Another consideration that might be relevant: is this a *new* issue or
>  something that also affects the current release (either as released or
>  with updates)? If something is a clear-cut blocker criterion violation
>  but isn't a regression *and* we're running late, using further release
>  delay as a forcing function feels like cutting off our nose to spite
>  our face.  
Both good points and in line with current practice, will add to a later
draft. Thanks!
-- 
Adam Williamson
Fedora QA Community Monkey
IRC: adamw | Twitter: AdamW_Fedora | XMPP: adamw AT happyassassin . net
http://www.happyassassin.net

[Kamil Paral]

The proposal sounds fine.

On Tue, Jul 18, 2017 at 3:48 PM, Matthew Miller <mattdm(a)fedoraproject.org&gt;
wrote:

>  Another consideration that might be relevant: is this a *new* issue
> or
>  something that also affects the current release (either as released or
>  with updates)? If something is a clear-cut blocker criterion violation
>  but isn't a regression *and* we're running late, using further release
>  delay as a forcing function feels like cutting off our nose to spite
>  our face.
>  
I'm not in favor of this one. Too many important bugs have been waived in
the past (even those easily fixable) just because "it's not a new bug". I
don't see why it should matter. Sure, we can use the existing data to
better estimate the impact (how often people complain about this, bug
duplicates, etc). But that's just better input data. It should not affect
the decision process. Or is the thinking process something like "users are
already used to suffer from this bug, and perhaps can even work around it,
so we don't need to try that hard to fix it"? I don't agree to that.

[Matthew Miller]

On Wed, Jul 19, 2017 at 12:24:16PM +0200, Kamil Paral wrote:
>  > Another consideration that might be relevant: is this a *new*
> issue or
>  > something that also affects the current release (either as released or
>  > with updates)? If something is a clear-cut blocker criterion violation
>  > but isn't a regression *and* we're running late, using further release
>  > delay as a forcing function feels like cutting off our nose to spite
>  > our face.
>  I'm not in favor of this one. Too many important bugs have been waived in
>  the past (even those easily fixable) just because "it's not a new bug". I
>  don't see why it should matter. Sure, we can use the existing data to
>  better estimate the impact (how often people complain about this, bug
>  duplicates, etc). But that's just better input data. It should not affect
>  the decision process. Or is the thinking process something like "users are
>  already used to suffer from this bug, and perhaps can even work around it,
>  so we don't need to try that hard to fix it"? I don't agree to that. 
We should definitely fix these things. I just think that delaying the
release is a very big hammer -- in a case where maybe a hammer isn't
even called for. Delaying the release means that users are denied all
sorts of other improvements, and developers can't get their stuff in
the hands of users. The release delay itself doesn't _help_ the fix in
any way. We're just using it as a forcing function, and I think that
causes more problems than it actually solves.


-- 
Matthew Miller
<mattdm(a)fedoraproject.org&gt;
Fedora Project Leader

[Kamil Paral]

On Wed, Jul 19, 2017 at 5:57 PM, Matthew Miller <mattdm(a)fedoraproject.org&gt;
wrote:

>  On Wed, Jul 19, 2017 at 12:24:16PM +0200, Kamil Paral wrote:
>  > > Another consideration that might be relevant: is this a *new* issue or
>  > > something that also affects the current release (either as released or
>  > > with updates)? If something is a clear-cut blocker criterion violation
>  > > but isn't a regression *and* we're running late, using further release
>  > > delay as a forcing function feels like cutting off our nose to spite
>  > > our face.
>  > I'm not in favor of this one. Too many important bugs have been waived in
>  > the past (even those easily fixable) just because "it's not a new
> bug". I
>  > don't see why it should matter. Sure, we can use the existing data to
>  > better estimate the impact (how often people complain about this, bug
>  > duplicates, etc). But that's just better input data. It should not affect
>  > the decision process. Or is the thinking process something like "users
>  are
>  > already used to suffer from this bug, and perhaps can even work around
>  it,
>  > so we don't need to try that hard to fix it"? I don't agree to that.
>
>  We should definitely fix these things. I just think that delaying the
>  release is a very big hammer -- in a case where maybe a hammer isn't
>  even called for. Delaying the release means that users are denied all
>  sorts of other improvements, and developers can't get their stuff in
>  the hands of users. The release delay itself doesn't _help_ the fix in
>  any way. We're just using it as a forcing function, and I think that
>  causes more problems than it actually solves.
>  
And I agree with you, perhaps with one small exception. It depends on your
definition of "help", but the delay *is* sometimes effective in a sense
that without the delay (and the force function), the fix would not appear
fast (in stable updates shortly after release) or even at all. It is not a
good way to approach things, but I don't see many other options.
Developers' time is limited and blocker bugs do set project priorities. I
agree that it should definitely be used very conservatively.

But all of that above is a separate problem. What I'd like to understand is
why you think existing bugs should be treated differently from new bugs.
What is the rationale? And if you want to treat them differently, then how?
Because if we accept new problem A as a blocker, but waive problem B
because it has already existed for some time, even though they have
completely equal impact on users, then without any other means to push for
B resolution during the lifetime of the fedora release, it's very likely
that the problem will not get fixed. Do you see PrioritizedBugs as an
efficient way to help this? Or something else?

[Matthew Miller]

On Thu, Jul 20, 2017 at 10:02:09AM +0200, Kamil Paral wrote:
>  But all of that above is a separate problem. What I'd like to
> understand is
>  why you think existing bugs should be treated differently from new bugs.
>  What is the rationale? And if you want to treat them differently, then how? 
I think they're *clearly* different when it comes to delaying the
release. If a bug is not currently affecting anyone, delaying stops it
from becoming a user problem. If a bug is already a user problem,
delaying doesn't help those people — and just hurts everyone else who
would benefit from the release.


>  Because if we accept new problem A as a blocker, but waive problem B
>  because it has already existed for some time, even though they have
>  completely equal impact on users, then without any other means to push for
>  B resolution during the lifetime of the fedora release, it's very likely
>  that the problem will not get fixed. Do you see PrioritizedBugs as an
>  efficient way to help this? Or something else? 
Prioritized Bugs is one avenue. I'm open to other suggestions. I think
in general packagers and maintainers *do* care about the kinds of bugs
that are likely to be in this situation.



-- 
Matthew Miller
<mattdm(a)fedoraproject.org&gt;
Fedora Project Leader

[Kamil Paral]

On Thu, Jul 20, 2017 at 4:22 PM, Matthew Miller <mattdm(a)fedoraproject.org&gt;
wrote:

>  On Thu, Jul 20, 2017 at 10:02:09AM +0200, Kamil Paral wrote:
>  > But all of that above is a separate problem. What I'd like to understand
>  is
>  > why you think existing bugs should be treated differently from new bugs.
>  > What is the rationale? And if you want to treat them differently, then
>  how?
>
>  I think they're *clearly* different when it comes to delaying the
>  release. If a bug is not currently affecting anyone, delaying stops it
>  from becoming a user problem. If a bug is already a user problem,
>  delaying doesn't help those people — and just hurts everyone else who
>  would benefit from the release.
>  
I'd argue that it does help - the bug gets fixed. The cost is maybe
delaying the release (if the bug is accepted as a blocker way ahead of the
milestone), or surely delaying the release (if the bug is accepted as a
blocker close to the milestone). The alternative is rejecting the blocker
right off the bat, which means the bug maybe gets fixed.

However, I think I misread your comment. I believed you're proposing we
reject bugs existing in stable releases as blockers at any point of the
development cycle. But you seem to have suggested we do this only if
they're discovered very shortly before the milestone deadline. Is that
correct? If so, I'm sorry for the confusion. In this case, I don't really
have a problem with this - it's the same approach as for any other bug
discovered a few hours before release. The impact is taken into
consideration, the difficulty of the fix is taken into consideration, and
also whether it already affects stable releases is taken into consideration
(why not, sure). If the bug is not critical, it's either pushed to the next
milestone, or pushed to the next release. That makes sense to me. If this
is the way it's formulated, I don't have a problem with it.

[Matthew Miller]

On Mon, Jul 24, 2017 at 11:03:34AM +0200, Kamil Paral wrote:
>  However, I think I misread your comment. I believed you're
> proposing we
>  reject bugs existing in stable releases as blockers at any point of the
>  development cycle. But you seem to have suggested we do this only if
>  they're discovered very shortly before the milestone deadline. Is that
>  correct? If so, I'm sorry for the confusion. In this case, I don't really 
Yes, correct. :)


-- 
Matthew Miller
<mattdm(a)fedoraproject.org&gt;
Fedora Project Leader

Blocker bug process proposal: waiving late-discovered blockers to next release (round 2)

[Adam Williamson]

On Mon, 2017-07-17 at 17:48 -0700, Adam Williamson wrote:
>  Hi, folks! 
<snip>

So there was some great feedback on the first version of this proposal;
here's the second draft, with all the suggestions considered and
applied. Note, given the misunderstanding between Kamil and Matt, I
added a little paragraph to specifically clarify that the list of
factors to consider really is just a list of things to *consider*, not
a checklist of criteria that we apply unthinkingly.

As I explained in the first mail, the proposal is to add a section to
https://fedoraproject.org/wiki/QA:SOP_blocker_bug_process , called
"Exceptional cases", as a sub-section of the 'Reviewing blocker bugs'
section. Here is the revised proposal for how the new section should
read:

##################

=== Exceptional cases ===

Generally speaking, any bug that is agreed to be a violation of the
[[Fedora Release Criteria|release criteria]] should be accepted as a
blocker bug for the next relevant milestone release.

However, as explained in the [[Fedora_Release_Life_Cycle|Fedora life
cycle page]] and the
[[Fedora_Release_Criteria#Release_Constraints|release criteria]], we
consider Fedora's release process not to be strictly based on time
''or'' strictly based on quality, but to take both into consideration.
This can mean that, in some exceptional circumstances, we may agree
that a bug constitutes a sufficient violation of the release criteria
that it would ordinarily be accepted as a blocker bug for the next
milestone release, but in fact accept it as a blocker bug for a later
milestone release.

There are currently two established circumstances in which this may
occur.

Firstly, it may occur if it is agreed to be very unlikely that the bug
can be fixed within a reasonable time frame for the release to
be made.

Secondly, it may occur if the bug is discovered and/or proposed as a
blocker very late in the release validation process. Sometimes, a
relatively less important blocker bug (such as a non-vital default
installed application on a release-blocking medium failing to run, for
instance) may only be found very near the end of the release validation
process, too late for it to be reasonably possible to fix it without
delaying the release. Again, we may make the determination that in such
a case it is preferable to go ahead with the release rather than delay
it to fix such a late-discovered bug.

All such cases must be evaluated and discussed by the usual parties
(usually at a blocker bug review meeting) and all relevant factors must
be taken into account, much like the discussion of a bug that is a
'conditional' violation of the release criteria. At least the following
will almost always be relevant:

* The severity and likely prevalence of the bug
* Whether the bug could, or should, have been discovered and/or
proposed as a blocker earlier
* Whether the bug affects the existing stable releases (if it does,
there is generally less benefit to be had by delaying the new release)
* How long the release in question has already been delayed
* Whether delaying the release may give us an opportunity to carry out
other desirable work
* The possible effects of the expected delay (to Fedora itself, and
also to other things influenced by Fedora's schedules, including
downstream projects)

Note that these factors should be carefully and intelligently
considered on a case-by-case basis. This isn't a checklist; we cannot
just say "oh, that bug existed in the previous release, therefore it's
not a blocker, done". It's just a list of some of the factors we
typically ''consider'' in making this determination.

It is expected that in almost all 'exceptional' cases, the bug will be
accepted as a blocker either for the very next milestone release, or
for the equivalent milestone for the next release (e.g. if this
'exceptional' provision is agreed to apply to a bug that otherwise
would have blocked {{FedoraVersion|long|next}} Final, it should be
accepted as a blocker either for {{FedoraVersion|long|next2}} Alpha or
{{FedoraVersion|long|next2}} Final).

#################
-- 
Adam Williamson
Fedora QA Community Monkey
IRC: adamw | Twitter: AdamW_Fedora | identi.ca: adamwfedora
http://www.happyassassin.net

Blocker bug process proposal: waiving late-discovered blockers to next release (round 2)

[Jan Kurik]

Thanks Adam for putting this together. I am definitely+1 to extend the
Blocker bug process with your proposal.

And there is one more topic related to this: how we should deal with
0day bugs found at the last moment before release ? Should not we have
a statement for Accepted0Day and AcceptedPreviousRelease blockers
saying that such bugs need to be verified and have enough karma before
relevant Go/No-Go meeting ? My question is base on the experience we
made during F26 release cycle, when we stopped already running
mirroring of a release as we realized the 0day fix will not be ready
at the release day. Having such a statement (and follow it) might save
the effort especially RelEng is putting into the release activities
after Go/No-Go meeting.

Regards,
Jan

On Thu, Aug 10, 2017 at 3:01 AM, Adam Williamson
<adamwill(a)fedoraproject.org&gt; wrote:
>  On Mon, 2017-07-17 at 17:48 -0700, Adam Williamson wrote:
> > Hi, folks!
>
>  <snip>
>
>  So there was some great feedback on the first version of this proposal;
>  here's the second draft, with all the suggestions considered and
>  applied. Note, given the misunderstanding between Kamil and Matt, I
>  added a little paragraph to specifically clarify that the list of
>  factors to consider really is just a list of things to *consider*, not
>  a checklist of criteria that we apply unthinkingly.
>
>  As I explained in the first mail, the proposal is to add a section to
>  https://fedoraproject.org/wiki/QA:SOP_blocker_bug_process , called
>  "Exceptional cases", as a sub-section of the 'Reviewing blocker bugs'
>  section. Here is the revised proposal for how the new section should
>  read:
>
>  ##################
>
>  === Exceptional cases ===
>
>  Generally speaking, any bug that is agreed to be a violation of the
>  [[Fedora Release Criteria|release criteria]] should be accepted as a
>  blocker bug for the next relevant milestone release.
>
>  However, as explained in the [[Fedora_Release_Life_Cycle|Fedora life
>  cycle page]] and the
>  [[Fedora_Release_Criteria#Release_Constraints|release criteria]], we
>  consider Fedora's release process not to be strictly based on time
>  ''or'' strictly based on quality, but to take both into consideration.
>  This can mean that, in some exceptional circumstances, we may agree
>  that a bug constitutes a sufficient violation of the release criteria
>  that it would ordinarily be accepted as a blocker bug for the next
>  milestone release, but in fact accept it as a blocker bug for a later
>  milestone release.
>
>  There are currently two established circumstances in which this may
>  occur.
>
>  Firstly, it may occur if it is agreed to be very unlikely that the bug
>  can be fixed within a reasonable time frame for the release to
>  be made.
>
>  Secondly, it may occur if the bug is discovered and/or proposed as a
>  blocker very late in the release validation process. Sometimes, a
>  relatively less important blocker bug (such as a non-vital default
>  installed application on a release-blocking medium failing to run, for
>  instance) may only be found very near the end of the release validation
>  process, too late for it to be reasonably possible to fix it without
>  delaying the release. Again, we may make the determination that in such
>  a case it is preferable to go ahead with the release rather than delay
>  it to fix such a late-discovered bug.
>
>  All such cases must be evaluated and discussed by the usual parties
>  (usually at a blocker bug review meeting) and all relevant factors must
>  be taken into account, much like the discussion of a bug that is a
>  'conditional' violation of the release criteria. At least the following
>  will almost always be relevant:
>
>  * The severity and likely prevalence of the bug
>  * Whether the bug could, or should, have been discovered and/or
>  proposed as a blocker earlier
>  * Whether the bug affects the existing stable releases (if it does,
>  there is generally less benefit to be had by delaying the new release)
>  * How long the release in question has already been delayed
>  * Whether delaying the release may give us an opportunity to carry out
>  other desirable work
>  * The possible effects of the expected delay (to Fedora itself, and
>  also to other things influenced by Fedora's schedules, including
>  downstream projects)
>
>  Note that these factors should be carefully and intelligently
>  considered on a case-by-case basis. This isn't a checklist; we cannot
>  just say "oh, that bug existed in the previous release, therefore it's
>  not a blocker, done". It's just a list of some of the factors we
>  typically ''consider'' in making this determination.
>
>  It is expected that in almost all 'exceptional' cases, the bug will be
>  accepted as a blocker either for the very next milestone release, or
>  for the equivalent milestone for the next release (e.g. if this
>  'exceptional' provision is agreed to apply to a bug that otherwise
>  would have blocked {{FedoraVersion|long|next}} Final, it should be
>  accepted as a blocker either for {{FedoraVersion|long|next2}} Alpha or
>  {{FedoraVersion|long|next2}} Final).
>
>  #################
>  --
>  Adam Williamson
>  Fedora QA Community Monkey
>  IRC: adamw | Twitter: AdamW_Fedora | identi.ca: adamwfedora
>  http://www.happyassassin.net
>  _______________________________________________
>  devel mailing list -- devel(a)lists.fedoraproject.org
>  To unsubscribe send an email to devel-leave(a)lists.fedoraproject.org 


-- 
Jan Kuřík
Platform & Fedora Program Manager
Red Hat Czech s.r.o., Purkynova 99/71, 612 45 Brno, Czech Republic

Blocker bug process proposal: waiving late-discovered blockers to next release (round 2)

[Adam Williamson]

On Thu, 2017-08-10 at 10:59 +0200, Jan Kurik wrote:
>  Thanks Adam for putting this together. I am definitely+1 to extend
> the
>  Blocker bug process with your proposal.
>  
>  And there is one more topic related to this: how we should deal with
>  0day bugs found at the last moment before release ? Should not we have
>  a statement for Accepted0Day and AcceptedPreviousRelease blockers
>  saying that such bugs need to be verified and have enough karma before
>  relevant Go/No-Go meeting ? My question is base on the experience we
>  made during F26 release cycle, when we stopped already running
>  mirroring of a release as we realized the 0day fix will not be ready
>  at the release day. Having such a statement (and follow it) might save
>  the effort especially RelEng is putting into the release activities
>  after Go/No-Go meeting. 
I think we could certainly stand to clarify the exact requirements
around Accepted0Day and AcceptedPreviousRelease blockers, yes. AFAIK
all we have for now is this in the blocker process SOP page:

"Accepted0Day is used for cases where the fix does not need to appear
in the final frozen release, but must be available as an update on
release day. AcceptedPreviousRelease is used for cases where the fix
must appear as an update for one or more stable releases."

And we're definitely missing some written-down policy about exactly
when the updates must be in exactly what state. I think we can do that
separately from this, though. I've got a lot on my plate ATM so it'd be
great if someone else could do this draft - perhaps you or Kamil (as I
know he's been interested in the question before)?
-- 
Adam Williamson
Fedora QA Community Monkey
IRC: adamw | Twitter: AdamW_Fedora | identi.ca: adamwfedora
http://www.happyassassin.net

Blocker bug process proposal: waiving late-discovered blockers to next release (round 2)

[Kamil Paral]

On Fri, Aug 11, 2017 at 12:51 AM, Adam Williamson <
adamwill(a)fedoraproject.org&gt; wrote:

>  On Thu, 2017-08-10 at 10:59 +0200, Jan Kurik wrote:
>  > Thanks Adam for putting this together. I am definitely+1 to extend the
>  > Blocker bug process with your proposal.
>  >
>  > And there is one more topic related to this: how we should deal with
>  > 0day bugs found at the last moment before release ? Should not we have
>  > a statement for Accepted0Day and AcceptedPreviousRelease blockers
>  > saying that such bugs need to be verified and have enough karma before
>  > relevant Go/No-Go meeting ? My question is base on the experience we
>  > made during F26 release cycle, when we stopped already running
>  > mirroring of a release as we realized the 0day fix will not be ready
>  > at the release day. Having such a statement (and follow it) might save
>  > the effort especially RelEng is putting into the release activities
>  > after Go/No-Go meeting.
>
>  I think we could certainly stand to clarify the exact requirements
>  around Accepted0Day and AcceptedPreviousRelease blockers, yes. AFAIK
>  all we have for now is this in the blocker process SOP page:
>
>  "Accepted0Day is used for cases where the fix does not need to appear
>  in the final frozen release, but must be available as an update on
>  release day. AcceptedPreviousRelease is used for cases where the fix
>  must appear as an update for one or more stable releases."
>
>  And we're definitely missing some written-down policy about exactly
>  when the updates must be in exactly what state. I think we can do that
>  separately from this, though. I've got a lot on my plate ATM so it'd be
>  great if someone else could do this draft - perhaps you or Kamil (as I
>  know he's been interested in the question before)?
>  
I tried to discuss this topic in the past wrt AcceptedPreviousRelease, and
I didn't receive many responses from relevant people. From what I did
receive, I created this description (part of the official SOP, so it should
be "in production"):
https://fedoraproject.org/wiki/QA:SOP_blocker_bug_process#Tracking_Accept...

It might seem that this only discusses when to close
AcceptedPreviousRelease bugs. However, having all blocker bugs closed is a
prerequisite to voting "Go". So this is effectively a policy of how to
handle AcceptedPreviousRelease blockers state during Go/NoGo. The key
sentence describing the required state is:

"Only when it is guaranteed that MirrorManager refers only to the push
containing the fixed package or newer pushes, we should mark this bug
as *CLOSED
ERRATA*. "

And track-previous-release-blocker.py script implements that (it's quite
tedious to check that manually). I guess we could use the same approach
(and the same script) even for Accepted0Day. I'm not exactly sure how
'updates' repo is handled before Go, whether some people with older repos
can get older metadata not containing the required fix. If it is guaranteed
that 'updates' repo on release day is always up-to-day for all users (i.e.
it's the very push to that repo, no older metadata references are in
repomd.xml), it's even easier to check the state - just make sure the fix
is in updates-pending tag and we're good to go.