Last minute blocker bugs

This is a resend.

In the last QA team meeting (04/29/2019) I volunteered to help with 
adding something to the blocker bug process to handle Last Minute 
Blocker Bugs.

I started by reading: 
https://lists.fedoraproject.org/archives/list/[e-mail]...

Followed by: https://fedoraproject.org/wiki/QA:SOP_blocker_bug_process

and: https://fedoraproject.org/wiki/QA:SOP_Blocker_Bug_Meeting

I agree that the provisions for how to handle last minute proposed 
blocker bugs should be in the Blocker Bug Process rather than Blocker 
Bug Meeting.



Here's my draft:

Exceptions for Last Minute Proposed Blocker Bugs

Last Minute Proposed Blocker Bugs must meet Blocker Bug Criteria before 
being processed per this section.

Last Minute Blocker Bugs shall be evaluated for being delayed to the 
next release cycle according to the criteria below. Last Minute Proposed 
Blocker Bugs that meet any of the listed criteria may be delayed to the 
next release cycle as Blocker Bugs if the Release Team agrees. Other 
Last Minute Proposed Blocker Bugs must be corrected before the current 
cycle Final Release.

1) Any bug that can not be fixed in a reasonable amount of time 
considering the current Release Cycle or due to complexity or resource 
constraints.

2) Any bug in non-vital system operation or release package installed 
application operation.


Delaying the current Release:

Delaying the current release cycle must take into account all of the 
following criteria.

1) Consider if the cause of the delay should have been caught earlier in 
the current cycle. What changes in process could help moving such 
discoveries earlier in the cycle?

2) Adding the current proposed delay to any prior delays, is the total 
delay becoming unacceptable in regard to Fedora release policy?

3) Will the current proposed delay enable other desirable work to be 
completed for the release?

4) Impact on down stream projects.


Looking forward to your feedback.


     Have a Great Day!

     Pat    (tablepc)

[Lukas Ruzicka]

Hello,

this was already been discussed a couple of years ago and the story can be
found here:
https://lists.fedoraproject.org/archives/list/[e-mail]...

I agree with how it is written, however, I sort of believe that this might
be a rabbit hole opening if mistreated.
Therefore, as an addition to Adam's draft, I would suggest there be *a
quorum* to make sure that the majority of stakeholders do really agree with
the blocker bug waiver. Let's say, the decision would take at least* ~10
people to vote, with the result of ~80 %*, so that it is clear that we want
it that way and the process cannot be misused to avoid bug fixing. Solid
arguments would be needed to go over the quorum and I think that is
reasonable.

Lukas


On Mon, May 13, 2019 at 5:27 PM pmkellly(a)frontier.com <pmkellly(a)frontier.com&gt;
wrote:

>  This is a resend.
>
>  In the last QA team meeting (04/29/2019) I volunteered to help with
>  adding something to the blocker bug process to handle Last Minute
>  Blocker Bugs.
>
>  I started by reading:
>
>
> https://lists.fedoraproject.org/archives/list/[e-mail]...
>
>  Followed by: https://fedoraproject.org/wiki/QA:SOP_blocker_bug_process
>
>  and: https://fedoraproject.org/wiki/QA:SOP_Blocker_Bug_Meeting
>
>  I agree that the provisions for how to handle last minute proposed
>  blocker bugs should be in the Blocker Bug Process rather than Blocker
>  Bug Meeting.
>
>
>
>  Here's my draft:
>
>  Exceptions for Last Minute Proposed Blocker Bugs
>
>  Last Minute Proposed Blocker Bugs must meet Blocker Bug Criteria before
>  being processed per this section.
>
>  Last Minute Blocker Bugs shall be evaluated for being delayed to the
>  next release cycle according to the criteria below. Last Minute Proposed
>  Blocker Bugs that meet any of the listed criteria may be delayed to the
>  next release cycle as Blocker Bugs if the Release Team agrees. Other
>  Last Minute Proposed Blocker Bugs must be corrected before the current
>  cycle Final Release.
>
>  1) Any bug that can not be fixed in a reasonable amount of time
>  considering the current Release Cycle or due to complexity or resource
>  constraints.
>
>  2) Any bug in non-vital system operation or release package installed
>  application operation.
>
>
>  Delaying the current Release:
>
>  Delaying the current release cycle must take into account all of the
>  following criteria.
>
>  1) Consider if the cause of the delay should have been caught earlier in
>  the current cycle. What changes in process could help moving such
>  discoveries earlier in the cycle?
>
>  2) Adding the current proposed delay to any prior delays, is the total
>  delay becoming unacceptable in regard to Fedora release policy?
>
>  3) Will the current proposed delay enable other desirable work to be
>  completed for the release?
>
>  4) Impact on down stream projects.
>
>
>  Looking forward to your feedback.
>
>
>       Have a Great Day!
>
>       Pat    (tablepc)
>  _______________________________________________
>  test mailing list -- test(a)lists.fedoraproject.org
>  To unsubscribe send an email to test-leave(a)lists.fedoraproject.org
>  Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
>  List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
>  List Archives:
>  https://lists.fedoraproject.org/archives/list/[e-mail]
>  

-- 

Lukáš Růžička

FEDORA QE, RHCE

Red Hat

<https://www.redhat.com>

Purkyňova 115

612 45 Brno - Královo Pole

lruzicka(a)redhat.com
TRIED AND PERSONALLY TESTED, ERGO TRUSTED. <https://redhat.com/trusted>

[Julen Landa Alustiza]

On both fc29 and fc30 cycles, nor the later blocker review meetings nor the
go/no-go ones did not have 10 participants, so needing a 80% agremeent with
a minimum of 10 votes would directly block on last minute bugs on those
scenarios.

I don't have a clear opinion on this, but for now I would prefer not having
an special hard quorum requirement

[Lukas Ruzicka]

I have suggested the quorum, but we can still discuss the exact numbers
(mine were just examples). Besides, the votes do not have to come from
people present in one particular IRC meeting, but

   - votes could be recorded in all such meetings (blocker bugs meeting,
   go-nogo, ...)
   - votes could be casted in Bugzilla and IRC presence does not need to be
   required.

The most important thing is, the majority agrees, not that we need to
organize voting every time we meet.

Lukas

On Tue, May 14, 2019 at 10:33 AM Julen Landa Alustiza <julen(a)zokormazo.info&gt;
wrote:

>  On both fc29 and fc30 cycles, nor the later blocker review meetings
> nor
>  the go/no-go ones did not have 10 participants, so needing a 80% agremeent
>  with a minimum of 10 votes would directly block on last minute bugs on
>  those scenarios.
>
>  I don't have a clear opinion on this, but for now I would prefer not
>  having an special hard quorum requirement
>  _______________________________________________
>  test mailing list -- test(a)lists.fedoraproject.org
>  To unsubscribe send an email to test-leave(a)lists.fedoraproject.org
>  Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
>  List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
>  List Archives:
>  https://lists.fedoraproject.org/archives/list/[e-mail]
>  

-- 

Lukáš Růžička

FEDORA QE, RHCE

Red Hat

<https://www.redhat.com>

Purkyňova 115

612 45 Brno - Královo Pole

lruzicka(a)redhat.com
TRIED AND PERSONALLY TESTED, ERGO TRUSTED. <https://redhat.com/trusted>

[Julen Landa Alustiza]

Lukas Ruzicka <lruzicka(a)redhat.com&gt; igorleak hau idatzi zuen (2019 mai. 14,
ar. 10:39):

>  I have suggested the quorum, but we can still discuss the exact
> numbers
>  (mine were just examples). Besides, the votes do not have to come from
>  people present in one particular IRC meeting, but
>
>     - votes could be recorded in all such meetings (blocker bugs meeting,
>     go-nogo, ...)
>     - votes could be casted in Bugzilla and IRC presence does not need to
>     be required.
>
>  The most important thing is, the majority agrees, not that we need to
>  organize voting every time we meet.
>
>  Lukas
>  
But we are talking about last minute blockers so potentially they'll be
discussed just on one meeting (latest blocker-review or GO/No GO) and we
will not have a big time window for BZ voting.

I agree that the waiving should be agreed by a good majority, but I would
prefer bot having any special quorum requeriments.

Scenario: Go/NO-Go is running and kparal finds a last minute bug. We don't
have time for BZ voting, nor we can't wait for quorum. We need to agreed a
decision during the running meeting, either waiving or blocking or it.
Well, I would go with standard discuss, vote & agreement process, nothing
special.

Otherwise we will finish just autoblocking on all last minute bockers
waiting for quorum, so waiving will not be possible.

>

[Lukas Ruzicka]

Yeah, you can be right, Julen, however the process, as we are having now,
would allow discarding Blocker Bugs possibly anytime, even if 3 or 4 people
would be present on a meeting. And I don't think it is correct to let 4
people decide.
In case of a qorum (and I am not saying it has to be 10 people), the few
people in the IRC meeting could always summon for more people to join, as
we always do when we need to talk to particular people. They usually
respond.

But to show some numbers, I have gone through the meetings minutes of F30
release and I realized that on average, there were more than 10 people
actively present on each meeting, the average Go/no Go presence is *17*.
See table.

= Meeting presence

== Go/No-go Meetings

|===
| Milestone | Date | Presence
| Beta | 3/21/19 | 14
| Beta | 3/28/19 | 16
| Final | 4/25/19 | 24
| Final | 4/26/19 | 14
| Average |       | 17
|===

== Blocker Review Meetings

|===
| Date | Presence
| 2/11/19 |  6
| 3/04/19 | 11
| 3/11/19 | 14
| 3/18/19 |  9
| 3/25/19 | 13
| 4/1/19  | 10
| 4/8/19  | 15
| 4/15/19 | 14
| 4/22/19 | 13
| Average |
11

|===


On Tue, May 14, 2019 at 12:21 PM Julen Landa Alustiza <julen(a)zokormazo.info&gt;
wrote:

>  Lukas Ruzicka <lruzicka(a)redhat.com&gt; igorleak hau idatzi zuen
> (2019 mai.
>  14, ar. 10:39):
>
> > I have suggested the quorum, but we can still discuss the exact numbers
> > (mine were just examples). Besides, the votes do not have to come from
> > people present in one particular IRC meeting, but
> >
> >    - votes could be recorded in all such meetings (blocker bugs meeting,
> >    go-nogo, ...)
> >    - votes could be casted in Bugzilla and IRC presence does not need to
> >    be required.
> >
> > The most important thing is, the majority agrees, not that we need to
> > organize voting every time we meet.
> >
> > Lukas
> >
>
>  But we are talking about last minute blockers so potentially they'll be
>  discussed just on one meeting (latest blocker-review or GO/No GO) and we
>  will not have a big time window for BZ voting.
>
>  I agree that the waiving should be agreed by a good majority, but I would
>  prefer bot having any special quorum requeriments.
>
>  Scenario: Go/NO-Go is running and kparal finds a last minute bug. We don't
>  have time for BZ voting, nor we can't wait for quorum. We need to agreed a
>  decision during the running meeting, either waiving or blocking or it.
>  Well, I would go with standard discuss, vote & agreement process, nothing
>  special.
>
>  Otherwise we will finish just autoblocking on all last minute bockers
>  waiting for quorum, so waiving will not be possible.
>
> > _______________________________________________
>  test mailing list -- test(a)lists.fedoraproject.org
>  To unsubscribe send an email to test-leave(a)lists.fedoraproject.org
>  Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
>  List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
>  List Archives:
>  https://lists.fedoraproject.org/archives/list/[e-mail]
>  

-- 

Lukáš Růžička

FEDORA QE, RHCE

Red Hat

<https://www.redhat.com>

Purkyňova 115

612 45 Brno - Královo Pole

lruzicka(a)redhat.com
TRIED AND PERSONALLY TESTED, ERGO TRUSTED. <https://redhat.com/trusted>

[Julen Landa Alustiza]

Lukas Ruzicka <lruzicka(a)redhat.com&gt; igorleak hau idatzi zuen (2019 mai. 14,
ar. 12:55):

>  Yeah, you can be right, Julen, however the process, as we are having
> now,
>  would allow discarding Blocker Bugs possibly anytime, even if 3 or 4 people
>  would be present on a meeting. And I don't think it is correct to let 4
>  people decide.
>  In case of a qorum (and I am not saying it has to be 10 people), the few
>  people in the IRC meeting could always summon for more people to join, as
>  we always do when we need to talk to particular people. They usually
>  respond.
>  
We can discuss about regular quorum rules of course, but here we were
talking about the last minute bug thing, are we talking about an special
rule for waiving or about changing the qorum rules for blocker-reviewing?
If the latter, we should discuss it on an independent thread, and sometimes
we do need small quorums. Many bz voted proposals are accepted/rejected
with just 3-4 +1s ans it helps a lot to polish the candidate to be able to
do that with some after last meeting bugs that need to be classified so
they can be pushed to the compose.


>  But to show some numbers, I have gone through the meetings minutes of
> F30
>  release and I realized that on average, there were more than 10 people
>  actively present on each meeting, the average Go/no Go presence is *17*.
>  See table.
>
>  = Meeting presence
>
>  == Go/No-go Meetings
>
>  |===
>  | Milestone | Date | Presence
>  | Beta | 3/21/19 | 14
>  | Beta | 3/28/19 | 16
>  | Final | 4/25/19 | 24
>  | Final | 4/26/19 | 14
>  | Average |       | 17
>  |===
>
>  == Blocker Review Meetings
>
>  |===
>  | Date | Presence
>  | 2/11/19 |  6
>  | 3/04/19 | 11
>  | 3/11/19 | 14
>  | 3/18/19 |  9
>  | 3/25/19 | 13
>  | 4/1/19  | 10
>  | 4/8/19  | 15
>  | 4/15/19 | 14
>  | 4/22/19 | 13
>  | Average | 11
>  
Nice job, thanks!!

But about the latest waiving thing, the ksieve bug that we found during the
4/25/29 go/no-go meeting was accepted with a +4/-1 result. That's 80% of
votes, but just 16,67% of the total audience from your matrix.

Anyhow I'm fine with that, even if we had waived it. We had to make a
agreement, and a vast majority of the participants agreed on the same way,
I'm fine with that.

[Lukas Ruzicka]

I am talking about the quorum, if there is a late discovered bug, that
would normally be considered a blocker, but because

a) it could not be fixed on time, and
b) it is not as serious as it would prevent people from using Fedora anyway

In this case, I think more people to decide is crucial because it keeps us
from letting bugs flow into releases *whenever *we might need to. If more
people decide that it is ok, then it is ok.

What if any number of people in an IRC meeting could decide it? Then one
person could possibly decide it, too? Or two? Or three? Or how many at
least? And that is the quorum. Or do we say, it should be some percentage
of the present? Ok.

Until now, we always used the "above 50%" which leaves a bitter after-taste
if you are in the "below 50" group. Therefore, I suggest more than 50. So
that it is clearly decided that we want that outcome.


That's 80% of votes, but just 16,67% of the total audience from your matrix.
>  
I do not understand which matrix do you mean?

The ksieve bug was reported by me in early February ... a long time before
Beta and Final. I did not propose it a blocker, because I thought, that
they would have fixed it a long time before Beta or Final. The did not.
However that is not a best example of an unexpected blocker jumping out of
nowhere. It was known for a long time.

I like that things are always decided by the community and with my
suggestion I cry for more community.

Lukas

[Julen Landa Alustiza]

Lukas Ruzicka <lruzicka(a)redhat.com&gt; igorleak hau idatzi zuen (2019 mai. 14,
ar. 14:12):

>  I am talking about the quorum, if there is a late discovered bug, that
>  would normally be considered a blocker, but because
>
>  a) it could not be fixed on time, and
>  b) it is not as serious as it would prevent people from using Fedora anyway
>
>  In this case, I think more people to decide is crucial because it keeps us
>  from letting bugs flow into releases *whenever *we might need to. If more
>  people decide that it is ok, then it is ok.
>
>  What if any number of people in an IRC meeting could decide it? Then one
>  person could possibly decide it, too? Or two? Or three? Or how many at
>  least? And that is the quorum. Or do we say, it should be some percentage
>  of the present? Ok.
>
>  Until now, we always used the "above 50%" which leaves a bitter
>  after-taste if you are in the "below 50" group. Therefore, I suggest more
>  than 50. So that it is clearly decided that we want that outcome.
>
>
>  That's 80% of votes, but just 16,67% of the total audience from your
> > matrix.
> >
>
>  I do not understand which matrix do you mean?
>  
The one on the previous mail. 4/25/19 meeting presence = 24. Positive votes
to block: 4. Voting people, 5. That makes a 80% of total votes but just
16,67% of total presence.


>  The ksieve bug was reported by me in early February ... a long time
> before
>  Beta and Final. I did not propose it a blocker, because I thought, that
>  they would have fixed it a long time before Beta or Final. The did not.
>  However that is not a best example of an unexpected blocker jumping out of
>  nowhere. It was known for a long time.
>  
Known or not, the bug was not proposed as blocker until aftet starting the
go/no-go meeting, so imo it can be considered as a latter bug. Actually we
resurrected this discussion due to that specific bug.

>  I like that things are always decided by the community and with my
>  suggestion I cry for more community.
>
>  Lukas
>
>
>
>  _______________________________________________
>  test mailing list -- test(a)lists.fedoraproject.org
>  To unsubscribe send an email to test-leave(a)lists.fedoraproject.org
>  Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
>  List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
>  List Archives:
>  https://lists.fedoraproject.org/archives/list/[e-mail]
>

[Ben Cotton]

I like the general idea of the original draft, but I would go for a simpler
route: bugs not nominated as a blocker by the scheduled start of the
Go/No-Go meeting are subject to not being considered for blocking the
release.

This gives us the flexibility to make the best judgment call that we can.
It doesn't prevent us from treating it as a blocker, but it also doesn't
require us to. I don't think we need to lawyer it any more than that (even
though I personally enjoy rules lawyering), because if we can't trust
ourselves to make the best decision, then we have bigger problems already.

-- 
Ben Cotton
He / Him / His
Fedora Program Manager
Red Hat
TZ=America/Indiana/Indianapolis