FreezeException process improvement - proposal

[Kamil Paral]

FreezeException proposal
========================

The FreezeException process (formerly NTH process) [1] has some deficiencies that I'd like to correct in this proposal.

[1] https://fedoraproject.org/wiki/QA:SOP_freeze_exception_bug_process


Background
----------

Currently anyone can propose any bug as a FreezeException. QA team (and other stakeholders) is then required to go review the proposal and decide whether the exception should be granted or not. Very often the FreezeException is requested not by developers themselves, but by users and testers. That means we have very partial data when deciding - usually we don't know:
* how difficult the issue is to fix
* how risky the fix is (whether it can break different parts of the program)
* whether developers are even willing to fix it in the remaining timeframe

Because of this, we often keep asking for more information, or just decide without having enough information.


The problem
-----------

We spend precious QA time evaluating many bugs that won't be fixed, not even touched by the developer. That is a wasted time and energy.

Also, the turnaround for FreezeExceptions is very slow. During the blocker bug meeting we spend most of the time discussing blocker bugs, because those are the more important ones. Because of that and because of the large numbers of FE bugs to go through, we often keep them unreviewed for a long time. That is a large setback for developers, because they don't know whether to start working on it or rather invest the time in something else (which has a higher chance for freeze exception grant).


The proposal
------------

I'd like to make the process more effective. I imagine both sides would benefit, QA and developers.

I'd like to break the concept that we review any bug that is tagged FE by anybody. I believe it makes sense to review it only after we know a developer is willing to fix it, or at least try to fix it. I'd like to use Tim Flink's form for proposing blockers and FEs [2] that is expected to be working for F19.

[2] http://tflink.fedorapeople.org/blockerbugs/mockups/propose_blocker.html

If the reporter wants to propose a Freeze Exception (activates the relevant checkbox), a new form block could appear. The user would be required to choose one of these options:

.-----------------------------------------------------------------------.
. (x) I'm a user or tester. I believe this bug should receive a freeze  .
.     exception. Please notify the developers and ask them to evaluate  .
.     the bug and post their opinion whether they are willing to try to .
.     fix this before FNN-milestone release.                            .
.                                                                       .
. ( ) I'm a developer and I'm willing to try to fix this before         .
.     FNN-milestone release. Please evaluate the bug and either grant   .
.     or reject the exception. I'm providing the details about fix      .
.     difficulty and risks involved in the justification below.         .
'-----------------------------------------------------------------------'

There could be a longer explanation below for each of the choices. The purpose is to distinguish whether someone is already willing to work on this or not.

If the first option is selected and the proposal submitted, a comment would be added to the Bugzilla ticket, like this:

****************************************************************************
* User NAME has proposed this issue as an freeze exception for Fedora NN   *
* MILESTONE release. He/she believes this is an important bug that might   *
* not be serious enough to be a blocker, but still is very inconvenient    *
* for our users. His/her justification is below:                           *
*                                                                          *
* [JUSTIFICATION]                                                          *
*                                                                          *
* You can read more about FreezeException process at [FreezeException SOP  *
* URL]. Please evaluate this bug and let us know if you're willing to try  *
* to fix this before that particular milestone. To do that, either use     *
* [Blocker/FE Proposal URL] or put "MILESTONEFreezeException" into Blocks  *
* field and add fix details (its difficulty and risks involved) as a new   *
* comment. We will review the available information and either grant or    *
* reject the exception. Thank you.                                         *
****************************************************************************

As you can see, this is the missing piece - asking developers to provide more information without QA spending time on it first just to find out there is not enough information yet. Also this solves the problem of reviewing bugs that won't go fixed anyway - at this stage, we haven't tagged it as MilestoneFreezeException yet, so it doesn't appear in our trackers. We are asking developers to do that if they are really willing to fix it.

We might still want to track these untaged bugs somehow, however, so that they are not completely invisible. For example we might need to poke some not-so-responsive developers manually (over IRC) if we think some of the suggested FEs is really important. For that reason I think we could tag these bugs with QAWatchList or some similar keyword and skim through it from time to time.

There is one additional benefit from automated messages like the one above - it can include all the relevant information and we can be sure we haven't forgotten about anything. Many of developers don't understand the Blocker/FE process properly, we have seen that in previous releases. That can be improved by providing all the details and wiki links, as you have seen in the example above. But we don't do that manually, because it's too tiresome, we would have to share the text snippets somehow between all QA members, etc. If a machine does that for us, we can really polish the message that gets printed into Bugzilla.

Now, back to the web form. If the reporter selects the second option ("I'm a developer"), another comment would get added to Bugzilla ticket, like this:

*****************************************************************************
* Developer NAME suggests he/she is willing to try to fix this issue in     *
* time before Fedora NN MILESTONE. His/her bug evaluation and justification *
* for granting the exception is below:                                      *
*                                                                           *
* [JUSTIFICATION]                                                           *
*                                                                           *
* The stakeholders will discuss it on a Blocker Bug Meeting and either      *
* grant or reject the exception with appropriate comment here. You can      *
* read more about the process at [FE SOP URL].                              *
*****************************************************************************

At this point, the bug would get tagged as MilestoneFreezeException, it would appear on our trackers, and we would discuss it on the next blocker bug meeting.


Summary
-------

Using web form for FE proposals and skipping the bug evaluation until we know somebody wants to work on it brings these benefits:

1. Less QA (and other stakeholders') time spent on blocker bug meetings.
2. Faster turnaround for information-complete FE bugs.
3. Faster notifications for developers whether they can start working on a particular bug or not.
4. Explanatory bugzilla comments with description of required steps and links to detailed guides.
5. More transparent FE process because of better provided information.

Possible drawbacks:

1. We need to set up another tracker bug if we want to follow FEs suggested by users but not yet proposed by developers.

Thoughts?


Future enhancements
-------------------

This is not directly related to this proposal, but I'd like to see automated Bugzilla comments even when we grant/reject the FE. I'm not fully sure how to do that, probably there would have to be a separate form with FAS login and only certain people allowed to access. But the automated comments could include:
* blocker bug meeting date, time and a hyperlink to its logs
* instructions to keep the fix small and contained, if possible
* instructions to change the bug status to MODIFIED or ON_QA once the fix is ready
* information what to do if you don't agree with FE rejection

All of this is missing in our present hand-crafted Bugzilla comments. Again, having these information there would help developers know what to do and improve the process in general.

Of course, these automated Bugzilla comments benefits also apply to Blocker Bugs, not just FreezeExceptions.