Backlog proposals, now and future - Special Bug Triage meeting, 2008-03-12 17:00UTC

Jon Stanley jonstanley at gmail.com
Tue Mar 11 04:10:48 UTC 2008


Hear ye, hear ye!  There will be a special meeting of the Fedora
BugZappers in our usual meeting slot, Wednesday 2008-03-12 17:00UTC in
#fedora-meeting on freenode.

The purpose of this meeting is to solicit input on proposals for
dealing with the current unmanageable backlog of bugs.  In the long
term, this backlog will cause Fedora irreparable harm, if it has not
already.  Our most valuable asset, the bug reporter, is feeling left
out in the cold.  Community triagers feel discouraged by what they see
as a insurmountable task, thereby making the problem feed on itself.
We have to act, and the time to do it is now.

To that end, I am proud to present two proposals,   One has to do with
dealing with the backlog that we have now, and the other has to do
with making sure we never get into this situation again -- ever.  We
believe that these proposals are the right thing to do, and now is the
right time to do them, right before a release.

I'd also like to give credit where it's due for these proposals.  The
primary author of both of them is John Poelstra, without whom many of
the things that have been accomplished to date would not have been.
In just a few short months, we've gone from having almost nothing
formalized to having a formal bug workflow, and having formal plans of
dealing with the backlog, both now and in the future.

It's important to note that these are PROPOSALS at the current time,
and have not been approved in any way.  I am expecting, and welcome,
impassioned debate on these proposals.  In the course of these
debates, however, lets be civil with one another - we're all
responsible adults here.  If you have comments or concerns about
either of the proposals, there's a comments section at the bottom of
both of them on the wiki, which I encourage you to use.  Also, please
come to the meeting and/or reply to this e-mail if you have comments.
Please try to back up changes to the proposals with a specific example
of where the proposed action is wrong/bad/whatever.

Also note that the framework of the proposals is there.  The exact
text to be found in bugs is still yet to be written, however that will
be written in the next week or so.  The text of both of the proposals
follows.  I realize that both this introduction and the proposals
themselves are extremely long, but please read them!  The wiki
versions of these proposals can be found at:

http://fedoraproject.org/wiki/BugZappers/HouseKeeping
http://fedoraproject.org/wiki/JohnPoelstra/BugzillaExtremeMakeOver

--- Housekeeping - the go-forward plan ---

||<tableclass="message notice">This page is under construction.  It
will hold our standard operating procecedures (SOP) for what needs to
be done at different times during the release cycle starting with
Fedora 9 Development.  March 2008||

= Bugzilla Maintenance SOP =

||<tableclass="message warning">This is a draft of the proposed
approach.  It will be circulated for review in Fedora, including
FESCo, prior to execution (2008-03-10)||


[[TableOfContents]]

== Background ==
The following page outlines the process for managing open Fedora bugs
during the release process.  We intend to proactively manage
unresolved bugzillas in a systematic orderly way which will introduce
predictability to the bugzilla maintenance cycle.  We also do not want
to disenfranchise one of our most valuable community resources: bug
reporters.

Historically we have a had a lot of stale open bugs.  In order to
remain focused as a project it is best to close out bugs we aren't
going to be able to resolve so that we focus on the bugs we will. And
if you think this is too aggressive you're welcome to keep your bugs
alive by continually moving them to the latest version.


== Bugzilla Product Versioning ==
 * Fedora will track bugs solely based on the version number of the
release or '''rawhide''': the release under development which has not
been released.
 * Fedora will '''not''' create separate ''fedora versions'' in
bugzilla  for individual test releases, including, but not limited to:
alpha, beta, preview release, etc.  Fedora has tried this structure in
the past, but found it difficult to maintain and its benefit
imperceptible.
 * Changes to this policy require review and approval by FESCo.

== Tracker Bugs ==
 * Fedora creates a series of tracker bugs at the beginning of each
new release cycle (1 day after GA of the previous release).  The
official tracker bugs are:
   1. Alpha
   1. Beta
   1. Preview Release
   1. GA
 * A list of tracker bugs is maintained at this page:
[https://bugzilla.redhat.com/buglist.cgi?query_format=advanced&short_desc_type=allwordssubstr&short_desc=&product=Fedora&version=&component=&query_format=advanced&bug_status=NEW&bug_status=ASSIGNED&bug_status=NEEDINFO&bug_status=MODIFIED&bug_status=ON_DEV&bug_status=ON_QA&bug_status=VERIFIED&bug_status=FAILS_QA&bug_status=RELEASE_PENDING&bug_status=POST&bug_status=PASSES_QA&long_desc_type=substring&long_desc=&bug_file_loc_type=allwordssubstr&bug_file_loc=&status_whiteboard_type=allwordssubstr&status_whiteboard=&fixed_in_type=allwordssubstr&fixed_in=&qa_whiteboard_type=allwordssubstr&qa_whiteboard=&keywords_type=allwords&keywords=Tracking&emailassigned_to1=1&emailtype1=exact&email1=&emailassigned_to2=1&emailreporter2=1&emailqa_contact2=1&emailcc2=1&emailtype2=exact&email2=&bugidtype=include&bug_id=&votes=&changedin=&chfieldfrom=&chfieldto=Now&chfieldvalue=&cmdtype=doit&order=Reuse+same+sort+as+last+time&field0-0-0=noop&type0-0-0=noop&value0-0-0=
here]

||<tableclass="message note">''Tracker'' bugs--commonly referred to as
''blocker'' bugs--are meta-bugillas used to monitor a group of bugs
that must be resolved before a specific release milestone and are
therefore considered to ''block'' the release.||


== Currently Supported Versions ==
 * Fedora 8: EOL one month after GA of Fedora 10 (''roughly'' November
30, 2008)
 * Fedora 7: EOL on 2008-05-29 (based on current schedule as of 2008-03-10)
 * All other versions of Fedora are ''unmaintained'' and thus ''End Of
Life'' (EOL).

||<tableclass="message note">An ''unmaintained release'' receives no
maintenance or updated packages.  Therefore bugs are not tracked
against these releases.  In the future we will seek an enhancement to
bugzilla to disable the ability to file bugs against unmaintained
versions as there is no value in doing so.||

== Release Processes ==
||<tableclass="message note">The following sections talk about
''submitting tickets to Red Hat Engineering Operations'' and ''running
scripts''.  The overarching idea here is to have standardized
processes and scripts to complete these tasks each release.  The
''scripts'' have not been written and are referred to for sake of
example. Red Hat Engineering Operations maintains Fedora's bugzilla.
||


=== First Day of Development ===
 * Create tracker bugs for test releases and GA of next release
 * Flush BugZappers/ActiveTriagers page and make way for the new
triagers for the next release
   * There are no term limits, but we want to flush the page each
release so it stays current without a lot of work.  We don't think
asking people to re-add their names once every six months is a big
deal.
   * Send out announcement to fedora-test-list at redhat.com asking
triagers for next release to add their names

=== Two Weeks Before Release Date ===
 * File a ticket with Red Hat Engineering Operations requesting
advanced preparation for the following:
   1. Creation of a new Fedora version the day before release day
   1. Ready the ''rawhide-mover'' script for mass-change of all bugs
currently open against the ''rawhide'' version to be mass moved to the
new Fedora version
      * For example all ''rawhide'' bugs open up until the official
release of ''Fedora 9''  will be changed to version ''9'' from
''rawhide''.
   1. Update text that refers to the latest Fedora version on these
pages (coordinate wording with marketing):
     a. https://bugzilla.redhat.com/
     a. https://bugzilla.redhat.com/enter_bug.cgi
   1. Create/update script ''eol-warning'' script for mass-change of
all bugs for the oldest supported version which will become ''end of
life'' (EOL) one month from GA date
     a. '''DECIDE''': Should these bugs be put in special state or
tagged in some way (e.g. keyword)?
     a. include text explaining that:
       i. one month of support remains
       i. please attempt to reproduce on most recent version of Fedora
released which is: '''FIXME'''
       i. if you discover this bug is present in the most recent
version please change the version to that release
       i. if you are using this bug for other purposes and wish for it
to remain open, change it to the latest Fedora version to avoid
auto-closure
       i. if this bug remains open on: '''FIXME''' it will
automatically be ''CLOSED:WONTFIX''
   1. Create/update script ''eol-closer'' script for mass-change of
'''all''' OPEN bugs for the release which will become ''unmaintained''
(EOL).  The script when run will:
     a. change status to CLOSED WONTFIX
     a. include text explaining that:
        i. this release is no longer supported--an updated package
will not be created.  As a project we are only capable of supporting
two releases at a time and there for our  efforts are focused on
currently supported versions.
        i. we would still appreciate your help if you could attempt to
reproduce this bug on the latest supported version which is:
'''FIXME''' or latest test release for the version under development
which is: '''FIXME''' . If issue still exists, please change the
bugzilla version to reflect where you reproduced the issue and reopen
the bug with a status of NEW.
        i. standard wording will be maintained on at the
Bugzappers/BugzillaBoilerPlate ('''TODO: CREATE''') page

=== One Week Before Release Date ===
 1. Ready the ''eol-warning'' script for mass-change of all bugs for
the oldest supported version which will become ''end of life'' (EOL)
one month from GA date:
   a. select all bugs !CLOSED (any status except CLOSED)
   a. select all bugs open for the version becoming EOL
   a. include text explaining that:
       i. one month of support remains
       i. please attempt to reproduce on most recent version of Fedora
which was just released and is: ____.  If the issue still exists,
please change the bugzilla version to reflect the current version and
note the package NVR.
       i. if this bug remains open on: _____ (date of EOL) it will
automatically be closed as WONTFIX
       i. standard wording will be maintained on at the
Bugzappers/BugzillaBoilerPlate ('''TODO: CREATE''') page

=== One Day Before Release Date ===
 1. Confirm that release day is for sure with: '''FIXME'''
 1. Enable new version in Bugzilla
 1. Request execution of ''rawhide-mover-script''

=== Day of Release ===
 1. Run ''eol-warning'' script
 1. File a ticket with Red Hat Engineering Operations requesting
advanced preparation of the ''eol-closer'' script for mass-change of
all bugs for versions which are which are EOL
     * '''DECIDE''': Should these bugs be put in special state or
tagged in some way (e.g. keyword)?
     * include text explaining that:
       a. one month of support remains
       a. please attempt to reproduce on most recent version of Fedora
released which is: ____
       a. if this bug remains open on: _____ (date of EOL) it will
automatically be closed as WONTFIX
 1. Run a quick query for '''all''' bugs that are !CLOSED (any state
except CLOSED) for all release which have reached End of Life (EOL).
    a. Add boilerplate text explaining that it is Fedora's policy that
all bugs all EOL releases remain in CLOSED state.  Also encourage the
reporter to attempt to reproduce the bug against the latest maintained
version of Fedora
    a. Change status of bug to CLOSED.
    a. standard wording will be maintained on at the
Bugzappers/BugzillaBoilerPlate ('''TODO: CREATE''') page

=== One Month after GA Release ===
 1. Confirm EOL date with: '''FIXME'''
 1. Run ''eol-closer'' script
 1. Update this page with correct maintained and EOL Fedora versions
 1. Update Bugzappers/BugzillaBoilerPlate ('''TODO: CREATE''') page
with correct maintained and EOL Fedora versions

=== Ongoing ===
 1. Monitor '''all''' bugs that are !CLOSED (any state except CLOSED)
for all release which are no longer maintained have reached End of
Life (EOL).
 1. Monitor and triage all bugs for currently maintained and rawhide
releases which have a bugzilla status of NEW or have been in status of
NEEDINFO for greater than thirty (30) days.  See
[:BugZappers/BugStatusWorkFlow bug life cycle] for more information
and policies.

== Bugzilla Script Lexicon ==
 1. ''rawhide-mover'' mass moves all bugs currently open against the
''rawhide'' version to the latest released Fedora version
 1. ''eol-warning'' posts a warning to all bugs for the oldest
supported version (GA minus 2) which will become ''end of life'' (EOL)
one month after the GA date
 1. ''eol-closer'' closes '''all''' bugs for release becoming EOL

== Comments and Concerns ==
 1.  Comment #1
 1.  Comment #2

--- Extreme Makeover - i.e. lightening the current load ---

= Bugzilla Extreme Make Over Fedora 9 =

||<tableclass="message warning">This is a draft of the proposed
approach.  It will be circulated for review prior to execution
(2008-02-22)||


[[TableOfContents]]

== Background ==
Fedora has a proliferation of open bugs, so much so, that we are
getting to the point (or may well have reached it) that it is becoming
unmanageable. It does not seem logical to believe that we can continue
on as we are and not do long term damage to the Fedora project.

The time to act is now!

By not addressing this arms race of bugs we put a brighter Fedora
future in jeopardy:
 1. We alienate community members that report bugs, but do not see a
timely response, if ever.
 1. As fewer people report bugs, the quality of Fedora releases has a
higher change of declining
 1. We discourage community bug triagers from joining a situation
where the results of their work cannot be seen because bug activity is
lost in the ocean of other bugs.
 1. We distract new bug triagers who want to focus time on the easiest
bugs (versions which are EOL) which provide very little value to
current work

We need to start ASAP so there is enough time to let the proposed
process below run its course prior to 2008-04-29 (GA for Fedora 9).

== Goal ==
 1. Completely execute the following proposal in stages by GA of Fedora 9
 1. Institute a [:BugZappers/HouseKeeping: go-foward plan] so we do
not have to repeat this process again--ever

== Open Bugs for EOL Versions ==
 * Fedora Core for ''versions'' 1 through 6

=== Overall strategy ===
 * Change status of '''all''' open bugs to NEEDINFO and close
'''all''' open bugs after waiting 30 days

=== Implementation ===
 1. START: now()
 1. Script executed by RHT Engineering Operations which queries bugs
according to following criteria and takes the following actions:
   a. Version of Fedora (1 to 6)
   a. Status !CLOSED (any bug not in a closed state)
   a. Ascertain if bug is a ''blocker'' or ''tracker'' bug
 1. Set bugs meeting criteria to ''NEEDINFO''
 1. If the bug is being used as a ''tracker'' or ''blocker'' bug add
special ''keyword'' of '''Tracking''' to identify this bug for proper
handling in the future
   a. Auto-change ''version'' to '''rawhide''' to extend life of Tracker
 1. Add a friendly comment addressing the following:
   a. apologize for untimely response to this issue
   a. explain present situation and go-forward plan
   a. request that, if at all possible, they reproduce the bug against
Fedora 8 or a Fedora 9 test release (version ''rawhide'')
   a. if bug exists in current version, change ''version'' of bug to
current version: '''8''' or '''rawhide'''
   a. if bug owner wishes (for whatever reason) for the bug to remain
''open'', the version must be ''changed'' to ''Fedora 8'' and status
of ''ASSIGNED''.
   a. if this bug remains open against EOL version it will be
auto-closed as WONTFIX 30 days from now
      i. regardless of current open state (NEW, ASSIGNED, NEEDINFO,
MODIFIED, etc.)
      i. there will no additional warnings or waiting period
 1. END: now() + 30 days

== Open Bugs for ''rawhide'' ==
 * Fedora bugs for ''version'' '''rawhide'''
 * open rawhide bugs are harder to address because it is not readily
apparent which ''released'' Fedora version they are most closely tied
to--the following sub-sections address this problem
 * A process has been proposed to avoid this problem with rawhide bugs
in the future at [:BugZappers/HouseKeeping: Good Bugzilla Hygiene]

=== Overall strategy ===
 * separate ''very stale rawhide'' bugs from ''relevant rawhide'' bugs
using the previous release schedule as a guide:
http://fedoraproject.org/wiki/Releases/HistoricalSchedules
 * stratify bugs and process in buckets resulting in bugs that will be
automatically closed OR manually triaged as part of the regular
processes
 * sync up rawhide bugs with the currently supported release going forward

=== Implementation ===
 * Script written and executed by RHT Engineering Operations which
selects bugs according to specified criteria and takes the specified
actions.

==== Stale Rawhide  ====
 1. START: now()
 1. Select all bugs meeting the following criteria:
    a. Version = ''rawhide''
    a. All states !CLOSED (Any status other than CLOSED)
    a. No comments or internal activity of any kind since GA of Fedora
7 (May 31, 2007): nine months of inactivity
       * Most likely these are bugs reported against releases that are EOL
       * admittedly this is a little aggressive, but risk is low; but
we need to address as many of these as possible--a bug can always be
reopened.
 1. If the bug is being used as a ''tracker'' or ''blocker'' bug add
special ''keyword'' of '''Tracking''' to identify this bug for proper
handling in the future
 1. Act on each bug meeting the specified criteria:
   a. Change bug status for all bugs meeting criteria to NEEDINFO
   a. Add ''whiteboard'' tag of ''bzcl34nup''
      * this helps to clearly tag bugs targeted for this exercise--the
date of last activity will no longer be an accurate field to trigger
off of after posting comments requesting action.
   a. Add a friendly comment addressing the following:
      i. apologize for untimely response to this issue
      i. explain present situation and go-forward plan
      i. request that, if at all possible, they reproduce the bug
against Fedora 8 or a Fedora 9 test release (''rawhide'')
      i. if the bug exists in current version, change ''version'' of
bug to current version (or leave as ''rawhide'') and change status to
'''NEW'''
         * NEW rawhide bugs will be picked up in subsequent step and
manually triaged
      i. if no action is taken this bug will be auto-closed as WONTFIX
30 days from now without any additional warnings or waiting period
      i. if bug is being used as a tracking bug add keyword of
'''Tracking''' and change status to ASSIGNED
 1. now() + 30 days
    a. Select all bugs meeting the following criteria:
       i. Version = ''rawhide''
       i. ''whiteboard'' tag of ''bzcl34nup''
       i. Status = ''NEEDINFO''
       i. Not a tracker bug (does NOT have ''Tracking'' keyword set)
    a. auto-close bugs as WONTFIX
    a. Add a friendly comment explaining the following:
      i. following up to request posted 30 days ago
      i. if bug exists in a currently supported version feel free to
reopen bug against that release

==== Possibly Relevant Rawhide ====
The easiest way to get these bugs into the pipeline for eventual
closeout or resolution is be to tie them to the Fedora 8 release

 1. START and END: now()
 1. Selection criteria
    a. ''rawhide'' version
    a. Opened AFTER May 31, 2007, but BEFORE November 1, 2007
    a. Any status other than CLOSED
    a. Not a tracker bug (does NOT have ''Tracking'' keyword set)
 1. Action:
    a. Change version to Fedora 8
    a. add a friendly note
      i. highlight version change
      i. explain reason for change
      i. include link to this proposal

==== Current Rawhide ====
 * Opened against ''rawhide'' version on or after November 1, 2007
 * Do '''nothing''' to these bugs--they will be addressed as part of
[:BugZappers/HouseKeeping: good bugzilla hygiene]
 * Once all of the above processes have run--take regular triage
actions on these bugs
   * Triagers review all ''rawhide'' bugs in NEW or NEEDINFO

== Concerns & Questions ==
 * YourWikiName
   1. Comment 1
   1. Comment 2

-- 
Jon Stanley
Fedora Bug Wrangler
jstanley at fedoraproject.org




More information about the devel mailing list