Draft for 'Bugzilla processes and procedures' mail to developers

[Adam Williamson]

Hi, guys. As discussed in this morning's meeting, I'm sending a draft of
a mail I propose to send to fedora-devel-list to ask for the opinions of
the developer group on what we should do about Bugzilla policies and
procedures. Does it look OK to everyone? Any suggestions to refine or
improve it? Thanks!

---------------------------------

Hi, -devel-list folks!

We in the Bugzappers team (part of the QA group) are working to revise
our Wiki space, and as part of that, various questions have arisen with
regards to Bugzilla procedures. A lot of the same issues have come up on
this list in the recent past.

In general, it seems like Fedora doesn't really have a properly defined
procedure for exactly how a bug should flow. Every maintainer, reporter
and triager has a slightly different idea of what each status or
resolution or keyword means, and when and by whom they should be
applied.

This is obviously confusing and problematic for any attempt to manage or
monitor Fedora bugs in a systematic way. We should have a consistent
procedure which works for reporters and developers and can be monitored
and managed by the Bugzappers group.

So, to arms!

General bug flow
----------------

This section covers things like "what do all the statuses mean" and
"what do all the resolutions mean" and "who changes what from what to
what, and when".

There is a very well-defined procedure for handling bugs in RHEL. The
first proposal is that we simply use the same procedure (and hence the
same statuses and resolutions) for Fedora. This has the advantage of
giving us a well-defined and tested process which some maintainers will
already be familiar with. The disadvantage of this is that the RHEL
process is probably in some ways over-engineered for Fedora: it uses
several stages managed by different teams which don't really exist so
far as Fedora is concerned, and relies on certain automatic steps which
aren't, AFAIK, actually automated for Fedora.

The second proposal would be to adopt some kind of simplified version of
the RHEL process, using a subset of the same statuses and resolutions,
usually to mean the same thing RHEL uses, with a less complex procedure
for getting all the way from file to fix.

The third proposal would be just to come up with a process for Fedora
from scratch, based probably on some kind of consensus around the
current practices used by various groups.

Specific issues
---------------

So, that's the overview. To get down to some specific issues, here's the
two big ones that have come up:

1) How to handle bugs that occur in multiple releases

There's no obvious One Good Fix for this one, because Bugzilla just
isn't designed to handle it. The two options most heavily discussed
within Bugzappers so far have been:

i) have reporters file separate bugs for each affected release that
someone cares about
ii) track all affected releases via one bug, using comments or keywords

Myself and Vincent Danen tried to come up with a flag-based system for
Mandriva to solve this problem once, but we couldn't really come up with
something that would work better than issue ii) above.

Personally I happen to prefer option i), but either can be made to work.
The most important thing is to pick one option or the other and apply it
consistently. This would mostly be the Bugzappers team's job, but
maintainers obviously have to know the policy chosen and work with it,
and Bugzappers don't want to try and just impose one choice or the
other, we want to know what would be most comfortable for maintainers.

2) What do we do with the Severity and Priority fields

At present these are pretty much roundly ignored by everyone (mostly on
the basis that they're initially set by reporters, who don't follow any
particular procedure, and so they don't convey any useful information).
I personally favour a policy whereby the Bugzappers group would set
these as a part of triage (their choices could be overridden later by
the package maintainer or RelEng). Severity should indicate the
importance of the issue *in the context of the package*, while Priority
indicates the importance of the issue *in the context of the
distribution as a whole*. So a crasher bug in a package only two people
in the world ever use may be High severity but Low priority, while say a
missing icon for Firefox might be Low severity but High priority. In my
experience, if these fields are consistently set by triagers according
to an agreed policy, they do convey valuable information to maintainers
and maintainers (and RelEng) find them useful.

Please, any feedback from active maintainers is most valuable - what I'm
trying to do here is find out what those who use Bugzilla at the sharp
end most need it to work like, to make their work most efficient. We in
Bugzappers see our mission as helping reporters to file useful reports
and maintainers to fix bugs as quickly and easily as possible, so we
want to set our policy based on what will work best for reporters and
maintainers.

Thanks!

----------------------------


-- 
Adam Williamson
Fedora QA Community Monkey
IRC: adamw | Fedora Talk: adamwill AT fedoraproject DOT org
http://www.happyassassin.net

[Christopher Beland]

On Tue, 2009-03-31 at 14:50 -0700, Adam Williamson wrote:
>  i) have reporters file separate bugs for each affected release that
>  someone cares about 
>  ii) track all affected releases via one bug, using comments or
>  keywords 
There is also something of a hybrid approach we were discussing: keep
one bug, until a fix is released.  If a fix does not come out for a
given release and someone wants to advocate for a backport, they can
file a separate bug at that time.


>  The first proposal is that we simply use the same procedure (and
>  hence the same statuses and resolutions) for Fedora....
>  The disadvantage of this is that the RHEL process is probably in
>  some ways over-engineered for Fedora: it uses several stages managed
>  by different teams which don't really exist so far as Fedora is
>  concerned, and relies on certain automatic steps which aren't,
>  AFAIK, actually automated for Fedora. 
Well, looking at this page:

https://bugzilla.redhat.com/page.cgi?id=fields.html

it seems that there are actually several states that RHEL doesn't use
that Fedora does.

Were there any other documents that we found that describe the RHEL
process in detail?  It might be nice to send out links to developers
to say, "And for proposal #1, what Fedora would adopt would be based
on *this*, after further discussion."

The only automatic transitions are:
 MODIFIED -> ON_QA
 ON_QA -> RELEASE_PENDING
 RELEASE_PENDING -> CLOSED/ERRATA

I think those are automated for supported Fedora releases, but not for
Rawhide?  Perhaps there is some room to improve automation so that
Rawhide changes that are supposed to fix a certain bug automatically
mark bugs CLOSED/RAWHIDE when the build gets released?  In the
meantime, it doesn't seem like a big downside to simply document that
developers (or whoever) needs to do that manually.


Two things that the above URL says aren't used in RHEL are
"WORKSFORME" and "UPSTREAM".  I guess I would have two questions to
answer to clarify this page for Fedora use.

1.) Who decides when to set a bug CLOSED/UPSTREAM, and what are the
criteria for doing so?  And specifically, should triagers be expected
to try to make this determination?

2.) Should WORKSFORME be banned?  It seems that for RHEL, the correct
answer is never "I tried it and it worked so there's probably not a
bug".  The alternative is to push reports back as NEEDINFO until they
are resolved as NOTABUG (if it turns out to be user error) or
CLOSED/RAWHIDE or the bug is isolated with the reporter's help.
Usually these cases are either 1.) there is some local state (user
configuration, hardware, etc.) that is triggering the bug or 2.)
different people are testing different versions.  Either way, further
testing can resolve the question.


"RAWHIDE" is also not used in RHEL.  Conversely, the only state I see
(correct me if I'm wrong) that's not used in Fedora is VERIFIED.


The (entirely sensible) way you laid out the question was: "We can
either 1.)  use exactly the same procedure for RHEL and Fedora 2.) use
a simplified version of RHEL for Fedora, or 3.) try to come up with
something from scratch for Fedora."

Based on the opinions expressed so far, I assume that there will be
general agreement that making the two reasonably similar would be
convenient. So the way I am thinking about the answer is: "What should
the differences between Fedora and RHEL bug flow procedures be, if
any?" The answer becomes clearer in my mind after seeing the actual
concrete differences in current practices laid out.  (Or at least the
key decisions that will have to be made.)

I don't know whether it's a good idea to get into details in the
initial message to fedora-devel-list, but hopefully the discussion
will output some concrete answers.

-B.

[Kevin Kofler]

Christopher Beland wrote:
>  2.) Should WORKSFORME be banned?  It seems that for RHEL, the
> correct
>  answer is never "I tried it and it worked so there's probably not a
>  bug".  The alternative is to push reports back as NEEDINFO until they
>  are resolved as NOTABUG (if it turns out to be user error) or
>  CLOSED/RAWHIDE or the bug is isolated with the reporter's help.
>  Usually these cases are either 1.) there is some local state (user
>  configuration, hardware, etc.) that is triggering the bug or 2.)
>  different people are testing different versions.  Either way, further
>  testing can resolve the question. 
The way I use WORKSFORME is "the reporter cannot reproduce the bug anymore
and neither can I (or I am the reporter in the first place), but I have no
idea what fixed it".

        Kevin Kofler

[Adam Williamson]

On Thu, 2009-04-02 at 07:07 +0200, Kevin Kofler wrote:
>  Christopher Beland wrote:
>  > 2.) Should WORKSFORME be banned?  It seems that for RHEL, the correct
>  > answer is never "I tried it and it worked so there's probably not a
>  > bug".  The alternative is to push reports back as NEEDINFO until they
>  > are resolved as NOTABUG (if it turns out to be user error) or
>  > CLOSED/RAWHIDE or the bug is isolated with the reporter's help.
>  > Usually these cases are either 1.) there is some local state (user
>  > configuration, hardware, etc.) that is triggering the bug or 2.)
>  > different people are testing different versions.  Either way, further
>  > testing can resolve the question.
>  
>  The way I use WORKSFORME is "the reporter cannot reproduce the bug anymore
>  and neither can I (or I am the reporter in the first place), but I have no
>  idea what fixed it". 
Same here, I've always used it for that too.
-- 
Adam Williamson
Fedora QA Community Monkey
IRC: adamw | Fedora Talk: adamwill AT fedoraproject DOT org
http://www.happyassassin.net

[John Poelstra]

Adam Williamson said the following on 03/31/2009 02:50 PM Pacific Time:
>  Hi, guys. As discussed in this morning's meeting, I'm sending
> a draft of
>  a mail I propose to send to fedora-devel-list to ask for the opinions of
>  the developer group on what we should do about Bugzilla policies and
>  procedures. Does it look OK to everyone? Any suggestions to refine or
>  improve it? Thanks!
>  
>  ---------------------------------
>  
>  Hi, -devel-list folks!
>  
>  We in the Bugzappers team (part of the QA group) are working to revise
>  our Wiki space, and as part of that, various questions have arisen with
>  regards to Bugzilla procedures. A lot of the same issues have come up on
>  this list in the recent past.
>  
>  In general, it seems like Fedora doesn't really have a properly defined
>  procedure for exactly how a bug should flow. Every maintainer, reporter
>  and triager has a slightly different idea of what each status or
>  resolution or keyword means, and when and by whom they should be
>  applied. 
I think you are overstating a problem that I'm not sure exists.  We have 
defined the states here:
https://fedoraproject.org/wiki/BugZappers/BugStatusWorkFlow
Why not improve on what is there?

When this was discussed in January 2008 the idea was to add some 
structure, but not force people to follow it exactly.  We don't have 
that many states so it is hard to do really bizarre things.  For the 
most part I think the maintainers are okay with what we have. If the 
BugZappers exist to help the maintainers would this change really help them?

>  This is obviously confusing and problematic for any attempt to manage
> or
>  monitor Fedora bugs in a systematic way. We should have a consistent
>  procedure which works for reporters and developers and can be monitored
>  and managed by the Bugzappers group. 
What would be gained from "managing or monitoring" Fedora bugs in a 
different way than they are currently?  In other words, by forcing 
people to follow a more stringent process what benefit will it provide? 
     Is there a sufficient return on investment?

>  
>  So, to arms!
>  
>  General bug flow
>  ----------------
>  
>  This section covers things like "what do all the statuses mean" and
>  "what do all the resolutions mean" and "who changes what from what to
>  what, and when".
>  
>  There is a very well-defined procedure for handling bugs in RHEL. The
>  first proposal is that we simply use the same procedure (and hence the
>  same statuses and resolutions) for Fedora. This has the advantage of
>  giving us a well-defined and tested process which some maintainers will
>  already be familiar with. The disadvantage of this is that the RHEL
>  process is probably in some ways over-engineered for Fedora: it uses
>  several stages managed by different teams which don't really exist so
>  far as Fedora is concerned, and relies on certain automatic steps which
>  aren't, AFAIK, actually automated for Fedora. 
The states are well defined on the bugzilla pages, but the actual 
procedures and handling of RHEL bugs is internal to Red Hat (and not 
likely something Fedora would want to follow, but maybe some of the RHEL 
maintainers would disagree ;-)

>  The second proposal would be to adopt some kind of simplified version
> of
>  the RHEL process, using a subset of the same statuses and resolutions,
>  usually to mean the same thing RHEL uses, with a less complex procedure
>  for getting all the way from file to fix.
>  
>  The third proposal would be just to come up with a process for Fedora
>  from scratch, based probably on some kind of consensus around the
>  current practices used by various groups. 
The fourth approach could be to use what we already have
https://fedoraproject.org/wiki/BugZappers/BugStatusWorkFlow and improve 
on it to fix what is thought to be broken.


>  Specific issues
>  ---------------
>  
>  So, that's the overview. To get down to some specific issues, here's the
>  two big ones that have come up:
>  
>  1) How to handle bugs that occur in multiple releases
>  
>  There's no obvious One Good Fix for this one, because Bugzilla just
>  isn't designed to handle it. The two options most heavily discussed
>  within Bugzappers so far have been:
>  
>  i) have reporters file separate bugs for each affected release that
>  someone cares about
>  ii) track all affected releases via one bug, using comments or keywords
>  
>  Myself and Vincent Danen tried to come up with a flag-based system for
>  Mandriva to solve this problem once, but we couldn't really come up with
>  something that would work better than issue ii) above.
>  
>  Personally I happen to prefer option i), but either can be made to work.
>  The most important thing is to pick one option or the other and apply it
>  consistently. This would mostly be the Bugzappers team's job, but
>  maintainers obviously have to know the policy chosen and work with it,
>  and Bugzappers don't want to try and just impose one choice or the
>  other, we want to know what would be most comfortable for maintainers.
>   
How big of a problem is this today and how will fixing it make Fedora 
better?  In other words is the current ambiguity causing problems?  If 
it is not, why change it?  My impression is that preference varies among 
maintainers and usually depends on the specific situation.


>  2) What do we do with the Severity and Priority fields
>  
>  At present these are pretty much roundly ignored by everyone (mostly on
>  the basis that they're initially set by reporters, who don't follow any
>  particular procedure, and so they don't convey any useful information).
>  I personally favour a policy whereby the Bugzappers group would set
>  these as a part of triage (their choices could be overridden later by
>  the package maintainer or RelEng). Severity should indicate the
>  importance of the issue *in the context of the package*, while Priority
>  indicates the importance of the issue *in the context of the
>  distribution as a whole*. So a crasher bug in a package only two people
>  in the world ever use may be High severity but Low priority, while say a
>  missing icon for Firefox might be Low severity but High priority. In my
>  experience, if these fields are consistently set by triagers according
>  to an agreed policy, they do convey valuable information to maintainers
>  and maintainers (and RelEng) find them useful.
>  
>  Please, any feedback from active maintainers is most valuable - what I'm
>  trying to do here is find out what those who use Bugzilla at the sharp
>  end most need it to work like, to make their work most efficient. We in
>  Bugzappers see our mission as helping reporters to file useful reports
>  and maintainers to fix bugs as quickly and easily as possible, so we
>  want to set our policy based on what will work best for reporters and
>  maintainers.
>   
What about turning this around and asking the maintiners what they see 
as the current problems with their interactions with bugzilla and how 
the bugzappers might help?  Just as an exmaple, what if most maintainers 
could care less about how multiple release bugs are filed, but really 
care about using the "priority field" then we could be more help to them 
by focusing on that.

I think it is great you want to tackle and clarify these things.  Having 
gone through a round myself with this process I guess I learned that 
some ambiguity wasn't as harmful as I first thought. :)

John

[Adam Williamson]

On Wed, 2009-04-01 at 21:22 -0700, John Poelstra wrote:

>  > Hi, -devel-list folks!
>  > 
>  > We in the Bugzappers team (part of the QA group) are working to revise
>  > our Wiki space, and as part of that, various questions have arisen with
>  > regards to Bugzilla procedures. A lot of the same issues have come up on
>  > this list in the recent past.
>  > 
>  > In general, it seems like Fedora doesn't really have a properly defined
>  > procedure for exactly how a bug should flow. Every maintainer, reporter
>  > and triager has a slightly different idea of what each status or
>  > resolution or keyword means, and when and by whom they should be
>  > applied.
>  
>  I think you are overstating a problem that I'm not sure exists.  We have 
>  defined the states here:
>  https://fedoraproject.org/wiki/BugZappers/BugStatusWorkFlow
>  Why not improve on what is there? 
>  I think it is great you want to tackle and clarify these things. 
> Having 
>  gone through a round myself with this process I guess I learned that 
>  some ambiguity wasn't as harmful as I first thought. :) 
You're right, of course. Somehow I'd forgotten about that flow.

So, I will revise the draft substantially. :) Here's my quick thoughts:

The obvious bit of hand-waviness in the graphic is the resolutions, we
don't define them (and it doesn't list some at all). DUPLICATE is
simple, and ERRATA and RAWHIDE are known: fixed bugs in official
releases are closed as ERRATA (should be done automatically), and fixed
bugs in Rawhide are closed as RAWHIDE (manually). Those we can write
down into that page without any discussion, I think.

We do, however, need to define what 'cantfix', 'wontfix',
'notabug',
and 'worksforme' are for. We should also explicitly state which
resolutions aren't used for Fedora (I think 'deferred',
'currentrelease'
and 'nextrelease' fit into this category) so they don't get used on
Fedora bugs by mistake.

It would really be nice, in fact, if we could have Bugzilla only show
the statuses and resolutions appropriate to the product the bug is filed
on...not sure if that's possible, though.

I will send a completely revised draft mail tomorrow, focussing on the
issues that are really valid.
-- 
Adam Williamson
Fedora QA Community Monkey
IRC: adamw | Fedora Talk: adamwill AT fedoraproject DOT org
http://www.happyassassin.net

[John Poelstra]

Adam Williamson said the following on 04/02/2009 01:29 PM Pacific Time:
>  On Wed, 2009-04-01 at 21:22 -0700, John Poelstra wrote:
>  
> >> Hi, -devel-list folks!
> >>
> >> We in the Bugzappers team (part of the QA group) are working to revise
> >> our Wiki space, and as part of that, various questions have arisen with
> >> regards to Bugzilla procedures. A lot of the same issues have come up on
> >> this list in the recent past.
> >>
> >> In general, it seems like Fedora doesn't really have a properly defined
> >> procedure for exactly how a bug should flow. Every maintainer, reporter
> >> and triager has a slightly different idea of what each status or
> >> resolution or keyword means, and when and by whom they should be
> >> applied.
> > I think you are overstating a problem that I'm not sure exists.  We have 
> > defined the states here:
> > https://fedoraproject.org/wiki/BugZappers/BugStatusWorkFlow
> > Why not improve on what is there?
>  
> > I think it is great you want to tackle and clarify these things.  Having 
> > gone through a round myself with this process I guess I learned that 
> > some ambiguity wasn't as harmful as I first thought. :)
>  
>  You're right, of course. Somehow I'd forgotten about that flow.
>  
>  So, I will revise the draft substantially. :) Here's my quick thoughts:
>  
>  The obvious bit of hand-waviness in the graphic is the resolutions, we
>  don't define them (and it doesn't list some at all). DUPLICATE is
>  simple, and ERRATA and RAWHIDE are known: fixed bugs in official
>  releases are closed as ERRATA (should be done automatically), and fixed
>  bugs in Rawhide are closed as RAWHIDE (manually). Those we can write
>  down into that page without any discussion, I think.
>  
>  We do, however, need to define what 'cantfix', 'wontfix',
> 'notabug',
>  and 'worksforme' are for. We should also explicitly state which
>  resolutions aren't used for Fedora (I think 'deferred',
> 'currentrelease'
>  and 'nextrelease' fit into this category) so they don't get used on
>  Fedora bugs by mistake. 
Yes, I agree these were never clearly defined on the wiki page and I 
can't remember why, though even now I'm wondering how important it is 
that we use the right reason and what we would use it for.

>  It would really be nice, in fact, if we could have Bugzilla only
> show
>  the statuses and resolutions appropriate to the product the bug is filed
>  on...not sure if that's possible, though. 
I can ask the Red Hat bugzilla team about this.

John

[Kevin Kofler]

Adam Williamson wrote:
>  We do, however, need to define what 'cantfix',
> 'wontfix', 'notabug',
>  and 'worksforme' are for. 
I've already mentioned what I use WORKSFORME for.

For the others, I thought it was pretty clear:
* CANTFIX = This is a bug, but it is impossible to fix in Fedora. Most
commonly because the bug is in some third-party (usually binary-only)
component which we do not even ship, e.g. the nvidia driver.
* NOTABUG = This is not a bug, but an intentional design feature.
* WONTFIX = Anything we won't fix for some other reason, e.g. because the
affected Fedora release(s) is/are no longer supported.

        Kevin Kofler

[Adam Williamson]

On Thu, 2009-04-02 at 13:29 -0700, Adam Williamson wrote:

>  > I think it is great you want to tackle and clarify these things.
>  Having 
>  > gone through a round myself with this process I guess I learned that 
>  > some ambiguity wasn't as harmful as I first thought. :)
>  
>  You're right, of course. Somehow I'd forgotten about that flow.
>  
>  So, I will revise the draft substantially. :) Here's my quick thoughts: 
I've revised my thinking on this again. It now seems to me that we don't
really need to discuss much. All we need to do is flesh out the existing
page to explain all the resolutions, and specifically state which
statuses and resolutions don't apply to Fedora, to avoid confusion with
things like 'NEXTRELEASE' and 'CURRENTRELEASE'.

The 'technical' fix - don't display statuses and resolutions which don't
apply to the product the bug is in - would be nice too, if we can get
it. I'll try and follow that up.

The only thing that still needs to be discussed is Priority and
Severity, as we genuinely don't currently have a policy or procedure for
those. So I'll write a new draft of a mail covering only that issue, and
revise the Wiki as outlined above.

Thanks all.
-- 
Adam Williamson
Fedora QA Community Monkey
IRC: adamw | Fedora Talk: adamwill AT fedoraproject DOT org
http://www.happyassassin.net