Reporting bugs upstream

[Mike A. Harris]

John Ellson wrote:

>> When I find a bug in fedora and it's obviously a bug upstream I report
>> it there. But do we have to report them in fedora bugzilla too and
>> reference the upstream bugreport? So if an other users files a bugreport
>> for the same bug and doesn't check upstream maintainer of the module
>> know it's already filed upstream. Or do we have to notify the maintainer
>> in other way?
>> I've searched the fedora wiki for this but I can't find any information
>> about this.
> 
> I think that Fedora users need a single point-of-contact for bug reporting.
> 
> Its tough enough to report bugs in the first place and the user can't be 
> expected
> to know all the internals of every package to know if its a Fedora 
> packaging bug
> or an upstream bug.

This particular topic is one in which it is extremely unlikely that
everyone will ever remotely agree upon, since different people are
going to look at the problem first and foremost, from the position
of how the "policy", affects themselves personally, wether there
a policy that is defacto, implied, explicit or otherwise.

Some people think very strongly and adamantly that "I am using Fedora
Core, and so I will report bugs in Fedora Core bugzilla and that
alone."  They have their own personal reasons which vary, but over
time I've witnessed people express their reasoning similar to what
you've described above.  More or less a "one stop shopping" viewpoint.

 From the viewpoint of getting an actual problem solved however, this
is not always the best way to report bugs.  There is no single rule,
and probably never will be however, so trying to apply a single piece
of logic to apply to all parts of the distribution, to all developers
and all packages, wont ever work.  Some of the software is actually
owned and maintained by Red Hat folk, directly at Red Hat, while other
stuff is maintained by Red Hat folk but in an upstream project.  Other
stuff is maintained solely by the upstream project, with a Red Hat
person or group of people also contributing occasionally to the upstream
project, and often fixing things in Red Hat bugzilla too.


 > Its incredibly frustrating and demoralizing to report a bug to Fedora,
 > only to have the Fedora maintainer close the bug with the response
 > that the user (what a dummy!) should have reported it upstream!
 >
 > Yes, this has happened to me.

It has happened to all of us I would suspect.  Again, because quite
often, the person experiencing a given problem finds Red Hat bugzilla
to be a known central location, where they may already have an account
set up in bugzilla, and it is simply the shortest path to ground.

On top of that, quite often when someone experiences a problem in a
particular piece of software, they may be quite upset and/or
frustrated about the issue.  Frustrated enough to file a bug report
to hopefully see the issue eventually fixed by someone, but also
quite often hoping to expend the absolute minimal amount of effort
to report the problem _anywhere_ as soon as possible, to make it
someone elses problem real fast, and wash their hands of it.

Quite often, that approach has some serious flaws.  While it is
perhaps the fastest way to report an issue and wash one's hands of
the problem, reporting a bug in Red Hat bugzilla, regardless of the
package it is in, etc. it very definitely is not always the fastest
way to get a solution to a particular problem.  Sometimes it takes
a bit more effort on the part of the reporter to report the issue
to the best place (or number of places) possible, to increase the
chances of the best people seeing the problem and having a chance
to fix it sooner rather than later.

While Red Hat employees do maintain all of the packages in the OS,
they are not necessarily experts in every aspect of the code in
every single package they maintain.  The kernel for example contains
a massive amount of code, and while we have a large number of kernel
engineers all of whom are kernel experts to one degree or another,
there are likely several pieces of the kernel which we do not have
an official "expert" in a given area.  While one or more of the kernel
guys may be able to theoretically dig into a given issue, they're
more likely to either leave the problem to someone else who _is_ more
of an expert in the given area, or to suggest to the reporter to
report the issue to the upstream maintainer of that area of the
kernel.

The same can be said for various other parts of the OS, but in
particular some of the extra large pieces, such as the kernel, X,
GNOME, KDE, OpenOffice, etc..


 > I propose that it be a policy that Fedora maintainers are themselves
 > responsible for forwarding the bug upsteam if necessary.

That does sometimes happen, but it is not something a bug reporter
should expect to happen.  Red Hat engineering manpower is a limited
resource, and our man hours are prioritized across quite a number
of tasks.  The amount of work that any given engineer "has" to do,
plus the amount of work that they could "potentially" do (such as
various random bugzilla bug reports), plus various meetings, and
other things that fill the day, is generally many times more than
the hours available.  For example, if we were to have 100 engineers
magically fall out of the sky and start fixing random bugs in
bugzilla, our existing engineers would still continue to have a full
workload, the new 100 people would have a full workload, and there
would easily be 10 times more work than that left to do, to which
there is nobody.

That means that as bugs land in Red Hat bugzilla, they simply
build up endlessly, and while the number of engineers increase over
time, the rate of bugs incoming is easily much greater than the
increase in engineer man hours as new hires come on board.

The end result, is that some bugs, will simply sit there and rot
forever, because there is not enough manpower to handle them all.
So, either development stops completely while everyone drops what
they are doing to go fix incoming bugs, and go report users bugs
to the proper upstream location acting like a proxy, or something
else has to give.

Another problem, is that even if there was a policy that Red Hat folk
had to report all bugs upstream personally like you suggest, that
causes some additional problems:

1) Some bugs are simply crackrock.  Sometimes the developer can discern
    that, however sometimes that is not possible for the developer to
    easily determine, in particular if it is outside of their personal
    area of expertise.

2) Some bugs are not directly reproduceable, and may not even be a bug.
    If the developer reports such issues upstream themselves, and does
    so often, and they turn out to be red herrings, it makes the
    _developer_ look like an ass.  The upstream developers are likely
    to be counting how many useless bugs Red Hat developer A reports
    upstream, as if it was the Red Hat person themselves having the
    problem.

3) It disconnects the 2 way communication link between the person
    having the problem (the original reporter) and the person looking
    into the problem (the upstream developer).  The upstream developer
    then may ask for more information or to test something.  Then there
    is a potential time lag between that occuring, the Red Hat developer
    noticing the bug email, and having time to respond to it, and to
    cut and paste the upstream maintainer's recommendations/questions
    into the Red Hat bug, and wait for the original reporter to respond
    again.  The number of hops that occur during the problem diagnosis
    is increased by one - the Red Hat person who ends up wasting their
    valuable and limited time being a proxy for a bug that may end up
    turning out to not be a bug at all, or may be a configuration
    mistake the user made or something.


This last issue is IMHO the most important.  A person reporting a bug
absolutely needs to be directly involved in 2 way communication with
the person or people who will be directly investigating the issue in
many (but not all) cases.  In particular this is true if the issue
in question is highly transient and/or unreproduceable, or only occurs
with rare/obscure hardware, or in complex circumstances that are
difficult or unlikely to be something to set up locally to diagnose
(or to justify the man-hours to the problem).

By reporting a problem directly to the bugzilla(s) and/or mailing list
or list(s) that are most likely to be seen by the largest number of
developers who might be able to solve a given problem, you establish
a direct 2 way communication link with the people who are most likely
to be able to help you, and to do so sooner rather than later, and you
also make the problem solving equation become a scalable quantity.

On the other hand, by reporting a bug in Red Hat bugzilla and washing
your hands of the issue, while that makes life easiest for you to
report it and forget about it, depending on the nature of the problem,
the Red Hat engineer determined severity and impact of the issue, the
priority of your issue in the eyes of the engineer compared to all
other issues on that person (or group)'s plate, may get your issue
investigated right away, perhaps next week, or if it is deemed lower
priority than a whackload of other stuff on everyone's plate, your
issue may simply sit in bugzilla and rot until a new OS release comes
out.  No, this doesn't sound wonderful, but it is a blunt reality
that there aren't enough people to investigate every single little
problem someone tosses into our bugzilla, so some get investigated,
and others sit there.  Report the problem to the place that is
most likely to get a quicker response (quite often the upstream
project), and your problem is more likely to get solved sooner
than later, even if it takes more effort on your part.

My take is that there will always be some people out there who take
a hard line view that they should never have to report something to
anywhere but Red Hat.  Nobody, including myself is likely to convince
all people out there otherwise, and so I wont try to.  I merely state
the facts, that if you solely rely on Red Hat bugzilla to report every
single problem you have, then you also have to accept the fact that
your problem is going to be fed through the available manpower resources
pop-straw, and individual and group prioritization inherent in our
jobs, which may get your issue investigated, or may leave it sitting
in bugzilla forever, or perhaps you'll be asked to report the issue
to a more appropriate place.

So, people will be people, and report whatever they want, to wherever
they want.  That is the nature of things.  And Red Hat folks will do
our jobs as we're paid to do, and prioritize things as best we can
realizing we can only solve a fraction issues compared to all that
end up getting reported.  Since we want to see more issues fixed for
people, we will try to help things scale by telling people to report
issues upstream, and leaving it up to them in some cases, and in
other cases doing it ourselves.  There are no hard rules to the
bug reporting game, and since each bug is different, each software
package is different, each project different, and each maintainer
and group is different, everyone will experience a variety of
different results.   ;o)

My words of advice to those that truly care about their bugs, is to
report it to the place that gives the highest likelyhood of hitting
the most people who are likely to be able to solve your problem.
Beyond that however, I certainly can't twist your arm, nor would
I want to.  Do whatever you prefer, and we'll do what we need to
do at the end of the day as well.



-- 
Mike A. Harris  *  Open Source Advocate  *  http://mharris.ca
                       Proud Canadian.