bugzilla.redhat.com vs upstream bug trackers

[Jeffrey Ollie]

OK, so this post is going to be rather long and rambling, and
hopefully respectful, but I'm passionate about this subject (and
Fedora).  The tl;dr summary is that there shouldn't be a single
standard for what we expect of packagers, especially in the context of
what to expect when bugs are filed against their packages on Red Hat's
bugzilla.  My view is that expectations are going to depend on the
criticality of the package to Fedora (kernel, installer, default
desktop stack) and the packager (are they being paid to maintain the
package in Fedora or are they volunteering their time).

Also, where some of us seem to be at odds is the definition of "proper
maintenance" for packages.  My definition:

1) Ensure that packages meet all of the packaging guidelines.
2) Fix packaging related bugs in a timely manner.
3) Incorporate new upstream releases, in accordance with packaging
guidelines (e.g. packages shouldn't be updated to a new major version
in a released version of Fedora).
4) Incorporate patches that fix security issues or critical bugs.

In my view these expectations imply that a packager has some
involvement with upstream.  I think that the level of involvement is
going to depend on the packager and the package.  I don't think that a
hard and fast rule can be developed to cover every case without
becoming ridiculously long and complex.  Other than generally
encouraging packagers to become involved with upstream development it
should be left up to the conscience of the packager.

In no way should packagers be expected to provide end-user support for
packages, be an expert in every aspect of a package, or be expected to
work with upstream to debug issues because the end user is unwilling
to do the work themselves (in essence becoming an upstream developer
themselves).

OK, so with that out of a way, I'm hopefully going to respectfully (if
wordily) respond to Jóhann's comments.

On Mon, Jun 17, 2013 at 10:55 AM, "Jóhann B. Guðmundsson"
<johannbg at gmail.com> wrote:
> On 06/17/2013 03:42 PM, Jeffrey Ollie wrote:
>
>> I maintain packages in Fedora because it allows me to get what I want
>> to do done, whether at work or at home.  Since I've done the work of
>> making these packages, why not share them with the Fedora community?
>
> Because if you cannot properly maintain the component in the distribution
> the community is better of without it.
>
>> 1)  There's a 99.999% chance that I don't have the resources (either
>> hardware or software) to reproduce the bug.
>
> Then you should not be maintaining that component

In some cases you may be right.  But in most cases I was the only
person to step up and package a particular piece of software.  Also,
in most cases I'm willing to step aside as the owner of a package if
someone wants to take over the responsibility.  In every case I've
been willing to take on co-maintainers to help out with the packaging
duties.

>> 2) There's a 100% chance that I don't have the time between work and
>> family obligations.
>
> We do not need unresponsive or poor maintained packages in the distribution
> and if there is really need or demand for the component you maintain,
> co-maintainers will appear or people to pick it up if you drop it so if you
> dont have the time to properly maintain your component(s) in the
> distribution then either find yourself co-maintainers or drop the package.

Again, our key disagreement here is on the definition of "proper
maintenance".  I have more than enough time to keep up with new
versions as they are released (most of the time) or to pull a patch
out of the upstream's version control and add it to the package. But
even if I had the hardware I don't have the kind of time it takes to
set up test environments to reproduce a bug, and then dig into the
code and find the bug, develop a fix and then test it.

>> 3) Even though I'm an excellent programmer, well versed in C and
>> Python, and decent in Perl, Ruby, et. al.  I probably don't have the
>> familiarity with the codebase to even know where to start looking for
>> a bug.
>
> If you aren't familiar with the component you are packaging and maintaining
> why are you doing it et all?

Well, let's take Asterisk.  First off, there are a lot of components
that require specific (and expensive) hardware like ISDN & POTS
adapters.  And if I had the Asterisk-specific hardware I'd need ISDN
or POTS lines to connect to which would mean sending lots of my money
to the local telco or spending lots of money on other telephony
hardware to set up a lab environment.  Then you've got to worry about
country-specific setups.  ISDN and POTS lines work differently
depending on what country you're in, and I've only had minimal
experience with such lines in the US and none at all in other
countries.

Asterisk provides support many VoIP protocols, each with their own
quirks.  A number of them only talk to proprietary hardware (which I
don't own).   Even if you're using SIP, it's one of those protocols
whose specification is fuzzy enough that every implementation that you
come across does things a little differently especially when users
start reconfiguring things for their particular needs.

Then there are all of the modules that Asterisk includes for providing
functionality like voice mail or conference calling.  Some are fairly
simple, but many have complex configurations and can interact with
other parts of the system in non-trivial ways.

Oh, and did I mention that Asterisk has *two* domain-specific
programming languages built into it (they are probably Turing-complete
too).  And it embeds Lua, has an embedded HTTP server for exposing
REST APIs, as well as subsystems that expose other APIs.

So outside of Digium (the company that controls upstream development),
there's practically *no one* that has the resources or knowledge to
provide support at the level that you seem to be expecting, especially
for complex or subtle cases.  And even inside Digium there isn't a
single individual that can meet your standards, which is why Digium
employs quite a few people to handle support and development.

Digium itself hasn't seen fit to get involved with Fedora, at least
not in an official way.  There are at least a couple ex-Digium
employees that are involved with Fedora, but they have their own
jobs/lives/interests.  One has stepped up to be a co-maintainer but
I've been quick enough with updates that he hasn't had much to do yet.
 I'm appreciative nonetheless.

If you look at the history of the Asterisk package, I spent over a
year getting the package through the review process, including
packaging related dependencies and I've spent the past 7+ years
maintaining it.  At times I've fallen behind a bit in the updates but
I don't think that you can seriously question my long-term commitment
to maintenance of the package.

In the past I've written patches and even whole modules for Asterisk
and had them accepted upstream, although with the pace of Asterisk
development and the amount of time that's elapsed there's probably not
much evidence of that left in current versions.  Asterisk has had a
lot of it's core code restructured in that time, and even more so in
the next major version that's in development so much of what I knew
about the internals is no longer relevant.

I follow the Asterisk development and security lists (I don't look at
every code review that's posted, but I do look at a few) to see what's
coming in the next release or to get notified of security issues.  In
a few cases I've packaged up new security fix releases and gotten them
into -testing even before the Red Hat/Fedora security team can create
the bugs in bugzilla.  I've also met a number of Digium employees
IRL...

I guess that I could change the package to include only the parts of
Asterisk that I'm familiar with, but that seems like a rather
anti-social thing to do.

So by my standards, I'm the best person *right now* to maintain the
Asterisk package.  In the future someone else may show up that's
willing to do end-user support or upstream development in addition to
package maintenance.  A number of people find having Asterisk packaged
in Fedora useful (no idea how many) but if there's more than just me I
think it's worth having the package in Fedora.

>> 4) Most software is complex enough that even configuration problems
>> are best handled by upstream because I'd be familiar with a small set
>> of configuration scenarios, but everyone's situation is unique and
>> understanding what exactly a configuration option does (especially in
>> edge cases) often requires an understanding of the code behind it.
>>
>> All of this means that I'm a speedbump in the way of getting the bug
>> fixed, at least until there's a patch that needs to be applied to the
>> package, or a new release to upgrade to.
>
> It's component's owner responsibility to be in touch and contact with
> upstream and the man in the middle role of the packager/maintainer can never
> be entirely gotten rid of.

Playing "man in the middle" is inefficient.  Unless it's an issue with
packaging or Fedora-specific patches the reporter should work with the
upstream developers to identify and fix the issue.  Once a fix is
available then it's the package maintainers responsibility to pull
that fix into Fedora (either as a patch or a new release).  That way
the upstream developers can work directly with the user that is having
the problem and ultimately every distribution benefits from the bug
fix.  The package maintainer should certainly be kept in the loop so
that they know an update/patch is imminent.

I personally think that the level of involvement/contact that a
package maintainer needs to have with upstream depends on the
component.  For core components like the kernel or the Gnome stack
expecting there to be a package (or packagers) that are intimately
involved upstream is reasonable.  It's also reasonable to expect that
you're not going to find those people unless they're being paid for
the work because of the large time commitment needed (and sometime
financial commitment, e.g. for hardware to test on or to fund travel
to a conference).  It's nice when someone dedicates a large chunk of
their free time to packaging software and working with upstream but I
don't think that you can expect it.

For complex non-core packages (like Asterisk), I think it's reasonable
to expect a packager to keep an eye on upstream development so that
they're aware of what's going on (pending new releases, especially new
releases that break compatibility or provide major new features,
security issues, etc.)

For everything else, it's less clear cut.  Guidelines can be given but
every case is going to be different and we'll need to work with
packagers to help them do their best.

--
Jeff Ollie