Investigation of the F23 mass rebuild

[Matthew Miller]

On Thu, Jul 02, 2015 at 01:47:11PM -0400, Adam Jackson wrote:
> > I agree. What can and should we do about it?
> Good question.  I'm not entirely sure, but I have opinions.

That's a good start... we can build from there in to plans.


> The binaries-in-/usr/share/doc thing is the sort of clearly obviously
> wrong thing that, ideally, would get your build rejected.  I would have
> hoped AutoQA or similar would be sufficiently potent for this by now.

I think that it probably is powerful enough, but to my knowledge we
haven't actually _enabled_ any sort of gating like this.

[...]


> Common to all of this is a certain reactive posture.  There's not a
> dashboard view of "sick packages".  Which could be useful along a number
> of axes, really.  How far behind is a package relative to its upstream's
> releases?  For a given sick package, how many packages depend on it?
> How idle has pkg git been relative to the incoming bug rate for a
> package?  The data exists, but we're not looking at it.  Obviously not
> all metrics are going to be comparable across packages, maybe for the
> kernel we want more of a moving average than a raw counter.

That seems useful. "Package Janitors' Dashboard" or something.

Hand-in-hand with reactive posture is our "high wall, chaos-filled
inside" model for enforcing the packaging guidelines. Package review
often consists of nitpicking with a fine-toothed comb, because except
for very egregious situations, that's the only time package quality is
ever looked at. 

This causes two undesirable situations: 

First, there's no way to let a "good enough" package in and have it
progress to excellent — it must be excellent at the start, because we
generally don't trust the package quality to do much but go down.
That's not to say that many packagers *don't* improve the quality of
the packages over time — I know I try to for the few I maintain
whenever I notice something, and I know many others do too. But there's
no regular mechanism for it, let alone accountability.

Second, once something is in, it can actively get worse. I can get a
package through package review 100% clean, and then the next day go and
edit it to do all sorts of horrible things, and odds are really good
that no one will call me on it. This generally isn't due to malice, but
the rules are complicated and the guidelines long — it's easy for all
but the most committed packagers to mess up simply by not being aware
of the right way to handle a situation.



> It's also worth remembering that the more developers we have, the fewer
> are generalists.  Clearly we don't have someone routinely inspecting
> packages to ensure CFLAGS is set properly, for example.  More to the
> point, when we do have systemic issues, we don't have people able or
> willing to dive into arbitrary packages and fix them, and we certainly
> don't have anyone _tasked_ with that.

Another example, not even considering fixing things that are broken:
sometimes there are big improvements we can make based on upstream
changes that just don't happen. Something that came up on the users'
list a few months ago is systemd's service restart behavior. Our
guidelines say:

  If you package a long-running service, please consider enabling
  systemd's automatic restart feature for it, to improve reliability by
  making sure the system automatically attempts recovering a failing
  daemon.

and I'm pretty sure this would be the best setting for most daemons we
ship. But, short of someone deciding to filing a change and signing up
a proven packager or mass-filing bugs, there's no one who has the
general responsibility of improving all the packages.



-- 
Matthew Miller
<mattdm at fedoraproject.org>
Fedora Project Leader