The question of rolling release?

[Josh Boyer]

On Wed, Jan 25, 2012 at 9:17 PM, Bryan Quigley <gquigs at gmail.com> wrote:
> I can understand exceptions for Firefox (but you don't want to switch
> to the enterprise slow release right?), and Wine, but...
>
> I've read it several times and I don't quite understand the major
> kernel version bumps.  3.2.1 just got released to Fedora 16, yet it
> started with 3.1.0.

It's pretty simple, really.  Basically, if we don't keep the kernel on at
least a somewhat recent release the amount of work required to support
that release grows beyond what we can realistically maintain.

Let's use F16 as an example (which also applies to F15 but we'll stick
with one release for now).  The 3.1.y stable series is now EOL upstream,
which means that there will be no more upstream maintenance on it at all.
No auto-backports, no security fixes, nothing.  So if we were to stick
with 3.1.y for the entirety of F16, we'd be looking at roughly another
year and a half of using a kernel that upstream absolutely does not care
about.  The Fedora kernel maintainers would need to weed through all of
the commits going into 3.2.y (until it goes EOL) and 3.3 and 3.4 to
figure out which fix security issues, which fix issues that hit enough
people to warrant a backport, etc.  Then we'd need to backport them which
starts out easy enough but as time goes on becomes increasingly difficult.
The rate of change in the upstream kernel is ridiculous (and I mean that
in both good and bad ways) so once you get a couple of releases removed
from what we're using the "simple" fixes are heavily dependent on code
changes and subsystem reworks that went in prior to that.

You might be saying "well... isn't that your job?"  The answer is, yes.
We do backports all the time, for fixes, hardware enablement, security
issues, etc.  However, by rebasing frequently we're only having to do that
from usually one release away.  You might also be saying "well... doesn't
RHEL do this kind of stuff too?"  Sure, probably (I don't work on RHEL so
I don't really know).  The difference there is in applicable manpower.
There are significantly more people working on the RHEL kernel than there
are Fedora.  We have 2 engineers and some great heavy lifing from a
couple of wireless developers and a few others for topic specific bugs.
(Which reminds me, we have an immediate opening if any of this mini-novel
on kernel maintenance doesn't make you run away screaming and you want to
apply...)

One might point out that F14 stuck with the 2.6.35-longterm kernel.  The
"longterm" kernels are meant to be a focal point for a number of
developers and distros to work with to help with the manpower and
backporting issues.  On paper that sounds awesome.  In practice, it
rarely works.  2.6.35.y has known unfixed security issues and backporting
things to it was essentially an exercise in frustration at the end.  It
hasn't had a new release in a long time.  The only longterm series I can
think of that somewhat works is 2.6.32.y.  If you think about it for a
second you'll notice that the two largest Enterprise Linux distributions
started out based on.. 2.6.32.  Correlation does not equal causation, but
I can't honestly think of any other reason that kernel continues to
interest people.  Even that longterm kernel is about to change upstream
maintainers because it's getting really long in the tooth now.

So by rebasing frequently, we limit the amount of work required to
actually fix issues that our users see.  We also get new hardware
enablement from the newer kernels, and new features.  Perhaps most
importantly, we stay relevant to upstream so that when we find bugs in
their work it's still fresh in their memories and we can possibly shame
them into providing fixes instead of being laughed at for using a kernel
that is 2 releases old.

Hopefully that helps explain what we're thinking when we go about doing
what we do.  As usual, sorry for being overly verbose.

josh