Product lifecycles and kernel impacts

[Josh Boyer]

On Thu, Nov 21, 2013 at 12:04 PM, Stephen Gallagher <sgallagh at redhat.com> wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> On 11/21/2013 09:18 AM, Josh Boyer wrote:
>> Hi All,
>>
>> I've seen a lot of discussion around release cycles and lifetimes.
>> I've not heard any specific requests for what impacts any of the
>> current ideas would have to the kernel, but that doesn't mean they
>> won't be coming.  We should probably start thinking about various
>> things we can support today, what we could possibly support in the
>> future, and what we'd need to do it.  I've CC'd the Product
>> liaisons so they can see this.  Please keep them on CC.
>>
>> The first scenario I can think of is probably some kind of
>> "long-term" release.  Exactly what long-term means here is
>> undefined, but we already support a Fedora release for 13 months or
>> so.  We do that today with rolling kernel releases.  We could
>> possibly continue, but I expect the Products to desire something
>> more "stable".
>>
>> What that implies is that we'd likely need to base on an upstream
>> longterm kernel.  That means:
>>
>> 1) No new kernel features. 2) No new hardware enablement outside of
>> things acceptable for an upstream longterm commit (basically just
>> adding a new USB/PCI id to an existing driver). 3) Bugfixes will
>> primarily come from the upstream longterm tree 4) There will be no
>> alternative kernels supported on the Fedora longterm release. 5)
>> There are no guarantees on bugfixes, response time, etc.
>>
>> If any of the above are desired, people are better off picking an
>> Enterprise distro where you have contracted support.
>>
>> So if that's the case, the feasibility of this is going to hinge a
>> lot on timing.  I would think from a kernel perspective we'd know
>> up-front when a Fedora longterm was going to happen, and we'd try
>> and align that release with the newest official longterm stable
>> kernel.  Those are maintained for 2 years, which seems to be a good
>> fit for a Fedora longterm release.  Any longer than that and you're
>> back in the Enterprise space again.
>>
>> The second scenario I can think of is products with mismatched
>> release cycles.  E.g. Server doing a 2 year longterm and developing
>> the Server.next release on top of Base, which releases every 6
>> months.  Or Workstation doing a release once a year while Base
>> moves every 6 months.  Etc.  It's hard to come up with a concrete
>> scenario since none of the products have gotten this far.  It's
>> worth noting that FESCo is basically disallowing this for the first
>> releases, so this is a secondary concern.
>>
>> I would propose that for Base, we continue our current method of
>> rebasing with each kernel release.  That has been working very
>> well for the past few Fedora releases, and gets us the most "bang
>> for the buck" in terms of features, hardware support, and fixes. A
>> year long release for e.g. Workstation would likely also desire the
>> newer support and features, so they could probably just update
>> their kernel in the same way.  This keeps our maintenance costs
>> down to today's and gives products flexibility.
>>
>> In the Server LTS + Server.next development scenario, I would
>> expect the LTS to use the longterm kernel as suggested above which
>> means whenever they cut over to LTS mode it would need to coincide
>> with an upstream longterm kernel.  That would be something we have
>> to watch and plan for.  Server.next would continue to use Base
>> until it was ready to cut over and repeat.  I hope that makes
>> sense.
>>
>
> Would you believe I was typing up an email to you earlier today (that
> I didn't finish) to suggest this exact thing?

I would believe it.  It's almost like I read the WG lists and pay
attention to stuff coming up there or something.

> My thought was that we would naturally tie the Fedora Server release
> cycle to the Kernel LTS cycle, as you suggest. This should keep the
> maintenance costs on the kernel down.

Down, yes.  Not non-existent but smaller.  And to be clear, I'm
referring to the official longterm kernels that GregKH supports and
uses for his LTSI stuff.  He commits to doing those for 2 years and so
far has done a good job with 3.0 and now 3.10.  In the past, others
have volunteered to do longterm kernels and they generally fail
because they lose interest.  We'll have to keep an eye on this because
if that becomes a problem, we aren't staffed to carry a longterm by
ourselves.

> I suspect that Server is the only one of the three products for which
> the LTS kernel makes sense, though. Both the Workstation and Cloud
> projects are likely going to want to take advantage of newer
> capabilities far more often.

Probably.  In fact, anything more than one Product doing an LTS would
really make me nervous unless it was using the same cycle and kernel.
I don't think doing disjoint cycles with different kernels for LTS is
feasible.

>> In terms of people, the only major change I see would be the
>> longterm addition.  That might be something we can tuck, but having
>> additional QA and maintainer resources would allow us to cover
>> development and support better.  If there was any major deviation
>> in which kernel was used in the various products, we'd likely need
>> either the Product teams themselves to pick up maintenance or
>> additional people on the kernel team to handle that.  The desire
>> here is to keep the kernel common among as many Products and
>> releases as we can.
>>
>
> What do you think you would need for a resource here, if we make the
> following assumptions (not final, just ideas):
>
> 1) The Fedora Server stable release is the only Product running the
> LTS kernel
>
> 2) We limit updates to the kernel package in the stable release to a
> regular schedule (excepting only security fixes). See my lifecycle
> proposal plan for how I suggest we might want to do Fedora Server 1.1,
> 1.2, etc. releases. If we limited kernel patch roll-ups to a six-month
> cycle, would that reduce your resource needs?

I'm not quite following you on this part.

If Server is doing an LTS, it follows the latest upstream longterm
kernel.  We agree there.  Which means that whenever upstream longterm
does a release (which is actually pretty regular), we do a kernel
update containing that.  E.g. 3.13.1, 3.13.2, 3.13.3, etc.  Those are
typically on the order of 1-3 weeks between releases.  Not months.
They also contain more than security fixes, but they are limited to
_fixes_ not features.  I would think that's acceptable.

So were you referring to Server.next (aka the development version)
with 6 month rollups?  I would think Server.next would just roll with
whatever is in Base until it cut off.  If you're talking about a
planned 6 month update bundle for the LTS of some kind, then I guess
you could do that.  I don't think it would change how often we update
the LTS kernel in koji and whatnot.  If you want bundled updates, we'd
probably just build the kernel and let the Server team pick it up in
whatever update form they want.

So from a maintainer resource standpoint, it would essentially be
another release to do with and a different kernel but mostly
hands-off.  If upstream lost interest in that particular longterm for
whatever reason, we'd likely have to pick up that support and we can't
currently do that.  Having another maintainer would help.  From a QA
standpoint, I'd probably push that back to the Product (Server) that
wanted to do the LTS, particularly if that Product wants to do some
kind of 6month roll-up release.  Make sense?

josh