When will we stop shipping WLAN improvements ahead of upstream in released Fedora version?

[John W. Linville]

On Sat, Jul 05, 2008 at 02:56:30PM +0200, Thorsten Leemhuis wrote:

> John (CCed), I really appreciate your work in the wireless area and 
> would like to use the opportunity to say "thanks for all you work", as 
> support for WLAN hardware in the Linux kernel improved a lot in the 
> upstream kernel and Fedora thx to your (and other linux wireless 
> developers) work over the last two years.

Thanks.  Normally my reward is to be kicked in the teeth when something
breaks -- usually by the same people who would be screaming "but
this is already fixed upstream" if I didn't push very recent patches,
but I digress...

> But we now for at least the second time in the past few weeks had/have
> a more-than-minor wireless breakage in a Fedora kernel for a released 
> distro (bug #453390 now; http://lwn.net/Articles/286558/ is discussing 
> the one some weeks ago; I think there was one more breakage not that 
> long ago, but I can't remember). I and many users (see for example 
> #453390) got hit by those problems. That's why I was wondering: what are 
> we at Fedora doing to prevent similar problems in the future?

For the record, bug 453390 was caused by me screwing-up a fixup for
build breakage between the current wireless code and the base 2.6.25
kernel in F9.  In other words, this was not the result of merging some
buggy upstream patch.  Instead it was the result of simple human error
on my part.  I would also point-out that cherry-picking individual
fixes often means rebasing that fix between upstream and the target
kernel, and doing so creates an opportunity for such "human error"
mistakes to creep-in _with_every_single_fix_.

If we want to cherry-pick individual wireless LAN fixes into Fedora
kernels then we need another monkey.  I already spend more time than
I should spare keeping Fedora kernels up-to-date as it is.  But all
the smiling faces are my reward...

> Three things spring to my mind and I just propose then here for 
> discussion; maybe something good comes out of it in the end:
> 
> - a karama of "+3" in bodhi seems not enough for a auto-move from 
> testing to stable (or even worse: straight to stable if enough people 
> tested the kernel and gave their +1 after the update got filed in bodhi 
> but *before* it actually hit fedora-testing) if there are no other 
> pressing issues (like security fixes). The kernel is a to complex beast; 
> more then 3 people should be needed to give a +1. And a bit of time 
> needs to pass to give enough people the opportunity to install, test and 
> report problems with new kernels. For the latest kernel it seems to me 
> that "to less time" really was the problem, otherwise the problem from 
> #453390 would have been noticed earlier

Something is definitely broken here.  I seem to recall beating the
drum for Karma in the not-too-distant past, when the required number
seemed to be up in the teens?  Who's bright idea was it to bring
this value down to +3?  My assumption had been that it was okay to
push these wireless bits because Bodhi would keep us from releasing
truly broken kernels.  If we are going to use +3 then my assumption
is clearly wrong and my practices have to change.

> - should we separate security updates and other kernel fixes in a better 
>  way to make sure those "other fixes" get proper testing before they 
> get send out to the users?

Sounds good, but I have no idea how to do that.  Does Fedora need a
"z-stream" a la RHEL?

> - John, having all those pending and not-yet-upstream-merged 
> improvements for wireless hardware in the Fedora kernel was something 
> good in the past when WLAN support in the kernel was quite 
> bad/incomplete. But the main and most important bits for proper wireless 
> hardware support seem to be in the upstream kernel now; sure, there will 
> always be improvements in the queue, but that's the same in most other 
> linux subsystems with drivers as well. So I'm wondering: isn't it time 
> now to finally stop shipping all those wireless-next bits (currently 
> quite some big patches; see:
> -rw-rw-r-- 1 thl thl    2484 14. Mär 17:06 
> linux-2.6-ms-wireless-receiver.patch
> -rw-rw-r-- 1 thl thl   39874  4. Jul 22:21 linux-2.6-wireless-fixups.patch
> -rw-rw-r-- 1 thl thl 2656652  4. Jul 22:21 linux-2.6-wireless-pending.patch
> -rw-rw-r-- 1 thl thl 4165718  4. Jul 22:21 linux-2.6-wireless.patch
> ) in released Fedora Version (e.g. 8 and 9 currently) when we start 
> shipping 2.6.26?

Perhaps it is worth explaing a bit about what these are:

-- linux-2.6-wireless.patch contains the stream of wireless patches
going from the base kernel's release (currently 2.6.25) and the next
upstream release (currently 2.6.26);

-- linux-2.6-wireless-pending.patch contains the stream of wireless
patches from the next upstream release to the following release
(currently 2.6.27);

-- linux-2.6-wireless-fixups.patch contains the changes required to
un-break the build after applying the previous two patches (a screw-up
here caused bug 453390).

So if you dropped the -wireless patch (i.e. the current contents
of the -pending patch) when F9 picks-up 2.6.26, you would be going
backwards in your wireless support.

FWIW, this practice started in Rawhide ages ago.  Then it continued
into F7 or so because wireless continued to lag expectations.
But at some point things got enough better for enough people that
now my judgment is widely questioned.  So I've been thinking for
some time that the process should change, but I was looking for a
graceful transition.

So, here is what I propose:

-- continue the current practice until 2.6.26 is released and the
2.6.27 merge window closes;

-- after that, continue the current practice for updating rawhide; but,

-- once F9 (and presumably F8) move to 2.6.26, move the -pending bits
to the -wireless.patch and do _not_ create a new -pending.patch for
2.6.28 bits;

-- once 2.6.27 is released, drop the -wireless patch and F9/F8 will
get no more wireless updates at all;

-- F10 will release with -wireless and -pending patches inherited from
rawhide, but they will "age out" following the process described for
F9/F8 above.

The alternative would be to simply remove the -pending (and possibly
-wireless) patches from F10 early after it is cut, but that seems
jittery to me.

I will defer to the judgment of the group -- as I've said I spend
a lot more time keeping Fedora up-to-date than I would like to
be doing.  Just don't expect me to transfer that effort over to
tireless cherry-picking of fixes, for I just do not have the time.

Thanks,

John

P.S.  This quote is based on ignorance by the person being quoted:

On Sat, Jul 05, 2008 at 05:50:52PM +0200, Thorsten Leemhuis wrote:

> Or, to abuse some words from someone else in the discussions around 
> separately packaged kernel modules for Fedora: "If they [the patches in 
> this case] are not good enough to get applied upstream why should they 
> be good enough for us?" There is a reason for the short merge window and 
> the longer stabilization phase upstream.

None of the patches in question meet this definition.  They are all
queued for upstream.  Some of them are queued for the next upstream
release rather than the current one due mostly to the upstream release
process's scheduling requirements.  But they are all upstream material.
The only patch I've added to Fedora that does not meet this definition
is the one for the at76_usb driver, which is a special case.  I'm happy
to discuss at76_usb separately if someone so wishes.

John
-- 
John W. Linville
linville at redhat.com