On Sat, Jul 05, 2008 at 02:56:30PM +0200, Thorsten Leemhuis wrote:
John (CCed), I really appreciate your work in the wireless area and
would like to use the opportunity to say "thanks for all you work", as
support for WLAN hardware in the Linux kernel improved a lot in the
upstream kernel and Fedora thx to your (and other linux wireless
developers) work over the last two years.
Thanks. Normally my reward is to be kicked in the teeth when something
breaks -- usually by the same people who would be screaming "but
this is already fixed upstream" if I didn't push very recent patches,
but I digress...
But we now for at least the second time in the past few weeks
a more-than-minor wireless breakage in a Fedora kernel for a released
distro (bug #453390 now; http://lwn.net/Articles/286558/
the one some weeks ago; I think there was one more breakage not that
long ago, but I can't remember). I and many users (see for example
#453390) got hit by those problems. That's why I was wondering: what are
we at Fedora doing to prevent similar problems in the future?
For the record, bug 453390 was caused by me screwing-up a fixup for
build breakage between the current wireless code and the base 2.6.25
kernel in F9. In other words, this was not the result of merging some
buggy upstream patch. Instead it was the result of simple human error
on my part. I would also point-out that cherry-picking individual
fixes often means rebasing that fix between upstream and the target
kernel, and doing so creates an opportunity for such "human error"
mistakes to creep-in _with_every_single_fix_.
If we want to cherry-pick individual wireless LAN fixes into Fedora
kernels then we need another monkey. I already spend more time than
I should spare keeping Fedora kernels up-to-date as it is. But all
the smiling faces are my reward...
Three things spring to my mind and I just propose then here for
discussion; maybe something good comes out of it in the end:
- a karama of "+3" in bodhi seems not enough for a auto-move from
testing to stable (or even worse: straight to stable if enough people
tested the kernel and gave their +1 after the update got filed in bodhi
but *before* it actually hit fedora-testing) if there are no other
pressing issues (like security fixes). The kernel is a to complex beast;
more then 3 people should be needed to give a +1. And a bit of time
needs to pass to give enough people the opportunity to install, test and
report problems with new kernels. For the latest kernel it seems to me
that "to less time" really was the problem, otherwise the problem from
#453390 would have been noticed earlier
Something is definitely broken here. I seem to recall beating the
drum for Karma in the not-too-distant past, when the required number
seemed to be up in the teens? Who's bright idea was it to bring
this value down to +3? My assumption had been that it was okay to
push these wireless bits because Bodhi would keep us from releasing
truly broken kernels. If we are going to use +3 then my assumption
is clearly wrong and my practices have to change.
- should we separate security updates and other kernel fixes in a
way to make sure those "other fixes" get proper testing before they
get send out to the users?
Sounds good, but I have no idea how to do that. Does Fedora need a
"z-stream" a la RHEL?
- John, having all those pending and not-yet-upstream-merged
improvements for wireless hardware in the Fedora kernel was something
good in the past when WLAN support in the kernel was quite
bad/incomplete. But the main and most important bits for proper wireless
hardware support seem to be in the upstream kernel now; sure, there will
always be improvements in the queue, but that's the same in most other
linux subsystems with drivers as well. So I'm wondering: isn't it time
now to finally stop shipping all those wireless-next bits (currently
quite some big patches; see:
-rw-rw-r-- 1 thl thl 2484 14. Mär 17:06
-rw-rw-r-- 1 thl thl 39874 4. Jul 22:21 linux-2.6-wireless-fixups.patch
-rw-rw-r-- 1 thl thl 2656652 4. Jul 22:21 linux-2.6-wireless-pending.patch
-rw-rw-r-- 1 thl thl 4165718 4. Jul 22:21 linux-2.6-wireless.patch
) in released Fedora Version (e.g. 8 and 9 currently) when we start
Perhaps it is worth explaing a bit about what these are:
-- linux-2.6-wireless.patch contains the stream of wireless patches
going from the base kernel's release (currently 2.6.25) and the next
upstream release (currently 2.6.26);
-- linux-2.6-wireless-pending.patch contains the stream of wireless
patches from the next upstream release to the following release
-- linux-2.6-wireless-fixups.patch contains the changes required to
un-break the build after applying the previous two patches (a screw-up
here caused bug 453390).
So if you dropped the -wireless patch (i.e. the current contents
of the -pending patch) when F9 picks-up 2.6.26, you would be going
backwards in your wireless support.
FWIW, this practice started in Rawhide ages ago. Then it continued
into F7 or so because wireless continued to lag expectations.
But at some point things got enough better for enough people that
now my judgment is widely questioned. So I've been thinking for
some time that the process should change, but I was looking for a
So, here is what I propose:
-- continue the current practice until 2.6.26 is released and the
2.6.27 merge window closes;
-- after that, continue the current practice for updating rawhide; but,
-- once F9 (and presumably F8) move to 2.6.26, move the -pending bits
to the -wireless.patch and do _not_ create a new -pending.patch for
-- once 2.6.27 is released, drop the -wireless patch and F9/F8 will
get no more wireless updates at all;
-- F10 will release with -wireless and -pending patches inherited from
rawhide, but they will "age out" following the process described for
The alternative would be to simply remove the -pending (and possibly
-wireless) patches from F10 early after it is cut, but that seems
jittery to me.
I will defer to the judgment of the group -- as I've said I spend
a lot more time keeping Fedora up-to-date than I would like to
be doing. Just don't expect me to transfer that effort over to
tireless cherry-picking of fixes, for I just do not have the time.
P.S. This quote is based on ignorance by the person being quoted:
On Sat, Jul 05, 2008 at 05:50:52PM +0200, Thorsten Leemhuis wrote:
Or, to abuse some words from someone else in the discussions around
separately packaged kernel modules for Fedora: "If they [the patches in
this case] are not good enough to get applied upstream why should they
be good enough for us?" There is a reason for the short merge window and
the longer stabilization phase upstream.
None of the patches in question meet this definition. They are all
queued for upstream. Some of them are queued for the next upstream
release rather than the current one due mostly to the upstream release
process's scheduling requirements. But they are all upstream material.
The only patch I've added to Fedora that does not meet this definition
is the one for the at76_usb driver, which is a special case. I'm happy
to discuss at76_usb separately if someone so wishes.
John W. Linville