Important kernel update should not break stuff

[Roman Kennke]

Hi folks,

Today something happened, that happens over and over again with Fedora,
and it makes me angry. I am running Fedora 17, and so far it worked well
with the initial kernel 3.3.x (except that it would panic on shutdown...
but that was not important to me, but still embarassing). Today I was
notified of an important security update in the kernel. Curiously, it
would update from 3.3 to 3.4 (a major version upgrade, which should not
happen in such a core package anyway, IMO). Reboot into the new kernel,
everything comes up --- until I want to actually want to read email,
surf web, or anything that requires my network. I am on an Intel Wifi
card, iwlwifi module. I *can* connect to the network, but everything is
suuuuuuper  slow or times-out every now and then. Completely unusable.
Reboot into the older kernel, things work well again. Now I am left with
the choice of running a new kernel w/o network or an unsecure kernel.
Thank you very much!

This sort of thing I would expect in rawhide/development builds, but not
in a supposed-to-be stable release. I can understand the underlying idea
of being on the bleeding edge, but I don't want to actually be bleeding.
At least the base system components should not undergo major version
updates. Security fixes should be backported to the software version
that is in the stable release (1 year release cycle shouldn't be too
demanding for this), and only security fixes and absolutely important
fixes should go into stable releases. (Not to mention that some fixes
that I *would* consider important enough to go into stable never end up
there.) If major version updates are really really necessary, they
should undergo serious testing. I cannot believe that I am the only one
on an Intel Wifi chip. The way it is now, Fedora feels like a constantly
rolling development version that is almost unusable (because any update,
even security, has a fairly high risk of breaking things) for day-to-day
work.

Bugzilla report:
https://bugzilla.redhat.com/show_bug.cgi?id=831571

Best regards,
Roman

[Roman Kennke]

>  Today something happened, that happens over and over again with
> Fedora,
>  and it makes me angry. I am running Fedora 17, and so far it worked well
>  with the initial kernel 3.3.x (except that it would panic on shutdown...
>  but that was not important to me, but still embarassing). Today I was
>  notified of an important security update in the kernel. Curiously, it
>  would update from 3.3 to 3.4 (a major version upgrade, which should not
>  happen in such a core package anyway, IMO). Reboot into the new kernel,
>  everything comes up --- until I want to actually want to read email,
>  surf web, or anything that requires my network. I am on an Intel Wifi
>  card, iwlwifi module. I *can* connect to the network, but everything is
>  suuuuuuper  slow or times-out every now and then. Completely unusable.
>  Reboot into the older kernel, things work well again. Now I am left with
>  the choice of running a new kernel w/o network or an unsecure kernel.
>  Thank you very much!
>  
>  This sort of thing I would expect in rawhide/development builds, but not
>  in a supposed-to-be stable release. I can understand the underlying idea
>  of being on the bleeding edge, but I don't want to actually be bleeding.
>  At least the base system components should not undergo major version
>  updates. Security fixes should be backported to the software version
>  that is in the stable release (1 year release cycle shouldn't be too
>  demanding for this), and only security fixes and absolutely important
>  fixes should go into stable releases. (Not to mention that some fixes
>  that I *would* consider important enough to go into stable never end up
>  there.) If major version updates are really really necessary, they
>  should undergo serious testing. I cannot believe that I am the only one
>  on an Intel Wifi chip. The way it is now, Fedora feels like a constantly
>  rolling development version that is almost unusable (because any update,
>  even security, has a fairly high risk of breaking things) for day-to-day
>  work.
>  
>  Bugzilla report:
>  https://bugzilla.redhat.com/show_bug.cgi?id=831571 
Since I just received an email in private pointing out that emails like
mine above might be discouraging and not helpful... let me apologize for
this. My intention is not to bash other people's best efforts, but
instead try to help out (otherwise I would not bother to diligently file
bugreports and mention my concerns on this list). I am willing to help
track down and fix the problem. However, I see a more general problem
and maybe we can turn this into a discussion how to address (or answer)
it.

- Why do we allow new major versions of core components into a stable
release? What sort of testing is performed before a major kernel update
hits Fedora stable?
- What is the policy with regards to risky changes (like unnecessary
feature updates, ABI changes, etc) in stable?
- How can problems like the one I described above be avoided? Is there
anything I and others can help with?

Roman

[Johannes Lips]

On Wed, Jun 13, 2012 at 1:00 PM, Roman Kennke <rkennke(a)redhat.com&gt; wrote:

>  > Today something happened, that happens over and over again with
> Fedora,
>  > and it makes me angry. I am running Fedora 17, and so far it worked well
>  > with the initial kernel 3.3.x (except that it would panic on shutdown...
>  > but that was not important to me, but still embarassing). Today I was
>  > notified of an important security update in the kernel. Curiously, it
>  > would update from 3.3 to 3.4 (a major version upgrade, which should not
>  > happen in such a core package anyway, IMO). Reboot into the new kernel,
>  > everything comes up --- until I want to actually want to read email,
>  > surf web, or anything that requires my network. I am on an Intel Wifi
>  > card, iwlwifi module. I *can* connect to the network, but everything is
>  > suuuuuuper  slow or times-out every now and then. Completely unusable.
>  > Reboot into the older kernel, things work well again. Now I am left with
>  > the choice of running a new kernel w/o network or an unsecure kernel.
>  > Thank you very much!
>  >
>  > This sort of thing I would expect in rawhide/development builds, but not
>  > in a supposed-to-be stable release. I can understand the underlying idea
>  > of being on the bleeding edge, but I don't want to actually be bleeding.
>  > At least the base system components should not undergo major version
>  > updates. Security fixes should be backported to the software version
>  > that is in the stable release (1 year release cycle shouldn't be too
>  > demanding for this), and only security fixes and absolutely important
>  > fixes should go into stable releases. (Not to mention that some fixes
>  > that I *would* consider important enough to go into stable never end up
>  > there.) If major version updates are really really necessary, they
>  > should undergo serious testing. I cannot believe that I am the only one
>  > on an Intel Wifi chip. The way it is now, Fedora feels like a constantly
>  > rolling development version that is almost unusable (because any update,
>  > even security, has a fairly high risk of breaking things) for day-to-day
>  > work.
>  >
>  > Bugzilla report:
>  > https://bugzilla.redhat.com/show_bug.cgi?id=831571
>
>  Since I just received an email in private pointing out that emails like
>  mine above might be discouraging and not helpful... let me apologize for
>  this. My intention is not to bash other people's best efforts, but
>  instead try to help out (otherwise I would not bother to diligently file
>  bugreports and mention my concerns on this list). I am willing to help
>  track down and fix the problem. However, I see a more general problem
>  and maybe we can turn this into a discussion how to address (or answer)
>  it.
>
>  - Why do we allow new major versions of core components into a stable
>  release? What sort of testing is performed before a major kernel update
>  hits Fedora stable?
>  - What is the policy with regards to risky changes (like unnecessary
>  feature updates, ABI changes, etc) in stable?
>  - How can problems like the one I described above be avoided? Is there
>  anything I and others can help with?
>
>  Roman
>
>
>  --
>  devel mailing list
>  devel(a)lists.fedoraproject.org
>  https://admin.fedoraproject.org/mailman/listinfo/devel
>  
I think the reason for shipping the latest upstream kernel is based on the
fact that backporting would be too much work.
http://fedoraproject.org/wiki/KernelRebases
Gives a good overview and probably prevents us from repeating arguments in
the discussion.

Johannes

[Roman Kennke]

Am Mittwoch, den 13.06.2012, 13:05 +0100 schrieb Johannes Lips:
>  
>  
>  On Wed, Jun 13, 2012 at 1:00 PM, Roman Kennke <rkennke(a)redhat.com&gt;
>  wrote:
>          > Today something happened, that happens over and over again
>          with Fedora,
>          > and it makes me angry. I am running Fedora 17, and so far it
>          worked well
>          > with the initial kernel 3.3.x (except that it would panic on
>          shutdown...
>          > but that was not important to me, but still embarassing).
>          Today I was
>          > notified of an important security update in the kernel.
>          Curiously, it
>          > would update from 3.3 to 3.4 (a major version upgrade, which
>          should not
>          > happen in such a core package anyway, IMO). Reboot into the
>          new kernel,
>          > everything comes up --- until I want to actually want to
>          read email,
>          > surf web, or anything that requires my network. I am on an
>          Intel Wifi
>          > card, iwlwifi module. I *can* connect to the network, but
>          everything is
>          > suuuuuuper  slow or times-out every now and then. Completely
>          unusable.
>          > Reboot into the older kernel, things work well again. Now I
>          am left with
>          > the choice of running a new kernel w/o network or an
>          unsecure kernel.
>          > Thank you very much!
>          >
>          > This sort of thing I would expect in rawhide/development
>          builds, but not
>          > in a supposed-to-be stable release. I can understand the
>          underlying idea
>          > of being on the bleeding edge, but I don't want to actually
>          be bleeding.
>          > At least the base system components should not undergo major
>          version
>          > updates. Security fixes should be backported to the software
>          version
>          > that is in the stable release (1 year release cycle
>          shouldn't be too
>          > demanding for this), and only security fixes and absolutely
>          important
>          > fixes should go into stable releases. (Not to mention that
>          some fixes
>          > that I *would* consider important enough to go into stable
>          never end up
>          > there.) If major version updates are really really
>          necessary, they
>          > should undergo serious testing. I cannot believe that I am
>          the only one
>          > on an Intel Wifi chip. The way it is now, Fedora feels like
>          a constantly
>          > rolling development version that is almost unusable (because
>          any update,
>          > even security, has a fairly high risk of breaking things)
>          for day-to-day
>          > work.
>          >
>          > Bugzilla report:
>          > https://bugzilla.redhat.com/show_bug.cgi?id=831571
>          
>          
>          Since I just received an email in private pointing out that
>          emails like
>          mine above might be discouraging and not helpful... let me
>          apologize for
>          this. My intention is not to bash other people's best efforts,
>          but
>          instead try to help out (otherwise I would not bother to
>          diligently file
>          bugreports and mention my concerns on this list). I am willing
>          to help
>          track down and fix the problem. However, I see a more general
>          problem
>          and maybe we can turn this into a discussion how to address
>          (or answer)
>          it.
>          
>          - Why do we allow new major versions of core components into a
>          stable
>          release? What sort of testing is performed before a major
>          kernel update
>          hits Fedora stable?
>          - What is the policy with regards to risky changes (like
>          unnecessary
>          feature updates, ABI changes, etc) in stable?
>          - How can problems like the one I described above be avoided?
>          Is there
>          anything I and others can help with?
>          
>          Roman
>          
>          
>          --
>          devel mailing list
>          devel(a)lists.fedoraproject.org
>          https://admin.fedoraproject.org/mailman/listinfo/devel
>  I think the reason for shipping the latest upstream kernel is based on
>  the fact that backporting would be too much work.
>  http://fedoraproject.org/wiki/KernelRebases
>  Gives a good overview and probably prevents us from repeating
>  arguments in the discussion. 
Ok, fair enough. The question remains, how can we avoid such bad things
to happen in the future? Should I regularily try out kernel builds on
their way to stable, and object to their stable-release when I find a
problem? And how would I do that? (I.e. how can I find out when a new
kernel is about to go to stable, and when to test it, etc) And what
about the other base components of the system? (Although, to be fair,
the kernel seems to be the most problematic one..)

Roman

[Stijn Hoop]

Hi,

On Wed, 13 Jun 2012 14:15:14 +0200
Roman Kennke <rkennke(a)redhat.com&gt; wrote:
>  Am Mittwoch, den 13.06.2012, 13:05 +0100 schrieb Johannes Lips:
>  > I think the reason for shipping the latest upstream kernel is based
>  > on the fact that backporting would be too much work.
>  > http://fedoraproject.org/wiki/KernelRebases
>  > Gives a good overview and probably prevents us from repeating
>  > arguments in the discussion.
>  
>  Ok, fair enough. The question remains, how can we avoid such bad
>  things to happen in the future? Should I regularily try out kernel
>  builds on their way to stable, and object to their stable-release
>  when I find a problem? And how would I do that? (I.e. how can I find
>  out when a new kernel is about to go to stable, and when to test it,
>  etc) And what about the other base components of the system?
>  (Although, to be fair, the kernel seems to be the most problematic
>  one..) 
Check

https://admin.fedoraproject.org/updates/kernel

for updates. Provide negative karma for them in Bodhi as well:

http://fedoraproject.org/wiki/QA/Updates_Testing

Regards,

--Stijn

[Roman Kennke]

Am Mittwoch, den 13.06.2012, 14:29 +0200 schrieb Stijn Hoop:
>  Hi,
>  
>  On Wed, 13 Jun 2012 14:15:14 +0200
>  Roman Kennke <rkennke(a)redhat.com&gt; wrote:
>  > Am Mittwoch, den 13.06.2012, 13:05 +0100 schrieb Johannes Lips:
>  > > I think the reason for shipping the latest upstream kernel is based
>  > > on the fact that backporting would be too much work.
>  > > http://fedoraproject.org/wiki/KernelRebases
>  > > Gives a good overview and probably prevents us from repeating
>  > > arguments in the discussion.
>  > 
>  > Ok, fair enough. The question remains, how can we avoid such bad
>  > things to happen in the future? Should I regularily try out kernel
>  > builds on their way to stable, and object to their stable-release
>  > when I find a problem? And how would I do that? (I.e. how can I find
>  > out when a new kernel is about to go to stable, and when to test it,
>  > etc) And what about the other base components of the system?
>  > (Although, to be fair, the kernel seems to be the most problematic
>  > one..)
>  
>  Check
>  
>  https://admin.fedoraproject.org/updates/kernel
>  
>  for updates. Provide negative karma for them in Bodhi as well:
>  
>  http://fedoraproject.org/wiki/QA/Updates_Testing 
Thanks everybody!

I subscribed to the above updates/kernel RSS feed and will try to test
kernels before they go to stable.

Would it make sense to require more karma than just the default 3?
Looking at:

https://admin.fedoraproject.org/updates/FEDORA-2012-8824/kernel-3.4.0-1.fc17

I see that there are 5 oks and 2 denys, which actually point to bug
reports, both sound fairly important. How does the karma system work if,
e.g., update requires +3, the update gets +4 and -1, and this -1 is
something that can be considered a release critical bug? data corruption
sounds quite release-critical? Is there a mechanism that prevents the
update to happen in this case?

Roman

[Josh Boyer]

On Wed, Jun 13, 2012 at 9:24 AM, Roman Kennke <rkennke(a)redhat.com&gt; wrote:
>  Would it make sense to require more karma than just the default 3?
>  Looking at:
>
>  https://admin.fedoraproject.org/updates/FEDORA-2012-8824/kernel-3.4.0-1.fc17
>
>  I see that there are 5 oks and 2 denys, which actually point to bug
>  reports, both sound fairly important. How does the karma system work if,
>  e.g., update requires +3, the update gets +4 and -1, and this -1 is
>  something that can be considered a release critical bug? data corruption
>  sounds quite release-critical? Is there a mechanism that prevents the
>  update to happen in this case? 
Good questions.

The person that submits the update gets emails for every comment added
to the update.  This particular one had a couple things that happened
though.

1) It got the requisite karma for stable rather quickly
2) Justin was on vacation when the negative karma was submitted.  Bodhi
only emails the update submitter and the rest of the kernel team didn't
notice.

I'm sure that it would have been pulled if Justin was actually around
or if the rest of the kernel team had remembered to go check the
update.  It's something that can be looked at in the future.

josh

[Adam Williamson]

On Wed, 2012-06-13 at 09:36 -0400, Josh Boyer wrote:
>  On Wed, Jun 13, 2012 at 9:24 AM, Roman Kennke
> <rkennke(a)redhat.com&gt; wrote:
>  > Would it make sense to require more karma than just the default 3?
>  > Looking at:
>  >
>  > https://admin.fedoraproject.org/updates/FEDORA-2012-8824/kernel-3.4.0-1.fc17
>  >
>  > I see that there are 5 oks and 2 denys, which actually point to bug
>  > reports, both sound fairly important. How does the karma system work if,
>  > e.g., update requires +3, the update gets +4 and -1, and this -1 is
>  > something that can be considered a release critical bug? data corruption
>  > sounds quite release-critical? Is there a mechanism that prevents the
>  > update to happen in this case?
>  
>  Good questions.
>  
>  The person that submits the update gets emails for every comment added
>  to the update.  This particular one had a couple things that happened
>  though.
>  
>  1) It got the requisite karma for stable rather quickly
>  2) Justin was on vacation when the negative karma was submitted.  Bodhi
>  only emails the update submitter and the rest of the kernel team didn't
>  notice.
>  
>  I'm sure that it would have been pulled if Justin was actually around
>  or if the rest of the kernel team had remembered to go check the
>  update.  It's something that can be looked at in the future. 
This can also serve as my quarterly reminder that Bodhi 2.0 is _still_
supposed to be coming, with much better feedback features. The simple
+/- points system in Bodhi 1.0 isn't really adequate for any number of
scenarios, including this one. I have posted before about what benefits
a better feedback system would have:
https://lists.fedoraproject.org/pipermail/devel/2011-November/159874.html
-- 
Adam Williamson
Fedora QA Community Monkey
IRC: adamw | Twitter: AdamW_Fedora | identi.ca: adamwfedora
http://www.happyassassin.net

[Roman Kennke]

Am Mittwoch, den 13.06.2012, 12:07 -0700 schrieb Adam Williamson:
>  On Wed, 2012-06-13 at 09:36 -0400, Josh Boyer wrote:
>  > On Wed, Jun 13, 2012 at 9:24 AM, Roman Kennke <rkennke(a)redhat.com&gt; wrote:
>  > > Would it make sense to require more karma than just the default 3?
>  > > Looking at:
>  > >
>  > > https://admin.fedoraproject.org/updates/FEDORA-2012-8824/kernel-3.4.0-1.fc17
>  > >
>  > > I see that there are 5 oks and 2 denys, which actually point to bug
>  > > reports, both sound fairly important. How does the karma system work if,
>  > > e.g., update requires +3, the update gets +4 and -1, and this -1 is
>  > > something that can be considered a release critical bug? data corruption
>  > > sounds quite release-critical? Is there a mechanism that prevents the
>  > > update to happen in this case?
>  > 
>  > Good questions.
>  > 
>  > The person that submits the update gets emails for every comment added
>  > to the update.  This particular one had a couple things that happened
>  > though.
>  > 
>  > 1) It got the requisite karma for stable rather quickly
>  > 2) Justin was on vacation when the negative karma was submitted.  Bodhi
>  > only emails the update submitter and the rest of the kernel team didn't
>  > notice.
>  > 
>  > I'm sure that it would have been pulled if Justin was actually around
>  > or if the rest of the kernel team had remembered to go check the
>  > update.  It's something that can be looked at in the future.
>  
>  This can also serve as my quarterly reminder that Bodhi 2.0 is _still_
>  supposed to be coming, with much better feedback features. The simple
>  +/- points system in Bodhi 1.0 isn't really adequate for any number of
>  scenarios, including this one. I have posted before about what benefits
>  a better feedback system would have:
>  https://lists.fedoraproject.org/pipermail/devel/2011-November/159874.html 
What you lay out there sounds useful indeed! I hope it's coming.

One thing that might be useful to think about is the way Debian handles
transitions. (Forgive me if the following is not 100% correct, I am only
remotely familiar with what I am saying.) Packages in testing (or
unstable) can only transition to the next more stable repository if
certain conditions are fulfilled. One is to be the at least X days,
another is to not have any (new) release critical bugs in it. (I think
there are more, but those two seem important.) In order to accomplish
this, the package update system (I guess it would be comparable to
Fedora's Bodhi) is linked to the bug DB, and bugs for a package can have
a severity of 'release critical' which basically blocks transitions of
the package. I think something similar could be useful for Fedora as
well, i.e. link Bodhi to bugzilla and react on certain things like
severity of bugs.

Of course, it still means that somebody actually needs to test it in
order to find and file bugs...

Roman

[Kevin Kofler]

Josh Boyer wrote:
>  The person that submits the update gets emails for every comment
> added
>  to the update.  This particular one had a couple things that happened
>  though.
>  
>  1) It got the requisite karma for stable rather quickly
>  2) Justin was on vacation when the negative karma was submitted.  Bodhi
>  only emails the update submitter and the rest of the kernel team didn't
>  notice.
>  
>  I'm sure that it would have been pulled if Justin was actually around
>  or if the rest of the kernel team had remembered to go check the
>  update.  It's something that can be looked at in the future. 
Why is karma automatism being enabled (by the submitters, it's optional!) on 
kernel updates in the first place? IMHO, kernels should always be pushed 
manually. (I actually consider karma automatism entirely broken, but the 
kernel is one of the worst packages to use it on!)

        Kevin Kofler

[Josh Boyer]

On Wed, Jun 13, 2012 at 8:15 AM, Roman Kennke <rkennke(a)redhat.com&gt; wrote:
> > I think the reason for shipping the latest upstream kernel is
> based on
> > the fact that backporting would be too much work.
> > http://fedoraproject.org/wiki/KernelRebases
> > Gives a good overview and probably prevents us from repeating
> > arguments in the discussion.
>
>  Ok, fair enough. The question remains, how can we avoid such bad things
>  to happen in the future? Should I regularily try out kernel builds on
>  their way to stable, and object to their stable-release when I find a
>  problem? And how would I do that? (I.e. how can I find out when a new 
Yes, testing would be very much appreciated.

>  kernel is about to go to stable, and when to test it, etc) And what
>  about the other base components of the system? (Although, to be fair,
>  the kernel seems to be the most problematic one..) 
The kernel maintainers push all updates through updates-testing for
this very reason.  It is very rare that one is pushed directly to
stable.  You can use 'yum update --enablerepo=updates-testing kernel'
(or similar commands) to only update the kernel from updates testing
if you wish.  Then you provide karma in Bodhi.

josh

[Frank Murphy]

On 13/06/12 13:15, Roman Kennke wrote:


>  Ok, fair enough. The question remains, how can we avoid such bad
> things
>  to happen in the future? Should I regularily try out kernel builds on
>  their way to stable, and object to their stable-release when I find a
>  problem? And how would I do that? (I.e. how can I find out when a new
>  kernel is about to go to stable, and when to test it, etc) 
Test the kernels when they arrive in testing-updates.
Well in advance of stable.
Give\Deny karma as appropriate.
based on what problems may occur.




-- 
Regards,
Frank
"Jack of all, fubars"

[Aleksandar Kurtakov]

----- Original Message -----
>  From: "Roman Kennke" <rkennke(a)redhat.com&gt;
>  To: "Development discussions related to Fedora"
> <devel(a)lists.fedoraproject.org&gt;
>  Sent: Wednesday, June 13, 2012 3:15:14 PM
>  Subject: Re: Important kernel update should not break stuff
>  
>  Am Mittwoch, den 13.06.2012, 13:05 +0100 schrieb Johannes Lips:
>  > 
>  > 
>  > On Wed, Jun 13, 2012 at 1:00 PM, Roman Kennke <rkennke(a)redhat.com&gt;
>  > wrote:
>  >         > Today something happened, that happens over and over
>  >         > again
>  >         with Fedora,
>  >         > and it makes me angry. I am running Fedora 17, and so far
>  >         > it
>  >         worked well
>  >         > with the initial kernel 3.3.x (except that it would panic
>  >         > on
>  >         shutdown...
>  >         > but that was not important to me, but still embarassing).
>  >         Today I was
>  >         > notified of an important security update in the kernel.
>  >         Curiously, it
>  >         > would update from 3.3 to 3.4 (a major version upgrade,
>  >         > which
>  >         should not
>  >         > happen in such a core package anyway, IMO). Reboot into
>  >         > the
>  >         new kernel,
>  >         > everything comes up --- until I want to actually want to
>  >         read email,
>  >         > surf web, or anything that requires my network. I am on
>  >         > an
>  >         Intel Wifi
>  >         > card, iwlwifi module. I *can* connect to the network, but
>  >         everything is
>  >         > suuuuuuper  slow or times-out every now and then.
>  >         > Completely
>  >         unusable.
>  >         > Reboot into the older kernel, things work well again. Now
>  >         > I
>  >         am left with
>  >         > the choice of running a new kernel w/o network or an
>  >         unsecure kernel.
>  >         > Thank you very much!
>  >         >
>  >         > This sort of thing I would expect in rawhide/development
>  >         builds, but not
>  >         > in a supposed-to-be stable release. I can understand the
>  >         underlying idea
>  >         > of being on the bleeding edge, but I don't want to
>  >         > actually
>  >         be bleeding.
>  >         > At least the base system components should not undergo
>  >         > major
>  >         version
>  >         > updates. Security fixes should be backported to the
>  >         > software
>  >         version
>  >         > that is in the stable release (1 year release cycle
>  >         shouldn't be too
>  >         > demanding for this), and only security fixes and
>  >         > absolutely
>  >         important
>  >         > fixes should go into stable releases. (Not to mention
>  >         > that
>  >         some fixes
>  >         > that I *would* consider important enough to go into
>  >         > stable
>  >         never end up
>  >         > there.) If major version updates are really really
>  >         necessary, they
>  >         > should undergo serious testing. I cannot believe that I
>  >         > am
>  >         the only one
>  >         > on an Intel Wifi chip. The way it is now, Fedora feels
>  >         > like
>  >         a constantly
>  >         > rolling development version that is almost unusable
>  >         > (because
>  >         any update,
>  >         > even security, has a fairly high risk of breaking things)
>  >         for day-to-day
>  >         > work.
>  >         >
>  >         > Bugzilla report:
>  >         > https://bugzilla.redhat.com/show_bug.cgi?id=831571
>  >         
>  >         
>  >         Since I just received an email in private pointing out that
>  >         emails like
>  >         mine above might be discouraging and not helpful... let me
>  >         apologize for
>  >         this. My intention is not to bash other people's best
>  >         efforts,
>  >         but
>  >         instead try to help out (otherwise I would not bother to
>  >         diligently file
>  >         bugreports and mention my concerns on this list). I am
>  >         willing
>  >         to help
>  >         track down and fix the problem. However, I see a more
>  >         general
>  >         problem
>  >         and maybe we can turn this into a discussion how to address
>  >         (or answer)
>  >         it.
>  >         
>  >         - Why do we allow new major versions of core components
>  >         into a
>  >         stable
>  >         release? What sort of testing is performed before a major
>  >         kernel update
>  >         hits Fedora stable?
>  >         - What is the policy with regards to risky changes (like
>  >         unnecessary
>  >         feature updates, ABI changes, etc) in stable?
>  >         - How can problems like the one I described above be
>  >         avoided?
>  >         Is there
>  >         anything I and others can help with?
>  >         
>  >         Roman
>  >         
>  >         
>  >         --
>  >         devel mailing list
>  >         devel(a)lists.fedoraproject.org
>  >         https://admin.fedoraproject.org/mailman/listinfo/devel
>  > I think the reason for shipping the latest upstream kernel is based
>  > on
>  > the fact that backporting would be too much work.
>  > http://fedoraproject.org/wiki/KernelRebases
>  > Gives a good overview and probably prevents us from repeating
>  > arguments in the discussion.
>  
>  Ok, fair enough. The question remains, how can we avoid such bad
>  things
>  to happen in the future? Should I regularily try out kernel builds on
>  their way to stable, and object to their stable-release when I find a
>  problem? And how would I do that? (I.e. how can I find out when a new
>  kernel is about to go to stable, and when to test it, etc) And what
>  about the other base components of the system? (Although, to be fair,
>  the kernel seems to be the most problematic one..) 
Try having updates-testing repo enabled test and provide karma via bodhi. +1 is as needed
as -1 as without it we never know whether people simply installed and it worked so they
stopped at that point. See http://fedoraproject.org/wiki/Bodhi for details. There is also
fedora-easy-karma to ease mass reporting via bodhi.
Note that this would not make it easier for you short term as you would spot the problems
just a bit earlier. But if enough people do that problems might even be spotted before you
update next time and the build untagged so you never see the broken build/update at all.
This kind of workflow is effective only if we manage to gather critical mass.

Regards,
Alex

>  
>  Roman
>  
>  
>  --
>  devel mailing list
>  devel(a)lists.fedoraproject.org
>  https://admin.fedoraproject.org/mailman/listinfo/devel

[Ian Malone]

On 13 June 2012 13:31, Aleksandar Kurtakov <akurtako(a)redhat.com&gt; wrote:

>  ----- Original Message -----
> > From: "Roman Kennke" <rkennke(a)redhat.com&gt;
> > To: "Development discussions related to Fedora"
> <devel(a)lists.fedoraproject.org&gt;
> > Sent: Wednesday, June 13, 2012 3:15:14 PM
> > Subject: Re: Important kernel update should not break stuff 
> > >         - How can problems like the one I described above
> be
> > >         avoided?
> > >         Is there
> > >         anything I and others can help with? 

> > Ok, fair enough. The question remains, how can we avoid such bad
> > things
> > to happen in the future? Should I regularily try out kernel builds on
> > their way to stable, and object to their stable-release when I find a
> > problem? And how would I do that? (I.e. how can I find out when a new
> > kernel is about to go to stable, and when to test it, etc) And what
> > about the other base components of the system? (Although, to be fair,
> > the kernel seems to be the most problematic one..)
>
>  Try having updates-testing repo enabled test and provide karma via bodhi. +1 is as needed
> as -1 as without it we never know whether people simply installed and it worked so they
> stopped at that point. See http://fedoraproject.org/wiki/Bodhi for details. There is also
> fedora-easy-karma to ease mass reporting via bodhi.
>  Note that this would not make it easier for you short term as you would spot the problems
> just a bit earlier. But if enough people do that problems might even be spotted before you
> update next time and the build untagged so you never see the broken build/update at all.
>  This kind of workflow is effective only if we manage to gather critical mass.
>  
I get the need for people to volunteer and test and that's fine. But
the thing I can't square here is why then we aren't all on
updates-testing all the time? The kernel is one of the few packages
you can guarantee everyone is using. We'd all like to know if an
upcoming kernel has problems that will affect us, but you can't know
that until you run it.

So the answer is to test it by being on updates-testing, but this
means you put yourself in the way of every possible kernel update and
open yourself to more potential problems. I don't have a solution for
that, but it's clearly something that needs some thought (as someone
who has quite a few times been stuck on old kernels and trying
updates-testing ones to see if they fix a newly introduced problem).

One question is why are regressions quite so common for some
components? iwlwifi for example seems to show up quite a bit.

-- 
imalone
http://ibmalone.blogspot.co.uk

[Bruno Wolff III]

On Wed, Jun 13, 2012 at 17:27:03 +0100,
   Ian Malone <ibmalone(a)gmail.com&gt; wrote:
> I get the need for people to volunteer and test and that's fine. But
> the thing I can't square here is why then we aren't all on
> updates-testing all the time? The kernel is one of the few packages
> you can guarantee everyone is using. We'd all like to know if an
> upcoming kernel has problems that will affect us, but you can't know
> that until you run it.
>
> So the answer is to test it by being on updates-testing, but this
> means you put yourself in the way of every possible kernel update and
> open yourself to more potential problems. I don't have a solution for
> that, but it's clearly something that needs some thought (as someone
> who has quite a few times been stuck on old kernels and trying
> updates-testing ones to see if they fix a newly introduced problem). 
Different people may be more comfortable with dealing with kernel problems 
then others. This can vary based on the persons skills and how they are 
using the affected machines.

Part of my contribution to Fedora is being a guinea pig using updates-testing 
and rawhide. However, dealing with these issues occasionally sucks up a lot 
of time and I can certainly see why others would need to not do this.

[Aleksandar Kurtakov]

----- Original Message -----
>  From: "Ian Malone" <ibmalone(a)gmail.com&gt;
>  To: "Development discussions related to Fedora"
> <devel(a)lists.fedoraproject.org&gt;
>  Sent: Wednesday, June 13, 2012 7:27:03 PM
>  Subject: Re: Important kernel update should not break stuff
>  
>  On 13 June 2012 13:31, Aleksandar Kurtakov <akurtako(a)redhat.com&gt;
>  wrote:
>  
>  > ----- Original Message -----
>  >> From: "Roman Kennke" <rkennke(a)redhat.com&gt;
>  >> To: "Development discussions related to Fedora"
>  >> <devel(a)lists.fedoraproject.org&gt;
>  >> Sent: Wednesday, June 13, 2012 3:15:14 PM
>  >> Subject: Re: Important kernel update should not break stuff
>  
>  >> >         - How can problems like the one I described above be
>  >> >         avoided?
>  >> >         Is there
>  >> >         anything I and others can help with?
>  
>  
>  >> Ok, fair enough. The question remains, how can we avoid such bad
>  >> things
>  >> to happen in the future? Should I regularily try out kernel builds
>  >> on
>  >> their way to stable, and object to their stable-release when I
>  >> find a
>  >> problem? And how would I do that? (I.e. how can I find out when a
>  >> new
>  >> kernel is about to go to stable, and when to test it, etc) And
>  >> what
>  >> about the other base components of the system? (Although, to be
>  >> fair,
>  >> the kernel seems to be the most problematic one..)
>  >
>  > Try having updates-testing repo enabled test and provide karma via
>  > bodhi. +1 is as needed as -1 as without it we never know whether
>  > people simply installed and it worked so they stopped at that
>  > point. See http://fedoraproject.org/wiki/Bodhi for details. There
>  > is also fedora-easy-karma to ease mass reporting via bodhi.
>  > Note that this would not make it easier for you short term as you
>  > would spot the problems just a bit earlier. But if enough people
>  > do that problems might even be spotted before you update next time
>  > and the build untagged so you never see the broken build/update at
>  > all.
>  > This kind of workflow is effective only if we manage to gather
>  > critical mass.
>  >
>  
>  I get the need for people to volunteer and test and that's fine. But
>  the thing I can't square here is why then we aren't all on
>  updates-testing all the time? The kernel is one of the few packages
>  you can guarantee everyone is using. We'd all like to know if an
>  upcoming kernel has problems that will affect us, but you can't know
>  that until you run it. 
If we means all people subscribed to fedora-devel I think everyone should run with
updates-testing enabled. This definition of we is the people that make Fedora happen so we
should always test our stuff - non-stop. Regular updates are for users. From my POV
everyone sending messages to fedora-devel is supposed to run with updates-testing enabled.
And excuses like "no time" and etc. doesn't make sense as this case is
showing that one can not simply care about his packages only because there is always
something else that your package collaborates with that needs testing too.

Alex

>  
>  So the answer is to test it by being on updates-testing, but this
>  means you put yourself in the way of every possible kernel update and
>  open yourself to more potential problems. I don't have a solution for
>  that, but it's clearly something that needs some thought (as someone
>  who has quite a few times been stuck on old kernels and trying
>  updates-testing ones to see if they fix a newly introduced problem).
>  
>  One question is why are regressions quite so common for some
>  components? iwlwifi for example seems to show up quite a bit.
>  
>  --
>  imalone
>  http://ibmalone.blogspot.co.uk
>  --
>  devel mailing list
>  devel(a)lists.fedoraproject.org
>  https://admin.fedoraproject.org/mailman/listinfo/devel

[Bruno Wolff III]

On Wed, Jun 13, 2012 at 13:16:24 -0400,
   Aleksandar Kurtakov <akurtako(a)redhat.com&gt; wrote:
> If we means all people subscribed to fedora-devel I think everyone should run with
> updates-testing enabled. This definition of we is the people that make Fedora happen so we
> should always test our stuff - non-stop. Regular updates are for users. From my POV
> everyone sending messages to fedora-devel is supposed to run with updates-testing enabled.
> And excuses like "no time" and etc. doesn't make sense as this case is
> showing that one can not simply care about his packages only because there is always
> something else that your package collaborates with that needs testing too. 
While I would enourage Fedora contributors to run with updates-testing 
available, I don't think everyone on the devel list needs to. We all have to 
make trade offs of time and the project may benefit more from some people 
spending their time on other things than testing updates before they get 
to stable. Note that not everyone subscribed to the devel list is a packager 
and that packagers have other ways to install their package updates without 
using the updates-testing repo.

[Nikola Pajkovsky]

Roman Kennke <rkennke(a)redhat.com&gt; writes:

>  Am Mittwoch, den 13.06.2012, 13:05 +0100 schrieb Johannes Lips:
> > 
> > 
> > On Wed, Jun 13, 2012 at 1:00 PM, Roman Kennke <rkennke(a)redhat.com&gt;
> > wrote:
> >         > Today something happened, that happens over and over again
> >         with Fedora,
> >         > and it makes me angry. I am running Fedora 17, and so far it
> >         worked well
> >         > with the initial kernel 3.3.x (except that it would panic on
> >         shutdown...
> >         > but that was not important to me, but still embarassing).
> >         Today I was
> >         > notified of an important security update in the kernel.
> >         Curiously, it
> >         > would update from 3.3 to 3.4 (a major version upgrade, which
> >         should not
> >         > happen in such a core package anyway, IMO). Reboot into the
> >         new kernel,
> >         > everything comes up --- until I want to actually want to
> >         read email,
> >         > surf web, or anything that requires my network. I am on an
> >         Intel Wifi
> >         > card, iwlwifi module. I *can* connect to the network, but
> >         everything is
> >         > suuuuuuper  slow or times-out every now and then. Completely
> >         unusable.
> >         > Reboot into the older kernel, things work well again. Now I
> >         am left with
> >         > the choice of running a new kernel w/o network or an
> >         unsecure kernel.
> >         > Thank you very much!
> >         >
> >         > This sort of thing I would expect in rawhide/development
> >         builds, but not
> >         > in a supposed-to-be stable release. I can understand the
> >         underlying idea
> >         > of being on the bleeding edge, but I don't want to actually
> >         be bleeding.
> >         > At least the base system components should not undergo major
> >         version
> >         > updates. Security fixes should be backported to the software
> >         version
> >         > that is in the stable release (1 year release cycle
> >         shouldn't be too
> >         > demanding for this), and only security fixes and absolutely
> >         important
> >         > fixes should go into stable releases. (Not to mention that
> >         some fixes
> >         > that I *would* consider important enough to go into stable
> >         never end up
> >         > there.) If major version updates are really really
> >         necessary, they
> >         > should undergo serious testing. I cannot believe that I am
> >         the only one
> >         > on an Intel Wifi chip. The way it is now, Fedora feels like
> >         a constantly
> >         > rolling development version that is almost unusable (because
> >         any update,
> >         > even security, has a fairly high risk of breaking things)
> >         for day-to-day
> >         > work.
> >         >
> >         > Bugzilla report:
> >         > https://bugzilla.redhat.com/show_bug.cgi?id=831571
> >         
> >         
> >         Since I just received an email in private pointing out that
> >         emails like
> >         mine above might be discouraging and not helpful... let me
> >         apologize for
> >         this. My intention is not to bash other people's best efforts,
> >         but
> >         instead try to help out (otherwise I would not bother to
> >         diligently file
> >         bugreports and mention my concerns on this list). I am willing
> >         to help
> >         track down and fix the problem. However, I see a more general
> >         problem
> >         and maybe we can turn this into a discussion how to address
> >         (or answer)
> >         it.
> >         
> >         - Why do we allow new major versions of core components into a
> >         stable
> >         release? What sort of testing is performed before a major
> >         kernel update
> >         hits Fedora stable?
> >         - What is the policy with regards to risky changes (like
> >         unnecessary
> >         feature updates, ABI changes, etc) in stable?
> >         - How can problems like the one I described above be avoided?
> >         Is there
> >         anything I and others can help with?
> >         
> >         Roman
> >         
> >         
> >         --
> >         devel mailing list
> >         devel(a)lists.fedoraproject.org
> >         https://admin.fedoraproject.org/mailman/listinfo/devel
> > I think the reason for shipping the latest upstream kernel is based on
> > the fact that backporting would be too much work.
> > http://fedoraproject.org/wiki/KernelRebases
> > Gives a good overview and probably prevents us from repeating
> > arguments in the discussion.
>
>  Ok, fair enough. The question remains, how can we avoid such bad things
>  to happen in the future? Should I regularily try out kernel builds on
>  their way to stable, and object to their stable-release when I find a
>  problem? And how would I do that? (I.e. how can I find out when a new
>  kernel is about to go to stable, and when to test it, etc) And what
>  about the other base components of the system? (Although, to be fair,
>  the kernel seems to be the most problematic one..) 
imo, kernel maintainers should have released 3.3.8 or 3.4.1, not 3.4.0
for f17

-- 
Nikola

[Bruno Wolff III]

On Wed, Jun 13, 2012 at 14:49:25 +0200,
   Nikola Pajkovsky <npajkovs(a)redhat.com&gt; wrote:
> imo, kernel maintainers should have released 3.3.8 or 3.4.1, not 3.4.0
> for f17 
3.4.2 is available, though the last I checked the build hadn't been pushed 
to testing. I don't know if that is an oversight or if they didn't feel 
the kernel was suitable for releasing.

The 3.4 kernels were being tested in rawhide before it was released to 
f17 and at least of us weren't seeing any regressions with them. So there 
was reason to believe they would work for people without problems.

[Josh Boyer]

On Wed, Jun 13, 2012 at 8:53 AM, Bruno Wolff III <bruno(a)wolff.to&gt; wrote:
>  On Wed, Jun 13, 2012 at 14:49:25 +0200,
>   Nikola Pajkovsky <npajkovs(a)redhat.com&gt; wrote:
> >
> >
> > imo, kernel maintainers should have released 3.3.8 or 3.4.1, not 3.4.0
> > for f17
>
>
>  3.4.2 is available, though the last I checked the build hadn't been pushed
>  to testing. I don't know if that is an oversight or if they didn't feel the
>  kernel was suitable for releasing. 
I believe it's a matter of getting caught up.  There are at least 2
more patches added after the 3.4.2 build that should probably get
included in the next update.

>  The 3.4 kernels were being tested in rawhide before it was released
> to f17
>  and at least of us weren't seeing any regressions with them. So there was
>  reason to believe they would work for people without problems. 
Yes, that is true.  However it also speaks to the extremely small usage
of rawhide and the relative lack of variety in both test hardware and
setups.  I have no magical solutions to those problems.  We simply need
more people testing rawhide kernels in general.

josh

[Jóhann B. Guðmundsson]

On 06/13/2012 01:39 PM, Josh Boyer wrote:
>  Yes, that is true.  However it also speaks to the extremely small
> usage
>  of rawhide and the relative lack of variety in both test hardware and
>  setups.  I have no magical solutions to those problems.  We simply need
>  more people testing rawhide kernels in general. 
A lot of us from the QA community who are not on rawhide are the running 
the rawhide kernels on F17
( I'm currently on 3.5.0-0.rc2.git0.1.fc18.x86_64 ) and I have not 
noticed ( hence not reported ) any issue.

I think the problem is actually the other way around those of us that 
are in the QA community aren't actually using official update GA kernel.

And getting reporters to actually stay on a GA release once rawhide gets 
in usable shape ( which by my experience is around alpha ) is a problem.
( one of the thing that proven tester concept was supposed to address to 
ensure we had people testing GA )

And to add insult to injury reporters on update-testing have been 
routinely in the past been redirected from the test list to the user 
list with any issue they have discovered.

Something I tried to address a while back amongst other things but 
certain people in our QA community want to keep the list rawhide only...

JBG

[M A Young]

On Wed, 13 Jun 2012, Nikola Pajkovsky wrote:

>  imo, kernel maintainers should have released 3.3.8 or 3.4.1, not
> 3.4.0
>  for f17 
I believe the 3.4.0 kernel package was effectively 3.4.1 anyway.

 	Michael Young

[Josh Boyer]

On Wed, Jun 13, 2012 at 9:14 AM, M A Young <m.a.young(a)durham.ac.uk&gt; wrote:
>  On Wed, 13 Jun 2012, Nikola Pajkovsky wrote:
>
> > imo, kernel maintainers should have released 3.3.8 or 3.4.1, not 3.4.0
> > for f17
>
>
>  I believe the 3.4.0 kernel package was effectively 3.4.1 anyway. 
Yes, it was.  Justin added the patches queued for the 3.4.1 release
before he submitted the build.

josh

[Adam Williamson]

On Wed, 2012-06-13 at 12:51 +0200, Roman Kennke wrote:

>  I cannot believe that I am the only one
>  on an Intel Wifi chip. 
I haven't yet read the rest of the thread, but I just wanted to point
out this common fallacy. As a general rule, all $VENDOR_$DEVICETYPE
devices do not act the same. You cannot assume that all Intel wifi
adapters are subject to the same bug you saw. You can't even assume that
the same adapter _in another system_ would be subject to the same bug.
This goes for everything: you can't assume all Intel video adapters
behave the same, or all AMD video adapters, or all Broadcom wireless
adapters, or _anything_.

I wish it _were_ true, believe me. It'd make testing (and I'm sure
development) a lot easier if all we had to do was have one of each type
of device from each manufacturer and then we could happily say 'okay,
this kernel works with every piece of hardware in existence'. Boy,
that'd be nice. Sadly, it just isn't the case at all.

Practical extensions from this rule:

1. Just because your NVIDIA graphics card doesn't work, don't assume
that the developers / testers are such idiots they didn't test (whatever
it is) on _any_ NVIDIA graphics cards at all.

2. Just because your Intel bluetooth adapter doesn't work, don't go
around telling people on forums, IRC etc that they can't use any Intel
bluetooth adapter with Fedora, because it's perfectly likely that lots
of others work just fine.

(Both of these are real-life examples I've seen more than once).

The only good grounds for acting as if you're subject to a bug which is
affecting a very broad class of hardware is if you have detailed reports
from multiple other people with devices in the same class of hardware,
detailed enough that it is clear they are all suffering from the same
bug. Your own personal experience is _never_ sufficient grounds for
concluding that the bug you're experiencing affects a much broader class
of devices.
-- 
Adam Williamson
Fedora QA Community Monkey
IRC: adamw | Twitter: AdamW_Fedora | identi.ca: adamwfedora
http://www.happyassassin.net

[Felix Miata]

On 2012/06/13 11:22 (GMT-0700) Adam Williamson composed:

>  Your own personal experience is _never_ sufficient grounds for
>  concluding that the bug you're experiencing affects a much broader class
>  of devices. 
Is "never" appropriate even if one's own experience is with 3 systems? 7 
systems? 13 systems? 40 systems? Never say never, or always. ;-)
-- 
"The wise are known for their understanding, and pleasant
words are persuasive." Proverbs 16:21 (New Living Translation)

  Team OS/2 ** Reg. Linux User #211409 ** a11y rocks!

Felix Miata  ***  http://fm.no-ip.com/

[Adam Williamson]

On Wed, 2012-06-13 at 14:33 -0400, Felix Miata wrote:
>  On 2012/06/13 11:22 (GMT-0700) Adam Williamson composed:
>  
>  > Your own personal experience is _never_ sufficient grounds for
>  > concluding that the bug you're experiencing affects a much broader class
>  > of devices.
>  
>  Is "never" appropriate even if one's own experience is with 3 systems? 7 
>  systems? 13 systems? 40 systems? Never say never, or always. ;-) 
*fist shake*
-- 
Adam Williamson
Fedora QA Community Monkey
IRC: adamw | Twitter: AdamW_Fedora | identi.ca: adamwfedora
http://www.happyassassin.net

[Kevin Kofler]

Felix Miata wrote:
>  Is "never" appropriate even if one's own experience is
> with 3 systems? 7
>  systems? 13 systems? 40 systems? Never say never, or always. ;-) 
That can widen the class of affected devices, but from there to "all Intel 
WiFi" is still a long stretch. (For a starter, last I checked they didn't 
even all use the same driver, but some legacy chipsets had legacy drivers. 
Then AFAIK there are different code paths in iwlwifi for different device 
generations, and also different firmware. And then of course different 
hardware behaves differently. Plus, WiFi issues also tend to depend on the 
exact configuration you're using, in particular on how the access point is 
set up. And finally there can be interaction with other hardware or drivers 
on the machine.)

        Kevin Kofler

[Roberto Ragusa]

On 06/14/2012 04:54 AM, Kevin Kofler wrote:
>  Felix Miata wrote:
> > Is "never" appropriate even if one's own experience is with 3 systems?
> 7
> > systems? 13 systems? 40 systems? Never say never, or always. ;-)
>  
>  That can widen the class of affected devices, but from there to "all Intel 
>  WiFi" is still a long stretch. (For a starter, last I checked they didn't 
>  even all use the same driver, but some legacy chipsets had legacy drivers. 
>  Then AFAIK there are different code paths in iwlwifi for different device 
>  generations, and also different firmware. And then of course different 
>  hardware behaves differently. Plus, WiFi issues also tend to depend on the 
>  exact configuration you're using, in particular on how the access point is 
>  set up. And finally there can be interaction with other hardware or drivers 
>  on the machine.) 
And do not forget the BIOS version, especially for suspend-related ACPI bugs.

-- 
   Roberto Ragusa    mail at robertoragusa.it