i686 kernel bug priority plan

[Josh Boyer]

Hello Fellow Contributors!

Our kernel team has been adrift in bugs roughly forever. The time we
spend on triage is time we don't get to spend fixing substantial bugs
on hardware lots of people are using, especially new/emergent
hardware. We have spoken previously about the difficulty of kernel bug
triage as a Fedora effort, and also about future kernel direction at
Flock 2014[1].

The Fedora kernel engineers are taking a new approach we think will
result in a better kernel overall. We can do more substantial fixes
for users, have more interaction upstream, and minimize burnout for
the team members. Not coincidentally, we think this will
correspondingly strengthen relations between Fedora and kernel
communities. We enjoy a healthy relationship there, but we're always
looking to do better.

If we pursue the goal of fixing bugs based on impact, we can enjoy
these benefits. We see impact as a combination of incidence rate and
criticality. But as with all things there are tradeoffs. One of the
tradeoffs we foresee is that i686 bugs are unlikely to rise to such a
high level of impact. Plenty of bugs are common across architectures,
and fixing those bugs will benefit i686 too.

This is an opportunity for community members for whom i686 is a
passion to join the team to participate in the bugfixing required. We
previously called out the need for assistance[2], but had no
substantial response. We hope that being transparent about our
priorities will prompt interested community members to help. If you
are interested to help, here are some resources to get started:

* Kernel team wiki page: https://fedoraproject.org/wiki/Kernel --
includes mailing list and IRC information
* Bugzilla query: http://tinyurl.com/k6zhz34 -- some example bugs that
could use your help

It's possible down the road that, if there is no community interest in
i686, the project might look at other options such as making i686 a
secondary architecture. This is not because we want to drive away
32-bit users; but we're passionate about making the Fedora kernel work
well for the majority of our user base. This prioritization helps us
get closer to that goal.

Our Fedora kernel engineers are devoted full time to supporting
Fedora. Just as any other community members, we have to choose where
to spend our cycles because those cycles are finite. With this
strategy we think we can make Fedora a better system overall, across
all our editions.

Thanks

josh, for the Fedora Kernel Team

[1] <http://www.youtube.com/watch?v=F9E0FF4XFDw>
[2] <http://jwboyer.livejournal.com/49909.html>

[drago01]

On Wed, Feb 25, 2015 at 2:15 PM, Josh Boyer <jwboyer(a)fedoraproject.org&gt; wrote:
>  Hello Fellow Contributors!
>
>  [...]
>  It's possible down the road that, if there is no community interest in
>  i686, the project might look at other options such as making i686 a
>  secondary architecture. This is not because we want to drive away
>  32-bit users; but we're passionate about making the Fedora kernel work
>  well for the majority of our user base. This prioritization helps us
>  get closer to that goal. 
Just to make this clear because this has suggestion has been brought
up multiple times ... while there might be less interests in running a
i686 kernel the story is very different for i686 user space (mostly
libraries but also applications like wine) even on a x86_64 host
kernel.

So don't draw a line from "no interests in i686 kernel" to "no
interesst in the i686 architecture and therefore it should be
secondary" .. its not as simple as with a completely isolated
architecture like ppc.

[Josh Boyer]

On Wed, Feb 25, 2015 at 8:30 AM, drago01 <drago01(a)gmail.com&gt; wrote:
>  On Wed, Feb 25, 2015 at 2:15 PM, Josh Boyer
> <jwboyer(a)fedoraproject.org&gt; wrote:
> > Hello Fellow Contributors!
> >
> > [...]
> > It's possible down the road that, if there is no community interest in
> > i686, the project might look at other options such as making i686 a
> > secondary architecture. This is not because we want to drive away
> > 32-bit users; but we're passionate about making the Fedora kernel work
> > well for the majority of our user base. This prioritization helps us
> > get closer to that goal.
>
>  Just to make this clear because this has suggestion has been brought
>  up multiple times ... while there might be less interests in running a
>  i686 kernel the story is very different for i686 user space (mostly
>  libraries but also applications like wine) even on a x86_64 host
>  kernel. 
Yes.  The paragraph didn't say kernel and it certainly didn't say that
the kernel team would be proposing demotion.  It's simply possible
that the rest of the project might look at the results that way.

>  So don't draw a line from "no interests in i686 kernel"
> to "no
>  interesst in the i686 architecture and therefore it should be
>  secondary" .. its not as simple as with a completely isolated
>  architecture like ppc. 
Actually, even your example is flawed.  ppc64 ran ppc userspace almost
exclusively until the RHEL6 era.  It's really kind of the same thing.
The major difference being that both of those architectures lacked
much in the way of users and support at the time they were demoted.

Anyway, thanks for your comment.  It is helpful.

josh

[Stephen Gallagher]

On Wed, 2015-02-25 at 14:30 +0100, drago01 wrote:
>  On Wed, Feb 25, 2015 at 2:15 PM, Josh Boyer <
>  jwboyer(a)fedoraproject.org&gt; wrote:
>  > Hello Fellow Contributors!
>  > 
>  > [...]
>  > It's possible down the road that, if there is no community 
>  > interest in i686, the project might look at other options such as 
>  > making i686 a secondary architecture. This is not because we want 
>  > to drive away 32-bit users; but we're passionate about making the 
>  > Fedora kernel work well for the majority of our user base. This 
>  > prioritization helps us get closer to that goal.
>  
>  Just to make this clear because this has suggestion has been brought 
>  up multiple times ... while there might be less interests in running 
>  a i686 kernel the story is very different for i686 user space 
>  (mostly libraries but also applications like wine) even on a x86_64 
>  host kernel.
>  
>  So don't draw a line from "no interests in i686 kernel" to "no 
>  interesst in the i686 architecture and therefore it should be
>  secondary" .. its not as simple as with a completely isolated 
>  architecture like ppc. 
Right, Josh isn't talking about retiring the i686 runtime 
(particularly in terms of multiarch support on x86_64). Merely that 
work on i686-specific bugs in the kernel are going to see less 
attention so they can focus their limited resources on bugs with a 
wider impact.

[Stephen Gallagher]

On Wed, 2015-02-25 at 08:15 -0500, Josh Boyer wrote:

<snip>
>  This is an opportunity for community members for whom i686 is a 
>  passion to join the team to participate in the bugfixing required. 
>  We previously called out the need for assistance[2], but had no
>  substantial response. We hope that being transparent about our 
>  priorities will prompt interested community members to help. If you 
>  are interested to help, here are some resources to get started: 
<snip>

Given our past history with passive requests for help, I think it 
might be beneficial to put a time limit on things. For example, if no 
i686 Kernel SIG steps up and agrees to handle bugs on that platform, 
we should strongly consider demoting i686 to secondary architecture 
for installation media at the Fedora 23 branch point (about six months 
from now).

Note: I'm only suggesting that we would demote the kernel and install 
media, *NOT* the 32-bit runtime libraries on an x86_64 system; we 
would still need to be building those.

[Peter Robinson]

> > This is an opportunity for community members for whom i686 is a
> > passion to join the team to participate in the bugfixing required.
> > We previously called out the need for assistance[2], but had no
> > substantial response. We hope that being transparent about our
> > priorities will prompt interested community members to help. If you
> > are interested to help, here are some resources to get started:
>  <snip>
>
>  Given our past history with passive requests for help, I think it
>  might be beneficial to put a time limit on things. For example, if no
>  i686 Kernel SIG steps up and agrees to handle bugs on that platform,
>  we should strongly consider demoting i686 to secondary architecture
>  for installation media at the Fedora 23 branch point (about six months
>  from now).
>
>  Note: I'm only suggesting that we would demote the kernel and install
>  media, *NOT* the 32-bit runtime libraries on an x86_64 system; we
>  would still need to be building those. 
How do you propose demoting to secondary only the kernel/install
without the userspace? Do you mean de-emphasise the i686 bit platform?

Peter

[Jóhann B. Guðmundsson]

On 02/25/2015 01:50 PM, Stephen Gallagher wrote:
>  i686 Kernel SIG steps up and agrees to handle bugs on that platform
Interesting so you are saying that the kernel sub community is 
effectively only x86_64

If you are going down that road you better ask for every <arch> kernel 
SIG to emerge or step up in the process.

Where does ARM fit in this plan

JBG

[Peter Robinson]

> > i686 Kernel SIG steps up and agrees to handle bugs on that
> platform
>
>
>  Interesting so you are saying that the kernel sub community is effectively
>  only x86_64
>
>  If you are going down that road you better ask for every <arch> kernel SIG
>  to emerge or step up in the process.
>
>  Where does ARM fit in this plan 
The ARM SIG has always been primary on the ARM kernel and associated issues.

Peter

[Peter Robinson]

On Wed, Feb 25, 2015 at 2:14 PM, Peter Robinson <pbrobinson(a)gmail.com&gt; wrote:
> >> i686 Kernel SIG steps up and agrees to handle bugs on that
> platform
> >
> >
> > Interesting so you are saying that the kernel sub community is effectively
> > only x86_64
> >
> > If you are going down that road you better ask for every <arch> kernel SIG
> > to emerge or step up in the process.
> >
> > Where does ARM fit in this plan
>
>  The ARM SIG has always been primary on the ARM kernel and associated issues. 
For that matter so have aarch64/ppc64/s390 SIGs

Peter

[Josh Boyer]

On Wed, Feb 25, 2015 at 9:15 AM, Peter Robinson <pbrobinson(a)gmail.com&gt; wrote:
>  On Wed, Feb 25, 2015 at 2:14 PM, Peter Robinson
> <pbrobinson(a)gmail.com&gt; wrote:
> >>> i686 Kernel SIG steps up and agrees to handle bugs on that platform
> >>
> >>
> >> Interesting so you are saying that the kernel sub community is effectively
> >> only x86_64
> >>
> >> If you are going down that road you better ask for every <arch> kernel SIG
> >> to emerge or step up in the process.
> >>
> >> Where does ARM fit in this plan
> >
> > The ARM SIG has always been primary on the ARM kernel and associated issues.
>
>  For that matter so have aarch64/ppc64/s390 SIGs 
Yes, and they are model examples of teams passionate about something
and working towards it.  I try and give all the credit for the success
of those architectures to those teams whenever possible.  The Fedora
kernel engineers do very very little for those kernels, so a hearty
thank you once more.

josh

[Jóhann B. Guðmundsson]

On 02/25/2015 02:15 PM, Peter Robinson wrote:
>  On Wed, Feb 25, 2015 at 2:14 PM, Peter Robinson
> <pbrobinson(a)gmail.com&gt; wrote:
> >>> i686 Kernel SIG steps up and agrees to handle bugs on that platform
> >>
> >> Interesting so you are saying that the kernel sub community is effectively
> >> only x86_64
> >>
> >> If you are going down that road you better ask for every <arch> kernel SIG
> >> to emerge or step up in the process.
> >>
> >> Where does ARM fit in this plan
> > The ARM SIG has always been primary on the ARM kernel and associated issues.
>  For that matter so have aarch64/ppc64/s390 SIGs 
Should not the kernel list be renamed to x86_64 to be in-line with and 
follow the tradition of the other architectures mailinglist setup?

JBG

[Peter Robinson]

> >>>> i686 Kernel SIG steps up and agrees to handle bugs on
> that platform
> >>>
> >>>
> >>> Interesting so you are saying that the kernel sub community is
> >>> effectively
> >>> only x86_64
> >>>
> >>> If you are going down that road you better ask for every <arch> kernel
> >>> SIG
> >>> to emerge or step up in the process.
> >>>
> >>> Where does ARM fit in this plan
> >>
> >> The ARM SIG has always been primary on the ARM kernel and associated
> >> issues.
> >
> > For that matter so have aarch64/ppc64/s390 SIGs
>
>
>  Should not the kernel list be renamed to x86_64 to be in-line with and
>  follow the tradition of the other architectures mailinglist setup? 
No, if you read the list archives there's discussions across all
architectures about the kernel there, the arch SIG lists cover
userspace, testing and other things other than just kernel.

Peter

[Stephen Gallagher]

On Wed, 2015-02-25 at 14:01 +0000, Peter Robinson wrote:
>  > > This is an opportunity for community members for whom i686
> is a 
>  > > passion to join the team to participate in the bugfixing 
>  > > required. We previously called out the need for assistance[2], 
>  > > but had no substantial response. We hope that being transparent 
>  > > about our priorities will prompt interested community members to 
>  > > help. If you are interested to help, here are some resources to 
>  > > get started:
>  > <snip>
>  > 
>  > Given our past history with passive requests for help, I think it 
>  > might be beneficial to put a time limit on things. For example, if 
>  > no i686 Kernel SIG steps up and agrees to handle bugs on that 
>  > platform, we should strongly consider demoting i686 to secondary 
>  > architecture for installation media at the Fedora 23 branch point 
>  > (about six months from now).
>  > 
>  > Note: I'm only suggesting that we would demote the kernel and 
>  > install media, *NOT* the 32-bit runtime libraries on an x86_64 
>  > system; we would still need to be building those.
>  
>  How do you propose demoting to secondary only the kernel/install 
>  without the userspace? Do you mean de-emphasise the i686 bit 
>  platform?
>   
Well, that was more a political and rel-eng statement than a technical 
one. Effectively, we'd not build the kernel package or do release-
blocking composes of i686 media. However, the Koji build-system would 
still have to be capable of producing i686 builds as a primary 
architecture. So yes, from a purely technical standpoint, it's a muddy 
statement.

[Bruno Wolff III]

On Wed, Feb 25, 2015 at 08:15:51 -0500,
  Josh Boyer <jwboyer(a)fedoraproject.org&gt; wrote:
> This is an opportunity for community members for whom i686 is a
> passion to join the team to participate in the bugfixing required. We
> previously called out the need for assistance[2], but had no
> substantial response. We hope that being transparent about our
> priorities will prompt interested community members to help. If you
> are interested to help, here are some resources to get started: 
I try to help with the bugs I run accross. I do bisects to try to find 
commits and have some interactions with upstream and report back when 
there are fixes.

I can't afford to replace my old hardware just now. I mostly use PAE 
kernels, except for a backup laptop that can't do them.

I am not good enough at working with kernel code to be able to do much 
more than than try to find the commit that triggers a problem and provide 
various logs to help with debugging.

[Jared K. Smith]

On Wed, Feb 25, 2015 at 8:50 AM, Stephen Gallagher <sgallagh(a)redhat.com&gt;
wrote:

>  we should strongly consider demoting i686 to secondary architecture
>  for installation media at the Fedora 23 branch point (about six months
>  from now).
>  
Is this intended only for 32-bit kernels on x86-based processors, or for
other 32-bit platforms such as ARM as well?  It'd be a shame to see arm7hl
support demoted to secondary arch, especially when there's a strong group
of contributors actively working on issues, and Aarch64 hardware is still
reasonably hard to obtain.

--
Jared Smith

[drago01]

On Thu, Feb 26, 2015 at 11:38 AM, Jared K. Smith
<jsmith(a)fedoraproject.org&gt; wrote:
>  On Wed, Feb 25, 2015 at 8:50 AM, Stephen Gallagher <sgallagh(a)redhat.com&gt;
>  wrote:
> >
> > we should strongly consider demoting i686 to secondary architecture
> > for installation media at the Fedora 23 branch point (about six months
> > from now).
>
>
>  Is this intended only for 32-bit kernels on x86-based processors, or for
>  other 32-bit platforms such as ARM as well?  It'd be a shame to see arm7hl
>  support demoted to secondary arch, especially when there's a strong group of
>  contributors actively working on issues, and Aarch64 hardware is still
>  reasonably hard to obtain. 
i686 has nothing to do with ARM. So when someone says "i686" he/she
means "32bit x86".

[Jared K. Smith]

On Thu, Feb 26, 2015 at 5:40 AM, drago01 <drago01(a)gmail.com&gt; wrote:

>  i686 has nothing to do with ARM. So when someone says
> "i686" he/she
>  means "32bit x86".
>  

Well yes -- that's what *I* take it to mean -- I just wanted to make sure
that's what everyone else took it to mean as well.  I just wanted to make
sure that the conversation really was around 32-bit x86 and not 32-bit in
general.

Sorry if I caused any confusion.

--
Jared Smith

[Josh Boyer]

On Thu, Feb 26, 2015 at 5:43 AM, Jared K. Smith
<jsmith(a)fedoraproject.org&gt; wrote:
>  On Thu, Feb 26, 2015 at 5:40 AM, drago01 <drago01(a)gmail.com&gt; wrote:
> >
> > i686 has nothing to do with ARM. So when someone says "i686" he/she
> > means "32bit x86".
>
>
>
>  Well yes -- that's what *I* take it to mean -- I just wanted to make sure that's
> what everyone else took it to mean as well.  I just wanted to make sure that the
> conversation really was around 32-bit x86 and not 32-bit in general. 
i686 means i686, not all of 32-bit.  I even gave the ARM and other
architecture SIGs kudos and thank yous as examples of how I'd like to
see i686 support grow earlier in the thread...

josh

[Dennis Gilmore]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On Wed, 25 Feb 2015 10:35:22 -0500
Stephen Gallagher <sgallagh(a)redhat.com&gt; wrote:

>  On Wed, 2015-02-25 at 14:01 +0000, Peter Robinson wrote:
>  > > > This is an opportunity for community members for whom i686 is a 
>  > > > passion to join the team to participate in the bugfixing 
>  > > > required. We previously called out the need for assistance[2], 
>  > > > but had no substantial response. We hope that being transparent 
>  > > > about our priorities will prompt interested community members
>  > > > to help. If you are interested to help, here are some resources
>  > > > to get started:
>  > > <snip>
>  > > 
>  > > Given our past history with passive requests for help, I think it 
>  > > might be beneficial to put a time limit on things. For example,
>  > > if no i686 Kernel SIG steps up and agrees to handle bugs on that 
>  > > platform, we should strongly consider demoting i686 to secondary 
>  > > architecture for installation media at the Fedora 23 branch point 
>  > > (about six months from now).
>  > > 
>  > > Note: I'm only suggesting that we would demote the kernel and 
>  > > install media, *NOT* the 32-bit runtime libraries on an x86_64 
>  > > system; we would still need to be building those.
>  > 
>  > How do you propose demoting to secondary only the kernel/install 
>  > without the userspace? Do you mean de-emphasise the i686 bit 
>  > platform?
>  > 
>  
>  Well, that was more a political and rel-eng statement than a
>  technical one. Effectively, we'd not build the kernel package or do
>  release- blocking composes of i686 media. However, the Koji
>  build-system would still have to be capable of producing i686 builds
>  as a primary architecture. So yes, from a purely technical
>  standpoint, it's a muddy statement.
>  
from a purely technical point of view it would be a complete and utter
mess. given how secondary arches work, we would be largely screwed
supporting such a model. at least without sitting down and majorly
redoing things.

Dennis
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v2
>
> iQIcBAEBAgAGBQJU8MkfAAoJEH7ltONmPFDRrl0P/istndwIaewWAqOS8kRsXggc
> G6BXc+v6/fLwuZDzAcCD9f4ZcmvmVslSREaNI5W+GjEIIY4mUgsxfXR9BQuGm3OH
> +NZi5JLBtRdby36sstEIjM1OaKdi3vhB/7WwxZH8D/e1BEtmd3RI5ERORjAbUvqB
> Ogm8nKmEOUMezqxWGu+sBYHMdoYIK0rf/PACJ4NzzNeJLrU1qtOwVif78E+dPZ0F
> FWUlB8YckTUVrYDHk2ww/iNhhgwKaLpR4bLZf8QnqKiqhozpSw2D/MxLGHTUHEqW
> 0/lpFHi87rXC2vjHcsgNzzRwIoSJnhExi3BWzijy4r66jh/lRIl5Gxc/uEn45ANB
> iseDJJHrhulqigxeq/4zpc3iPpBssOrZmfWtc+tBjKHaZBjWPZ8YF38xN2Aa1uHl
> a/d7rh/XhBtYgs2BmvxkTd/sALfRONK8RVG9aCL8rY8Upck1ENfPx9DvCf1duLeO
> zwcnpC8ZeGv4JKMC5Mn6PdFqthPZUxaYlkZ+lYlE8+I3EnXOF0o31+bPdpQgvn/i
> EUuY1BBpldhmLe8GWZpqfABS6WoX9ErPTGqpB4iAZ4L2hgSn4PWnxCbdjc0Oak4w
> 4dOqTQ/rAHABcjpL07pJ2F+EMKFbDTxh5yCjTCsuIWRT9G6wSYg58dFC9Veu+gN1
> KoZifUI74JXo+NjGDmWv
> =uxsR
> -----END PGP SIGNATURE-----
>