The GNU C Library will be rebased in F21 to match glibc 2.20.

[Carlos O'Donell]

Fedora,

This is a reminder that the glibc team will be rebasing
glibc in F21 to match glibc 2.20.

The plan remains largely as was written here:
http://fedoraproject.org/wiki/Changes/GLIBC220

Only glibc 2.20 has ABI guarantees, and therefore we
will move to 2.20 before F21 goes to GA to ensure that
F21 has a strong ABI guarantee.

I expect this to be the process forever going forward:

* Rawhide tracks glibc master.
* Fedora release is branched from Rawhide.
* glibc release is made upstream.
* Fedora branch is rebased on glibc upstream release
  to include ABI guarantees.
* Fedora release goes GA.

The changes in the rebase should be minor since we've
been tracking master the whole time.

Cheers,
Carlos.

[Matthew Miller]

On Wed, Jul 30, 2014 at 01:18:57AM -0400, Carlos O'Donell wrote:
>  I expect this to be the process forever going forward:
>  
>  * Rawhide tracks glibc master.
>  * Fedora release is branched from Rawhide.
>  * glibc release is made upstream.
>  * Fedora branch is rebased on glibc upstream release
>    to include ABI guarantees.
>  * Fedora release goes GA.
>  
>  The changes in the rebase should be minor since we've
>  been tracking master the whole time. 
Looks reasonable. Thanks for the update.

-- 
Matthew Miller
<mattdm(a)fedoraproject.org&gt;
Fedora Project Leader

[Rahul Sundaram]

Hi


On Wed, Jul 30, 2014 at 1:18 AM, Carlos O'Donell wrote:

>  I expect this to be the process forever going forward:
>
>  * Rawhide tracks glibc master.
>  * Fedora release is branched from Rawhide.
>  * glibc release is made upstream.
>  * Fedora branch is rebased on glibc upstream release
>    to include ABI guarantees.
>  * Fedora release goes GA.
>
>  The changes in the rebase should be minor since we've
>  been tracking master the whole time.
>  
I remember a similar process causing problems in Rawhide earlier and Jared
Smith talking with the glibc team to ensure that only regular releases hit
rawhide.

Rahul

[Matthew Miller]

On Wed, Jul 30, 2014 at 03:16:30PM -0400, Rahul Sundaram wrote:
>  > * Rawhide tracks glibc master.
>  > * Fedora release is branched from Rawhide.
>  > * glibc release is made upstream.
>  > * Fedora branch is rebased on glibc upstream release
>  >   to include ABI guarantees.
>  > * Fedora release goes GA.
>  > The changes in the rebase should be minor since we've
>  > been tracking master the whole time.
>  I remember a similar process causing problems in Rawhide earlier and Jared
>  Smith talking with the glibc team to ensure that only regular releases hit
>  rawhide. 
It would be nice if we're careful in staging ABI changes in the glibc master
into Rawhide so it doesn't break. (Possibly something we can do with
Taskotron?)


-- 
Matthew Miller
<mattdm(a)fedoraproject.org&gt;
Fedora Project Leader

[Siddhesh Poyarekar]

On Wed, Jul 30, 2014 at 03:36:34PM -0400, Matthew Miller wrote:
>  > > The changes in the rebase should be minor since we've
>  > > been tracking master the whole time.
>  > I remember a similar process causing problems in Rawhide earlier and Jared
>  > Smith talking with the glibc team to ensure that only regular releases hit
>  > rawhide. 
The problem with that approach is that lots of bugs go unnoticed until
very late in rawhide, resulting in those bugs being caught and fixed
only post-release.  Besides, this process is different in principle
from the earlier one because the earlier process simply cut the latest
master into a Fedora release just days before Fedora went gold.

In the current process the glibc release Fedora gets is decided well
in advance, that is when the Fedora release branch is cut from
rawhide.  Combine that with the fact that glibc upstream freezes weeks
before it releases and you'll see that the glibc release that makes it
into a Fedora release is eventually well tested in rawhide *and* is
not ancient.

>  It would be nice if we're careful in staging ABI changes in the
> glibc master
>  into Rawhide so it doesn't break. (Possibly something we can do with
>  Taskotron?) 
ABI changes are a minor problem with respect to glibc because as a
principle, we don't break ABI.  That is, any ABI breakages that happen
are usually bugs.  We try to ensure that with a basic ABI checker in
the glibc testsuite and in future we'll be looking to enhance our ABI
checking capabilities.  Taskotron (based on a quick read from the
wiki) seems to be a good framework for this.

The bigger problem is bugs that brick your system because unlike
kernel, there is currently no way to simply reboot into an older
known-to-work glibc.  We had one such situation last week and IIRC the
last such situation before that was close to two years ago, so while
it is serious, it is not frequent.

That said, we're also thinking of ways to recover from such botched
updates.

Siddhesh

[Rahul Sundaram]

Hi


On Wed, Jul 30, 2014 at 11:12 PM, Siddhesh Poyarekar  wrote:

>  The problem with that approach is that lots of bugs go unnoticed until
>  very late in rawhide, resulting in those bugs being caught and fixed
>  only post-release.  ABI.  That is, any ABI breakages that happen
>  are usually bugs.  We try to ensure that with a basic ABI checker in
>  the glibc testsuite and in future we'll be looking to enhance our ABI
>  checking capabilities.  Taskotron (based on a quick read from the
>  wiki) seems to be a good framework for this.
>  
Yes.  My concern is that glibc is using Rawhide as their continuous
integration sandbox to shake out bugs as opposed to doing it elsewhere and
just taking care of integration of releases when they are ready.   If this
viewed as a gap that needs to be fixed, that's fine for now.

Rahul

[Richard W.M. Jones]

On Thu, Jul 31, 2014 at 09:03:45AM -0400, Rahul Sundaram wrote:
>  Hi
>  
>  
>  On Wed, Jul 30, 2014 at 11:12 PM, Siddhesh Poyarekar  wrote:
>  
>  >
>  > The problem with that approach is that lots of bugs go unnoticed until
>  > very late in rawhide, resulting in those bugs being caught and fixed
>  > only post-release.  ABI.  That is, any ABI breakages that happen
>  > are usually bugs.  We try to ensure that with a basic ABI checker in
>  > the glibc testsuite and in future we'll be looking to enhance our ABI
>  > checking capabilities.  Taskotron (based on a quick read from the
>  > wiki) seems to be a good framework for this.
>  >
>  
>  Yes.  My concern is that glibc is using Rawhide as their continuous
>  integration sandbox to shake out bugs as opposed to doing it elsewhere and
>  just taking care of integration of releases when they are ready. 
I'm actually not concerned about Rawhide being used like this.  That's
kind of the purpose of Rawhide.

However the previous problem(s -- multiple) was glibc using
non-Rawhide for integration testing, especially just while we were
trying to stablise Fedora for a release:

https://lists.fedoraproject.org/pipermail/devel/2011-October/158440.html
https://lists.fedoraproject.org/pipermail/devel/2011-October/158662.html

But 2.20 is a stable branch, right?  So this shouldn't be a concern now.

Rich.

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
virt-builder quickly builds VMs from scratch
http://libguestfs.org/virt-builder.1.html

[Siddhesh Poyarekar]

On Thu, Jul 31, 2014 at 03:06:00PM +0100, Richard W.M. Jones wrote:
>  However the previous problem(s -- multiple) was glibc using
>  non-Rawhide for integration testing, especially just while we were
>  trying to stablise Fedora for a release:
>  
>  https://lists.fedoraproject.org/pipermail/devel/2011-October/158440.html
>  https://lists.fedoraproject.org/pipermail/devel/2011-October/158662.html 
We're absolutely not going to rebase master in non-rawhide.  Our
intention is to get glibc 2.20 release out before Fedora turns beta.

>  But 2.20 is a stable branch, right?  So this shouldn't be a
> concern
>  now. 
Yes, it is in what we call, a slushy freeze right now.  Changes going
in are mostly regression fixes and other fixes that should not have
any generated code impact.  There is however one change that the s/390
folks will find painful - s/390 broke ABI in 2.19 without a proper
justification and that is now going to be reverted in 2.20 (and in
2.19, but we don't really care about 2.19 right now).

Siddhesh

[Josh Boyer]

On Thu, Jul 31, 2014 at 12:34 PM, Siddhesh Poyarekar
<siddhesh(a)redhat.com&gt; wrote:
>  On Thu, Jul 31, 2014 at 03:06:00PM +0100, Richard W.M. Jones wrote:
> > However the previous problem(s -- multiple) was glibc using
> > non-Rawhide for integration testing, especially just while we were
> > trying to stablise Fedora for a release:
> >
> > https://lists.fedoraproject.org/pipermail/devel/2011-October/158440.html
> > https://lists.fedoraproject.org/pipermail/devel/2011-October/158662.html
>
>  We're absolutely not going to rebase master in non-rawhide.  Our
>  intention is to get glibc 2.20 release out before Fedora turns beta.
>
> > But 2.20 is a stable branch, right?  So this shouldn't be a concern
> > now.
>
>  Yes, it is in what we call, a slushy freeze right now.  Changes going
>  in are mostly regression fixes and other fixes that should not have
>  any generated code impact.  There is however one change that the s/390
>  folks will find painful - s/390 broke ABI in 2.19 without a proper
>  justification and that is now going to be reverted in 2.20 (and in
>  2.19, but we don't really care about 2.19 right now). 
To expand on this just a bit, the s/390 ABI break was done by the
s/390 glibc port maintainers and was done intentionally.  This wasn't
a "oops, we accidentally broke the ABI" bug.  It was something the
s/390 maintainers decided was of minimal concern and it turns out they
were wrong.  In my experience, glibc goes above and beyond maintaining
their software.  This s/390 incident is an anomaly in an otherwise
very controlled and well-run software project.

josh

[Richard W.M. Jones]

On Thu, Jul 31, 2014 at 01:41:25PM -0400, Josh Boyer wrote:
>  On Thu, Jul 31, 2014 at 12:34 PM, Siddhesh Poyarekar
>  > There is however one change that the s/390
>  > folks will find painful - s/390 broke ABI in 2.19 without a proper
>  > justification and that is now going to be reverted in 2.20 (and in
>  > 2.19, but we don't really care about 2.19 right now).
>  
>  To expand on this just a bit, the s/390 ABI break was done by the
>  s/390 glibc port maintainers and was done intentionally.  This wasn't
>  a "oops, we accidentally broke the ABI" bug.  It was something the
>  s/390 maintainers decided was of minimal concern and it turns out they
>  were wrong.  In my experience, glibc goes above and beyond maintaining
>  their software.  This s/390 incident is an anomaly in an otherwise
>  very controlled and well-run software project. 
(FYI) LWN article on this subject:

https://lwn.net/Articles/605607/

Rich.

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
Fedora Windows cross-compiler. Compile Windows programs, test, and
build Windows installers. Over 100 libraries supported.
http://fedoraproject.org/wiki/MinGW

[Dan Horák]

On Thu, 31 Jul 2014 19:19:49 +0100
"Richard W.M. Jones" <rjones(a)redhat.com&gt; wrote:

>  On Thu, Jul 31, 2014 at 01:41:25PM -0400, Josh Boyer wrote:
>  > On Thu, Jul 31, 2014 at 12:34 PM, Siddhesh Poyarekar
>  > > There is however one change that the s/390
>  > > folks will find painful - s/390 broke ABI in 2.19 without a proper
>  > > justification and that is now going to be reverted in 2.20 (and in
>  > > 2.19, but we don't really care about 2.19 right now).
>  > 
>  > To expand on this just a bit, the s/390 ABI break was done by the
>  > s/390 glibc port maintainers and was done intentionally.  This
>  > wasn't a "oops, we accidentally broke the ABI" bug.  It was
>  > something the s/390 maintainers decided was of minimal concern and
>  > it turns out they were wrong.  In my experience, glibc goes above
>  > and beyond maintaining their software.  This s/390 incident is an
>  > anomaly in an otherwise very controlled and well-run software
>  > project.
>  
>  (FYI) LWN article on this subject:
>  
>  https://lwn.net/Articles/605607/ 
yeah, it can happen, the hardest thing was to come up with a solution
that would allow to follow the regular Rawhide build process using
koji-shadow.


		Dan

[Florian Weimer]

On 07/31/2014 07:41 PM, Josh Boyer wrote:
>  To expand on this just a bit, the s/390 ABI break was done by the
>  s/390 glibc port maintainers and was done intentionally.  This wasn't
>  a "oops, we accidentally broke the ABI" bug.  It was something the
>  s/390 maintainers decided was of minimal concern and it turns out they
>  were wrong. 
And we couldn't have avoided that if we used only released glibc 
versions in rawhide because the 2.19 release had the ABI change, too. 
It wasn't just in the git master branch.

-- 
Florian Weimer / Red Hat Product Security

[Siddhesh Poyarekar]

On Thu, Jul 31, 2014 at 09:03:45AM -0400, Rahul Sundaram wrote:
>  Yes.  My concern is that glibc is using Rawhide as their continuous
>  integration sandbox to shake out bugs as opposed to doing it elsewhere and
>  just taking care of integration of releases when they are ready.   If this
>  viewed as a gap that needs to be fixed, that's fine for now. 
CI is indeed a gap that needs to be fixed, but rebasing in rawhide
isn't nearly as fatal as it seems.  The upstream glibc commit policy
is quite stringent in that every commit is required to pass an
extensive (but obviously not exhaustive) testsuite and as a result it
is usually safe (as safe one would consider rawhide) to rebase from
the latest commit.  I've been doing regular rebases for a while now
(almost a year AFAICT) and IIRC we have had just one serious problem
last week where i686 boxes got bricked.  In the end it was found to be
a gcc bug that miscompiled some code in glibc.

That said, we still need a nicer way to back out from a broken glibc
update into the previous known-to-work glibc similar to how the kernel
does it and not need a rescue disk for it.

Siddhesh

[Bruno Wolff III]

On Thu, Jul 31, 2014 at 23:05:21 +0530,
  Siddhesh Poyarekar <siddhesh(a)redhat.com&gt; wrote:
> the latest commit.  I've been doing regular rebases for a while now
> (almost a year AFAICT) and IIRC we have had just one serious problem
> last week where i686 boxes got bricked.  In the end it was found to be
> a gcc bug that miscompiled some code in glibc. 
If you figured out what was going on in time you could even recover 
with a yum downgrade in you had a root shell open (which I was able to do).
So not everyone using i686 got their machines bricked.

[Peter Robinson]

On Wed, Jul 30, 2014 at 8:36 PM, Matthew Miller
<mattdm(a)fedoraproject.org&gt; wrote:
>  On Wed, Jul 30, 2014 at 03:16:30PM -0400, Rahul Sundaram wrote:
> > > * Rawhide tracks glibc master.
> > > * Fedora release is branched from Rawhide.
> > > * glibc release is made upstream.
> > > * Fedora branch is rebased on glibc upstream release
> > >   to include ABI guarantees.
> > > * Fedora release goes GA.
> > > The changes in the rebase should be minor since we've
> > > been tracking master the whole time.
> > I remember a similar process causing problems in Rawhide earlier and Jared
> > Smith talking with the glibc team to ensure that only regular releases hit
> > rawhide.
>
>  It would be nice if we're careful in staging ABI changes in the glibc master
>  into Rawhide so it doesn't break. (Possibly something we can do with
>  Taskotron?) 
Ultimately glibc as a core library should be following what the rest
of the packages / features in development change process are required
to do IE anything that might change the ABI should have landed before
the mass rebuild so it can be addressed as part of that and hence be
in a stable state at the same time as the other features are due to be
complete.

The fact that a core library that's stability is critical to the
distribution as a whole doesn't bother to adhere to this and while
having gone through the hoops of putting in a feature change basically
then proceeded to completely ignore the requirements of said process
(ie they dumped a bunch of stuff on the wiki and haven't bothered to
revisit it and update it since) is some what pathetic in my mind.

What happens if between now and when 2.20 goes GA there's a
requirement of an ABI break and we need to rebuild a chuck of the
distribution? It doesn't affect them as they don't need to do the work
to make that happen.

Peter

[Siddhesh Poyarekar]

On Thu, Jul 31, 2014 at 02:13:44PM +0100, Peter Robinson wrote:
>  The fact that a core library that's stability is critical to the
>  distribution as a whole doesn't bother to adhere to this and while
>  having gone through the hoops of putting in a feature change basically
>  then proceeded to completely ignore the requirements of said process
>  (ie they dumped a bunch of stuff on the wiki and haven't bothered to
>  revisit it and update it since) is some what pathetic in my mind. 
As an upstream policy, glibc does not break ABI[1] and we've tried to
adhere to this ever since I started contributing to glibc (which is ~2
years ago).  Any ABI break that happens is a bug and the ABI impact
Carlos talks about is mainly due to such bugs being discovered in
rawhide, which again is not very often.

Siddhesh

[1] We did unknowingly break ABI in S/390 in 2.19 and that will likely
    be fixed in 2.20 and require at least a partial rebuild (perl and
    its modules), but you would have needed the mass rebuild even if
    we would have directly introduced a 2.20 release.