First stage of glibc recvmsg/sendmsg ABI revert landed in rawhide

[Florian Weimer]

glibc upstream, during development of the 2.24 release, introduced new 
symbol versions recvmsg(a)GLIBC_2.24, sendmsg(a)GLIBC_2.24 (and 
recvmmsg(a)GLIBC_2.24, sendmmsg(a)GLIBC_2.24 on 64-bit architectures), in 
order to fix some minor POSIX compliance issue.  (POSIX and the Linux 
kernel disagree about the width of some fields in struct msghdr.)  These 
changes landed in rawhide as part of glibc-2.23.90-19.fc25.

This change caused quite a few issues (chrony stopped building, Address 
Sanitizer interception of these functions was affected, probably more). 
Considering that the deviation from POSIX was really minor, this was 
considered a poor trade-off, and the patch and ABI change was eventually 
reverted upstream.

We cannot implement the ABI reversal immediately in rawhide because that 
would break existing binaries, and we don't want to do a bootstrap or 
mass rebuild for this.  Therefore, glibc-2.23.90-21.fc25 turns the new 
symbols into compatibility symbols.  As a result, new binaries will be 
linked against the old symbol versions, as before, and old binaries 
continue to work.

(glibc-2.23.90-22.fc25 adds some polishing, but this version is 
functionally equivalent to the previous one.)

Based on Sunday's compose, the list of packages which need to rebuild is 
fairly small:

                  source
----------------------------------------
  NetworkManager-1.2.2-2.fc25
  OpenIPMI-2.0.22-1.fc25
  OpenImageIO-1.6.14-1.fc25
  boost-1.60.0-7.fc25
  chrony-2.4-1.fc25
  collectd-5.5.1-12.fc25
  cryptsetup-1.7.2-1.fc25
  firefox-47.0-4.fc25
  freerdp-2.0.0-9.git.aa15327.fc25
  gdb-7.11.1-75.fc25
  glib2-2.49.1-2.fc25
  glibc-2.23.90-20.fc25
  gnutls-3.4.13-1.fc25
  haproxy-1.6.5-2.fc25
  hostapd-2.5-4.fc25
  java-1.8.0-openjdk-1.8.0.92-1.b14.fc25
  libguestfs-1.33.35-1.fc25
  libnice-0.1.13-6.fc25
  libvirt-1.3.5-1.fc25
  lxc-2.0.1-1.fc25
  mono-4.4.0-5.fc25
  nftables-0.6-1.fc25
  ntp-4.2.6p5-41.fc25
  openssh-7.2p2-7.fc25
  php-5.6.23-0.1.RC1.fc25
  python-twisted-16.1.1-3.fc25
  qt5-qtbase-5.6.1-2.fc25
  qt5-qtwebengine-5.6.1-1.fc25
  qt5-qtwebkit-5.6.1-1.b889f46git.fc25
  root-6.06.04-2.fc25
  samba-4.4.4-1.fc25
  thunderbird-45.1.1-2.fc25
  tigervnc-1.6.0-6.fc25
  weechat-1.5-1.fc25
  wine-1.9.11-1.fc25
(35 rows)

I'll wait some time to see how many of those pick up the old ABI due to 
regular development.  Probably around mid-July, I will start pinging 
developers if rebuilds are missing.  We must remove the compatibility 
symbol before the release, so I'll have to enlist help of provenpackages 
eventually to trigger missing rebuilds.

Thanks,
Florian

[Richard W.M. Jones]

On Mon, Jun 13, 2016 at 07:09:33AM +0200, Florian Weimer wrote:
>  glibc upstream, during development of the 2.24 release, introduced
>  new symbol versions recvmsg(a)GLIBC_2.24, sendmsg(a)GLIBC_2.24 (and
>  recvmmsg(a)GLIBC_2.24, sendmmsg(a)GLIBC_2.24 on 64-bit architectures),
>  in order to fix some minor POSIX compliance issue.  (POSIX and the
>  Linux kernel disagree about the width of some fields in struct
>  msghdr.)  These changes landed in rawhide as part of
>  glibc-2.23.90-19.fc25. 
This may explain why some old code we had that sent file descriptors
around using SCM_RIGHTS suddenly broke last week in Rawhide.  I was
going to investigate it, but once I had a look at our code, I ended up
completely rewriting it instead, and that appears to have fixed it,
but perhaps that was just as a side effect of relinking (the old code
needed to be rewritten anyway).

Here was the change I made, in case it's interesting:
https://github.com/libguestfs/libguestfs/commit/69e5410e6264f58082883beb5...

Libvirt also broke with some kind of file descriptor passing error:

https://bugzilla.redhat.com/show_bug.cgi?id=1344016

No one has really looked at the libvirt problem, but Cole pointed me
at this thread.

I tried just installing glibc 2.23.90-22, but even after a reboot that
did not appear to fix libvirt.

Recompiling libvirt won't be a problem.

>   libguestfs-1.33.35-1.fc25
>   libvirt-1.3.5-1.fc25 
I will do these two.

Rich.

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
virt-builder quickly builds VMs from scratch
http://libguestfs.org/virt-builder.1.html

[Florian Weimer]

On 06/13/2016 03:26 PM, Richard W.M. Jones wrote:
>  I tried just installing glibc 2.23.90-22, but even after a reboot
> that
>  did not appear to fix libvirt. 
If you don't rebuild, the code will use the GLIBC_2.24 symbols, which 
have the POSIX conformance enhancement.

Florian

[Richard W.M. Jones]

On Mon, Jun 13, 2016 at 03:37:26PM +0200, Florian Weimer wrote:
>  On 06/13/2016 03:26 PM, Richard W.M. Jones wrote:
>  >I tried just installing glibc 2.23.90-22, but even after a reboot that
>  >did not appear to fix libvirt.
>  
>  If you don't rebuild, the code will use the GLIBC_2.24 symbols,
>  which have the POSIX conformance enhancement. 
Thanks - in any case I have rebuilt libvirt against glibc -22 now.

Rich.

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
virt-builder quickly builds VMs from scratch
http://libguestfs.org/virt-builder.1.html

[Daniel P. Berrange]

On Mon, Jun 13, 2016 at 07:09:33AM +0200, Florian Weimer wrote:
>  glibc upstream, during development of the 2.24 release, introduced
> new
>  symbol versions recvmsg(a)GLIBC_2.24, sendmsg(a)GLIBC_2.24 (and
>  recvmmsg(a)GLIBC_2.24, sendmmsg(a)GLIBC_2.24 on 64-bit architectures), in order
>  to fix some minor POSIX compliance issue.  (POSIX and the Linux kernel
>  disagree about the width of some fields in struct msghdr.)  These changes
>  landed in rawhide as part of glibc-2.23.90-19.fc25.
>  
>  This change caused quite a few issues (chrony stopped building, Address
>  Sanitizer interception of these functions was affected, probably more).
>  Considering that the deviation from POSIX was really minor, this was
>  considered a poor trade-off, and the patch and ABI change was eventually
>  reverted upstream. 
Do you have a pointer to the glibc change, or specific details about
what exactly changed. It looks like the change broke libvirt usage
of SCM_RIGHTS, and despite fact that you're reverting it in glibc,
I'm wondering if there is non-standards compliant usage in libvirt
that we should be fixing regardless.

Regards,
Daniel
-- 
|: http://berrange.com      -o-    http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org              -o-             http://virt-manager.org :|
|: http://autobuild.org       -o-         http://search.cpan.org/~danberr/ :|
|: http://entangle-photo.org       -o-       http://live.gnome.org/gtk-vnc :|

[Florian Weimer]

On 06/13/2016 03:52 PM, Daniel P. Berrange wrote:
>  On Mon, Jun 13, 2016 at 07:09:33AM +0200, Florian Weimer wrote:
> > glibc upstream, during development of the 2.24 release, introduced new
> > symbol versions recvmsg(a)GLIBC_2.24, sendmsg(a)GLIBC_2.24 (and
> > recvmmsg(a)GLIBC_2.24, sendmmsg(a)GLIBC_2.24 on 64-bit architectures), in order
> > to fix some minor POSIX compliance issue.  (POSIX and the Linux kernel
> > disagree about the width of some fields in struct msghdr.)  These changes
> > landed in rawhide as part of glibc-2.23.90-19.fc25.
> >
> > This change caused quite a few issues (chrony stopped building, Address
> > Sanitizer interception of these functions was affected, probably more).
> > Considering that the deviation from POSIX was really minor, this was
> > considered a poor trade-off, and the patch and ABI change was eventually
> > reverted upstream.
>
>  Do you have a pointer to the glibc change, or specific details about
>  what exactly changed. It looks like the change broke libvirt usage
>  of SCM_RIGHTS, and despite fact that you're reverting it in glibc,
>  I'm wondering if there is non-standards compliant usage in libvirt
>  that we should be fixing regardless. 
The upstream commit with the revert is here:

 
https://sourceware.org/git/?p=glibc.git;a=commit;h=78880cc185dc521855a580...

I assume the error comes from this code at gnulib/lib/passfd.c:

       /* fake errno: at end the file is not available */
       errno = len ? EACCES : ENOTCONN;

So the ENOTCONN isn't real.

It looks the most likely explanation is that glibc temporarily had the 
struct cmsghdr change in the header file, but no compensating rewriting 
code in the library.  The padding field could be random garbage and 
passed to the kernel, which observes an impossible length value as a result.

Florian

Second stage of glibc recvmsg/sendmsg ABI revert landed in rawhide

[Florian Weimer]

On 06/13/2016 07:09 AM, Florian Weimer wrote:
>  glibc upstream, during development of the 2.24 release, introduced
> new
>  symbol versions recvmsg(a)GLIBC_2.24, sendmsg(a)GLIBC_2.24 (and
>  recvmmsg(a)GLIBC_2.24, sendmmsg(a)GLIBC_2.24 on 64-bit architectures), in
>  order to fix some minor POSIX compliance issue.  (POSIX and the Linux
>  kernel disagree about the width of some fields in struct msghdr.)  These
>  changes landed in rawhide as part of glibc-2.23.90-19.fc25.
>
>  This change caused quite a few issues (chrony stopped building, Address
>  Sanitizer interception of these functions was affected, probably more).
>  Considering that the deviation from POSIX was really minor, this was
>  considered a poor trade-off, and the patch and ABI change was eventually
>  reverted upstream.
>
>  We cannot implement the ABI reversal immediately in rawhide because that
>  would break existing binaries, and we don't want to do a bootstrap or
>  mass rebuild for this.  Therefore, glibc-2.23.90-21.fc25 turns the new
>  symbols into compatibility symbols.  As a result, new binaries will be
>  linked against the old symbol versions, as before, and old binaries
>  continue to work. 
I have removed the compatibility symbols from rawhide, after all the 
necessary rebuilds happened on primary.  I double-checked aarch64, 
ppc64, ppc64le, s390x as well and asked for additional rebuilds where 
appropriate.  (Some packages were built against different glibc versions 
than on primary.)  As a result, except on ppc64/ppc64le, all compat 
symbol references are gone from the build root.  ppc64/ppc64le current 
has unrelated issues and falls farther and farther beyond primary, which 
is why I didn't want to delay the compat removal (and alignment with the 
glibc upstream ABI) until ppc64/ppc64le catches up eventually.  I do not 
expect any problems as long as rebuilds happen roughly in the order they 
happened on primary.

Symptoms of an issue related to the compat symbol removal include error 
messages about undefined GLIBC_2.24 symbols, particularly sendmsg, 
sendmmsg, recvmsg, recvmmsg.  RPM dependencies do not express this 
information because they are not sufficiently fine-grained.

Florian