Auto-generated dependencies for builds against rawhide glibc snapshots

[Florian Weimer]

I've pushed a new glibc build to rawhide (glibc-2.33.9000-29.fc35) that
auto-generates versioned dependencies on glibc if symbols within the
under-development symbol version are used, where the ELF-derived RPM
dependencies are inaccurate.  Given the change to the startup code, this
affects all programs in this release cycle, and the libdl and libpthread
changes mean that most shared objects trigger the versioned dependency
as well.

The dependencies are conservative in the sense that you might a
dependency on a more recent glibc version than that is actually
required by the symbols used.

For the technical background, here's what I put into a comment within
the dependency generator itself:

# A glibc development snapshot (say version 2.33.9000) may define
# symbols in its under-development symbol version (GLIBC_2.34).  RPM
# automatically derives RPM dependencies such as
# libc.so.6(GLIBC_2.34)(64bit) from that.  While the GLIBC_2.34
# version is under development, these dependencies may be inaccurate
# and could be satisfied by glibc RPM package versions that lack the
# symbols because they were created from an earlier development
# snapshot that had some other GLIBC_2.34 symbols.  Therefore, if the
# latest, under-development ELF symbol version is detected, this
# dependency generator adds an explicit RPM dependencies on the glibc
# packaging version against which an RPM package is built.

I've tested this to the best of my abilities, and it seems to work as
expected, but given that this new territory for me, there could be some
issues (like the race condition in the dependency generator discussed in
that other thread).

Thanks,
Florian

[Miro Hrončok]

On 27. 06. 21 20:54, Florian Weimer wrote:
>  I've pushed a new glibc build to rawhide
> (glibc-2.33.9000-29.fc35) that
>  auto-generates versioned dependencies on glibc if symbols within the
>  under-development symbol version are used, where the ELF-derived RPM
>  dependencies are inaccurate.  Given the change to the startup code, this
>  affects all programs in this release cycle, and the libdl and libpthread
>  changes mean that most shared objects trigger the versioned dependency
>  as well.
>  
>  The dependencies are conservative in the sense that you might a
>  dependency on a more recent glibc version than that is actually
>  required by the symbols used.
>  
>  For the technical background, here's what I put into a comment within
>  the dependency generator itself:
>  
>  # A glibc development snapshot (say version 2.33.9000) may define
>  # symbols in its under-development symbol version (GLIBC_2.34).  RPM
>  # automatically derives RPM dependencies such as
>  # libc.so.6(GLIBC_2.34)(64bit) from that.  While the GLIBC_2.34
>  # version is under development, these dependencies may be inaccurate
>  # and could be satisfied by glibc RPM package versions that lack the
>  # symbols because they were created from an earlier development
>  # snapshot that had some other GLIBC_2.34 symbols.  Therefore, if the
>  # latest, under-development ELF symbol version is detected, this
>  # dependency generator adds an explicit RPM dependencies on the glibc
>  # packaging version against which an RPM package is built.
>  
>  I've tested this to the best of my abilities, and it seems to work as
>  expected, but given that this new territory for me, there could be some
>  issues (like the race condition in the dependency generator discussed in
>  that other thread). 
I've noticed two potential issues, but both are unlikely to cause trouble.

The generator is installed in %{_fileattrsdir} and %{_rpmconfigdir}, 
directories owned by the rpm package. Technically, glibc-devel should now 
either require rpm or co-own the directories. But since it is rpm we are 
talking about, this is unlikely to cause any trouble as is.

The generator uses elfutils and grep. Both should be required by glibc-devel 
(at least "if rpm-build"). Currently elfutils is required by rpm-build and grep

is in the default buidroot, so this will not break any time soon, but it might 
break in the future.

-- 
Miro Hrončok
-- 
Phone: [phone]
IRC: mhroncok

[Richard W.M. Jones]

On Sun, Jun 27, 2021 at 08:54:20PM +0200, Florian Weimer wrote:
>  I've pushed a new glibc build to rawhide
> (glibc-2.33.9000-29.fc35) that
>  auto-generates versioned dependencies on glibc if symbols within the
>  under-development symbol version are used, where the ELF-derived RPM
>  dependencies are inaccurate.  Given the change to the startup code, this
>  affects all programs in this release cycle, and the libdl and libpthread
>  changes mean that most shared objects trigger the versioned dependency
>  as well. 
I noticed this change has affected the build of a C library that I did
just now:

https://koji.fedoraproject.org/koji/buildinfo?buildID=1777714

  $ rpm -qR libnbd-1.9.2-1.fc35
  glibc >= 2.33.9000-36.fc35
  ...

This means of course that the library can't be installed without
upgrading to bleeding-edge glibc.  I understand that it's Rawhide, but
still that seems a bit excessive?!  (For context we often install
hand-picked Rawhide packages in older Fedora in order to gain specific
new features - in this case optimized nbdcopy.)

>  The dependencies are conservative in the sense that you might a
>  dependency on a more recent glibc version than that is actually
>  required by the symbols used. 
Is there a way to find out if the dependency is real or conservative?

According to nm -D we're using these symbols from future glibc:

                 U pthread_getspecific@(a)GLIBC_2.34
                 U pthread_key_create@(a)GLIBC_2.34
                 U pthread_key_delete@(a)GLIBC_2.34
                 U pthread_setspecific@(a)GLIBC_2.34

Obviously these symbols are nothing new, but I suppose this might be
related to -lpthread no longer being a separate library?

>  For the technical background, here's what I put into a comment
> within
>  the dependency generator itself:
>  
>  # A glibc development snapshot (say version 2.33.9000) may define
>  # symbols in its under-development symbol version (GLIBC_2.34).  RPM 
This seems applicable.

>  # automatically derives RPM dependencies such as
>  # libc.so.6(GLIBC_2.34)(64bit) from that.  While the GLIBC_2.34
>  # version is under development, these dependencies may be inaccurate
>  # and could be satisfied by glibc RPM package versions that lack the
>  # symbols because they were created from an earlier development
>  # snapshot that had some other GLIBC_2.34 symbols.  Therefore, if the
>  # latest, under-development ELF symbol version is detected, this
>  # dependency generator adds an explicit RPM dependencies on the glibc
>  # packaging version against which an RPM package is built. 
This explains what happened in this case (I guess?) but still leaves
me wondering why pthread_* APIs could be affected by this.

Rich.

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
virt-df lists disk usage of guests without needing to install any
software inside the virtual machine.  Supports Linux and Windows.
http://people.redhat.com/~rjones/virt-df/

[Florian Weimer]

* Richard W. M. Jones:

>  On Sun, Jun 27, 2021 at 08:54:20PM +0200, Florian Weimer wrote:
> > I've pushed a new glibc build to rawhide (glibc-2.33.9000-29.fc35) that
> > auto-generates versioned dependencies on glibc if symbols within the
> > under-development symbol version are used, where the ELF-derived RPM
> > dependencies are inaccurate.  Given the change to the startup code, this
> > affects all programs in this release cycle, and the libdl and libpthread
> > changes mean that most shared objects trigger the versioned dependency
> > as well.
>
>  I noticed this change has affected the build of a C library that I did
>  just now:
>
>  https://koji.fedoraproject.org/koji/buildinfo?buildID=1777714
>
>    $ rpm -qR libnbd-1.9.2-1.fc35
>    glibc >= 2.33.9000-36.fc35
>    ...
>
>  This means of course that the library can't be installed without
>  upgrading to bleeding-edge glibc.  I understand that it's Rawhide, but
>  still that seems a bit excessive?!  (For context we often install
>  hand-picked Rawhide packages in older Fedora in order to gain specific
>  new features - in this case optimized nbdcopy.) 
That wouldn't work anyway for this rawhide cycle because so many new
symbol versions had to be added (including
__libc_start_main@(a)GLIBC_2.34, which is used by *all* main programs).

> > The dependencies are conservative in the sense that you might a
> > dependency on a more recent glibc version than that is actually
> > required by the symbols used.
>
>  Is there a way to find out if the dependency is real or conservative?
>
>  According to nm -D we're using these symbols from future glibc:
>
>                   U pthread_getspecific@(a)GLIBC_2.34
>                   U pthread_key_create@(a)GLIBC_2.34
>                   U pthread_key_delete@(a)GLIBC_2.34
>                   U pthread_setspecific@(a)GLIBC_2.34
>
>  Obviously these symbols are nothing new, but I suppose this might be
>  related to -lpthread no longer being a separate library? 
Correct, older glibc won't be able to load this version of the library
because libpthread isn't linked in anymore.  If the processes that needs
libnbd has linked in libpthread, it might have worked by accident on
older glibc without the new symbol versions.  But that would be very
obscure failure mode.  So we added new symbol versions to make it
explicit that a new glibc version is needed.

Basically, that caused the dependency switched from
libpthread.so.0(GLIBC_2.2.5)(64bit) to libc.so.6(GLIBC_2.34)(64bit).

And because of the __libc_start_main change (which we had to do for
security hardening purposes), I thought it was not such a significant
additional burden to do it for the former libpthread symbols, too.
Which is why I don't look too hard at fancy ways to avoid the symbol
version change.

In the next rawhide cycle, I expect that the auto-generated versioned
dependency applies to fewer builds.

Thanks,
Florian

[Richard W.M. Jones]

Fair enough, thanks for the explanation it sounds reasonable.

Rich.

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
virt-p2v converts physical machines to virtual machines.  Boot with a
live CD or over the network (PXE) and turn machines into KVM guests.
http://libguestfs.org/virt-v2v