TL;DR: glibc 2.34 snapshots are coming. libpthread as a separate library creates problems, so we are removing it. There may be some hickups. Full updates with “dnf update” are recommended.
We expect that we will soon be able to import glibc upstream snapshots regularly into Fedora 35 and CentOS Stream 9. We did such regular imports for the past couple of Fedora rawhide releases, but the situation will be slightly different, as explained below. The snapshots will also land in CentOS Stream 9, after a delay as the result of a different testing pipeline.
glibc 2.33 and earlier accidentally provided an very generic ROP gadget in the statically linked startup code that is present in every program (even dynamically linked ones). We have removed the ROP gadget and moved that particular initialization code into the dynamically linked part, but that means that the interface between the startup code and the dynamically loaded glibc parts had to change. As a result, any program linked against glibc 2.34 will not run with glibc 2.33 or any earlier version. (Some of you may remember the memcpy@GLIBC_2.14 issue, it was similar.) This version requirement will be properly reflected in RPM dependencies.
glibc 2.34 removes libpthread as a separate library. This is based on the observation that most processes load libpthread anyway. (On this system, I count 141 out of 159 processes that load libpthread.) One particularly thorny issue is that certain NSS modules depend on libpthread, and if the main program is not linked (indirectly) against libpthread, loading such NSS modules effectively loads libpthread via dlopen, which is something that glibc has never supported well. Furthermore, the availability of some pthread_* functions without -lpthread has been a source of confusion to developers, resulting in both overlinking and underlinking.
We started the libpthread transition in glibc 2.33 when we added the __libc_single_threaded variable as a replacement for the weak symbol hacks that some libraries use to detect single-threaded processes. (For example, libstdc++ used to do this to avoid using atomic instructions in std::shared_ptr.) Backwards compatibility for dynamically linked binaries should be preserved (as usual), but we know of one issue on ppc64le, where the weak symbol hacks resulted in slightly corrupt binaries. Upstream glibc did not accept the proposed backwards compatibility enhancement so far. Instead, we will rebuild affected distribution binaries with a binutils version which handles weak symbols slightly differently, avoiding the corruption. As we plan to perform these rebuilds before the new glibc lands in the buildroot, the requirement to upgrade to the new binaries before or at the same time of the upgrade to the glibc upstream snapshot will NOT be reflected in RPM dependencies. A full upgrade using “dnf update” will work, however.
In addition to the removal of libpthread, I also hope to remove libdl and librt. This means that new GLIBC_2.34 symbols will be added to libc.so.6 (e.g., timer_create@GLIBC_2.34). The librt removal in particular will probably not land with the first imported snapshot. This is unfortunate because RPM does not have a way to express this: there's just a blanket Provides: libc.so.6(GLIBC_2.34), which will already be included in the first snapshot that adds the symbol __lib_start_main@@GLIBC_2.34 (whether or not the RPM package includes timer_create@GLIBC_2.34 as well). Some tools in the fedpkg/centpkg/mock sphere try to install builds from the Koji buildroot into installations from composes, without upgrading everything to the current buildroot, so it's possible that glibc won't be upgraded to match the requirements of RPMs directly imported from the buildroot. Developers encountered similar issues with glibc snapshot imports (e.g., around the symbol pthread_getattr_np). Our RPM infrastructure does not have per-symbol dependencies, so there isn't much we can do about it at the packaging level. It's a transitory issue during rawhide/CentOS Stream 9 development; the finished release will add all GLIBC_2.34 symbols in one upgrade, so end users won't see it in a Fedora 34 to Fedora 35 upgrade or a CentOS Stream 8 to CentOS Stream 9 upgrade.
We think there is value in providing access to these snapshots early, and will try to make the transitions as smooth as possible, within the constraints outlined above.
Thanks, Florian
glibc-2.33.9000-18.fc35 with the first set of changes has been tagged into Fedora rawhide.
Thanks, Florian
TL;DR: glibc 2.34 snapshots are coming. libpthread as a separate library creates problems, so we are removing it. There may be some hickups. Full updates with “dnf update” are recommended.
We expect that we will soon be able to import glibc upstream snapshots regularly into Fedora 35 and CentOS Stream 9. We did such regular imports for the past couple of Fedora rawhide releases, but the situation will be slightly different, as explained below. The snapshots will also land in CentOS Stream 9, after a delay as the result of a different testing pipeline.
glibc 2.33 and earlier accidentally provided an very generic ROP gadget in the statically linked startup code that is present in every program (even dynamically linked ones). We have removed the ROP gadget and moved that particular initialization code into the dynamically linked part, but that means that the interface between the startup code and the dynamically loaded glibc parts had to change. As a result, any program linked against glibc 2.34 will not run with glibc 2.33 or any earlier version. (Some of you may remember the memcpy@GLIBC_2.14 issue, it was similar.) This version requirement will be properly reflected in RPM dependencies.
glibc 2.34 removes libpthread as a separate library. This is based on the observation that most processes load libpthread anyway. (On this system, I count 141 out of 159 processes that load libpthread.) One particularly thorny issue is that certain NSS modules depend on libpthread, and if the main program is not linked (indirectly) against libpthread, loading such NSS modules effectively loads libpthread via dlopen, which is something that glibc has never supported well. Furthermore, the availability of some pthread_* functions without -lpthread has been a source of confusion to developers, resulting in both overlinking and underlinking.
We started the libpthread transition in glibc 2.33 when we added the __libc_single_threaded variable as a replacement for the weak symbol hacks that some libraries use to detect single-threaded processes. (For example, libstdc++ used to do this to avoid using atomic instructions in std::shared_ptr.) Backwards compatibility for dynamically linked binaries should be preserved (as usual), but we know of one issue on ppc64le, where the weak symbol hacks resulted in slightly corrupt binaries. Upstream glibc did not accept the proposed backwards compatibility enhancement so far. Instead, we will rebuild affected distribution binaries with a binutils version which handles weak symbols slightly differently, avoiding the corruption. As we plan to perform these rebuilds before the new glibc lands in the buildroot, the requirement to upgrade to the new binaries before or at the same time of the upgrade to the glibc upstream snapshot will NOT be reflected in RPM dependencies. A full upgrade using “dnf update” will work, however.
In addition to the removal of libpthread, I also hope to remove libdl and librt. This means that new GLIBC_2.34 symbols will be added to libc.so.6 (e.g., timer_create@GLIBC_2.34). The librt removal in particular will probably not land with the first imported snapshot. This is unfortunate because RPM does not have a way to express this: there's just a blanket Provides: libc.so.6(GLIBC_2.34), which will already be included in the first snapshot that adds the symbol __lib_start_main@@GLIBC_2.34 (whether or not the RPM package includes timer_create@GLIBC_2.34 as well). Some tools in the fedpkg/centpkg/mock sphere try to install builds from the Koji buildroot into installations from composes, without upgrading everything to the current buildroot, so it's possible that glibc won't be upgraded to match the requirements of RPMs directly imported from the buildroot. Developers encountered similar issues with glibc snapshot imports (e.g., around the symbol pthread_getattr_np). Our RPM infrastructure does not have per-symbol dependencies, so there isn't much we can do about it at the packaging level. It's a transitory issue during rawhide/CentOS Stream 9 development; the finished release will add all GLIBC_2.34 symbols in one upgrade, so end users won't see it in a Fedora 34 to Fedora 35 upgrade or a CentOS Stream 8 to CentOS Stream 9 upgrade.
We think there is value in providing access to these snapshots early, and will try to make the transitions as smooth as possible, within the constraints outlined above.
Thanks, Florian
On 16/06/2021 11:17, Florian Weimer wrote:
glibc-2.33.9000-18.fc35 with the first set of changes has been tagged into Fedora rawhide.
Please take a look at this: https://bugzilla.redhat.com/show_bug.cgi?id=1977878
-
Florian Weimer -
Vitaly Zaitsev