Hi,
heads-up: systemd 250-rc1 has been built in rawhide.
It's a fairly big release (140 paragraphs in the NEWS file entry) [1], but there are also some packaging changes that are relevant for Fedora.
There's a bunch of new configuration settings (RestrictNetworkInterfaces=, StartupAllowedCPUs=, StartupAllowedMemoryNodes=, [Condition|Assert][Memory|CPU|IO]Pressure=, ExecSearchPath=, RuntimeRandomizedExtraSec=, TTYRows=/TTYColumns=, [Service] ExitType=, [Automount] ExtraOptions=), new units (systemd-boot-update.service, factory-reset.target), support for TPM2-backed credential storage, file system integrity, and a lot of new features in systemd-networkd, systemd-analyze, systemd-dissect, systemd-homed, systemd-coredump, sd-boot, and various other smaller features and enhancements. Most tools now provide json output for easy postprocessing.
The credential storage stuff could be relevant in the light of the recent discussion about logins in emergency mode.
There are some dependency changes in packaging in Fedora. We changed most of the crypto code to use openssl instead of gcrypt. We also changed various libraries to be dlopened when actually used, instead of being dynamically linked. The goal is to reduce the required dependencies. This means that various libraries that were previously always pulled in, are not listed as Recommends by the various subpackages:
$ rpm -q --recommends systemd | grep '^lib' libbpf.so.0()(64bit) libbpf.so.0(LIBBPF_0.4.0)(64bit) libcryptsetup.so.12()(64bit) libcryptsetup.so.12(CRYPTSETUP_2.4)(64bit) libdw.so.1()(64bit) libdw.so.1(ELFUTILS_0.186)(64bit) libelf.so.1()(64bit) libelf.so.1(ELFUTILS_1.7)(64bit) libidn2.so.0()(64bit) libidn2.so.0(IDN2_0.0.0)(64bit) libpcre2-8.so.0()(64bit) libpwquality.so.1()(64bit) libpwquality.so.1(LIBPWQUALITY_1.0)(64bit) libqrencode.so.4()(64bit) libxkbcommon(x86-64)
Recommends are normally installed, but if you're using 'dnf --setopt install_weak_deps=False' or similar, please make sure that you install those libraries too if appropriate.
libgcrypt is pulled in by libsystemd.so, which means it is pretty much everywhere. The plan is to switch it to being dlopened, because it is only used for one minor feature irrelevant to almost all users.
Also, I moved more stuff into the -udev subpackage: systemd-integritysetup, systemd-veritysetup, systemd-binfmt, systemd-sysctl, systemd-coredump. IIUC, none of those are useful in containers, so they fit better in -udev which is supposed to be installed on real hardware or VMs.
systemd-resolved has also seen work, but it mostly has been behind-the-scenes refactoring. All of the crypto code had been ported from libgcrypt to openssl, but hopefully that change will be completely invisible. There have been some bugfixes too, but not enough. I hope to clear some of the outstanding issues before v250-final.
If there are regressions, please file a bug or let me know otherwise.
[1] https://raw.githubusercontent.com/systemd/systemd/v250-rc1/NEWS
Zbyszek
On 10/12/2021 08:34, Zbigniew Jędrzejewski-Szmek wrote:
We also changed various libraries to be dlopened when actually used, instead of being dynamically linked. The goal is to reduce the required dependencies.
What happens if these libraries are not installed an cannot be dl-opened?
Am 10.12.21 um 10:40 schrieb Vitaly Zaitsev via devel:
On 10/12/2021 08:34, Zbigniew Jędrzejewski-Szmek wrote:
We also changed various libraries to be dlopened when actually used, instead of being dynamically linked. The goal is to reduce the required dependencies.
dl-opening instead of dynamic linkage doesn't remove these deps.
It only hides away the deps and moves them from "packaging" to run-time.
What happens if these libraries are not installed an cannot be dl-opened?
Programs will fail at run-time.
Openly said, I feel this is a massive regression and design flaw.
Ralf
On 10/12/2021 13:34, Ralf Corsépius wrote:
Programs will fail at run-time. Openly said, I feel this is a massive regression and design flaw.
+1. And crash on init-system level will fail the whole system.
All library dependencies should be tracked by linker.
dlopen() should be used only for loading plugins or some restricted libraries, like openh264 or ffmpeg-libs.
On Fr, 10.12.21 10:40, Fedora Development ML (devel@lists.fedoraproject.org) wrote:
On 10/12/2021 08:34, Zbigniew Jędrzejewski-Szmek wrote:
We also changed various libraries to be dlopened when actually used, instead of being dynamically linked. The goal is to reduce the required dependencies.
What happens if these libraries are not installed an cannot be dl-opened?
We handle that gracefully. i.e. the specific feature is disabled, but everything else stil works as it should.
here's an example: systemd-cryptenroll allows you to enroll additional keys into LUKS2 volumes for systemd-cryptsetup to honour. Besides pkcs11/fido2/tpm2 stuff it also supports "recovery keys", which are pretty much the same as a regular password, but are computer generated rather than user chosen, and thus have guaranteed high entropy. Since users have to read the generated key off screen and write it down/store it away somewhere, we will normally show you a QR code in the terminal you can quickly scan it off screen with. This is implemented via dlopen()ing libqrencode. If you install that lib, you'll get the QR code + text string displayed. If not, you'll just get the text string.
And the other cases where we use dlopen() are pretty much the same: the specific feature is skipped or made unavailable, but the basic behaviour remains the same. When we really really need something, then we'll use regular share library deps as always, not dlopen(). We use dlopen() only for the stuff that isn't a "core" feature, i.e. essential for the program to do its thing.
It's an excercise in minimizing our footprint (for containers, embedded devices) , but still providing a rich feature set for those who want it, without requiring recompilation, and whith automatic upgrading to the full feature set, simply by dropping in a few more of the optional deps.
Lennart
-- Lennart Poettering, Berlin
On Fri, Dec 10, 2021 at 03:00:29PM +0100, Lennart Poettering wrote:
It's an excercise in minimizing our footprint (for containers, embedded devices) , but still providing a rich feature set for those who want it, without requiring recompilation, and whith automatic upgrading to the full feature set, simply by dropping in a few more of the optional deps.
This seems like a big step forward in flexiblity in using the same binaries for a lot of different situations. Cool!
On Fri, Dec 10, 2021 at 10:40:33AM +0100, Vitaly Zaitsev via devel wrote:
On 10/12/2021 08:34, Zbigniew Jędrzejewski-Szmek wrote:
We also changed various libraries to be dlopened when actually used, instead of being dynamically linked. The goal is to reduce the required dependencies.
What happens if these libraries are not installed an cannot be dl-opened?
The functionality that requires those libraries will not be available. An easy example is libqrencode: some systemd tools (journalctl, homectl, cryptenroll) will print recovery keys and such as qrcode on the terminal for easy transfer to a cellphone, in addition to the usual text output. If the library is not present, the qrcode is not printed.
Another example is systemd-coredump: it uses libdw and libelf to analyze a core file and generate a traceback. If those libs cannot be loaded, the crash is reported without it.
Other examples: libpwquality is used to warn about too-easy passwords, without it the warning does not happen, libpcre2 is used for 'journalctl --grep', and without it the --grep option just returns an error, etc.
Systemd allows almost such dependencies to be turned on/off at compile time. In Fedora we compile with everything on, which means that systemd would pull in a lot of dependencies. In particular, most systemd binaries are link to libsystemd-shared.so, which in turn is linked to most of those libraries. This means that pulling in even a single binary pulls in many many library dependencies, and even if we were to split systemd into multiple subpackages, we wouldn't avoid pulling in the dependencies.
Using dlopen allows us to make the requirements more flexible: for example, in the initrd you certainly don't need qrcodes or pwquality checks. Thus for the initrd we just don't install those libraries.
We have been doing this for a while. I think the first was libxkbcommon (which is used by localed to check keyboard mapping configurations), back in 2016 (https://github.com/systemd/systemd/commit/5de344704d). When this done correctly, people mostly don't notice.
Some other replies in this thread assumed that the programs will crash. I have no idea why you would assume that. We're not complete idiots.
Zbyszek
Am 10.12.21 um 15:13 schrieb Zbigniew Jędrzejewski-Szmek:
Some other replies in this thread assumed that the programs will crash. I have no idea why you would assume that.
If properly implemented, such programs typically won't crash, they will "just not work" oder "error out".
Functionally, dlopening libraries, instead of dynamically linking really does not provide any advantages.
We're not complete idiots.With all due respect, tt's not the first time systemd communicates the
attitude of blindly imitating MS-Windows. Whether you like it or not, this appears to be one of these cases.
Ralf
On 12/10/21 08:15, Ralf Corsépius wrote:
Am 10.12.21 um 15:13 schrieb Zbigniew Jędrzejewski-Szmek:
Some other replies in this thread assumed that the programs will crash. I have no idea why you would assume that.
If properly implemented, such programs typically won't crash, they will "just not work" oder "error out".
Functionally, dlopening libraries, instead of dynamically linking really does not provide any advantages.
Maybe you need to read the other messages in the thread again. It's the difference between crashing or gracefully reducing functionality while avoiding hard dependencies.
We're not complete idiots.With all due respect, tt's not the first time systemd communicates the
attitude of blindly imitating MS-Windows. Whether you like it or not, this appears to be one of these cases.
There's no need to be rude and you aren't being respectful. This has nothing to do with Windows. I don't even understand what that comparison is supposed to mean in this context.
On 12/10/21 09:13, Zbigniew Jędrzejewski-Szmek wrote:
On Fri, Dec 10, 2021 at 10:40:33AM +0100, Vitaly Zaitsev via devel wrote: What happens if these libraries are not installed an cannot be dl-opened? The functionality that requires those libraries will not be available.
This is quite clever, +1.
It occurred to me that a downside might be in mysterious functionality variances in case of latent bugs. Say, something happens to libpcre2 (file gets damaged, or ABI incompatibility creeps in, etc) so that suddenly 'journalctl --grep' stops working. How would one find out about the cause? From reading the journalctl code, it seems to me that pcre2-dlopen.c:dlopen_pcre2() is called when --grep is used, and it logs the failure, which I think addresses my concern.
So the apps using this technique would have to be written carefully, to only dlopen() (and therefore also print possible error messages) when the optional functionality is called for, right? If the dlopen()s were just shotgunned at the startup, it could result in a lot of chatter.
If this technique became widely used, as it probably should, maybe there ought to be a common way of finding out what is and isn't available. For instance, the apps could have a special common option, like 'journalctl --capabilities' returning
...
libpcre2 loaded, --grep enabled
libfancy NOT loaded, --FANCY disabled
..
Or, there could be a system-wide status matrix of executables vs libraries, listing the outcome of recent dlopen()s, so that a query like 'capabilites journalctl' would return the status above (except that such general utility would not in general be able to print what app functionality is tied to the particular library)
On Fri, Dec 10, 2021 at 01:58:28PM -0500, przemek klosowski via devel wrote:
On 12/10/21 09:13, Zbigniew Jędrzejewski-Szmek wrote:
On Fri, Dec 10, 2021 at 10:40:33AM +0100, Vitaly Zaitsev via devel wrote: What happens if these libraries are not installed an cannot be dl-opened? The functionality that requires those libraries will not be available.
This is quite clever, +1.
It occurred to me that a downside might be in mysterious functionality variances in case of latent bugs. Say, something happens to libpcre2 (file gets damaged, or ABI incompatibility creeps in, etc) so that suddenly 'journalctl --grep' stops working. How would one find out about the cause? From reading the journalctl code, it seems to me that pcre2-dlopen.c:dlopen_pcre2() is called when --grep is used, and it logs the failure, which I think addresses my concern.
So the apps using this technique would have to be written carefully, to only dlopen() (and therefore also print possible error messages) when the optional functionality is called for, right? If the dlopen()s were just shotgunned at the startup, it could result in a lot of chatter.
In general, we do this because we think the advantages outweigh the issues, but this does mean that we're happy with the implementation. Right now the function signatures are copied into our code, and there's boilerplate code, and in general the whole system is a bit clunky.
In practice it works fine because the libraries we link to are stable system libraries that just don't break API or ABI compatibility. We also import just a handful of symbols. But this is not a general solution — it'd be horrible for any library whose API evolves or many functions need to be used.
Ideally, we would have support in the linker to do this automatically. We would specify that certain libraries should be linked weakly, and the linker or compiler would generate stubs that'd be resolved to either NULL or the actual symbol at runtime. I.e. just like calling statically or dynamically linked libraries requires no modifications in the code, and is entirely handled by linker commandline options.
If this technique became widely used, as it probably should, maybe there ought to be a common way of finding out what is and isn't available. For instance, the apps could have a special common option, like 'journalctl --capabilities' returning
...
libpcre2 loaded, --grep enabled
libfancy NOT loaded, --FANCY disabled
That's a reasonable request. Right now it'd need to be implemented manually, i.e. we'd need to remember that e.g. journalctl --grep dlopens libpcre2, and add call the same load function in the function implementing --capabilities. This is certainly doable, but I hope we could make this all more automatic… Maybe we should just add this --version output. It already prints +SUPPORTED -UNSUPPORTED.
Zbyszek
Zbigniew Jędrzejewski-Szmek wrote:
This means that various libraries that were previously always pulled in, are not listed as Recommends by the various subpackages:
[...]
Recommends are normally installed, but if you're using 'dnf --setopt install_weak_deps=False' or similar, please make sure that you install those libraries too if appropriate.
Was "not" supposed to be "now"? Otherwise these statements don't make sense together.
Björn Persson
On Fri, Dec 10, 2021 at 12:16:37PM +0100, Björn Persson wrote:
Zbigniew Jędrzejewski-Szmek wrote:
This means that various libraries that were previously always pulled in, are not listed as Recommends by the various subpackages:
[...]
Recommends are normally installed, but if you're using 'dnf --setopt install_weak_deps=False' or similar, please make sure that you install those libraries too if appropriate.
Was "not" supposed to be "now"? Otherwise these statements don't make sense together.
Yes, "now". Sorry for the typo.
Zbyszek
On Fri, Dec 10, 2021 at 07:34:42AM +0000, Zbigniew Jędrzejewski-Szmek wrote:
Also, I moved more stuff into the -udev subpackage: systemd-integritysetup, systemd-veritysetup, systemd-binfmt, systemd-sysctl, systemd-coredump. IIUC, none of those are useful in containers, so they fit better in -udev which is supposed to be installed on real hardware or VMs.
Could we keep udev to being for udev and introduce a different package for extras? This change affects libguestfs which only wants udev, not systemd or anything else.
(Ideally, package udev as a separate project of course.)
Rich.
On Fri, Dec 10, 2021 at 05:03:45PM +0000, Richard W.M. Jones wrote:
On Fri, Dec 10, 2021 at 07:34:42AM +0000, Zbigniew Jędrzejewski-Szmek wrote:
Also, I moved more stuff into the -udev subpackage: systemd-integritysetup, systemd-veritysetup, systemd-binfmt, systemd-sysctl, systemd-coredump. IIUC, none of those are useful in containers, so they fit better in -udev which is supposed to be installed on real hardware or VMs.
Could we keep udev to being for udev and introduce a different package for extras? This change affects libguestfs which only wants udev, not systemd or anything else.
Hmm, I don't think this changes anything for libguestfs. The systemd-udev rpm requires systemd, so if have systemd-udev, this is a noop.
Zbyszek
On Sat, Dec 11, 2021 at 5:43 AM Zbigniew Jędrzejewski-Szmek zbyszek@in.waw.pl wrote:
On Fri, Dec 10, 2021 at 05:03:45PM +0000, Richard W.M. Jones wrote:
On Fri, Dec 10, 2021 at 07:34:42AM +0000, Zbigniew Jędrzejewski-Szmek wrote:
Also, I moved more stuff into the -udev subpackage: systemd-integritysetup, systemd-veritysetup, systemd-binfmt, systemd-sysctl, systemd-coredump. IIUC, none of those are useful in containers, so they fit better in -udev which is supposed to be installed on real hardware or VMs.
Could we keep udev to being for udev and introduce a different package for extras? This change affects libguestfs which only wants udev, not systemd or anything else.
Hmm, I don't think this changes anything for libguestfs. The systemd-udev rpm requires systemd, so if have systemd-udev, this is a noop.
Please put systemd-coredump back in the main systemd package. Unlike the others, this is actually required functionality for backtrace stuff, especially if you're running OS containers (nspawn, ubi-init, etc.).
On Wed, Dec 15, 2021 at 07:13:49AM -0500, Neal Gompa wrote:
On Sat, Dec 11, 2021 at 5:43 AM Zbigniew Jędrzejewski-Szmek zbyszek@in.waw.pl wrote:
On Fri, Dec 10, 2021 at 05:03:45PM +0000, Richard W.M. Jones wrote:
On Fri, Dec 10, 2021 at 07:34:42AM +0000, Zbigniew Jędrzejewski-Szmek wrote:
Also, I moved more stuff into the -udev subpackage: systemd-integritysetup, systemd-veritysetup, systemd-binfmt, systemd-sysctl, systemd-coredump. IIUC, none of those are useful in containers, so they fit better in -udev which is supposed to be installed on real hardware or VMs.
Could we keep udev to being for udev and introduce a different package for extras? This change affects libguestfs which only wants udev, not systemd or anything else.
Hmm, I don't think this changes anything for libguestfs. The systemd-udev rpm requires systemd, so if have systemd-udev, this is a noop.
Please put systemd-coredump back in the main systemd package. Unlike the others, this is actually required functionality for backtrace stuff, especially if you're running OS containers (nspawn, ubi-init, etc.).
The handler for coredumps (/proc/sys/kernel/core_pattern) is not namespaced, so coredumps reported in a container are handled on the host (*). Are you sure that your coredump processing happens in the container? How does the pattern look like and how is the whole thing set up?
(*) In principle systemd-coredump on the host could feed the core file back to some handler invoked in the container. We discussed this a few times in the past and it would be quite useful for some types of containers. But I'm not aware of this being implemented anywhere.
Zbyszek
Hi Zbyszek,
On 10. 12. 21 8:34, Zbigniew Jędrzejewski-Szmek wrote:
Hi,
heads-up: systemd 250-rc1 has been built in rawhide.
If there are regressions, please file a bug or let me know otherwise.
just a minor thing about systemd cryptsetup plugins, but please take a look at: https://bugzilla.redhat.com/show_bug.cgi?id=2031873
Regards O.