I'm not pretending these will solve everything, but they should make attacks a little harder in future.
(1) We should routinely delete autoconf-generated cruft from upstream projects and regenerate it in %prep. It is easier to study the real source rather than dig through the convoluted, generated shell script in an upstream './configure' looking for back doors.
For most projects, just running "autoreconf -fiv" is enough.
Yes, there are some projects that depend on a specific or old version of autoconf. We should fix those. But that doesn't need to delay us from using autoreconf on many projects today.
In the xz case this wouldn't have been enough, it turns out we would also have to delete m4/build-to-host.m4, which then autoreconf regenerates. I don't fully understand why that is.
(2) We should discourage gnulib as much as possible.
In libvirt we took the decision a few years ago to remove gnulib. It's extremely convoluted and almost no one understands how it really works. It's written in obscure m4 macros and shell script.
It's also not necessary for Linux since gnulib is mainly about porting to non-Linux platforms. There are better ways to do this.
In the xz case it was a gnulib-derived script which was modified to do the initial injection (original: https://git.savannah.gnu.org/gitweb/?p=gnulib.git;a=blob;f=m4/build-to-host....).
(3) We should have a "security path", like "critical path".
sshd is linked to a lot of libraries:
/lib64/libaudit.so.1 audit-libs /lib64/libc.so.6 glibc /lib64/libcap-ng.so.0 libcap-ng /lib64/libcap.so.2 libcap /lib64/libcom_err.so.2 libcom_err /lib64/libcrypt.so.2 libxcrypt /lib64/libcrypto.so.3 openssl-libs /lib64/libeconf.so.0 libeconf /lib64/libgcc_s.so.1 libgcc /lib64/libgssapi_krb5.so.2 krb5-libs /lib64/libk5crypto.so.3 krb5-libs /lib64/libkeyutils.so.1 keyutils-libs /lib64/libkrb5.so.3 krb5-libs /lib64/libkrb5support.so.0 krb5-libs /lib64/liblz4.so.1 lz4-libs /lib64/liblzma.so.5 xz-libs /lib64/libm.so.6 glibc /lib64/libpam.so.0 pam-libs /lib64/libpcre2-8.so.0 pcre2 /lib64/libresolv.so.2 glibc /lib64/libselinux.so.1 libselinux /lib64/libsystemd.so.0 systemd-libs /lib64/libz.so.1 zlib / zlib-ng /lib64/libzstd.so.1 zstd
Should we have a higher level of attention to these packages? We already have "critical path", but that's a broad category now. These seem like they are "security path" packages, an intentionally small subset associated with very secure services which are enabled by default.
These are just my thoughts on a Saturday morning. Feedback welcome of course.
Rich.
W dniu 30.03.2024 o 10:37, Richard W.M. Jones pisze:
(1) We should routinely delete autoconf-generated cruft from upstream projects and regenerate it in %prep. It is easier to study the real source rather than dig through the convoluted, generated shell script in an upstream './configure' looking for back doors.
For most projects, just running "autoreconf -fiv" is enough.
Yes, there are some projects that depend on a specific or old version of autoconf. We should fix those. But that doesn't need to delay us from using autoreconf on many projects today.
In the xz case this wouldn't have been enough, it turns out we would also have to delete m4/build-to-host.m4, which then autoreconf regenerates. I don't fully understand why that is.
There were also projects where configure script was generated long time ago and then edited by hand. Usually because no one in project knew m4.
Hey Richard,
Should we have a higher level of attention to these packages? We already have "critical path", but that's a broad category now. These seem like they are "security path" packages, an intentionally small subset associated with very secure services which are enabled by default.
It sounds like a good plan to put certain dependencies on a critical path. Perhaps anything that is used by packages included in the various editions of Fedora that allow for remote access (even if disabled by default) could fall under that path?
We could also try to ensure that packages do not contain any binary blobs and instead require generation scripts for those that we can run ourselves.
Regards,
Simon
On Sat, Mar 30 2024 at 09:37:44 AM +00:00:00, Richard W.M. Jones rjones@redhat.com wrote:
In the xz case this wouldn't have been enough, it turns out we would also have to delete m4/build-to-host.m4, which then autoreconf regenerates. I don't fully understand why that is.
I agree that running autoreconf on our packages makes sense to start doing. Still, to avoid this backdoored m4 file, we would have needed to stop using release tarballs altogether and switch to using git tags directly instead. That would at least force the malicious attacker to commit their code to version control, making it slightly harder to hide the attack.
Michael
Once upon a time, Michael Catanzaro mcatanzaro@redhat.com said:
I agree that running autoreconf on our packages makes sense to start doing. Still, to avoid this backdoored m4 file, we would have needed to stop using release tarballs altogether and switch to using git tags directly instead. That would at least force the malicious attacker to commit their code to version control, making it slightly harder to hide the attack.
Using a signed tarball is ideally better than a git tag (it's an extra level of author attestation)... but where both are available, it'd be nice to have a way to denote in the spec file that there are two URLs for the source. In this case, if both URLs were listed and something could be run to automatically fetch and compare them, the attack code would have been flagged.
Just because something is public in a git tag doesn't mean somebody else reviewed it and caught an attack (after all, in this case, part of it was committed to git and at least one other maintainer didn't notice anything odd).
On Sat, Mar 30, 2024 at 07:25:50AM -0500, Chris Adams wrote:
Once upon a time, Michael Catanzaro mcatanzaro@redhat.com said:
I agree that running autoreconf on our packages makes sense to start doing. Still, to avoid this backdoored m4 file, we would have needed to stop using release tarballs altogether and switch to using git tags directly instead. That would at least force the malicious attacker to commit their code to version control, making it slightly harder to hide the attack.
Using a signed tarball is ideally better than a git tag (it's an extra level of author attestation)... but where both are available, it'd be nice to have a way to denote in the spec file that there are two URLs for the source. In this case, if both URLs were listed and something could be run to automatically fetch and compare them, the attack code would have been flagged.
Tarball production should be reproducible. Then the maintainer can both make a signature locally and make it public, and users can download the auto-generated tarball.
In fact, github tarball generation is stable. A few years ago they tried to improve the compression method (i.e. .tar would be still identical, but .tar.gz would be different), and after a huge outcry this was reverted. They still haven't officially said that it's stable, but let's hope that it never changes, at least not without a suitable advance warning.
Zbyszek
On Sat, Mar 30, 2024 at 03:23:55PM +0000, Zbigniew Jędrzejewski-Szmek wrote:
On Sat, Mar 30, 2024 at 07:25:50AM -0500, Chris Adams wrote:
Once upon a time, Michael Catanzaro mcatanzaro@redhat.com said:
I agree that running autoreconf on our packages makes sense to start doing. Still, to avoid this backdoored m4 file, we would have needed to stop using release tarballs altogether and switch to using git tags directly instead. That would at least force the malicious attacker to commit their code to version control, making it slightly harder to hide the attack.
Using a signed tarball is ideally better than a git tag (it's an extra level of author attestation)... but where both are available, it'd be nice to have a way to denote in the spec file that there are two URLs for the source. In this case, if both URLs were listed and something could be run to automatically fetch and compare them, the attack code would have been flagged.
Tarball production should be reproducible. Then the maintainer can both make a signature locally and make it public, and users can download the auto-generated tarball.
In fact, github tarball generation is stable. A few years ago they tried to improve the compression method (i.e. .tar would be still identical, but .tar.gz would be different), and after a huge outcry this was reverted. They still haven't officially said that it's stable, but let's hope that it never changes, at least not without a suitable advance warning.
I used the following to check all 193 github tarballs provided for systemd (those are autogenerated):
git clone --bare https://github.com/systemd/systemd dir=systemd.git tags=$(git --git-dir=$dir tag|rg '^v\d+(.\d+)?$'|sed 's/^v//'|sort -g)
for v in $tags; do echo $v
if [[ v =~ . ]]; then wget https://github.com/systemd/systemd-stable/archive/v$v/systemd-$v.tar.gz git --git-dir=$dir archive v$v --prefix=systemd-stable-$v/ | gzip >systemd-$v.local.tar.gz else wget https://github.com/systemd/systemd/archive/v$v/systemd-$v.tar.gz git --git-dir=$dir archive v$v --prefix=systemd-$v/ | gzip >systemd-$v.local.tar.gz fi
cmp systemd-$v.local.tar.gz systemd-$v.tar.gz || break
rm systemd-$v.local.tar.gz systemd-$v.tar.gz echo done
Fortunately, they all match ;)
Zbyszek
On Sat, 2024-03-30 at 15:23 +0000, Zbigniew Jędrzejewski-Szmek wrote:
On Sat, Mar 30, 2024 at 07:25:50AM -0500, Chris Adams wrote:
Once upon a time, Michael Catanzaro mcatanzaro@redhat.com said:
I agree that running autoreconf on our packages makes sense to start doing. Still, to avoid this backdoored m4 file, we would have needed to stop using release tarballs altogether and switch to using git tags directly instead. That would at least force the malicious attacker to commit their code to version control, making it slightly harder to hide the attack.
Using a signed tarball is ideally better than a git tag (it's an extra level of author attestation)... but where both are available, it'd be nice to have a way to denote in the spec file that there are two URLs for the source. In this case, if both URLs were listed and something could be run to automatically fetch and compare them, the attack code would have been flagged.
Tarball production should be reproducible. Then the maintainer can both make a signature locally and make it public, and users can download the auto-generated tarball.
In fact, github tarball generation is stable. A few years ago they tried to improve the compression method (i.e. .tar would be still identical, but .tar.gz would be different), and after a huge outcry this was reverted. They still haven't officially said that it's stable, but let's hope that it never changes, at least not without a suitable advance warning.
I do not know if this changed in the last couple of years, but tarballs in github are not stable (or weren't up to a couple years ago), they are cached for a period of time, so they may look stable in busy projects where you have regular downloads that keep the cache alive, but they are *regenerated* from the tag for seldom downloaded tarballs. And when that happens then hashes change.
Simo.
Dne 30. 03. 24 v 1:25 odp. Chris Adams napsal(a):
Using a signed tarball is ideally better than a git tag (it's an extra level of author attestation).
In this case signed tarball would not help at all. And git-tag would prevent this attack.
On Sat, Mar 30, 2024 at 11:47 AM Miroslav Suchý msuchy@redhat.com wrote:
Dne 30. 03. 24 v 1:25 odp. Chris Adams napsal(a):
Using a signed tarball is ideally better than a git tag (it's an extra level of author attestation).
In this case signed tarball would not help at all. And git-tag would prevent this attack.
Only because that person didn't think to check it in and tag it. They very well could have since they had direct commit access.
Richard W.M. Jones wrote on Sat, Mar 30, 2024 at 09:37:44AM +0000:
In the xz case this wouldn't have been enough, it turns out we would also have to delete m4/build-to-host.m4, which then autoreconf regenerates. I don't fully understand why that is.
I'm not 100% sure about the theory, but it looks like `autoreconf -fi` looks at the 'serial' in the first line of the script. The one bundled in the xz sources was marked very high: # build-to-host.m4 serial 30 But the one in the system (as of f39) is only 'serial 1'.
Artificially lowering the serial back to 0 in the file and running `autoreconf -fi` again properly reinstall the one from the system over it, but anything higher will keep it...
So if we want to go this route, we should remove the full m4 dir, but unfortunately I've seen quite a few projects that depend on non-standard m4 scripts so we'll need to fix these as we go... (At which point I'd suggest it's probably faster to convert it all to meson or another new shiny, and saner, build system, but getting upstreams to agree will be fun)
(2) We should discourage gnulib as much as possible. [...] In the xz case it was a gnulib-derived script which was modified to do the initial injection (original: https://git.savannah.gnu.org/gitweb/?p=gnulib.git;a=blob;f=m4/build-to-host....).
(Honestly I did compare the backdoored script and the real one this morning and I would be hard pressed to say if either is backdoored just looking at either version... Admitedly it was 3AM when I looked at it, but I don't think it's just a late hour problem)
(3) We should have a "security path", like "critical path".
sshd is linked to a lot of libraries:
/lib64/libaudit.so.1 audit-libs /lib64/libc.so.6 glibc /lib64/libcap-ng.so.0 libcap-ng /lib64/libcap.so.2 libcap /lib64/libcom_err.so.2 libcom_err /lib64/libcrypt.so.2 libxcrypt /lib64/libcrypto.so.3 openssl-libs /lib64/libeconf.so.0 libeconf /lib64/libgcc_s.so.1 libgcc /lib64/libgssapi_krb5.so.2 krb5-libs /lib64/libk5crypto.so.3 krb5-libs /lib64/libkeyutils.so.1 keyutils-libs /lib64/libkrb5.so.3 krb5-libs /lib64/libkrb5support.so.0 krb5-libs /lib64/liblz4.so.1 lz4-libs /lib64/liblzma.so.5 xz-libs /lib64/libm.so.6 glibc /lib64/libpam.so.0 pam-libs /lib64/libpcre2-8.so.0 pcre2 /lib64/libresolv.so.2 glibc /lib64/libselinux.so.1 libselinux /lib64/libsystemd.so.0 systemd-libs /lib64/libz.so.1 zlib / zlib-ng /lib64/libzstd.so.1 zstd
Should we have a higher level of attention to these packages? We already have "critical path", but that's a broad category now. These seem like they are "security path" packages, an intentionally small subset associated with very secure services which are enabled by default.
Before making each of these safer we should make sshd not link with so many things in the first place. On oss-security, Solar Designer made a lot of good points about it (around here: https://www.openwall.com/lists/oss-security/2024/03/29/27 , but the full thread is interesting)
But that's specific to sshd, and this problem could happen with any network-facing service: I guess sshd is a juciy target because it runs as root (whereas backdooring e.g. nginx would run as www user and not be as interesting), but in practice that could be bad enough. (hm, well, your point about 'enabled by default' strikes home though - we definitely could pay more attention to these first)
Dominique Martinet wrote:
I'm not 100% sure about the theory, but it looks like `autoreconf -fi` looks at the 'serial' in the first line of the script. The one bundled in the xz sources was marked very high: # build-to-host.m4 serial 30 But the one in the system (as of f39) is only 'serial 1'.
Artificially lowering the serial back to 0 in the file and running `autoreconf -fi` again properly reinstall the one from the system over it, but anything higher will keep it...
So if we want to go this route, we should remove the full m4 dir, but unfortunately I've seen quite a few projects that depend on non-standard m4 scripts so we'll need to fix these as we go...
Well, it all depends on whether those m4 scripts are really source code or whether they are autoimported from somewhere like gnulib. True source code needs to be retained, anything that can be autoimported should be autoimported at build time. (And upstreams should stop using imported copylibs to begin with, but that is a different story.)
(At which point I'd suggest it's probably faster to convert it all to meson or another new shiny, and saner, build system, but getting upstreams to agree will be fun)
CMake! :-)
(2) We should discourage gnulib as much as possible. [...] In the xz case it was a gnulib-derived script which was modified to do the initial injection (original: https://git.savannah.gnu.org/gitweb/?p=gnulib.git;a=blob;f=m4/build-to-host....).
(Honestly I did compare the backdoored script and the real one this morning and I would be hard pressed to say if either is backdoored just looking at either version... Admitedly it was 3AM when I looked at it, but I don't think it's just a late hour problem)
That is exactly the problem with autotools code, almost nobody understands what the heck it does, almost everybody just copies and pastes somebody else's snippet hoping it does not do bad things. And gnulib is a particularly ugly piece of the puzzle.
Before making each of these safer we should make sshd not link with so many things in the first place.
Indeed. E.g., Arch Linux does not transitively link sshd against liblzma. Fedora does because of this innocuous-looking patch: https://src.fedoraproject.org/rpms/openssh/blob/rawhide/f/openssh-7.4p1-syst... which is what ultimately allowed this to happen. This drags in libsystemd for sd_notify, and libsystemd is linked to way too much stuff including liblzma. Either we need a split libsdnotify that contains only sd_notify, or we should just stop using sd_notify at all. It increases the attack surface of daemons a lot just to allow the service to be "Type=notify" rather than one of the other available approaches. Arch Linux is also systemd-based nowadays, but still does not link OpenSSH against libsystemd.
Kevin Kofler
On Sat, Mar 30, 2024 at 8:07 AM Kevin Kofler via devel devel@lists.fedoraproject.org wrote:
Dominique Martinet wrote:
I'm not 100% sure about the theory, but it looks like `autoreconf -fi` looks at the 'serial' in the first line of the script. The one bundled in the xz sources was marked very high: # build-to-host.m4 serial 30 But the one in the system (as of f39) is only 'serial 1'.
Artificially lowering the serial back to 0 in the file and running `autoreconf -fi` again properly reinstall the one from the system over it, but anything higher will keep it...
So if we want to go this route, we should remove the full m4 dir, but unfortunately I've seen quite a few projects that depend on non-standard m4 scripts so we'll need to fix these as we go...
Well, it all depends on whether those m4 scripts are really source code or whether they are autoimported from somewhere like gnulib. True source code needs to be retained, anything that can be autoimported should be autoimported at build time. (And upstreams should stop using imported copylibs to begin with, but that is a different story.)
(At which point I'd suggest it's probably faster to convert it all to meson or another new shiny, and saner, build system, but getting upstreams to agree will be fun)
CMake! :-)
Funnily enough, xz can also be built with CMake. :)
I don't know whether the CMake scripts are complete enough to switch to yet, but we should consider doing it.
(2) We should discourage gnulib as much as possible. [...] In the xz case it was a gnulib-derived script which was modified to do the initial injection (original: https://git.savannah.gnu.org/gitweb/?p=gnulib.git;a=blob;f=m4/build-to-host....).
(Honestly I did compare the backdoored script and the real one this morning and I would be hard pressed to say if either is backdoored just looking at either version... Admitedly it was 3AM when I looked at it, but I don't think it's just a late hour problem)
That is exactly the problem with autotools code, almost nobody understands what the heck it does, almost everybody just copies and pastes somebody else's snippet hoping it does not do bad things. And gnulib is a particularly ugly piece of the puzzle.
Before making each of these safer we should make sshd not link with so many things in the first place.
Indeed. E.g., Arch Linux does not transitively link sshd against liblzma. Fedora does because of this innocuous-looking patch: https://src.fedoraproject.org/rpms/openssh/blob/rawhide/f/openssh-7.4p1-syst... which is what ultimately allowed this to happen. This drags in libsystemd for sd_notify, and libsystemd is linked to way too much stuff including liblzma. Either we need a split libsdnotify that contains only sd_notify, or we should just stop using sd_notify at all. It increases the attack surface of daemons a lot just to allow the service to be "Type=notify" rather than one of the other available approaches. Arch Linux is also systemd-based nowadays, but still does not link OpenSSH against libsystemd.
This is related to philosophical differences between Arch and Fedora.
That said, it probably makes sense for sd_notify to be its own library instead of being part of libsystemd. I'm sure systemd upstream will consider it now. :)
On Sat, Mar 30, 2024 at 08:23:48AM -0400, Neal Gompa wrote:
On Sat, Mar 30, 2024 at 8:07 AM Kevin Kofler via devel devel@lists.fedoraproject.org wrote:
(At which point I'd suggest it's probably faster to convert it all to meson or another new shiny, and saner, build system, but getting upstreams to agree will be fun)
CMake! :-)
Meson outclasses CMake in functionality, clarity, and brevity. I doesn't make sense to consider switching to CMake at this point.
This drags in libsystemd for sd_notify, and libsystemd is linked to way too much stuff including liblzma. Either we need a split libsdnotify that contains only sd_notify, or we should just stop using sd_notify at all. It increases the attack surface of daemons a lot just to allow the service to be "Type=notify" rather than one of the other available approaches. Arch Linux is also systemd-based nowadays, but still does not link OpenSSH against libsystemd.
This is related to philosophical differences between Arch and Fedora.
That said, it probably makes sense for sd_notify to be its own library instead of being part of libsystemd. I'm sure systemd upstream will consider it now. :)
I don't think a separate library would be particularly useful. sd_notify() as used by sshd just writes a static string to a unix socket. This can be implemented in ~10 lines of C, including error handling.
If that is the _only_ thing needed from libsystemd, then open-coding it is a good option. I guess it'd make sense for systemd to provide a header file that provides a reference documentation as an inline function.
One thing which is happening, mostly for unrelated reasons, it that systemd code is being changed to use dlopen() for various libraries which are usually not needed at runtime. The primary motivation for this is to omit such libraries from the initrd. But this also helps with overlinking.
In systemd git main, libsystemd is only linked to libc, libcap, and libgcrypt + libgpg-error. A pull request to convert that that last pair to dlopen is under review.
Zbyszek
On Sat, Mar 30, 2024 at 02:44:32PM +0000, Zbigniew Jędrzejewski-Szmek wrote:
On Sat, Mar 30, 2024 at 08:23:48AM -0400, Neal Gompa wrote:
On Sat, Mar 30, 2024 at 8:07 AM Kevin Kofler via devel devel@lists.fedoraproject.org wrote:
(At which point I'd suggest it's probably faster to convert it all to meson or another new shiny, and saner, build system, but getting upstreams to agree will be fun)
CMake! :-)
Meson outclasses CMake in functionality, clarity, and brevity. I doesn't make sense to consider switching to CMake at this point.
Which is the better build system doesn't much matter here, as they all let you run shell scripts so source level backdoors could be inserted in all of them.
The big difference they all have over autoconf is that they don't usually distribute a giant obfuscated (basically binary) shell script.
We can fix autoconf now by running autoreconf and we don't need to boil the ocean to do so.
This drags in libsystemd for sd_notify, and libsystemd is linked to way too much stuff including liblzma. Either we need a split libsdnotify that contains only sd_notify, or we should just stop using sd_notify at all. It increases the attack surface of daemons a lot just to allow the service to be "Type=notify" rather than one of the other available approaches. Arch Linux is also systemd-based nowadays, but still does not link OpenSSH against libsystemd.
This is related to philosophical differences between Arch and Fedora.
That said, it probably makes sense for sd_notify to be its own library instead of being part of libsystemd. I'm sure systemd upstream will consider it now. :)
I don't think a separate library would be particularly useful. sd_notify() as used by sshd just writes a static string to a unix socket. This can be implemented in ~10 lines of C, including error handling.
If that is the _only_ thing needed from libsystemd, then open-coding it is a good option. I guess it'd make sense for systemd to provide a header file that provides a reference documentation as an inline function.
I think systemd should do this, just so we have one implementation, and not loads of buggy ones.
One thing which is happening, mostly for unrelated reasons, it that systemd code is being changed to use dlopen() for various libraries which are usually not needed at runtime. The primary motivation for this is to omit such libraries from the initrd. But this also helps with overlinking.
In systemd git main, libsystemd is only linked to libc, libcap, and libgcrypt + libgpg-error. A pull request to convert that that last pair to dlopen is under review.
Also a good change, thanks.
Rich.
On Sat, Mar 30, 2024 at 12:10 PM Richard W.M. Jones rjones@redhat.com wrote:
On Sat, Mar 30, 2024 at 02:44:32PM +0000, Zbigniew Jędrzejewski-Szmek wrote:
On Sat, Mar 30, 2024 at 08:23:48AM -0400, Neal Gompa wrote:
On Sat, Mar 30, 2024 at 8:07 AM Kevin Kofler via devel devel@lists.fedoraproject.org wrote:
(At which point I'd suggest it's probably faster to convert it all to meson or another new shiny, and saner, build system, but getting upstreams to agree will be fun)
CMake! :-)
Meson outclasses CMake in functionality, clarity, and brevity. I doesn't make sense to consider switching to CMake at this point.
Which is the better build system doesn't much matter here, as they all let you run shell scripts so source level backdoors could be inserted in all of them.
The big difference they all have over autoconf is that they don't usually distribute a giant obfuscated (basically binary) shell script.
We can fix autoconf now by running autoreconf and we don't need to boil the ocean to do so.
This drags in libsystemd for sd_notify, and libsystemd is linked to way too much stuff including liblzma. Either we need a split libsdnotify that contains only sd_notify, or we should just stop using sd_notify at all. It increases the attack surface of daemons a lot just to allow the service to be "Type=notify" rather than one of the other available approaches. Arch Linux is also systemd-based nowadays, but still does not link OpenSSH against libsystemd.
This is related to philosophical differences between Arch and Fedora.
That said, it probably makes sense for sd_notify to be its own library instead of being part of libsystemd. I'm sure systemd upstream will consider it now. :)
I don't think a separate library would be particularly useful. sd_notify() as used by sshd just writes a static string to a unix socket. This can be implemented in ~10 lines of C, including error handling.
If that is the _only_ thing needed from libsystemd, then open-coding it is a good option. I guess it'd make sense for systemd to provide a header file that provides a reference documentation as an inline function.
I think systemd should do this, just so we have one implementation, and not loads of buggy ones.
One thing which is happening, mostly for unrelated reasons, it that systemd code is being changed to use dlopen() for various libraries which are usually not needed at runtime. The primary motivation for this is to omit such libraries from the initrd. But this also helps with overlinking.
In systemd git main, libsystemd is only linked to libc, libcap, and libgcrypt + libgpg-error. A pull request to convert that that last pair to dlopen is under review.
Also a good change, thanks.
Note that dlopen() doesn't fix the problem of the giant libsystemd in the first place. It just obfuscates the true dependency graph of libsystemd.
Long ago (I think like ~10 years ago), libsystemd was actually several separate smaller libraries. Perhaps we could consider asking upstream to switch back to that model?
Neal Gompa wrote:
Note that dlopen() doesn't fix the problem of the giant libsystemd in the first place. It just obfuscates the true dependency graph of libsystemd.
At least it (hopefully) means liblzma will not be opened if you do not use an API that needs liblzma. But it makes it even harder to tell whether liblzma will end up being loaded or not.
Long ago (I think like ~10 years ago), libsystemd was actually several separate smaller libraries. Perhaps we could consider asking upstream to switch back to that model?
+1
Kevin Kofler
On Sat, Mar 30, 2024 at 06:58:05PM +0100, Kevin Kofler via devel wrote:
Neal Gompa wrote:
Note that dlopen() doesn't fix the problem of the giant libsystemd in the first place. It just obfuscates the true dependency graph of libsystemd.
At least it (hopefully) means liblzma will not be opened if you do not use an API that needs liblzma. But it makes it even harder to tell whether liblzma will end up being loaded or not.
Long ago (I think like ~10 years ago), libsystemd was actually several separate smaller libraries. Perhaps we could consider asking upstream to switch back to that model?
+1
This comes up occasionally, and the main reason to not do this is that systemd code has a lot of helpers that are used across a lot of the codebase and we end up with recursive calls between many of the different "components". So if tried to split things, we'd either need to remove a lot of polish, or copy code, or have the shared code duplicated. Some of those split-out libraries would probably end up embedding almost all of some of the other ones. libsystemd now consists of 12 parts (sd-bus, sd-daemon, sd-device, sd-event, sd-hwdb, sd-id128, sd-journal, sd-login, sd-netlink, sd-network, sd-path, sd-resolve) and it'd be a lot of work to untangle, and the overall footprint would likely grow.
Zbyszek
On 2024-03-30 09:12, Neal Gompa wrote:
Note that dlopen() doesn't fix the problem of the giant libsystemd in the first place. It just obfuscates the true dependency graph of libsystemd.
This isn't my area of expertise, but I am curious:
Why doesn't dlopen() solve the problem? As best I understand it, liblzma was able to steal one (or more) of the symbols from libcrypto.so.3 because it ran constructors at a point in time when the GOT was still writable. After loading shared objects is complete, that table is made read-only. If dlopen() is used after the program starts, then even if the library is loaded, it can't steal symbols in the table any more.
Or do I misunderstand this entirely?
On Tue, Apr 02, 2024 at 02:00:52AM -0700, Gordon Messmer wrote:
On 2024-03-30 09:12, Neal Gompa wrote:
Note that dlopen() doesn't fix the problem of the giant libsystemd in the first place. It just obfuscates the true dependency graph of libsystemd.
This isn't my area of expertise, but I am curious:
Why doesn't dlopen() solve the problem? As best I understand it, liblzma was able to steal one (or more) of the symbols from libcrypto.so.3 because it ran constructors at a point in time when the GOT was still writable. After loading shared objects is complete, that table is made read-only. If dlopen() is used after the program starts, then even if the library is loaded, it can't steal symbols in the table any more.
Or do I misunderstand this entirely?
You understood correctly ;)
As you wrote, dlopen() is done after RELRO has taken effect.
Also, dlopen() is only called lazily when the required library is to be used for the first time… So in this particular case, the program could call sd_notify(), but no dlopen() call would be done.
Zbyszek
* Gordon Messmer:
Why doesn't dlopen() solve the problem? As best I understand it, liblzma was able to steal one (or more) of the symbols from libcrypto.so.3 because it ran constructors at a point in time when the GOT was still writable. After loading shared objects is complete, that table is made read-only. If dlopen() is used after the program starts, then even if the library is loaded, it can't steal symbols in the table any more.
The rsa_pkcs1_ossl_meth variable that contains a function pointer to the actual implementation resides in read-write memory:
static RSA_METHOD rsa_pkcs1_ossl_meth = { "OpenSSL PKCS#1 RSA", rsa_ossl_public_encrypt, rsa_ossl_public_decrypt, /* signature verification */ rsa_ossl_private_encrypt, /* signing */ rsa_ossl_private_decrypt, rsa_ossl_mod_exp, BN_mod_exp_mont, /* XXX probably we should not use Montgomery * if e == 3 */ rsa_ossl_init, rsa_ossl_finish, RSA_FLAG_FIPS_METHOD, /* flags */ NULL, 0, /* rsa_sign */ 0, /* rsa_verify */ NULL, /* rsa_keygen */ NULL /* rsa_multi_prime_keygen */ };
The missing “const” keyword is probably an oversight, but I don't think it matters because the variable is not used directly. All access happens through deefault_RSA_meth:
static const RSA_METHOD *default_RSA_meth = &rsa_pkcs1_ossl_meth;
void RSA_set_default_method(const RSA_METHOD *meth) { default_RSA_meth = meth; }
const RSA_METHOD *RSA_get_default_method(void) { return default_RSA_meth; }
const RSA_METHOD *RSA_PKCS1_OpenSSL(void) { return &rsa_pkcs1_ossl_meth; }
And that's clearly required to be read-write. RSA_set_default_method is a documented interface, too. So I don't think GOT patching was required to achieve the intended effect.
Furthermore, it's possible to undo the effect of RELRO with mprotect.
Using dlopen, liblzma would not have been loaded at run time, but it's not clear if it had mattered because liblzma would still have been loaded by RPM, both at build and install time.
Thanks, Florian
Am 30.03.24 um 15:44 schrieb Zbigniew Jędrzejewski-Szmek:
Meson outclasses CMake in functionality, clarity, and brevity. I doesn't make sense to consider switching to CMake at this point.
While I do agree on clarity and brevity, I don't on functionality.
Meson doesn't allow you do create your own functions. While one should try to avoid them in build systems, there are cases where you need them. I work on a project where they are needed, but it also wouldn't make sense to upstream it because it's too project specific, and creating a loop out of every call like Meson's FAQ recommends (https://mesonbuild.com/FAQ.html#why-doesnt-meson-have-user-defined-functions...) would create one heck of spagetthi code.
So yeah, there are edge cases where I would recommend CMake over Meson purely without looking at the rest of the ecosystem.
Another is ofc that a quite big chunk of the C and C++ ecosystem uses CMake and integration between build systems kinda sucks (but the CMake developers try to work on making it better by working together with other projects).
But I don't think this is the right place to talk about which build system to switch to FOR OTHER PROJECTS.
(But if we are at it, ever looked at Zig as a buildsystem for a C or C++ project? I know some companies which do this at this point.)
Sincerely
Kilian Hanich
Kilian Hanich via devel wrote:
Meson doesn't allow you do create your own functions. While one should try to avoid them in build systems, there are cases where you need them. I work on a project where they are needed, but it also wouldn't make sense to upstream it because it's too project specific, and creating a loop out of every call like Meson's FAQ recommends (https://mesonbuild.com/FAQ.html#why-doesnt-meson-have-user-defined-functions...) would create one heck of spagetthi code.
Yes, that is exactly what makes Meson all hardcoded and not extensible. If you need to do anything that the Meson developers did not think of, you end up having to work around the build system (using external commands, or a Meson file generator that preprocesses macros, or some other ugly hack) instead of with the build system or just having to switch to a better build system (such as CMake ;-) ).
Using loops instead of functions or macros also misses one important point: the function or macro can be shared between projects and even eventually upstreamed to the build system (both of which happen regularly in the CMake world).
By the way, CMake allows defining both macros and actual functions, and macros are what you want to use in most cases. The main reason functions were introduced is to allow recursion (which is a two-edged sword because it makes the language Turing-complete with all its implications).
Kevin Kofler
Zbigniew Jędrzejewski-Szmek wrote:
Meson outclasses CMake in functionality,
LOL, how so? Everything in Meson is hardcoded, you have very little flexibility (but still enough to plant a backdoor if that is what you want, because you can just invoke a shell script or any other external command). CMake is completely extensible with a complete programming language (it is Turing-complete if and only if you use recursive functions; if you avoid FUNCTION or at least recursive ones, then it is guaranteed to always terminate and still usually expressive enough, but Turing-completeness through recursion is there if you really need it).
clarity, and brevity.
That is very much in the eye of the beholder, and it also depends on whether you use modern CMake or legacy constructs that are often overly verbose. Meson does not have so much legacy stuff simply because 1. it is much newer (which also means it is less mature) and 2. it cares less about backwards compatibility (which is not a good thing, backwards compatibility is essential if you want to avoid upstreams bundling old copies of build tools).
I doesn't make sense to consider switching to CMake at this point.
CMake being what the whole Qt and KDE community uses nowadays (even Qt itself switched to it with Qt 6), it very much makes sense to switch to CMake now more than ever.
The main reason Meson is a thing at all (and not just one person's little- known project as it initially was) is that GNOME did not want to use the same build tool KDE was already using (even though they had the exact same issues with autotools).
I don't think a separate library would be particularly useful. sd_notify() as used by sshd just writes a static string to a unix socket. This can be implemented in ~10 lines of C, including error handling.
Copy&paste is exactly what we do not want here. (It can easily be done wrong, either accidentally or deliberately, introducing security issues, and any security issue discovered in the copied&pasted code has to be fixed in all the copies.) If daemons are likely to need only one function (sd_notify), then a library containing only that one function is exactly what is needed, no matter how small it is.
If that is the _only_ thing needed from libsystemd, then open-coding it is a good option. I guess it'd make sense for systemd to provide a header file that provides a reference documentation as an inline function.
That is an option which is better than copy&paste, but it still means that all users will need to be rebuilt if a bug in that header file is fixed (as for a static library, but unlike a shared library).
One thing which is happening, mostly for unrelated reasons, it that systemd code is being changed to use dlopen() for various libraries which are usually not needed at runtime. The primary motivation for this is to omit such libraries from the initrd. But this also helps with overlinking.
In systemd git main, libsystemd is only linked to libc, libcap, and libgcrypt + libgpg-error. A pull request to convert that that last pair to dlopen is under review.
That helps somewhat (it would have prevented this backdoor from working), but it also makes things even less transparent: How do we know whether some random sd_foobarify() function or some random foobard linked against libsystemd will (always or ever (and when?)) end up dlopening liblzma or not?
Distribution packagers tend to dislike dlopen due to the hidden dependencies it introduces.
Kevin Kofler
On Sat, Mar 30, 2024 at 06:56:27PM +0100, Kevin Kofler via devel wrote:
Zbigniew Jędrzejewski-Szmek wrote:
Meson outclasses CMake in functionality,
LOL, how so? Everything in Meson is hardcoded, you have very little flexibility (but still enough to plant a backdoor if that is what you want, because you can just invoke a shell script or any other external command). CMake is completely extensible with a complete programming language (it is Turing-complete if and only if you use recursive functions; if you avoid FUNCTION or at least recursive ones, then it is guaranteed to always terminate and still usually expressive enough, but Turing-completeness through recursion is there if you really need it).
Pretty much every build system is Turing-complete, and pretty much every build system allows scripts to be called. Upstreams can do crazy stuff with any build system too.
I think Meson hits the sweet spot where simple projects can be expressed in very few very readable lines. And common configuration tests that need to be done are still very readable.
The handling of dependencies and options is very declarative. (Meson is also evolving, in early versions it was less so.)
It'd be interesting to see some real statistics, e.g. over all Fedora packages, but in my experience, CMake projects tend to have many more files with lots of lines.
clarity, and brevity.
That is very much in the eye of the beholder, and it also depends on whether you use modern CMake or legacy constructs that are often overly verbose.
True ;) In _my_ eye, it's prettier. I guess we'll not achieve consensus.
One particular issue I have with CMake as a downstream maintainer is it's often very hard to override linking or compilation options or when the project is using one of the cmake find scripts that gets something wrong… It's possible that those projects are "doing cmake wrong", but it seems that CMake makes it easy to do things wrong.
I doesn't make sense to consider switching to CMake at this point.
CMake being what the whole Qt and KDE community uses nowadays (even Qt itself switched to it with Qt 6), it very much makes sense to switch to CMake now more than ever.
Right, but if one doesn't work on Qt, then this argument doesn't do much.
[snip]
One thing which is happening, mostly for unrelated reasons, it that systemd code is being changed to use dlopen() for various libraries which are usually not needed at runtime. The primary motivation for this is to omit such libraries from the initrd. But this also helps with overlinking.
In systemd git main, libsystemd is only linked to libc, libcap, and libgcrypt + libgpg-error. A pull request to convert that that last pair to dlopen is under review.
That helps somewhat (it would have prevented this backdoor from working), but it also makes things even less transparent: How do we know whether some random sd_foobarify() function or some random foobard linked against libsystemd will (always or ever (and when?)) end up dlopening liblzma or not?
Distribution packagers tend to dislike dlopen due to the hidden dependencies it introduces.
This is true. It'd be great if the compilers and linkers supported a "weak dependency" model, where we'd use the normal call syntax and things would be the same as with a normal shared library link, but if that library is not found, the program would still load and run. Then we could still autogenerate those dependencies on optional libraries, but using Recommends instead of Requires.
Right now the dependency handling needs to be handled manually, which is error-prone and annoying.
Zbyszek
On Sat, Mar 30, 2024 at 10:15:47PM +0000, Zbigniew Jędrzejewski-Szmek wrote:
One particular issue I have with CMake as a downstream maintainer is it's often very hard to override linking or compilation options or when the project is using one of the cmake find scripts that gets something wrong… It's possible that those projects are "doing cmake wrong", but it seems that CMake makes it easy to do things wrong.
I just had to interact with CMake, so let's use that as example: I'm rebuilding rpm.rpm with some local patches using 'fedpkg local' in F40. The build fails with: Directory not found: /home/zbyszek/rpmbuild/BUILDROOT/rpm-4.19.1.1-2.fc41.x86_64/usr/lib64/python3.12/site-packages/rpm
Hmm, why? Oh, rpm uses cmake, and cmake has it's own special detection of python, and it found /usr/bin/python3.13t that I have installed, and subsequently it got all the paths wrong. How do I override this? ('cmake -LAH' doesn't yield anything useful.)
(When compiling a python module, any other alternative would be better. The build uses %python3, i.e. /usr/bin/python3.12, so there's no need to detect anything, the location of python is known and all this binary can be called to print all paths. Otherwise, either call 'python3-config' or 'pkgconf python', don't do you own reimplementation.)
Zbyszek
That's why you should never build packages outside of mock.
Kevin Kofler
On Sun, Apr 7 2024 at 13:52:26 +00:00:00, Zbigniew Jędrzejewski-Szmek zbyszek@in.waw.pl wrote:
On Sat, Mar 30, 2024 at 10:15:47PM +0000, Zbigniew Jędrzejewski-Szmek wrote:
One particular issue I have with CMake as a downstream maintainer is it's often very hard to override linking or compilation options or when the project is using one of the cmake find scripts that gets something wrong… It's possible that those projects are "doing cmake wrong", but it seems that CMake makes it easy to do things wrong.
I just had to interact with CMake, so let's use that as example: I'm rebuilding rpm.rpm with some local patches using 'fedpkg local' in F40. The build fails with: Directory not found: /home/zbyszek/rpmbuild/BUILDROOT/rpm-4.19.1.1-2.fc41.x86_64/usr/lib64/python3.12/site-packages/rpm
Hmm, why? Oh, rpm uses cmake, and cmake has it's own special detection of python, and it found /usr/bin/python3.13t that I have installed, and subsequently it got all the paths wrong. How do I override this? ('cmake -LAH' doesn't yield anything useful.)
(When compiling a python module, any other alternative would be better. The build uses %python3, i.e. /usr/bin/python3.12, so there's no need to detect anything, the location of python is known and all this binary can be called to print all paths. Otherwise, either call 'python3-config' or 'pkgconf python', don't do you own reimplementation.)
Zbyszek
I wrote:
On Sun, Apr 7 2024 at 13:52:26 +00:00:00, Zbigniew Jędrzejewski-Szmek zbyszek@in.waw.pl wrote:
Hmm, why? Oh, rpm uses cmake, and cmake has it's own special detection of python, and it found /usr/bin/python3.13t that I have installed, and subsequently it got all the paths wrong.
That's why you should never build packages outside of mock.
PS: Autotools also loves to autodetect random libraries that happen to be installed on the system. It is in no way specific to CMake.
How do I override this? ('cmake -LAH' doesn't yield anything useful.)
Usually -DSOME_VARIABLE=/some/path is the way, look in FindPython.cmake for the variables it uses. (First, try to figure out whether RPM is using a system-installed FindPython or its own custom version, so you look at the correct version.) But the safest (for all build systems) is to always build in a mock chroot with only the expected BuildRequires installed, as I have written.
Kevin Kofler
On Mon, Apr 08, 2024 at 12:03:19AM +0200, Kevin Kofler via devel wrote:
I wrote:
On Sun, Apr 7 2024 at 13:52:26 +00:00:00, Zbigniew Jędrzejewski-Szmek zbyszek@in.waw.pl wrote:
Hmm, why? Oh, rpm uses cmake, and cmake has it's own special detection of python, and it found /usr/bin/python3.13t that I have installed, and subsequently it got all the paths wrong.
That's why you should never build packages outside of mock.
That's another way of saying "it's broken" ;] Mock is great, but I'm doing local development and a local build against my envirnoment is what I need.
PS: Autotools also loves to autodetect random libraries that happen to be installed on the system. It is in no way specific to CMake.
How do I override this? ('cmake -LAH' doesn't yield anything useful.)
Usually -DSOME_VARIABLE=/some/path is the way, look in FindPython.cmake for the variables it uses. (First, try to figure out whether RPM is using a system-installed FindPython or its own custom version, so you look at the correct version.)
Exactly. I'm sure it doable, but CMake ecosystem somehow doesn't want to integrate with the Linux userspace in the normal way.
Zbyszek
On Mon, Apr 8, 2024 at 5:37 AM Zbigniew Jędrzejewski-Szmek zbyszek@in.waw.pl wrote:
On Mon, Apr 08, 2024 at 12:03:19AM +0200, Kevin Kofler via devel wrote:
I wrote:
On Sun, Apr 7 2024 at 13:52:26 +00:00:00, Zbigniew Jędrzejewski-Szmek zbyszek@in.waw.pl wrote:
Hmm, why? Oh, rpm uses cmake, and cmake has it's own special detection of python, and it found /usr/bin/python3.13t that I have installed, and subsequently it got all the paths wrong.
That's why you should never build packages outside of mock.
That's another way of saying "it's broken" ;] Mock is great, but I'm doing local development and a local build against my envirnoment is what I need.
PS: Autotools also loves to autodetect random libraries that happen to be installed on the system. It is in no way specific to CMake.
How do I override this? ('cmake -LAH' doesn't yield anything useful.)
Usually -DSOME_VARIABLE=/some/path is the way, look in FindPython.cmake for the variables it uses. (First, try to figure out whether RPM is using a system-installed FindPython or its own custom version, so you look at the correct version.)
Exactly. I'm sure it doable, but CMake ecosystem somehow doesn't want to integrate with the Linux userspace in the normal way.
You know that pkgconfig *also* uses variables too, right? It's perfectly "normal" to do it this way. Arguably, the fact that every variable is known and saved in the CMakeLists and CMakeCache data for you to read is a boon, because if you need to know a variable to override, you can find it easily in the logs.
And this your statement on normalcy is silly since we don't *have* a normal way. The GNU Build System (Autotools), Meson, and CMake all do things differently, and all three have significant share in our packages. Projects that use plain Makefiles are even *more* non-standard, as they do whatever they want too.
On Mon, Apr 08, 2024 at 05:52:07AM -0400, Neal Gompa wrote:
On Mon, Apr 8, 2024 at 5:37 AM Zbigniew Jędrzejewski-Szmek zbyszek@in.waw.pl wrote:
On Mon, Apr 08, 2024 at 12:03:19AM +0200, Kevin Kofler via devel wrote:
I wrote:
On Sun, Apr 7 2024 at 13:52:26 +00:00:00, Zbigniew Jędrzejewski-Szmek zbyszek@in.waw.pl wrote:
Hmm, why? Oh, rpm uses cmake, and cmake has it's own special detection of python, and it found /usr/bin/python3.13t that I have installed, and subsequently it got all the paths wrong.
That's why you should never build packages outside of mock.
That's another way of saying "it's broken" ;] Mock is great, but I'm doing local development and a local build against my envirnoment is what I need.
PS: Autotools also loves to autodetect random libraries that happen to be installed on the system. It is in no way specific to CMake.
How do I override this? ('cmake -LAH' doesn't yield anything useful.)
Usually -DSOME_VARIABLE=/some/path is the way, look in FindPython.cmake for the variables it uses. (First, try to figure out whether RPM is using a system-installed FindPython or its own custom version, so you look at the correct version.)
Exactly. I'm sure it doable, but CMake ecosystem somehow doesn't want to integrate with the Linux userspace in the normal way.
You know that pkgconfig *also* uses variables too, right? It's perfectly "normal" to do it this way. Arguably, the fact that every variable is known and saved in the CMakeLists and CMakeCache data for you to read is a boon, because if you need to know a variable to override, you can find it easily in the logs.
So… this is what I'm talking about: there is no obvious way to figure out what to set. Looking at the logs and trying to figure out some variables from that is not very attractive.
Of course I know pgkconfig uses variables, I don't know what you are trying to say.
I think CMake hits the "sour spot" here in a few different ways:
- because of the historical mindset, instead of using standard mechanisms that everybody else uses, it implements custom detection, - this autodetection very often does things wrong, - because of the approach with Find* scripts, each package is different and each detection script has a custom interface, so overriding things is a lot of work, - the complicated language that is partially variable-based, partially macro-based is not easy to follow.
autotools has './configure --help', meson has all options in `meson_options.txt' and also 'meson configure' will print all options, but with CMake the options are buried in Find* and not easy to figure out.
And this your statement on normalcy is silly since we don't *have* a normal way. The GNU Build System (Autotools), Meson, and CMake all do things differently, and all three have significant share in our packages. Projects that use plain Makefiles are even *more* non-standard, as they do whatever they want too.
Yeah, I know we have a bazillion build systems. And all have to work and the packagers need to use and understand all of them. Nevertheless, for me, CMake and autotools are outdated technologies that shouldn't be used in new projects.
Zbyszek
Zbigniew Jędrzejewski-Szmek wrote:
So… this is what I'm talking about: there is no obvious way to figure out what to set. Looking at the logs and trying to figure out some variables from that is not very attractive.
The comments at the top of the relevant Find*.cmake module are the best source for which variables you are supposed to set directly.
But there is also cmake-gui that can show you all the available options in a pretty Qt GUI.
Nevertheless, for me, CMake and autotools are outdated technologies that shouldn't be used in new projects.
And for me, Meson is just a poor Not Invented Here imitation of CMake, with fewer features and in a slower programming language. :-)
And the kind of automagic you complain about is something all 3 major build systems do (and plenty of obscure ones, too). Maybe not for the specific case of the Python executable, but there are plenty of other cases where autotools and Meson also do automagic, which is why building outside of a chroot is such a bad idea.
Kevin Kofler
On Sa, 30.03.24 18:56, Fedora Development ML (devel@lists.fedoraproject.org) wrote:
In systemd git main, libsystemd is only linked to libc, libcap, and libgcrypt + libgpg-error. A pull request to convert that that last pair to dlopen is under review.
That helps somewhat (it would have prevented this backdoor from working), but it also makes things even less transparent: How do we know whether some random sd_foobarify() function or some random foobard linked against libsystemd will (always or ever (and when?)) end up dlopening liblzma or not?
Distribution packagers tend to dislike dlopen due to the hidden dependencies it introduces.
Well, this is certainly a valid point, but the solution is not to make all deps hard ones. Instead, in order to just make these deps visible I see multiple better alternatives:
1. teach Linux/ELF something inspired by MacOS's weak deps. i.e. allow ELF programs declare that some symbols shall be backed by some lib, but make it a weak dep that is only resolved on demand, and gracefully is set to NULL if the library cannot be found. If we had that, we'd stop using dlopen() for systemd's deps, since all we do is basically reimplement this concept manually.
2. without Linux/ELF supporting the above we could still teach systemd and other tools in similar situations to declare the dlopen deps they have in some ELF note or section, so that it can be processed by initrd generators, automatic dep generators in dpkg/rpm, ldd-like tools and everything else that wants this info. This would require some C macro magic, but could be added in probably just a few lines of codes added to relevant projects.
Lennart
-- Lennart Poettering, Berlin
* Lennart Poettering:
On Sa, 30.03.24 18:56, Fedora Development ML (devel@lists.fedoraproject.org) wrote:
In systemd git main, libsystemd is only linked to libc, libcap, and libgcrypt + libgpg-error. A pull request to convert that that last pair to dlopen is under review.
That helps somewhat (it would have prevented this backdoor from working), but it also makes things even less transparent: How do we know whether some random sd_foobarify() function or some random foobard linked against libsystemd will (always or ever (and when?)) end up dlopening liblzma or not?
Distribution packagers tend to dislike dlopen due to the hidden dependencies it introduces.
Well, this is certainly a valid point, but the solution is not to make all deps hard ones. Instead, in order to just make these deps visible I see multiple better alternatives:
- teach Linux/ELF something inspired by MacOS's weak deps. i.e. allow ELF programs declare that some symbols shall be backed by some lib, but make it a weak dep that is only resolved on demand, and gracefully is set to NULL if the library cannot be found. If we had that, we'd stop using dlopen() for systemd's deps, since all we do is basically reimplement this concept manually.
How exactly is this implemented? Is the object loaded on the first function call? I'd be worried that this made initialization even more complicated than it is today.
Loading the object unconditionally (prior to the function call) doesn't make a difference for objects which are always installed on the system. They would always be loaded.
Thanks, Florian
Kevin Kofler via devel wrote:
Dominique Martinet wrote:
Before making each of these safer we should make sshd not link with so many things in the first place.
Indeed. E.g., Arch Linux does not transitively link sshd against liblzma. Fedora does because of this innocuous-looking patch: https://src.fedoraproject.org/rpms/openssh/blob/rawhide/f/openssh-7.4p1-syst... which is what ultimately allowed this to happen. This drags in libsystemd for sd_notify, and libsystemd is linked to way too much stuff including liblzma. Either we need a split libsdnotify that contains only sd_notify, or we should just stop using sd_notify at all.
Upstream openssh-portable has a proposed patch which simply implements the sdnotify protocol directly. That would provide the benefits with none of the over-linking risk.
https://bugzilla.mindrot.org/show_bug.cgi?id=2641#c13
It could use some review from distro folks familiar with sshd systemd integration.
(The wider point about splitting the sdnotify functionality is still quite useful, to avoid everyone re-implementing the same thing and possibly adding bugs in _that_ process.)
On Sat, Mar 30, 2024 at 1:07 PM Kevin Kofler via devel < devel@lists.fedoraproject.org> wrote:
Before making each of these safer we should make sshd not link with so many things in the first place.
Indeed. E.g., Arch Linux does not transitively link sshd against liblzma. Fedora does because of this innocuous-looking patch:
https://src.fedoraproject.org/rpms/openssh/blob/rawhide/f/openssh-7.4p1-syst... which is what ultimately allowed this to happen. This drags in libsystemd for sd_notify, and libsystemd is linked to way too much stuff including liblzma. Either we need a split libsdnotify that contains only sd_notify, or we should just stop using sd_notify at all. It increases the attack surface of daemons a lot just to allow the service to be "Type=notify" rather than one of the other available approaches. Arch Linux is also systemd-based nowadays, but still does not link OpenSSH against libsystemd.
We have an upstream-adjusted version of this patch, see https://bugzilla.mindrot.org/show_bug.cgi?id=2641 I'm OK to bring the updated version of this script to Fedora as soon as it is finalized.
On 2024-03-30 11:52, Dmitry Belyavskiy wrote:
We have an upstream-adjusted version of this patch, see https://bugzilla.mindrot.org/show_bug.cgi?id=2641 I'm OK to bring the updated version of this script to Fedora as soon as it is finalized.
I proposed https://src.fedoraproject.org/rpms/openssh/pull-request/72 , which uses the implementation from https://www.freedesktop.org/software/systemd/man/devel/sd_notify.html#Notes
On Wednesday, 3 April 2024 01:34:00 CEST Gordon Messmer wrote:
On 2024-03-30 11:52, Dmitry Belyavskiy wrote:
We have an upstream-adjusted version of this patch, see https://bugzilla.mindrot.org/show_bug.cgi?id=2641 I'm OK to bring the updated version of this script to Fedora as soon as it is finalized.
I proposed https://src.fedoraproject.org/rpms/openssh/pull-request/72 , which uses the implementation from https://www.freedesktop.org/software/systemd/man/devel/sd_notify.html#Notes
Thanks for the contribution, but please use https://bugzilla.mindrot.org/ show_bug.cgi?id=2641#c13 instead.
I wrote:
That is exactly the problem with autotools code, almost nobody understands what the heck it does, almost everybody just copies and pastes somebody else's snippet hoping it does not do bad things. And gnulib is a particularly ugly piece of the puzzle.
PS: Here is a pretty good post summarizing the issues with autotools, both generally and in the context of the xz vulnerability: https://felipec.wordpress.com/2024/04/04/xz-backdoor-and-autotools-insanity/
Kevin Kofler
On Sat, Mar 30, 2024 at 08:22:06PM +0900, Dominique Martinet wrote:
the initial injection (original: https://git.savannah.gnu.org/gitweb/?p=gnulib.git;a=blob;f=m4/build-to-host....).
(Honestly I did compare the backdoored script and the real one this morning and I would be hard pressed to say if either is backdoored just looking at either version... Admitedly it was 3AM when I looked at it, but I don't think it's just a late hour problem)
Right! Definitely not a 3am problem :-/
(3) We should have a "security path", like "critical path".
...
Before making each of these safer we should make sshd not link with so many things in the first place. On oss-security, Solar Designer made a lot of good points about it (around here: https://www.openwall.com/lists/oss-security/2024/03/29/27 , but the full thread is interesting)
Agreed.
Rich.
Richard W.M. Jones wrote:
(1) We should routinely delete autoconf-generated cruft from upstream projects and regenerate it in %prep. It is easier to study the real source rather than dig through the convoluted, generated shell script in an upstream './configure' looking for back doors.
For most projects, just running "autoreconf -fiv" is enough.
Yes, there are some projects that depend on a specific or old version of autoconf. We should fix those. But that doesn't need to delay us from using autoreconf on many projects today.
I have always said that. We do not allow other prebuilt binary blobs and require rebuilding everything from source. I do not see how the obfuscated autogenerated shell scripts from the autotools are any different. Sadly, I was ignored. Now you folks (Fedora) see where this leads. Told You So.
In the xz case this wouldn't have been enough, it turns out we would also have to delete m4/build-to-host.m4, which then autoreconf regenerates. I don't fully understand why that is.
Looks like autoreconf does not work as advertised. With -f, it is supposed to regenerate all files that it knows how to regenerate. It clearly does not do what it is supposed to and ought to be fixed.
(2) We should discourage gnulib as much as possible.
In libvirt we took the decision a few years ago to remove gnulib. It's extremely convoluted and almost no one understands how it really works. It's written in obscure m4 macros and shell script.
It's also not necessary for Linux since gnulib is mainly about porting to non-Linux platforms. There are better ways to do this.
In the xz case it was a gnulib-derived script which was modified to do the initial injection (original: https://git.savannah.gnu.org/gitweb/?p=gnulib.git;a=blob;f=m4/build-to-host....).
There too, I agree completely. Also because gnulib is a "copylib", which is a very broken concept. A true library would not be subject to this kind of "take the file from the copylib and inject bad code into it" attack.
(3) We should have a "security path", like "critical path".
sshd is linked to a lot of libraries:
/lib64/libaudit.so.1 audit-libs /lib64/libc.so.6 glibc /lib64/libcap-ng.so.0 libcap-ng /lib64/libcap.so.2 libcap /lib64/libcom_err.so.2 libcom_err /lib64/libcrypt.so.2 libxcrypt /lib64/libcrypto.so.3 openssl-libs /lib64/libeconf.so.0 libeconf /lib64/libgcc_s.so.1 libgcc /lib64/libgssapi_krb5.so.2 krb5-libs /lib64/libk5crypto.so.3 krb5-libs /lib64/libkeyutils.so.1 keyutils-libs /lib64/libkrb5.so.3 krb5-libs /lib64/libkrb5support.so.0 krb5-libs /lib64/liblz4.so.1 lz4-libs /lib64/liblzma.so.5 xz-libs /lib64/libm.so.6 glibc /lib64/libpam.so.0 pam-libs /lib64/libpcre2-8.so.0 pcre2 /lib64/libresolv.so.2 glibc /lib64/libselinux.so.1 libselinux /lib64/libsystemd.so.0 systemd-libs /lib64/libz.so.1 zlib / zlib-ng /lib64/libzstd.so.1 zstd
Should we have a higher level of attention to these packages? We already have "critical path", but that's a broad category now. These seem like they are "security path" packages, an intentionally small subset associated with very secure services which are enabled by default.
I think the issue is that there is just too much stuff in critpath these days. Whole desktop environments and all their transitive dependencies probably ought to not be in there. If at all, I would put the display manager in there, maybe the window manager, and no further.
Kevin Kofler
On Sat, Mar 30, 2024 at 7:54 AM Kevin Kofler via devel devel@lists.fedoraproject.org wrote:
Richard W.M. Jones wrote:
(1) We should routinely delete autoconf-generated cruft from upstream projects and regenerate it in %prep. It is easier to study the real source rather than dig through the convoluted, generated shell script in an upstream './configure' looking for back doors.
For most projects, just running "autoreconf -fiv" is enough.
Yes, there are some projects that depend on a specific or old version of autoconf. We should fix those. But that doesn't need to delay us from using autoreconf on many projects today.
I have always said that. We do not allow other prebuilt binary blobs and require rebuilding everything from source. I do not see how the obfuscated autogenerated shell scripts from the autotools are any different. Sadly, I was ignored. Now you folks (Fedora) see where this leads. Told You So.
This is not helpful in the slightest and the tone is not appreciated at all.
That said, this is being tracked by the Packaging Committee: https://pagure.io/packaging-committee/issue/1350
In the xz case this wouldn't have been enough, it turns out we would also have to delete m4/build-to-host.m4, which then autoreconf regenerates. I don't fully understand why that is.
Looks like autoreconf does not work as advertised. With -f, it is supposed to regenerate all files that it knows how to regenerate. It clearly does not do what it is supposed to and ought to be fixed.
(2) We should discourage gnulib as much as possible.
In libvirt we took the decision a few years ago to remove gnulib. It's extremely convoluted and almost no one understands how it really works. It's written in obscure m4 macros and shell script.
It's also not necessary for Linux since gnulib is mainly about porting to non-Linux platforms. There are better ways to do this.
In the xz case it was a gnulib-derived script which was modified to do the initial injection (original: https://git.savannah.gnu.org/gitweb/?p=gnulib.git;a=blob;f=m4/build-to-host....).
There too, I agree completely. Also because gnulib is a "copylib", which is a very broken concept. A true library would not be subject to this kind of "take the file from the copylib and inject bad code into it" attack.
Yes, we should scrutinize things like this. Though I will note that we didn't actually suffer an attack through this venue with library code, just the build scripts. Generally, people do not pay attention to build scripts, and that was how this slipped by for so long. But even so, Autotools is particularly difficult to understand and I don't think we would have ordinarily caught it anyway.
(3) We should have a "security path", like "critical path".
sshd is linked to a lot of libraries:
/lib64/libaudit.so.1 audit-libs /lib64/libc.so.6 glibc /lib64/libcap-ng.so.0 libcap-ng /lib64/libcap.so.2 libcap /lib64/libcom_err.so.2 libcom_err /lib64/libcrypt.so.2 libxcrypt /lib64/libcrypto.so.3 openssl-libs /lib64/libeconf.so.0 libeconf /lib64/libgcc_s.so.1 libgcc /lib64/libgssapi_krb5.so.2 krb5-libs /lib64/libk5crypto.so.3 krb5-libs /lib64/libkeyutils.so.1 keyutils-libs /lib64/libkrb5.so.3 krb5-libs /lib64/libkrb5support.so.0 krb5-libs /lib64/liblz4.so.1 lz4-libs /lib64/liblzma.so.5 xz-libs /lib64/libm.so.6 glibc /lib64/libpam.so.0 pam-libs /lib64/libpcre2-8.so.0 pcre2 /lib64/libresolv.so.2 glibc /lib64/libselinux.so.1 libselinux /lib64/libsystemd.so.0 systemd-libs /lib64/libz.so.1 zlib / zlib-ng /lib64/libzstd.so.1 zstd
Should we have a higher level of attention to these packages? We already have "critical path", but that's a broad category now. These seem like they are "security path" packages, an intentionally small subset associated with very secure services which are enabled by default.
I think the issue is that there is just too much stuff in critpath these days. Whole desktop environments and all their transitive dependencies probably ought to not be in there. If at all, I would put the display manager in there, maybe the window manager, and no further.
I don't know if the security path package list would help any, since we still have no one to do security work.
That said, I agree that pretty much every display manager and compositor for every Fedora variant should be critpath'd.
Neal Gompa wrote:
This is not helpful in the slightest and the tone is not appreciated at all.
Well, I have been arguing against this exception (exempting prebuilt autotools output) from the "no prebuilt blobs" rule for years, and it saddens me that something like this had to happen for Fedora to finally realize that that exception has always been a bad idea.
That said, this is being tracked by the Packaging Committee: https://pagure.io/packaging-committee/issue/1350
Finally, thanks! (But you filed that only now after this incident, see above. Still, thanks that you are bringing this up now!)
Yes, we should scrutinize things like this. Though I will note that we didn't actually suffer an attack through this venue with library code, just the build scripts. Generally, people do not pay attention to build scripts, and that was how this slipped by for so long. But even so, Autotools is particularly difficult to understand and I don't think we would have ordinarily caught it anyway.
I definitely agree there, build scripts are indeed an attractive target for backdoor authors, and autotools is indeed a big part of the problem.
The whole architecture of autotools was designed for a situation that is mostly obsolete these days: people running some proprietary Unix with some buggy implementation of a Bourne-like shell and no centralized build tools who want to just untar a tarball and build it with only what they already have installed (the buggy shell). Hence all this concept of shipping prebuilt obfuscated shell blobs full of workarounds for bugs in ancient shell implementations. Nowadays, people are either running GNU/Linux, where centralized build tools such as CMake or Meson are readily installable from the repository (and where most builds are done by distributions, for whom it is just a matter of adding, e.g., "BuildRequires: cmake"), or Microsoft Windows, where an environment that can run autotools scripts (e.g., MSYS2) is NOT part of the system and actually as hard or harder to install than something like CMake. So, nowadays, pregenerating shell scripts is a completely outdated and unhelpful way of working.
That said, I agree that pretty much every display manager and compositor for every Fedora variant should be critpath'd.
Well, where we disagree is that I actually want to see LESS stuff in critpath, not more. It cannot be scrutinized well enough because there is just too much stuff in it. E.g., at times, we had MySQL/MariaDB in critpath because Akonadi required it. (Nowadays, Akonadi actually recommends using SQLite instead.) That just does not make sense.
Kevin Kofler
On Sat, Mar 30, 2024 at 8:53 AM Kevin Kofler via devel devel@lists.fedoraproject.org wrote:
Neal Gompa wrote:
This is not helpful in the slightest and the tone is not appreciated at all.
Well, I have been arguing against this exception (exempting prebuilt autotools output) from the "no prebuilt blobs" rule for years, and it saddens me that something like this had to happen for Fedora to finally realize that that exception has always been a bad idea.
That said, this is being tracked by the Packaging Committee: https://pagure.io/packaging-committee/issue/1350
Finally, thanks! (But you filed that only now after this incident, see above. Still, thanks that you are bringing this up now!)
Yes, we should scrutinize things like this. Though I will note that we didn't actually suffer an attack through this venue with library code, just the build scripts. Generally, people do not pay attention to build scripts, and that was how this slipped by for so long. But even so, Autotools is particularly difficult to understand and I don't think we would have ordinarily caught it anyway.
I definitely agree there, build scripts are indeed an attractive target for backdoor authors, and autotools is indeed a big part of the problem.
The whole architecture of autotools was designed for a situation that is mostly obsolete these days: people running some proprietary Unix with some buggy implementation of a Bourne-like shell and no centralized build tools who want to just untar a tarball and build it with only what they already have installed (the buggy shell). Hence all this concept of shipping prebuilt obfuscated shell blobs full of workarounds for bugs in ancient shell implementations. Nowadays, people are either running GNU/Linux, where centralized build tools such as CMake or Meson are readily installable from the repository (and where most builds are done by distributions, for whom it is just a matter of adding, e.g., "BuildRequires: cmake"), or Microsoft Windows, where an environment that can run autotools scripts (e.g., MSYS2) is NOT part of the system and actually as hard or harder to install than something like CMake. So, nowadays, pregenerating shell scripts is a completely outdated and unhelpful way of working.
And in CMake's favor, there's a huge ecosystem of helpers and integrations that make it easier for people to understand what CMake is doing as it's being developed, built, and shipped. That makes it much more attractive than Autotools simply because the knowledge and the tooling is there for developers and users to dig into it.
That said, I agree that pretty much every display manager and compositor for every Fedora variant should be critpath'd.
Well, where we disagree is that I actually want to see LESS stuff in critpath, not more. It cannot be scrutinized well enough because there is just too much stuff in it. E.g., at times, we had MySQL/MariaDB in critpath because Akonadi required it. (Nowadays, Akonadi actually recommends using SQLite instead.) That just does not make sense.
I didn't say every single one we package, just the ones shipped in deliverables. I think it's worth making sure those are on the critpath.
Neal Gompa wrote:
And in CMake's favor, there's a huge ecosystem of helpers and integrations that make it easier for people to understand what CMake is doing as it's being developed, built, and shipped. That makes it much more attractive than Autotools simply because the knowledge and the tooling is there for developers and users to dig into it.
Well, to be fair, I have also seen (more than once!) arcane stuff being done in CMake, with almost a whole new build system being built on top of CMake (tons of custom macros implementing things such as bundled libraries, before CMake had native support for that), which was not particularly easy to understand either.
If you use CMake the way it was intended, a CMakeLists.txt is a lot more readable than even a configure.ac and Makefile.am, let alone the generated blobs autoconf spits out based on those. But there is potential for abuse, too.
That said, I do not believe completely banning custom functions and macros as Meson does is a workable solution.
Kevin Kofler
Kevin Kofler wrote:
This is not helpful in the slightest and the tone is not appreciated at all.
Well, I have been arguing against this exception (exempting prebuilt autotools output) from the "no prebuilt blobs" rule for years, and it saddens me that something like this had to happen for Fedora to finally realize that that exception has always been a bad idea.
[…]
CMIIW, but it would not have made any difference as the source code had been shipped as part of the tar ball and auto(re)conf would have happily integrated it into the next build. I suspect that a modification to CMakeLists.txt and its includes would not have been detected either; even a daring, but obvious change in the 30000+ lines of source itself might have gone unnoticed.
A major factor seems to have been the discrepancy between "the source code" at GitHub & Co. that was probably scrutinized by many eyes and the shipped, but different artifact. So one step (as a inter-distribution effort) could be to continuously automatically compare shipped artifacts with their "make dist" equivalents and publishing the results.
Tim
On Sat, Mar 30, 2024 at 09:09:35AM -0400, Neal Gompa wrote:
And in CMake's favor, there's a huge ecosystem of helpers and integrations that make it easier for people to understand what CMake is doing as it's being developed, built, and shipped.
That is actually a weakness:
On Sat, Mar 30, 2024 at 01:38:45PM +0000, Tim Landscheidt wrote:
Kevin Kofler wrote:
Well, I have been arguing against this exception (exempting prebuilt autotools output) from the "no prebuilt blobs" rule for years, and it saddens me that something like this had to happen for Fedora to finally realize that that exception has always been a bad idea.
CMIIW, but it would not have made any difference as the source code had been shipped as part of the tar ball and auto(re)conf would have happily integrated it into the next build. I suspect that a modification to CMakeLists.txt and its includes would not have been detected either; even a daring, but obvious change in the 30000+ lines of source itself might have gone unnoticed.
CMake for many years fought against pkgconf and pushed people towards copying those scripts into sources. It is still very common for projects using CMake to come with a whole directory of badly written detection scripts that each replace a single-line pkgconf invocation.
And of course nobody has time to look into those scripts, making it easy to smuggle something through there.
Zbyszek
On Sat, Mar 30 2024 at 02:55:21 PM +00:00:00, Zbigniew Jędrzejewski-Szmek zbyszek@in.waw.pl wrote:
CMake for many years fought against pkgconf and pushed people towards copying those scripts into sources. It is still very common for projects using CMake to come with a whole directory of badly written detection scripts that each replace a single-line pkgconf invocation.
And of course nobody has time to look into those scripts, making it easy to smuggle something through there.
It's still better than Autotools, though. If a project doesn't want to switch to Meson for whatever reason, then CMake is a reasonable alternative.
I agree that CMake is not as good as Meson, and that CMake find modules are inferior to pkg-config. The CMake developers are working on replacing find modules with CPS [1] which is intended to be a replacement for pkg-config that will work better on non-Linux platforms, where pkg-config is not always adequate. It looks like that work has maybe stalled? but if successful that would fix the problem with find modules.
On Sat, Mar 30, 2024 at 10:02:42AM -0500, Michael Catanzaro wrote:
On Sat, Mar 30 2024 at 02:55:21 PM +00:00:00, Zbigniew Jędrzejewski-Szmek zbyszek@in.waw.pl wrote:
CMake for many years fought against pkgconf and pushed people towards copying those scripts into sources. It is still very common for projects using CMake to come with a whole directory of badly written detection scripts that each replace a single-line pkgconf invocation.
And of course nobody has time to look into those scripts, making it easy to smuggle something through there.
It's still better than Autotools, though. If a project doesn't want to switch to Meson for whatever reason, then CMake is a reasonable alternative.
I agree that CMake is not as good as Meson, and that CMake find modules are inferior to pkg-config.
But then we shouldn't describe them as equivalent alternatives ;) If we say "switch to a modern build systemd, e.g. cmake or meson", people will randomly choose on or the other and since the whole CMake ecosystem is built around find modules, we'll end with a bazillion of those.
Instead we should say: "Use meson. If you can't for some reason, consider CMake, but come talk to us first."
The CMake developers are working on replacing find modules with CPS [1] which is intended to be a replacement for pkg-config that will work better on non-Linux platforms, where pkg-config is not always adequate. It looks like that work has maybe stalled? but if successful that would fix the problem with find modules.
Ack. But from our point of view, this wouldn't be great. We already have pkgconf files for almost everything and CPS files for nothing… In fact, we should probably make the effort to add pkgconf files for the few libraries that don't have it to make it completely standard and expected.
Zbyszek
On Sat, Mar 30, 2024 at 11:36 AM Zbigniew Jędrzejewski-Szmek zbyszek@in.waw.pl wrote:
On Sat, Mar 30, 2024 at 10:02:42AM -0500, Michael Catanzaro wrote:
On Sat, Mar 30 2024 at 02:55:21 PM +00:00:00, Zbigniew Jędrzejewski-Szmek zbyszek@in.waw.pl wrote:
CMake for many years fought against pkgconf and pushed people towards copying those scripts into sources. It is still very common for projects using CMake to come with a whole directory of badly written detection scripts that each replace a single-line pkgconf invocation.
And of course nobody has time to look into those scripts, making it easy to smuggle something through there.
It's still better than Autotools, though. If a project doesn't want to switch to Meson for whatever reason, then CMake is a reasonable alternative.
I agree that CMake is not as good as Meson, and that CMake find modules are inferior to pkg-config.
But then we shouldn't describe them as equivalent alternatives ;) If we say "switch to a modern build systemd, e.g. cmake or meson", people will randomly choose on or the other and since the whole CMake ecosystem is built around find modules, we'll end with a bazillion of those.
Instead we should say: "Use meson. If you can't for some reason, consider CMake, but come talk to us first."
Meson's own module instability and lack of extensibility make it unsuitable for a wide range of projects, especially complex ones. The lack of stability in Meson itself is so bad that Meson upgrades break GNOME, libvirt, and others. And the lack of extensibility is an anti-feature. It means that Meson cannot scale to the infinite world of project needs because everything has to be bent around it or hacks need to be written in the projects to work around its weaknesses.
No way would I personally recommend it. I'm not going to go as far as to recommend one explicitly over the other from a distribution perspective, but personally I would never choose Meson anymore.
-- 真実はいつも一つ！/ Always, there's only one truth!
On Sat, Mar 30, 2024 at 11:53:07AM -0400, Neal Gompa wrote:
On Sat, Mar 30, 2024 at 11:36 AM Zbigniew Jędrzejewski-Szmek zbyszek@in.waw.pl wrote:
On Sat, Mar 30, 2024 at 10:02:42AM -0500, Michael Catanzaro wrote:
On Sat, Mar 30 2024 at 02:55:21 PM +00:00:00, Zbigniew Jędrzejewski-Szmek zbyszek@in.waw.pl wrote:
CMake for many years fought against pkgconf and pushed people towards copying those scripts into sources. It is still very common for projects using CMake to come with a whole directory of badly written detection scripts that each replace a single-line pkgconf invocation.
And of course nobody has time to look into those scripts, making it easy to smuggle something through there.
It's still better than Autotools, though. If a project doesn't want to switch to Meson for whatever reason, then CMake is a reasonable alternative.
I agree that CMake is not as good as Meson, and that CMake find modules are inferior to pkg-config.
But then we shouldn't describe them as equivalent alternatives ;) If we say "switch to a modern build systemd, e.g. cmake or meson", people will randomly choose on or the other and since the whole CMake ecosystem is built around find modules, we'll end with a bazillion of those.
Instead we should say: "Use meson. If you can't for some reason, consider CMake, but come talk to us first."
Meson's own module instability and lack of extensibility make it unsuitable for a wide range of projects, especially complex ones. The lack of stability in Meson itself is so bad that Meson upgrades break GNOME, libvirt, and others. And the lack of extensibility is an anti-feature. It means that Meson cannot scale to the infinite world of project needs because everything has to be bent around it or hacks need to be written in the projects to work around its weaknesses.
No way would I personally recommend it. I'm not going to go as far as to recommend one explicitly over the other from a distribution perspective, but personally I would never choose Meson anymore.
Also we haven't found the meson lead developer to be very open to fixing an obvious bug that affects Fedora/RISC-V, even after providing and carefully testing a patch for it. It wasn't encouraging.
Rich.
-- 真実はいつも一つ！/ Always, there's only one truth! -- _______________________________________________ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Zbigniew Jędrzejewski-Szmek wrote:
In fact, we should probably make the effort to add pkgconf files for the few libraries that don't have it to make it completely standard and expected.
That is a particularly bad idea. Downstream .pc files are a nuisance. If someone develops upstream software on Fedora, they will end up relying on those .pc files, thinking they are a supported way to link that library, then only after releasing the code, finding out that those .pc files do not exist upstream or in any other distribution (or worse, other distributions may have their own, incompatible, downstream .pc file for the same library).
I have already been in that situation as a developer. It just sucked.
If a library does not support pkg-config upstream, you MUST NOT use pkg- config to find it. It is not portable. So shipping such a downstream .pc file advertises a false interface to developers.
Kevin Kofler
On Sat, Mar 30, 2024 at 07:28:43PM +0100, Kevin Kofler via devel wrote:
Zbigniew Jędrzejewski-Szmek wrote:
In fact, we should probably make the effort to add pkgconf files for the few libraries that don't have it to make it completely standard and expected.
That is a particularly bad idea. Downstream .pc files are a nuisance.
I agree, I never suggested doing this downstream. This kind of thing needs to happen upstream so that other upstreams can start depending on it.
A pkgconf file is very simple to add, it's the kind of a thing that can be provided as a pull request by an external contributor, in particular a downstream maintainer.
Zbyszek
Zbigniew Jędrzejewski-Szmek wrote:
CMake for many years fought against pkgconf and pushed people towards copying those scripts into sources. It is still very common for projects using CMake to come with a whole directory of badly written detection scripts that each replace a single-line pkgconf invocation.
Find*.cmake scripts for many common libraries are actually shipped with CMake itself. More and more libraries even ship their own *Config.cmake which makes a Find*.cmake script entirely unnecessary.
For less common libraries, you are probably better off simply using the CMake pkg-config integration, which exists. Assuming of course that that library ships a .pc file to begin with, otherwise, pkg-config is not going to help you. (And when I write "pkg-config", I mean either the original pkg- config or pkgconf, the build system should not need to care about the difference.)
The reason CMake upstream recommends against using pkg-config directly, and instead to use it at most as a source of hints for CMake's find_* commands (e.g., find_library) in a Find*.cmake script, is that using pkg-config on Windows is often frowned upon. So typically, the way it works is that someone first writes a Find*.cmake that finds the library in the standard paths using find_library, then extends it to invoke pkg-config using the CMake pkg-config integration under "if (UNIX)" and to add the result as a path hint to the already written find_library call.
And of course nobody has time to look into those scripts, making it easy to smuggle something through there.
You are right that bundled Find*.cmake scripts are a problem.
Kevin Kofler
Tim Landscheidt writes:
A major factor seems to have been the discrepancy between "the source code" at GitHub & Co. that was probably scrutinized by many eyes and the shipped, but different artifact. So one step (as a inter-distribution effort) could be to continuously automatically compare shipped artifacts with their "make dist" equivalents and publishing the results.
There are several versions of autoconf, automake, and libtool in wide use.
There are also many libraries that ship autotools macros for inclusion in code that builds against those libraries. Those macros don't change very often, but they do change.
You will need to match the version for everything in the build environment in order to meaningfully compare autotools-generated shellcode.
On Sat, 2024-03-30 at 12:53 +0100, Kevin Kofler via devel wrote:
I think the issue is that there is just too much stuff in critpath these days. Whole desktop environments and all their transitive dependencies probably ought to not be in there. If at all, I would put the display manager in there, maybe the window manager, and no further.
I wrote a mail about this a while ago. The problem is really that the "critical path" concept has changed somewhat over time, and gotten a bit overloaded.
The original idea of critical path was to require special testing attention for it. Back in Ye Earlie Days, critpath packages had all kinds of special rules around them, including requiring +2 or +3 (it's a long time ago, I forget) from "proven testers" (remember those?)
*Most* of that has now gone. The only significant of critpath for manual testing in the current update policy is that critpath packages have a longer minimum wait in updates-testing (14 days vs. 7 days, at least after a certain point in the release cycle). They do not have higher karma requirements (at least, not by policy; Bodhi doesn't actually implement the policy correctly ATM, but I'm fixing that). The karma minima defined in the updates policy are currently identical at all points in the cycle for critpath and non-critpath updates. The "proven testers" concept was put on ice long ago.
The primary 'meaning' of critpath these days is that it triggers openQA testing, and critpath updates are gated on openQA testing. I set things up this way really just because it was convenient, and as is the way of things, now it's kinda baked in.
We probably want to look at separating out the concepts a bit. It's certainly technically possible, it just requires some work. The 'releng.py' script that "generates the critical path" is really just a comps-informed depsolver that spits out JSON. It could generate all kinds of groups besides "critical path" groups. We'd just have to wire them up to *mean*...whatever we want them to mean.
On Sat, Mar 30, 2024 at 9:38 AM Richard W.M. Jones rjones@redhat.com wrote:
I'm not pretending these will solve everything, but they should make attacks a little harder in future.
Thanks for starting the discussion.
A well resourced supply chain attack is probably not preventable (no matter how many eyes are looking). That does not mean we should not try to minimize the likelihood of future such attacks.
(3) We should have a "security path", like "critical path".
.....
Should we have a higher level of attention to these packages? We already have "critical path", but that's a broad category now. These seem like they are "security path" packages, an intentionally small subset associated with very secure services which are enabled by default.
Obligatory xkcd:
What I do think we should start with is look at the list of dependencies in the list of whatever we can agree are security critical packages (running as root and opening network ports is always a good start) and dependencies which are not supported by a large-ish organization (even if only informal), with a set of experienced developers, and sufficiently funded to continue support of the package, and has a good security reporting and response process in place.
xz would not seem to meet that vague hand waving criteria, and so it should either be replaced with something else (if possible) or get it (or in this case, likely its new team) resourced to its level of importance in the ecosystem.
I expect, with a critical eye, other such projects will be identified.
The response to Heartbleed was (among other things), resourcing OpenSSL. If a decision is made that xz needs to stay as part of the critical chain, it needs to be resourced too (although, as others have suggested, removing xz from that chain may be a better choice).
Gary Buhrmaster wrote:
What I do think we should start with is look at the list of dependencies in the list of whatever we can agree are security critical packages (running as root and opening network ports is always a good start) and dependencies which are not supported by a large-ish organization (even if only informal), with a set of experienced developers, and sufficiently funded to continue support of the package, and has a good security reporting and response process in place.
What if, as in the case of SELinux, said "large-ish organization" is exactly the kind of organization one would expect to plant a backdoor like this?
Also, a "large-ish organization" can be secretly contacted by the intelligence agencies of the country it resides in and tasked to implement secret backdoors for them. It has happened with large proprietary software providers, so why could it not happen with a large organization developing Free Software?
Projects done by a "large-ish organization" are NOT immune to this kind of attack. It would just be executed differently, not as a hostile takeover by one "motivated new maintainer" as for an individual hobbyist project like xz.
Kevin Kofler
Dne 30. 03. 24 v 10:37 dop. Richard W.M. Jones napsal(a):
I'm not pretending these will solve everything, but they should make attacks a little harder in future.
4) Fetch build artifacts before executing tests
https://github.com/rpm-software-management/mock/issues/1352
(3) We should have a "security path", like "critical path".
Generally good idea. But several packages that JiaT75 GH-starred were:
* doxygen - when you infect this, you have open path to 700 Fedora packages, including gcc.
* squashfs-tools - when you infect this, you have open path to all images (just example, not sure if our toolchain use this or -ng version).
So the security patch should be much wider.
I like the idea of the security path as well, where all packages in that path have upstream subject to higher security standards (that means helping them to achieve it as well), and greater defense downstream in any way possible.
Two things that came to mind I shared in another channel: * no binary blobs in the upstream, or no blob referred to in the source built, or referred to in the build tools * diffoscope should show no difference except file stats between the tar.gz being pulled by the spec, and the source brought with a git clone.
Both things could be automated with tools.
On 3/30/24 08:58, Miroslav Suchý wrote:
Dne 30. 03. 24 v 10:37 dop. Richard W.M. Jones napsal(a):
I'm not pretending these will solve everything, but they should make attacks a little harder in future.
- Fetch build artifacts before executing tests
https://github.com/rpm-software-management/mock/issues/1352
(3) We should have a "security path", like "critical path".
Generally good idea. But several packages that JiaT75 GH-starred were:
- doxygen - when you infect this, you have open path to 700 Fedora
packages, including gcc.
- squashfs-tools - when you infect this, you have open path to all
images (just example, not sure if our toolchain use this or -ng version).
So the security patch should be much wider.
On Sat, Mar 30, 2024 at 09:20:28AM -0700, Carlos Rodriguez-Fernandez wrote:
I like the idea of the security path as well, where all packages in that path have upstream subject to higher security standards (that means helping them to achieve it as well), and greater defense downstream in any way possible.
Two things that came to mind I shared in another channel:
- no binary blobs in the upstream, or no blob referred to in the source
built, or referred to in the build tools
I think there's some useful points here, but this would need to be qualified and/or made more flexible to be applied.
For example, systemd repo has fuzzer case files, which are a mix of text, mojibake, and actual binary protocol samples. For example, dhcp input packets, dns packets. There are also other ~binary test files, for example corrupted journal files.
The tests are defined via meson.build, so those files are "referred to in the build tools", and would not be allowed by the above definition. But if we dropped those, we'd lose very valuable testing of the codebase.
- diffoscope should show no difference except file stats between the tar.gz
being pulled by the spec, and the source brought with a git clone.
Here, OTOH, I'd just say that the tarball generated from a git clone should be bit-identical to the tar.gz pulled in by the spec.
Both things could be automated with tools.
Zbyszek
Zbigniew Jędrzejewski-Szmek wrote:
I think there's some useful points here, but this would need to be qualified and/or made more flexible to be applied.
For example, systemd repo has fuzzer case files, which are a mix of text, mojibake, and actual binary protocol samples. For example, dhcp input packets, dns packets. There are also other ~binary test files, for example corrupted journal files.
The tests are defined via meson.build, so those files are "referred to in the build tools", and would not be allowed by the above definition. But if we dropped those, we'd lose very valuable testing of the codebase.
On the other hand, "test files" are exactly how the payload of this backdoor was disguised! So a policy that deletes all binary test files or even all test files altogether would have prevented this backdoor.
Kevin Kofler
On Sat, Mar 30, 2024 at 08:00:29PM +0100, Kevin Kofler via devel wrote:
Zbigniew Jędrzejewski-Szmek wrote:
I think there's some useful points here, but this would need to be qualified and/or made more flexible to be applied.
For example, systemd repo has fuzzer case files, which are a mix of text, mojibake, and actual binary protocol samples. For example, dhcp input packets, dns packets. There are also other ~binary test files, for example corrupted journal files.
The tests are defined via meson.build, so those files are "referred to in the build tools", and would not be allowed by the above definition. But if we dropped those, we'd lose very valuable testing of the codebase.
On the other hand, "test files" are exactly how the payload of this backdoor was disguised! So a policy that deletes all binary test files or even all test files altogether would have prevented this backdoor.
Well, if we made a rule that "binary" files are not allowed, the attacker would e.g. commit some minified bash that generates whatever output is needed. The real problem was the complex and unreadable build system that made it easy to embed _multiple_ obfuscated components for the attack.
To trust code, it needs to be reviewed. And if the code is reviewed, and the build system is sane, it's usually fairly easy to say "oh, this is a sample and the only way it's used is as input to a fuzzer binary", or "this sample is used as input to a unit test", etc.
A policy against binary files would hinder this particular implementation of the attack, but the attacker had full control of the repo, so they would be able to arrange the attack around such a policy without too much trouble.
Zbyszek
hi Zbyszek, how did you review the corrupted journal files committed in systemd? Can you know for certain that they do not contain any backdoor or anything illegal or unlicensed?
On Sun, Mar 31, 2024 at 09:07:21AM -0000, François Rigault wrote:
hi Zbyszek, how did you review the corrupted journal files committed in systemd? Can you know for certain that they do not contain any backdoor or anything illegal or unlicensed?
The licensing and legal side is easy: those files are produced by a program that we wrote (journald), so copyright and patents don't apply. In principle there could be some privileged information in those files, but it was disclosed by the person who submitted a pull request with those files, so at this point distributing this information wouldn't make further difference. And also the person submitting them accept the license which allows redistribution.
If there's a backdoor: those files are read by a program which is supposed to be resilient against broken input. We execute this program under multiple sanitizers over this input file. So we're doing pretty strong testing that the input is parsed correctly (or refused).
I wrote a bit more abou this in other part of the thread [1]. I think that while it's theoretically possible to do something malicious with bad fuzzer samples, it'd be very very do pull off.
[1] https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org/...,
Zbyszek
To echo
To trust code, it needs to be reviewed. If the code is reviewed, and the build system is sane, [..]
I deduce from your response that the binary tests committed in systemd were not reviewed neither by co-maintainers nor by downstream package maintainers.
I understand that the build system used by systemd makes it much less probable that some binary blob used in a test obfuscates something that could be used for other purposes outside the test; still, wouldn't you agree it would be a good practice to make sure everyone is able to review everything in the source code repository?
On Mon, Apr 01, 2024 at 08:46:39AM -0000, François Rigault wrote:
To echo
To trust code, it needs to be reviewed. If the code is reviewed, and the build system is sane, [..]
I deduce from your response that the binary tests committed in systemd were not reviewed neither by co-maintainers nor by downstream package maintainers.
Yes, some of those blobs are treated as opaque.
I understand that the build system used by systemd makes it much less probable that some binary blob used in a test obfuscates something that could be used for other purposes outside the test; still, wouldn't you agree it would be a good practice to make sure everyone is able to review everything in the source code repository?
It's a trade-off. We can include a useful test case (e.g. a journal file that causes journalctl to busyloop or crash), to verify that the issue was fixed and that we don't regress, or we can reject the file and forego the test.
With a reasonable build system, it's fairly easy to figure out how the file is used, and I think it's entirely reasonable to review _that_.
OTOH, figuring out what effect that file would have if (hypothetically) used as input to a different tool or whether it might embed some code which might be extracted somehow is hard. But I really think that the risk is low. Also, consider that systemd has 2500 .c and .h files with 875k lines… It's not like you can review that in a weekend.
Zbyszek
On Mon, 1 Apr 2024 at 04:47, François Rigault rigault.francois@gmail.com wrote:
To echo
To trust code, it needs to be reviewed. If the code is reviewed, and the build system is sane, [..]
I deduce from your response that the binary tests committed in systemd were not reviewed neither by co-maintainers nor by downstream package maintainers.
Are we talking about the blobs which contained the malware? Those blobs were not in systemd, they were in the compression library. You are going to see weird blobs in any compression library because compression needs to test how it does against different data types.. especially binary data because you need to see if you improved or worsened the compression with a change. The places you are going to see binary data which may not make any 'sense' are: compression, anything graphics related, sound and general archiving tools. You will find actually 'binaries' in various parts of any compilation set because you may be linking or 'embedding' it into code that will run something else.
But that is just where you are just sticking plain binary data. You can uuencode, base64 or many other things and put it into regular code in places where you might need to run some stuff. They went after ssh most likely because the main target was internet based attacks. However they could have gone a slightly different path and used dnf/apt (rpm/dpkg) as the target application and placed additional scripts to open systems up somewhere. This could be done with a nice bit of C code which usually does one thing but for what would look like legitimate reasons does something else after a certain date or code sequence. Anyone who has tried to read through an obfuscated C contest entry can see where this is going.
I understand that the build system used by systemd makes it much less probable that some binary blob used in a test obfuscates something that could be used for other purposes outside the test; still, wouldn't you agree it would be a good practice to make sure everyone is able to review everything in the source code repository? -- _______________________________________________ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Those blobs were not in systemd,
that was not my point, nevertheless putting it this way: nobody knows.
For the example about compression methods you could generate your binary using a piece of code, that can be reviewed (maybe using a fixed seed as inspired by https://git.rootprojects.org/root/xz/commit/6e636819e8f070330d835fce46289a3f... btw!). If you want to test systemd against a broken journal then can't you commit a valid journal (that can be reviewed) and some code that generates a corrupted one?
The obfuscated C code is a different problem - at least it can be reviewed/audited and the maintainer can ask to simplify it.
My point is that everything should get reviewed before merge. I would hope that, as a lesson learnt from this attack, no unreviewed "corrupted binary" exist anymore in any project, since really, nobody knows what they actually contain.
On Mon, Apr 01, 2024 at 02:23:19PM -0000, François Rigault wrote:
Those blobs were not in systemd,
that was not my point, nevertheless putting it this way: nobody knows.
For the example about compression methods you could generate your binary using a piece of code, that can be reviewed (maybe using a fixed seed as inspired by https://git.rootprojects.org/root/xz/commit/6e636819e8f070330d835fce46289a3f... btw!). If you want to test systemd against a broken journal then can't you commit a valid journal (that can be reviewed) and some code that generates a corrupted one?
The obfuscated C code is a different problem - at least it can be reviewed/audited and the maintainer can ask to simplify it.
My point is that everything should get reviewed before merge. I would hope that, as a lesson learnt from this attack, no unreviewed "corrupted binary" exist anymore in any project, since really, nobody knows what they actually contain.
Or (if production of the binary is expensive or can be fiddly), commit the binary but include a script to generate it that can be run by others to check that the included binary is legit.
Call it "Reproducible Tests" to go along with reproducible builds.
Cryptography has the same concept now, learning from the Dual EC DBRG backdoor: https://en.wikipedia.org/wiki/Nothing-up-my-sleeve_number
So "nothing-up-my-sleeve scripts" could be another moniker.
Dne 30. 03. 24 v 22:14 Zbigniew Jędrzejewski-Szmek napsal(a):
On Sat, Mar 30, 2024 at 08:00:29PM +0100, Kevin Kofler via devel wrote:
Zbigniew Jędrzejewski-Szmek wrote:
I think there's some useful points here, but this would need to be qualified and/or made more flexible to be applied.
For example, systemd repo has fuzzer case files, which are a mix of text, mojibake, and actual binary protocol samples. For example, dhcp input packets, dns packets. There are also other ~binary test files, for example corrupted journal files.
The tests are defined via meson.build, so those files are "referred to in the build tools", and would not be allowed by the above definition. But if we dropped those, we'd lose very valuable testing of the codebase.
On the other hand, "test files" are exactly how the payload of this backdoor was disguised! So a policy that deletes all binary test files or even all test files altogether would have prevented this backdoor.
Well, if we made a rule that "binary" files are not allowed
I think that we should really not talk about binary files and talk e.g. about pre-generated files instead. This chapter in guidelines is already trying to cover this:
https://docs.fedoraproject.org/en-US/packaging-guidelines/what-can-be-packag...
But maybe we should rewrite the "No inclusion of pre-built binaries or libraries" chapter above to be more generic.
Vít
, the attacker would e.g. commit some minified bash that generates whatever output is needed. The real problem was the complex and unreadable build system that made it easy to embed _multiple_ obfuscated components for the attack.
To trust code, it needs to be reviewed. And if the code is reviewed, and the build system is sane, it's usually fairly easy to say "oh, this is a sample and the only way it's used is as input to a fuzzer binary", or "this sample is used as input to a unit test", etc.
A policy against binary files would hinder this particular implementation of the attack, but the attacker had full control of the repo, so they would be able to arrange the attack around such a policy without too much trouble.
Zbyszek
devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Once upon a time, Zbigniew Jędrzejewski-Szmek zbyszek@in.waw.pl said:
Here, OTOH, I'd just say that the tarball generated from a git clone should be bit-identical to the tar.gz pulled in by the spec.
That might change with zlib-ng... expecting compressed data to match is probably not a reasonable expectation, as there are several that can generate slightly different output at the same level but with other different options (such as multiple threads).
I think it's probably more reasonable to say the compressed file should pass a test by the compressor (e.g. gzip -t) and the uncompressed data should match.
It was unreasonable before zlib-ng too [0], but zlib-ng does make it a slightly bigger issue.
[0] https://github.com/rpm-software-management/createrepo_c/pull/336
Miroslav Suchý wrote:
- Fetch build artifacts before executing tests
Or better: Do not execute tests to begin with! rm -rf test in %prep and NEVER run tests during builds. Even when the tests are all legitimate, all it does is slow down the build (e.g., compare glibc build times without and with tests) and every so often break it because the test, not the software, is broken. And a claimed "test file" is what allowed the payload to be snuck in here.
Unit tests are something for upstream developers. They should NEVER be run in a distribution build.
Kevin Kofler
Am 30.03.24 um 20:11 schrieb Kevin Kofler via devel:
Or better: Do not execute tests to begin with! rm -rf test in %prep and NEVER run tests during builds. Even when the tests are all legitimate, all it does is slow down the build (e.g., compare glibc build times without and with tests) and every so often break it because the test, not the software, is broken. And a claimed "test file" is what allowed the payload to be snuck in here.
Unit tests are something for upstream developers. They should NEVER be run in a distribution build.
As a developer I need to say: If your build also runs your tests implicitly even if you just want to make a normal build of your software, your build setup is broken. That should just not happen.
But "rm -rf test" or something like that may not be possible. Quite some projects (or just straight up languages) these days have their tests in the same file as the code they test.
Kilian Hanich
Dear Kevin,
On Sat, Mar 30, 2024 at 8:12 PM Kevin Kofler via devel < devel@lists.fedoraproject.org> wrote:
Miroslav Suchý wrote:
- Fetch build artifacts before executing tests
Or better: Do not execute tests to begin with! rm -rf test in %prep and NEVER run tests during builds. Even when the tests are all legitimate, all it does is slow down the build (e.g., compare glibc build times without and with tests) and every so often break it because the test, not the software, is broken. And a claimed "test file" is what allowed the payload to be snuck in here.
It's a terrible idea. Sorry.
Unit tests are something for upstream developers. They should NEVER be run in a distribution build.
The first thesis is completely wrong. Having, say, a 30+ downstream patches and declining to run upstream tests is the most effective way to break a gazillion use-cases.
But the fuzzing tests look quite dangerous to me here and now. No one can review a corpse of binary files :(
On Sat, Mar 30, 2024 at 08:51:03PM +0100, Dmitry Belyavskiy wrote:
Dear Kevin,
On Sat, Mar 30, 2024 at 8:12 PM Kevin Kofler via devel < devel@lists.fedoraproject.org> wrote:
Miroslav Suchý wrote:
- Fetch build artifacts before executing tests
Or better: Do not execute tests to begin with! rm -rf test in %prep and NEVER run tests during builds. Even when the tests are all legitimate, all it does is slow down the build (e.g., compare glibc build times without and with tests) and every so often break it because the test, not the software, is broken. And a claimed "test file" is what allowed the payload to be snuck in here.
It's a terrible idea. Sorry.
Yes, it's a terrible idea. Also it doesn't solve anything. Because if you can inject code into the tarball to inject code into the tests, then you can also inject code into the output binaries, but you can also inject it into configuration tests.
Our official policy is to run tests during build, because that helps to catch many silly mistakes and miscompilations. If we stop doing that, we'll close one avanue of attack, leaving a few ~equivalent avanues of attack open which share the characteristic that if one is accessible to the attacker, most likely all are.
Unit tests are something for upstream developers. They should NEVER be run in a distribution build.
The first thesis is completely wrong. Having, say, a 30+ downstream patches and declining to run upstream tests is the most effective way to break a gazillion use-cases.
But the fuzzing tests look quite dangerous to me here and now. No one can review a corpse of binary files :(
Meh, the primary purpose of such tests is to stress the code under sanitizers. Those test cases are not added randomly, they are added by upstream developers when they solve an issue. To construct an attack, somebody would need to construct a sample that does not trigger any failures with sanitizers, but actually causes a misbehaviour allowing taking control of execution. Then this control would need to be used to modify the build artifacts (because presumably doing something in the sandboxed build environment is not particularly interesting), and _then_ the attacker would need to convince the maintainers to commit the sample. This is all theoretically possible, but pretty hard to pull off.
Again, I think that the important aspect is reviewability, not some simple rule about the format of files.
Zbyszek
Once upon a time, Kevin Kofler kevin.kofler@chello.at said:
Unit tests are something for upstream developers. They should NEVER be run in a distribution build.
Upstream developers aren't testing in every Fedora release (which whatever the current compiler+library+app combo is), so unit tests should always be run when available in the Fedora environment.
On Sat, Mar 30, 2024 at 04:30:53PM -0500, Chris Adams wrote:
Once upon a time, Kevin Kofler kevin.kofler@chello.at said:
Unit tests are something for upstream developers. They should NEVER be run in a distribution build.
Upstream developers aren't testing in every Fedora release (which whatever the current compiler+library+app combo is), so unit tests should always be run when available in the Fedora environment.
Don't forget architectures. One of the reasons I use to justify getting my company's open source projects in distributions like Fedora and Debian is to surface issues when compiling against various compiler+library+app combo as you noted but as importantly, architecture specific issues.
Preserving the build artifacts so they can't be tampered with during the test run is probably worthwhile though.
Best regards,
On Sat, Mar 30, 2024 at 08:11:38PM +0100, Kevin Kofler via devel wrote:
Unit tests are something for upstream developers. They should NEVER be run in a distribution build.
Even in the few little packages I'm still responsible for, I sometimes see unit test failures. The developer ran the tests, but not on S390. Or, with a different timezone database than current in Fedora. Or etc.
On Mon, Apr 1, 2024 at 17:11:46 -0400, Matthew Miller via devel wrote: On Sat, Mar 30, 2024 at 08:11:38PM +0100, Kevin Kofler via devel wrote:
Unit tests are something for upstream developers. They should NEVER be run in a distribution build.
Even in the few little packages I'm still responsible for, I sometimes see unit test failures. The developer ran the tests, but not on S390. Or, with a different timezone database than current in Fedora. Or etc.
IMHO, there's no good way to *programmatically* protect ourselves from a malicious upstream on which we depend. If their goal is to compromise us, they will work around whatever programmatic/technical measures we happen to have in place at the time they decide to launch their attack.
Any potential defense against this sort of thing will have to be *social*, and/or *process* based. Packagers should get to know (as best as possible) their upstream maintainers and developers -- by reaching out over upstream's dev fora, by meeting up at events and conferences, etc. Packagers should hopefully be familiar with the human *and* technical situation of upstream, and have a chance to notice when things go "weird".
Just another $0.02 from the peanut gallery...
Cheers, --Gabriel
Once upon a time, Gabriel Somlo gsomlo@gmail.com said:
IMHO, there's no good way to *programmatically* protect ourselves from a malicious upstream on which we depend. If their goal is to compromise us, they will work around whatever programmatic/technical measures we happen to have in place at the time they decide to launch their attack.
Yeah. This was clearly an attack targeted at Fedora and Debian; trying to fix the specific point of entry is a losing battle, as at the end of the day, Fedora will still be taking code from upstreams and distributing it to systems far and wide. The particular use of test and autoconf files to try to hide the attack may be novel, but there are other ways it could have been done. If there's easy and minimal-impact ways to help (which I haven't really seen suggested), that's good to look at, but putting lots of effort into how tests are run or wholesale changes to configuration seem to not be all that useful.
However, it's a good trigger to review Fedora's security approach in general (like 2FA use).
Chris, The specific points of entry were evading the strength of open source: many skilled eyes. Therefore, there is value in programmatically enforcing that everything used as an input in a build must have been exposed to *normal opensource workflows*. It is a very simple principle, yet very powerful: let's flag everything that can evade review: basically, tar.gz must be equal in content to what it is browseable on the git repository, and also blobs must be flagged for additional analysis, because released tar.gz and blobs are not normally reviewed on a opensource workflow.
Those problems can be codified in a tool (which it is already done in a proof of concept to show that it is simple to do and possible), or the approach can be changed all together to something like source-git?
On 4/1/24 18:47, Chris Adams wrote:
Yeah. This was clearly an attack targeted at Fedora and Debian; trying to fix the specific point of entry is a losing battle, as at the end of the day, Fedora will still be taking code from upstreams and distributing it to systems far and wide. The particular use of test and autoconf files to try to hide the attack may be novel, but there are other ways it could have been done. If there's easy and minimal-impact ways to help (which I haven't really seen suggested), that's good to look at, but putting lots of effort into how tests are run or wholesale changes to configuration seem to not be all that useful.
However, it's a good trigger to review Fedora's security approach in general (like 2FA use).
Chris Adams wrote:
However, it's a good trigger to review Fedora's security approach in general (like 2FA use).
Using such an issue that made it through upstream 2FA and would also have made it through any 2FA enforcement in Fedora as an excuse to force 2FA on us is just pure nonsense.
Kevin Kofler
I personally would very much agree with enforcing the use of 2fa on the Fedora Account System. Maybe take that opportunity to make it a bit more user friendly? (Such as the fkinit prompt requiring the 2fa code being added at the end of your password -- to be clear I think the 2fa code should be separate)
On Tue, Apr 2, 2024 at 3:06 PM Kevin Kofler via devel < devel@lists.fedoraproject.org> wrote:
Chris Adams wrote:
However, it's a good trigger to review Fedora's security approach in general (like 2FA use).
Using such an issue that made it through upstream 2FA and would also have made it through any 2FA enforcement in Fedora as an excuse to force 2FA on us is just pure nonsense.
Kevin Kofler-- _______________________________________________ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
On Tue, Apr 2, 2024 at 3:55 PM Steve Cossette farchord@gmail.com wrote:
I personally would very much agree with enforcing the use of 2fa on the Fedora Account System. Maybe take that opportunity to make it a bit more user friendly? (Such as the fkinit prompt requiring the 2fa code being added at the end of your password -- to be clear I think the 2fa code should be separate)
On Tue, Apr 02, 2024 at 04:38:25PM -0400, Stephen Gallagher wrote:
On Tue, Apr 2, 2024 at 3:55 PM Steve Cossette farchord@gmail.com wrote:
I personally would very much agree with enforcing the use of 2fa on the Fedora Account System. Maybe take that opportunity to make it a bit more user friendly? (Such as the fkinit prompt requiring the 2fa code being added at the end of your password -- to be clear I think the 2fa code should be separate)
I agree that fixing the mismatch in prompts might be nice, but why does having 2fa seperate make things any better? I mean, it's one more return you get to hit. ;)
And... I am not sure about moving the handling of passwords to a bash script from a kinit prompt.
kevin
On Tue, Apr 2, 2024 at 7:41 PM Kevin Fenzi kevin@scrye.com wrote:
On Tue, Apr 02, 2024 at 04:38:25PM -0400, Stephen Gallagher wrote:
On Tue, Apr 2, 2024 at 3:55 PM Steve Cossette farchord@gmail.com wrote:
I personally would very much agree with enforcing the use of 2fa on the Fedora Account System. Maybe take that opportunity to make it a bit more user friendly? (Such as the fkinit prompt requiring the 2fa code being added at the end of your password -- to be clear I think the 2fa code should be separate)
I agree that fixing the mismatch in prompts might be nice, but why does having 2fa seperate make things any better? I mean, it's one more return you get to hit. ;)
And... I am not sure about moving the handling of passwords to a bash script from a kinit prompt.
The kinit is already being run inside a bash script, so if bash is compromised with a keylogger, you've already lost the game... I'm not sure how this is worse.
Yeah, it's an extra keystroke, but I think there's value in helping the user provide the input in the proper format. Right now it's confusing (particularly since the kinit prompt gives bad information that we have to warn about).
On Wed, Apr 03, 2024 at 07:27:12AM -0400, Stephen Gallagher wrote:
On Tue, Apr 2, 2024 at 7:41 PM Kevin Fenzi kevin@scrye.com wrote:
On Tue, Apr 02, 2024 at 04:38:25PM -0400, Stephen Gallagher wrote:
On Tue, Apr 2, 2024 at 3:55 PM Steve Cossette farchord@gmail.com wrote:
I personally would very much agree with enforcing the use of 2fa on the Fedora Account System. Maybe take that opportunity to make it a bit more user friendly? (Such as the fkinit prompt requiring the 2fa code being added at the end of your password -- to be clear I think the 2fa code should be separate)
I agree that fixing the mismatch in prompts might be nice, but why does having 2fa seperate make things any better? I mean, it's one more return you get to hit. ;)
And... I am not sure about moving the handling of passwords to a bash script from a kinit prompt.
The kinit is already being run inside a bash script, so if bash is compromised with a keylogger, you've already lost the game... I'm not sure how this is worse.
Well, I meant more that now $PASSWORD has your password where before kinit was the only thing you input your password into. :) So, if someone does say a 'sh -x fkinit' to look at something, their password will show up, but it's probibly fine.
Yeah, it's an extra keystroke, but I think there's value in helping the user provide the input in the proper format. Right now it's confusing (particularly since the kinit prompt gives bad information that we have to warn about).
Sure.
kevin
Matthew Miller,
Unit tests, even though in theory developer should mock dependencies to isolate their code to the maximum, in reality, it is not that clear cut. Therefore, those unit tests do serve to some extent as a validation that their code works with the system libraries and platforms present in the targeted configuration. I think it is valuable to run them during the rpm builds, and contribute upstream when they break.
On 4/1/24 14:11, Matthew Miller wrote:
On Sat, Mar 30, 2024 at 08:11:38PM +0100, Kevin Kofler via devel wrote:
Unit tests are something for upstream developers. They should NEVER be run in a distribution build.
Even in the few little packages I'm still responsible for, I sometimes see unit test failures. The developer ran the tests, but not on S390. Or, with a different timezone database than current in Fedora. Or etc.
Matthew Miller wrote:
I sometimes see unit test failures. The developer ran the tests, but not on S390.
Why would I want a test failure on such an exotic architecture to fail my build? The only reason Fedora supports that architecture at all is pressure from IBM. Basically nobody uses it.
Kevin Kofler
On Tue, Apr 02, 2024 at 08:59:32PM +0200, Kevin Kofler via devel wrote:
Matthew Miller wrote:
I sometimes see unit test failures. The developer ran the tests, but not on S390.
Why would I want a test failure on such an exotic architecture to fail my build?
The architecture is weird enough (big endian!) that it may show your code has various incorrect assumptions. We found one the other day actually:
https://sourceware.org/pipermail/binutils/2024-January/132096.html
Rich.
The only reason Fedora supports that architecture at all is pressure from IBM. Basically nobody uses it.
Kevin Kofler-- _______________________________________________ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Hi,
It was sheer luck that the exploit was discovered and major distros haven't yet included it in their stable releases. It's quite possible and plausible it could have reached RHEL, Debian, Ubuntu, SLES and other distros and it's almost reached Fedora 40.
I don't know how to talk to RedHat/IBM/FSF/Ubuntu and all the big players behind Open Source/Linux but I want to raise a very important issue.
There's near zero accountability for the tens of thousands of packages included in Linux distros, often by maintainers who have no resources, qualifications or even know any programming languages to spot the "bad" code and raise an alarm. Upstream packages are pushed into Linux distros without considerationand that's it.
That's all completely unacceptable on multiple levels. Security is a joke as a result of this considering the infamous "Jia Tan" who was almost the sole maintainer of XZ for over two years.
I propose this issue to be tackled in a centralized way by the collaboration of major distros.
There must be a website or a central authority which includes known to be good/safe/verified/vetted open source packages along with e.g. SHA256/384/512/whatever hashes of the source tarballs. In addition, the source tarballs (not their compressed versions because people may use different compressors and compression settings) and their hashes must be digitally signed or have the appropriate PGP signatures from the trusted parties.
Some parties must be assigned trust to be able to push new packages to this repository. Each push must be verified by at least two independent parties, let's say RedHat and Ubuntu or Ubuntu and Arch, it doesn't matter. The representatives of these parties must be people whose whereabouts are known to confirm who they physically are. No nicknames allowed.
This website must also have/allow a revocation mechanism for situations like this.
Now Fedora/Arch/Debian/Ubuntu/whatever distros can build packages knowing they are safe to use.
If that's the wrong place to come up with this proposal, please forward it to the people who are responsible for making such decisions. I'm not willing to dig through the dirt to understand how the Fedora project works, who is responsible for what, and what are the appropriate communication channels. If you care, you'll simply forward my message. Thanks a lot.
Best regards, Artem
On Sat, 30 Mar 2024 at 13:26, Artem S. Tashkinov via devel < devel@lists.fedoraproject.org> wrote:
I propose this issue to be tackled in a centralized way by the collaboration of major distros.
There must be a website or a central authority which includes known to be good/safe/verified/vetted open source packages along with e.g. SHA256/384/512/whatever hashes of the source tarballs. In addition, the source tarballs (not their compressed versions because people may use different compressors and compression settings) and their hashes must be digitally signed or have the appropriate PGP signatures from the trusted parties.
Some parties must be assigned trust to be able to push new packages to this repository. Each push must be verified by at least two independent parties, let's say RedHat and Ubuntu or Ubuntu and Arch, it doesn't matter. The representatives of these parties must be people whose whereabouts are known to confirm who they physically are. No nicknames allowed.
This website must also have/allow a revocation mechanism for situations like this.
Now Fedora/Arch/Debian/Ubuntu/whatever distros can build packages knowing they are safe to use.
If that's the wrong place to come up with this proposal, please forward it to the people who are responsible for making such decisions. I'm not willing to dig through the dirt to understand how the Fedora project works, who is responsible for what, and what are the appropriate communication channels. If you care, you'll simply forward my message. Thanks a lot.
There is no one who makes such decisions for any of the distros. Most of the distributions make decisions by consensus of hundreds of individuals who read a list and come to the conclusion that they are 'going to dig through the dirt' to make something happen or not. For changes like what you propose, you need groups of people to work for years to get all the agreements in place, get the various tooling adapted, and work out all the personalities involved. It will usually start with an email like this, and then various disagreements about how it will never work, and then some group of people to actually try to get something like it to work somewhere. At which point, the next round of 'well did you think about..' problems arrive and either the people are able to fix them or the idea gets shelved until later.
Best regards, Artem -- _______________________________________________ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
I'm not sure my proposal has been understood at all.
This website/authority is a sort of advisory board where each member's participation is 100% voluntary and distros are free to **ignore** it altogether.
What this website will contain is just a nice list of vetted open source packages, versions and their hashes, signed by at least two independent parties (people or orgs, doesn't matter), that's it. Who's going to populate this website, is up to people to decide.
This is just fundamentally not how Free Software works.
Fundamentally I don't understand your comment at all. The proposal of mine is not there to limit anyone's freedom, it's to provide guarantees that certain packages have been vetted (checked and verified to be malware free), and you are safe to use it.
Actually it's a huge stinking problem for a **ton** of open source users who want to install certain packages that their distros don't have. It's especially relevant for Fedora given it's a basically a precursor of RedHat and it cannot contain a ton of packages related to software patents.
As a result of it, BTW, your users blindly trust RPMFusion. A seemingly absolutely shady website with no official ties to RedHat, which guarantees neither that the packages it builds are malware free, nor that there are any actual people responsible for them. If there are RPMFusion maintainers here, I'm not here to insult you, I'm just relaying the status quo. RPMFusion does not look legit. I stopped using it over a decade ago because I simply cannot understand why I should trust it. If RedHat denies anything patent related, there's zero legal obligations for RedHat if someone starts spreading malware via it. That sucks.
Back to the topic.
Then you have to painstakingly scour the web for distros already using this package and check whether their have the same version with a hash. Then you download the package and verify the hash and pray to God the distro has at least given a cursory look to this package, so it's actually safe to install.
I guess I'm not coming from @fedora.org or @redhat.com, so my proposal is "anti-freedom".
Sorry for wasting your time. You have not even provided the very basic counter-arguments why my proposal makes no sense.
RedHat absolutely can start this initiative. You have all the means and resources, and I'm not talking about something super complex or expensive. For all I know, it could be the most basic website running on top of SQLite which costing the company $50 a month to run.
And of course, without this website, distros will continue to valiantly include upstream packages and get royally screwed and screw their poor users because a ton of your maintainers have neither the time/resources, nor qualifications to check whether the code you happily push to users is malware free.
I guess we'll have to have a few more accidents like this before someone will come up with a similar solution only not coming from me personally, because I'm a no one and just rending the air.
Sorry for intervening, Artem
On Sat, 30 Mar 2024 at 15:29, Artem S. Tashkinov via devel < devel@lists.fedoraproject.org> wrote:
I'm not sure my proposal has been understood at all.
Probably not.. proposals like this need to be thought about, reviewed and thought about. Some people who like to say NO to various things will of course voice their NO as soon as they see it.
Back to the topic.
Then you have to painstakingly scour the web for distros already using this package and check whether their have the same version with a hash. Then you download the package and verify the hash and pray to God the distro has at least given a cursory look to this package, so it's actually safe to install.
I guess I'm not coming from @fedora.org or @redhat.com, so my proposal is "anti-freedom".
No. You are taking one comment from a person who is known to yell at everyone and doesn't work for Red Hat as 'being an official comment'. Most everyone on this list does not speak for Fedora at all. This is the part of the proposal that I was trying to bring up earlier that needs work: 1. There is NO one who speaks for most distributions. There is a group of many people with many different opinions who speak for a part of a distribution. They work together as best they can, but they are all going to have differing opinions and any proposal has to understand that changing 400 people's minds (for the number of active devs in Fedora) takes time. 2. Red Hat sponsors Fedora but doesn't control what Fedora does. There are many developers who are paid by Red Hat but only work on Fedora in their spare time so Red Hat may 'influence' but can not command what they do. Debian and Arch are even further along the anarchy meter for who gets to decide what happens where. 3. Getting distros to work together is hard. People choose their distros like they choose their sports teams. When they see another distro doing something, their first reaction is to do the opposite. This is why it takes a long time to get changes done and it takes a lot of people time to make it happen.
Sorry for wasting your time. You have not even provided the very basic counter-arguments why my proposal makes no sense.
It isn't a waste of time, but I see this response with a lot of good proposals which get any criticism. You are going to get criticism, you are going to get people yelling about stupid things in any proposal you make. People don't change their minds quickly and there is a lot of meat-space circuitry to try and make sure it isn't easy. To get a proposal through, people need to be able to understand that most of the time the first thing they are going to get is NO. Some of it is because people have a lot going on in life and they need space and time to see something for what its worth. Other times they have seen a lot of empty proposals and are trying to see if the person making it is going to actually do something about it or not.
RedHat absolutely can start this initiative. You have all the means and resources, and I'm not talking about something super complex or expensive. For all I know, it could be the most basic website running on top of SQLite which costing the company $50 a month to run.
And of course, without this website, distros will continue to valiantly include upstream packages and get royally screwed and screw their poor users because a ton of your maintainers have neither the time/resources, nor qualifications to check whether the code you happily push to users is malware free.
I guess we'll have to have a few more accidents like this before someone will come up with a similar solution only not coming from me personally, because I'm a no one and just rending the air.
Sorry for intervening, Artem -- _______________________________________________ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
This approach
let's delete autoconf-generated cruft from upstream projects and regenerate it in %prep
To me sounds woefully inappropriate for the task at hand. You remove a single attack vector while completely overlooking that many of your maintainers don't have the qualifications to vet the included code even if it's autoconf-cruft free.
AFAIK, the bad code discovered in XZ was on its way to the GIT repo, could have been included in a slightly different way and if not for the wonderful discovery by the Microsoft software engineer of all people, we'd be dealing with a whole much more terrible outcome.
The source code must be vetted. In a perfect world, considering the entire source code is available, an API calls map for each release could be built/extracted, so that whenever a new version is published by the upstream, a new map for the new release could easily show at least all the new API calls that the project now uses to easily discover whether new features correspond to what the project maintainer published in their changelog. Probably another "anti-freedom" proposal.
And of course, would be great to employ all the methods of automated software verification, like static analysis, sandboxing, fuzzying, etc. I'm just dreaming.
Let's pretend this incident is entirely about autoconf stuff, and not about a software project being hijacked by hostile actors. And of course, you're absolutely sure this hasn't already happened in other widely used projects and will never happen again.
Again, sorry for intervening. I just freaked out when I realized my Linux box might have been compromised after almost believing in the persistent myth of "multiple eyes" (five?) scanning open source software for malware. It's not been the case, the entire proposal that we are discussing here is not talking about it either. I'm confused.
Regards, Artem
Hi Artem,
I disagree that the idea is not appropriate. Ensuring that the tar.gz you are getting is exactly what it is in the git repo reduces a lot of risk. So, it makes a lot of sense to get rid of the practice of distributing tar.gz with pregenerated build scripts not present in the git repo that has not been reviewed, without an audit trail implicit in the git history, and without a clear non-repudiateable authorship. Ignoring them and regenerating them is a step forward.
I bet the JT user would not have been able to put the attack in the git source code without Lasse seeing it, or other developers, or tools eventually seeing it, sooner than if it had been obfuscated in the "release" downloads (as it was).
Regarding the multiple eyes, that is not the goal, but the goal is multiple *skilled* eyes.
At the end of the day, this is not about building a bullet-proof system since that's unrealistic, but a system that makes it very hard by setting up tools and processes that mitigate the risk significantly.
Regards, Carlos.
On 3/30/24 12:43, Artem S. Tashkinov via devel wrote:
And of course, would be great to employ all the methods
Artem S. Tashkinov via devel wrote:
There must be a website or a central authority which includes known to be good/safe/verified/vetted open source packages along with e.g. SHA256/384/512/whatever hashes of the source tarballs. In addition, the source tarballs (not their compressed versions because people may use different compressors and compression settings) and their hashes must be digitally signed or have the appropriate PGP signatures from the trusted parties.
Some parties must be assigned trust to be able to push new packages to this repository. Each push must be verified by at least two independent parties, let's say RedHat and Ubuntu or Ubuntu and Arch, it doesn't matter. The representatives of these parties must be people whose whereabouts are known to confirm who they physically are. No nicknames allowed.
This is just fundamentally not how Free Software works.
Kevin Kofler
It's not how free software works, but there are some interesting projects working on (distributed, not centrally managed) code review systems that are kind of similar in spirit to what OP describes.
https://github.com/crev-dev/crev https://github.com/crev-dev/cargo-crev https://mozilla.github.io/cargo-vet/
That is, individuals and organizations can publish the results of their code audits publicly in a standardized format, tied to a package artifact, and a downstream consumer could denote which individuals and organizations they trust to perform said audits.
It's technically possible and not an entirely ridiculous idea, it's just economically challenging. How do you find enough people willing and able to audit a package (including e.g. autotools build scripts), in multiple, to the level of scrutiny that would be required to catch something like this.
On Sat, Mar 30, 2024 at 09:07:19PM -0000, Daniel Alley wrote:
It's not how free software works, but there are some interesting projects working on (distributed, not centrally managed) code review systems that are kind of similar in spirit to what OP describes.
https://github.com/crev-dev/crev https://github.com/crev-dev/cargo-crev https://mozilla.github.io/cargo-vet/
That is, individuals and organizations can publish the results of their code audits publicly in a standardized format, tied to a package artifact, and a downstream consumer could denote which individuals and organizations they trust to perform said audits.
It's technically possible and not an entirely ridiculous idea, it's just economically challenging. How do you find enough people willing and able to audit a package (including e.g. autotools build scripts), in multiple, to the level of scrutiny that would be required to catch something like this.
I think such cross-checks already happen in practice. When distro packages are updated, maintainers will do _some_ review. We can't force them to full reviews every time, and it wouldn't even make sense, but we can be certain that if anything malicious is found, notifications too the other distros will go out immediately.
Or to put in a different way: if distribution publishes a package, we implicitly know that they did the review of that package in that version to the level they consider acceptable. We if asked them to make an additional signature, it wouldn't add much.
As a maintainer, the level of review I do for updates varies a lot: for some upstreams I'll do a patch-by-patch review and maybe use diffoscope to see what changed in the internals, for some upstreams I'll scan the NEWS file, and for other upstreams I'll not even do that, if I know what is hapenning in the project and I trust the quality and integrity. If there was a "task force" auditing packages, I think it should apply some variant of that rule: focus on packages which are a) important, b) have few maintainers, c) have lots of ugly internals, d) raise suspicion for whatever other reason. This would probably yield better results then trying to apply the same level of scrutiny accross the board.
--
That said, one useful variant of the auditing proposal would be to check that different distributions actually have the same source tarballs, for the same versions of the same packages. This is something that could scripted and occasionally executed. It'd be very suspicious if it turned out that the "upstream tarball" differs in Fedora or Arch or Debian or Suse or any other distro.
Zbyszek
Dne 30. 03. 24 v 18:26 Artem S. Tashkinov via devel napsal(a):
Hi,
It was sheer luck that the exploit was discovered and major distros haven't yet included it in their stable releases. It's quite possible and plausible it could have reached RHEL, Debian, Ubuntu, SLES and other distros and it's almost reached Fedora 40.
I don't know how to talk to RedHat/IBM/FSF/Ubuntu and all the big players behind Open Source/Linux but I want to raise a very important issue.
There's near zero accountability for the tens of thousands of packages included in Linux distros, often by maintainers who have no resources, qualifications or even know any programming languages to spot the "bad" code and raise an alarm. Upstream packages are pushed into Linux distros without considerationand that's it.
That's all completely unacceptable on multiple levels. Security is a joke as a result of this considering the infamous "Jia Tan" who was almost the sole maintainer of XZ for over two years.
I propose this issue to be tackled in a centralized way by the collaboration of major distros.
If I was JT, I would applaud this proposal. This would give me an opportunity to infiltrate such powerful body and either
1) close my eyes above some of the reviewed content from the right parties or
2) have some nice proposal such as "do you think your code is correct and won't you include rather this specially crafted piece of code?"
But since I am not JT, I prefer the current decentralized approach.
Vít
There must be a website or a central authority which includes known to be good/safe/verified/vetted open source packages along with e.g. SHA256/384/512/whatever hashes of the source tarballs. In addition, the source tarballs (not their compressed versions because people may use different compressors and compression settings) and their hashes must be digitally signed or have the appropriate PGP signatures from the trusted parties.
Some parties must be assigned trust to be able to push new packages to this repository. Each push must be verified by at least two independent parties, let's say RedHat and Ubuntu or Ubuntu and Arch, it doesn't matter. The representatives of these parties must be people whose whereabouts are known to confirm who they physically are. No nicknames allowed.
This website must also have/allow a revocation mechanism for situations like this.
Now Fedora/Arch/Debian/Ubuntu/whatever distros can build packages knowing they are safe to use.
If that's the wrong place to come up with this proposal, please forward it to the people who are responsible for making such decisions. I'm not willing to dig through the dirt to understand how the Fedora project works, who is responsible for what, and what are the appropriate communication channels. If you care, you'll simply forward my message. Thanks a lot.
Best regards, Artem -- _______________________________________________ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
On 2024-03-30 02:37, Richard W.M. Jones wrote:
(3) We should have a "security path", like "critical path".
sshd is linked to a lot of libraries:
I really don't want to start a systemd thread, but... the xz, lz4, and zstd libraries are pulled in by libsystemd, merely so that sshd can call "sd_notify" (https://src.fedoraproject.org/rpms/openssh/blob/rawhide/f/openssh-7.4p1-syst...), which raises a couple of questions.
The first one that comes to mind is: Is the increased attack surface incurred by linking to these additional libraries worth the value provided by calling "sd_notify", or should that patch be dropped to improve sshd security?
The second is: Is libsystemd too large? I could very easily be misreading it, but it looks like at least some of src/libsystemd/sd-journal is used by journald, including the compression bits. Do those really belong in libsystemd? If they need to be shared components, could the journald components be split out to reduce the size of libsystemd? (That is, to avoid linking to the compression libs?)
Moving on to a broader topic:
The write up describing the back door indicates that the malicious xz library "changes the value of RSA_public_decrypt@....plt to point to its own code." So the back door has pointed one of the symbols that should point to a page mapped to OpenSSL's libcrypto.so.3 to a page mapped to liblzma.so.5, instead.
Would it be possible to audit the value of a process's symbols at runtime to look for this kind of shenanigans? Could this type of auditing be added to functional tests or rpminspect?
On 2024-03-30 13:18, Gordon Messmer wrote:
The write up describing the back door indicates that the malicious xz library "changes the value of RSA_public_decrypt@....plt to point to its own code." So the back door has pointed one of the symbols that should point to a page mapped to OpenSSL's libcrypto.so.3 to a page mapped to liblzma.so.5, instead.
Would it be possible to audit the value of a process's symbols at runtime to look for this kind of shenanigans? Could this type of auditing be added to functional tests or rpminspect?
As a proof of concept, I extended GEF a tiny bit: https://github.com/gordonmessmer/gef
Now gdb can print the GOT with the paths providing the memory section containing a function. For example, on a Debian 12 system with liblzma 5.6:
---
got RSA
GOT protection: Full RelRO | GOT functions: 463
[0x555957c780f8] RSA_set0_key@OPENSSL_3.0.0 → 0x7f7bce676c90 : /usr/lib/x86_64-linux-gnu/libcrypto.so.3 [0x555957c78218] RSA_public_decrypt@OPENSSL_3.0.0 → 0x7f7bce948510 : /usr/lib/x86_64-linux-gnu/liblzma.so.5 [0x555957c782a8] EVP_PKEY_set1_RSA@OPENSSL_3.0.0 → 0x7f7bce618f30 : /usr/lib/x86_64-linux-gnu/libcrypto.so.3 ...
---
If the full table were collected and logged in functional testing, and compared from build to build, this seems like it could detect this class of attack. RSA_public_decrypt has clearly moved from libcrypto.so.3 to liblzma.so.5.
Is this worth pursuing?
On 2024-04-01 23:59, Gordon Messmer wrote:
Now gdb can print the GOT with the paths providing the memory section containing a function. For example, on a Debian 12 system with liblzma 5.6:
Purely as trivia, and as I haven't seen it discussed elsewhere, the malware steals a different set of symbols on Fedora, where RSA_public_decrypt doesn't seem to appear in the GOT at all. On Fedora 40:
gef➤ got RSA
GOT protection: Full RelRO | GOT functions: 503
[0x556ac0b94ff8] RSA_set0_key@OPENSSL_3.0.0 → 0x7f4e95dafce0 : /usr/lib64/libcrypto.so.3.2.1 [0x556ac0b951c0] RSA_bits@OPENSSL_3.0.0 → 0x7f4e95daf0a0 : /usr/lib64/libcrypto.so.3.2.1 [0x556ac0b951e0] EVP_PKEY_set1_RSA@OPENSSL_3.0.0 → 0x7f4e960e23b0 : /usr/lib64/liblzma.so.5.6.1 [0x556ac0b95310] RSA_set0_crt_params@OPENSSL_3.0.0 → 0x7f4e95dafea0 : /usr/lib64/libcrypto.so.3.2.1 [0x556ac0b953c8] RSA_size@OPENSSL_3.0.0 → 0x7f4e95daf0b0 : /usr/lib64/libcrypto.so.3.2.1 [0x556ac0b95518] RSA_new@OPENSSL_3.0.0 → 0x7f4e95db3330 : /usr/lib64/libcrypto.so.3.2.1 [0x556ac0b95778] RSA_get0_crt_params@OPENSSL_3.0.0 → 0x7f4e95dae490 : /usr/lib64/libcrypto.so.3.2.1 [0x556ac0b95870] RSA_free@OPENSSL_3.0.0 → 0x7f4e95db2f00 : /usr/lib64/libcrypto.so.3.2.1 [0x556ac0b95b90] RSA_get0_key@OPENSSL_3.0.0 → 0x7f4e960e1ac0 : /usr/lib64/liblzma.so.5.6.1 [0x556ac0b95c00] RSA_get0_factors@OPENSSL_3.0.0 → 0x7f4e95dae470 : /usr/lib64/libcrypto.so.3.2.1 [0x556ac0b95c88] EVP_PKEY_get1_RSA@OPENSSL_3.0.0 → 0x7f4e95d59710 : /usr/lib64/libcrypto.so.3.2.1 [0x556ac0b95da0] RSA_get_ex_data@OPENSSL_3.0.0 → 0x7f4e95db3440 : /usr/lib64/libcrypto.so.3.2.1 [0x556ac0b95e50] RSA_set0_factors@OPENSSL_3.0.0 → 0x7f4e95dafdc0 : /usr/lib64/libcrypto.so.3.2.1 [0x556ac0b95f00] RSA_blinding_on@OPENSSL_3.0.0 → 0x7f4e95db17f0 : /usr/lib64/libcrypto.so.3.2.1
On Tue, Apr 02, 2024 at 12:45:18AM -0700, Gordon Messmer wrote:
On 2024-04-01 23:59, Gordon Messmer wrote:
Now gdb can print the GOT with the paths providing the memory section containing a function. For example, on a Debian 12 system with liblzma 5.6:
Purely as trivia, and as I haven't seen it discussed elsewhere, the malware steals a different set of symbols on Fedora, where RSA_public_decrypt doesn't seem to appear in the GOT at all. On Fedora 40:
gef➤ got RSA
GOT protection: Full RelRO | GOT functions: 503
[0x556ac0b94ff8] RSA_set0_key@OPENSSL_3.0.0 → 0x7f4e95dafce0 : /usr/lib64/libcrypto.so.3.2.1 [0x556ac0b951c0] RSA_bits@OPENSSL_3.0.0 → 0x7f4e95daf0a0 : /usr/lib64/libcrypto.so.3.2.1 [0x556ac0b951e0] EVP_PKEY_set1_RSA@OPENSSL_3.0.0 → 0x7f4e960e23b0 : /usr/lib64/liblzma.so.5.6.1 [0x556ac0b95310] RSA_set0_crt_params@OPENSSL_3.0.0 → 0x7f4e95dafea0 : /usr/lib64/libcrypto.so.3.2.1 [0x556ac0b953c8] RSA_size@OPENSSL_3.0.0 → 0x7f4e95daf0b0 : /usr/lib64/libcrypto.so.3.2.1 [0x556ac0b95518] RSA_new@OPENSSL_3.0.0 → 0x7f4e95db3330 : /usr/lib64/libcrypto.so.3.2.1 [0x556ac0b95778] RSA_get0_crt_params@OPENSSL_3.0.0 → 0x7f4e95dae490 : /usr/lib64/libcrypto.so.3.2.1 [0x556ac0b95870] RSA_free@OPENSSL_3.0.0 → 0x7f4e95db2f00 : /usr/lib64/libcrypto.so.3.2.1 [0x556ac0b95b90] RSA_get0_key@OPENSSL_3.0.0 → 0x7f4e960e1ac0 : /usr/lib64/liblzma.so.5.6.1 [0x556ac0b95c00] RSA_get0_factors@OPENSSL_3.0.0 → 0x7f4e95dae470 : /usr/lib64/libcrypto.so.3.2.1 [0x556ac0b95c88] EVP_PKEY_get1_RSA@OPENSSL_3.0.0 → 0x7f4e95d59710 : /usr/lib64/libcrypto.so.3.2.1 [0x556ac0b95da0] RSA_get_ex_data@OPENSSL_3.0.0 → 0x7f4e95db3440 : /usr/lib64/libcrypto.so.3.2.1 [0x556ac0b95e50] RSA_set0_factors@OPENSSL_3.0.0 → 0x7f4e95dafdc0 : /usr/lib64/libcrypto.so.3.2.1 [0x556ac0b95f00] RSA_blinding_on@OPENSSL_3.0.0 → 0x7f4e95db17f0 : /usr/lib64/libcrypto.so.3.2.1
Since no one else replied yet, this is a nice bit of analysis.
Rich.
On Tue, Apr 02, 2024 at 04:32:24PM +0100, Richard W.M. Jones wrote:
On Tue, Apr 02, 2024 at 12:45:18AM -0700, Gordon Messmer wrote:
On 2024-04-01 23:59, Gordon Messmer wrote:
Now gdb can print the GOT with the paths providing the memory section containing a function. For example, on a Debian 12 system with liblzma 5.6:
Since no one else replied yet, this is a nice bit of analysis.
+1. I put the tool on my TODO list of things to look into.
Zbyszek
On 2024-04-04 06:10, Zbigniew Jędrzejewski-Szmek wrote:
+1. I put the tool on my TODO list of things to look into.
When you get that time, I've also opened the following PR that includes a proof-of-concept test
https://src.fedoraproject.org/rpms/openssh/pull-request/73
It's sloppy at the moment. The GEF extension should probably be packaged and not bundled in the test. If this is an interesting approach it should probably be a shared test instead, and used in various other packages that commonly listed on network ports. But I think this at least illustrates what I'm suggesting.
Gordon Messmer wrote:
Purely as trivia, and as I haven't seen it discussed elsewhere, the malware steals a different set of symbols on Fedora, where RSA_public_decrypt doesn't seem to appear in the GOT at all.
This proves again that this is a very targeted attack that carefully analyzed the individual targeted distributions, the distributions whose packaging tools the build script attempts to detect were not just picked because they are known to link OpenSSH to liblzma, but also individually tested and targeted.
Kevin Kofler
On 2024-04-01 23:59, Gordon Messmer wrote:
On 2024-03-30 13:18, Gordon Messmer wrote:
The write up describing the back door indicates that the malicious xz library "changes the value of RSA_public_decrypt@....plt to point to its own code." So the back door has pointed one of the symbols that should point to a page mapped to OpenSSL's libcrypto.so.3 to a page mapped to liblzma.so.5, instead.
Would it be possible to audit the value of a process's symbols at runtime to look for this kind of shenanigans? Could this type of auditing be added to functional tests or rpminspect?
As a proof of concept, I extended GEF a tiny bit: https://github.com/gordonmessmer/gef
I spent a little more time extending GEF further, as a new "got-audit" command. The command will report an error if two or more libraries appear to export conflicting symbols. It will also report an error if a symbol in the GOT points to a shared object that doesn't appear to export that symbol. For all symbols in the GOT, it reports a mapping between the symbol and the path where that symbol is mapped.
I'll work on a functional test for the openssh package. I think the naive approach is to simply record the known-good output of the audit in a file in the test's directory, run the "got-audit" command, and compare the two files. Any difference is an error.
I haven't started on that yet, but the "port-forward" test seems fairly small and simple, so I'll try writing something similar, unless anyone has suggestions otherwise.
On Sa, 30.03.24 13:18, Gordon Messmer (gordon.messmer@gmail.com) wrote:
On 2024-03-30 02:37, Richard W.M. Jones wrote:
(3) We should have a "security path", like "critical path".
sshd is linked to a lot of libraries:
I really don't want to start a systemd thread, but... the xz, lz4, and zstd libraries are pulled in by libsystemd, merely so that sshd can call "sd_notify" (https://src.fedoraproject.org/rpms/openssh/blob/rawhide/f/openssh-7.4p1-syst...), which raises a couple of questions.
The first one that comes to mind is: Is the increased attack surface incurred by linking to these additional libraries worth the value provided by calling "sd_notify", or should that patch be dropped to improve sshd security?
As discussed elsewhere: these deps are now dlopen() in systemd git main. So they are basically gone, unless you actually use one of the API calls that need the code.
The second is: Is libsystemd too large? I could very easily be misreading it, but it looks like at least some of src/libsystemd/sd-journal is used by journald, including the compression bits. Do those really belong in libsystemd?
There are various programs that use sd-journal to read journal files. It's after all *the* place to you get your logs from these days. And journal files can be compressed. We only use one compressor when writing, but we have changed the compression library (i.e. nowadays use zstd), and need to provide read compatibility. Hence the other compression libs.
If they need to be shared components, could the journald components be split out to reduce the size of libsystemd? (That is, to avoid linking to the compression libs?)
This creates many other problems. For an explanation see the various comments here: https://github.com/systemd/systemd/issues/32028
Also, I don't think we should get hung up too much on the libsystemd thing. I know people like to hit on systemd, but the attacker could have used various other vehicle libs for their goal, too. I mean, sshd uses PAM, and that pull in variety of things through its modules, and that's just very hard to properly review even when one just focusses on the default PAM config on Fedora.
Lennart
-- Lennart Poettering, Berlin
On 2024-04-02 03:42, Lennart Poettering wrote:
Also, I don't think we should get hung up too much on the libsystemd thing. I know people like to hit on systemd,
I know, and one of the problems that results from having just a torrent of undeserved criticism is that it naturally predisposes the maintainers to reflexively disregard all criticism. To be clear: I *like* systemd. One of the things I like about it is that while the project is fairly large, the individual functions/services are modular and mostly independent of each other. But that isn't really true of libsystemd. I understand that there are some dependencies between different components of libsystemd, but sd-daemon doesn't seem to have any dependencies on sd-journal, for example.
but the attacker could have used various other vehicle libs for their goal, too. I mean, sshd uses PAM, and that pull in variety of things through its modules
PAM modules aren't a problem for the same reason that the compression libraries aren't a problem now that they're dlopen()ed.
On Sat, 2024-03-30 at 09:37 +0000, Richard W.M. Jones wrote:
I'm not pretending these will solve everything, but they should make attacks a little harder in future.
I don't disagree with Richard's list. However...more in regards to some of the grandiose ideas in later posts than Richard's list...I think we're in danger of building castles in the sky while not cleaning up the poop in our backyard, here.
Before we start in on the grand fantasies about converting the world off autotools or banning binaries in repos or centralized source depots authenticated by a committee of Top People, can we remember:
1. We *still don't have compulsory 2FA for Fedora packagers*. We *still don't have compulsory 2FA for Fedora packagers*. *WE STILL DON'T HAVE COMPULSORY 2FA FOR FEDORA PACKAGERS*.
2. Our process for vetting packagers is, let's face it, from a security perspective almost *comically* patchy. There are 140 sponsors in the packager FAS group. Any one of those people - or someone who compromises any one of those 140 accounts - can grant any other person on earth Fedora packager status. Our policy on how they should do this is https://docs.fedoraproject.org/en-US/package-maintainers/How_to_Sponsor_a_Ne... . The words "trust" and "identity" do not appear in it. There is, AFAIK, no policy or procedure by which inactive sponsors have this power removed. There is no mandatory 2FA policy for sponsors.
3. We have no mechanism to flag when J. Random Packager adds "Supplements: glibc" to their random leaf node package. As a reminder, *we are a project that allows 1,601 minimally-vetted people to deliver arbitrary code executed as root on hundreds of thousands of systems*, and this mechanism allows any one of those people to cause the package they have complete control over to be automatically pulled in as a dependency on virtually every single one of those systems.
4. Our main auth system was written years ago by someone who no longer contributes and nobody is really actively maintaining it[0].
These are just the ones that leap to my tired mind at this moment. I'm sure we can think of many more things we should probably look at before we start pontificating (or, worse, lecturing) about how things should be done upstream of us.
[0]: https://pagure.io/ipsilon/commits/master
Dne 31. 03. 24 v 10:58 dop. Adam Williamson napsal(a):
- Westill don't have compulsory 2FA for Fedora packagers. We *still
don't have compulsory 2FA for Fedora packagers*. *WE STILL DON'T HAVE COMPULSORY 2FA FOR FEDORA PACKAGERS*.
Fair enough.
Let's do something about it: https://pagure.io/fesco/issue/3186
On 31/03/2024 10:58, Adam Williamson wrote:
- We *still don't have compulsory 2FA for Fedora packagers*. We *still
don't have compulsory 2FA for Fedora packagers*. *WE STILL DON'T HAVE COMPULSORY 2FA FOR FEDORA PACKAGERS*.
This finally motivated me to enable 2fa for my Fedora acount...
An important note, the docs [0] should probably mention `fkinit -u <username>`. It's much easier than using kinit.
[0]: https://docs.fedoraproject.org/en-US/fedora-accounts/user/#pkinit
Adam Williamson wrote:
- We *still don't have compulsory 2FA for Fedora packagers*. We *still
don't have compulsory 2FA for Fedora packagers*. *WE STILL DON'T HAVE COMPULSORY 2FA FOR FEDORA PACKAGERS*.
This 2FA nonsense needs to stop! GitHub has enforced compulsory 2FA for contributors for a while, starting with "important" projects, then getting stricter and stricter. It has done absolutely nothing to stop this attack. How could it, when the backdoor was apparently introduced by the authorized maintainer? (Or if not, the attacker must have had access to their 2FA secret as well.) So, 2FA DOES NOT SOLVE THIS PROBLEM! STOP FORCING 2FA ON US! And especially DO NOT abuse this incident as an excuse to force 2FA down our throats, since 2FA DOES NOT SOLVE THIS PROBLEM. Sorry for being repetitive, but you were, too. THIS 2FA NONSENSE NEEDS TO STOP!
- Our process for vetting packagers is, let's face it, from a security
perspective almost *comically* patchy. There are 140 sponsors in the packager FAS group. Any one of those people - or someone who compromises any one of those 140 accounts - can grant any other person on earth Fedora packager status. Our policy on how they should do this is https://docs.fedoraproject.org/en-US/package-maintainers/How_to_Sponsor_a_Ne... . The words "trust" and "identity" do not appear in it. There is, AFAIK, no policy or procedure by which inactive sponsors have this power removed. There is no mandatory 2FA policy for sponsors.
We already have a manpower problem, how is removing sponsors going to improve the situation?
- We have no mechanism to flag when J. Random Packager adds
"Supplements: glibc" to their random leaf node package. As a reminder, *we are a project that allows 1,601 minimally-vetted people to deliver arbitrary code executed as root on hundreds of thousands of systems*, and this mechanism allows any one of those people to cause the package they have complete control over to be automatically pulled in as a dependency on virtually every single one of those systems.
This would get noticed pretty quickly, when that package comes up in update transactions for no reason. I believe this has never happened so far. It is just too obvious.
Kevin Kofler
On 31/03/2024 13:03, Kevin Kofler via devel wrote:
This 2FA nonsense needs to stop! GitHub has enforced compulsory 2FA for contributors for a while, starting with "important" projects, then getting stricter and stricter. It has done absolutely nothing to stop this attack. How could it, when the backdoor was apparently introduced by the authorized maintainer? (Or if not, the attacker must have had access to their 2FA secret as well.) So, 2FA DOES NOT SOLVE THIS PROBLEM! STOP FORCING 2FA ON US! And especially DO NOT abuse this incident as an excuse to force 2FA down our throats, since 2FA DOES NOT SOLVE THIS PROBLEM. Sorry for being repetitive, but you were, too. THIS 2FA NONSENSE NEEDS TO STOP!
2FA for Fedora packagers doesn't solve *this* issue, but that wasn't Adam's point. What Adam is saying is that we're in danger of focusing too much on a specific issue while we should spent our time and energy on the general security aspect of Fedora. 2FA isn't nonsense, it strengthens security by a lot. A compromised (proven)packager account can do a lot of harm and can take a while to be noticed. If this would happen to us, Fedora's reputation would tank immediately. Mint is still regarded as a insecure distro (in my circles) for things that happened before I even entered the linux scene...
Like it or not, this is 2024 and passwords are not as secure as they used to be. Yelling about it isn't going to solve anything. Meanwhile, enabling 2FA helps A LOT even if used incorrectly (e.g. storing it in the same keepassxc database).
On Sun, Mar 31, 2024 at 7:36 AM Arthur Bols arthur@bols.dev wrote:
On 31/03/2024 13:03, Kevin Kofler via devel wrote:
This 2FA nonsense needs to stop! GitHub has enforced compulsory 2FA for contributors for a while, starting with "important" projects, then getting stricter and stricter. It has done absolutely nothing to stop this attack. How could it, when the backdoor was apparently introduced by the authorized maintainer? (Or if not, the attacker must have had access to their 2FA secret as well.) So, 2FA DOES NOT SOLVE THIS PROBLEM! STOP FORCING 2FA ON US! And especially DO NOT abuse this incident as an excuse to force 2FA down our throats, since 2FA DOES NOT SOLVE THIS PROBLEM. Sorry for being repetitive, but you were, too. THIS 2FA NONSENSE NEEDS TO STOP!
2FA for Fedora packagers doesn't solve this issue, but that wasn't Adam's point. What Adam is saying is that we're in danger of focusing too much on a specific issue while we should spent our time and energy on the general security aspect of Fedora. 2FA isn't nonsense, it strengthens security by a lot. A compromised (proven)packager account can do a lot of harm and can take a while to be noticed. If this would happen to us, Fedora's reputation would tank immediately. Mint is still regarded as a insecure distro (in my circles) for things that happened before I even entered the linux scene...
Like it or not, this is 2024 and passwords are not as secure as they used to be. Yelling about it isn't going to solve anything. Meanwhile, enabling 2FA helps A LOT even if used incorrectly (e.g. storing it in the same keepassxc database).
At this point, I'm used to MFA for stuff (and I use a password manager that handles 2FA OTPs too), but the Fedora implementation of MFA is uniquely bad because we have to do a lot in the terminal, and our MFA implementation sucks for terminal usage.
If MFA is turned on:
1. The Fedora account integration in GNOME breaks 2. You need to concatenate password and OTP for getting a krb5 session ticket 3. The recovery mechanism involves GPG signed emails
The experience using 2FA for Fedora accounts is sufficiently unpleasant that I really don't want to use it.
On 31/03/2024 13:42, Neal Gompa wrote:
At this point, I'm used to MFA for stuff (and I use a password manager that handles 2FA OTPs too), but the Fedora implementation of MFA is uniquely bad because we have to do a lot in the terminal, and our MFA implementation sucks for terminal usage.
If MFA is turned on:
- The Fedora account integration in GNOME breaks
- You need to concatenate password and OTP for getting a krb5 session ticket
- The recovery mechanism involves GPG signed emails
The experience using 2FA for Fedora accounts is sufficiently unpleasant that I really don't want to use it.
Thank you, these are valid points that I was mostly unaware of.
1 and 2 (I use a mooltipass, so it's just one extra click) aren't a problem for me and hopefully I'll never need the 3rd, but I can see how this could have a big impact on other users.
On Sun, Mar 31, 2024 at 07:42:24AM -0400, Neal Gompa wrote:
At this point, I'm used to MFA for stuff (and I use a password manager that handles 2FA OTPs too), but the Fedora implementation of MFA is uniquely bad because we have to do a lot in the terminal, and our MFA implementation sucks for terminal usage.
If MFA is turned on:
- The Fedora account integration in GNOME breaks
To clarify, goa cannot get a new token for you, but once you have gotten one with your otp, it will renew it for you until it's renewal time is over.
- You need to concatenate password and OTP for getting a krb5 session ticket
Yep. I think this is being worked on...
- The recovery mechanism involves GPG signed emails
Yep. Or... you can enroll multiple otps. You only need one to be working. You can enroll more and keep backup ones.
The experience using 2FA for Fedora accounts is sufficiently unpleasant that I really don't want to use it.
:(
kevin
On Sun, 2024-03-31 at 07:42 -0400, Neal Gompa wrote:
On Sun, Mar 31, 2024 at 7:36 AM Arthur Bols arthur@bols.dev wrote:
On 31/03/2024 13:03, Kevin Kofler via devel wrote:
This 2FA nonsense needs to stop! GitHub has enforced compulsory 2FA for contributors for a while, starting with "important" projects, then getting stricter and stricter. It has done absolutely nothing to stop this attack. How could it, when the backdoor was apparently introduced by the authorized maintainer? (Or if not, the attacker must have had access to their 2FA secret as well.) So, 2FA DOES NOT SOLVE THIS PROBLEM! STOP FORCING 2FA ON US! And especially DO NOT abuse this incident as an excuse to force 2FA down our throats, since 2FA DOES NOT SOLVE THIS PROBLEM. Sorry for being repetitive, but you were, too. THIS 2FA NONSENSE NEEDS TO STOP!
2FA for Fedora packagers doesn't solve this issue, but that wasn't Adam's point. What Adam is saying is that we're in danger of focusing too much on a specific issue while we should spent our time and energy on the general security aspect of Fedora. 2FA isn't nonsense, it strengthens security by a lot. A compromised (proven)packager account can do a lot of harm and can take a while to be noticed. If this would happen to us, Fedora's reputation would tank immediately. Mint is still regarded as a insecure distro (in my circles) for things that happened before I even entered the linux scene...
Like it or not, this is 2024 and passwords are not as secure as they used to be. Yelling about it isn't going to solve anything. Meanwhile, enabling 2FA helps A LOT even if used incorrectly (e.g. storing it in the same keepassxc database).
At this point, I'm used to MFA for stuff (and I use a password manager that handles 2FA OTPs too), but the Fedora implementation of MFA is uniquely bad because we have to do a lot in the terminal, and our MFA implementation sucks for terminal usage.
If MFA is turned on:
- The Fedora account integration in GNOME breaks
- You need to concatenate password and OTP for getting a krb5 session ticket
- The recovery mechanism involves GPG signed emails
The experience using 2FA for Fedora accounts is sufficiently unpleasant that I really don't want to use it.
Copying this comment here from the FESCo ticket at Kevin's request - please follow up here, not there:
I think the points above and others made later about areas where the 2FA experience on Fedora are bad are absolutely correct and justified. I also think there is a solid argument that we should do this anyway.
I think the vulnerability associated with having packager (and provenpackager? Do we require 2FA for provenpackager yet? I wasn't actually sure, when I wrote my list post) accounts without 2FA is sufficiently horrendous that we just cannot accept it. Yes, our implementation of 2FA is suboptimal and would frustrate people. Is that worse than the consequences of letting ourselves be compromised? Imagine how brutal the response would be if it came to light that Fedora had been compromised through exploitation of single-factor authentication. It would not be kind. It would be a huge and lasting stain on our reputation. People would say, justifiably so, that it was absolutely unacceptable for us to be allowing single-factor authentication for contributors to a general-purpose operating system in 2024. It is.
We now have incontrovertible evidence that extremely sophisticated attackers are willing to mount extremely sophisticated attacks on the supply chain of which we are a significant component. We are busy Monday morning quarterbacking about extremely complex ways to try and counteract an attack of that sophistication. Meanwhile we are still leaving this huge vulnerability to much less sophisticated attacks, which has been known to be one for literal decades at this point, open. We know that much less sophisticated attackers exploit single-factor authentication for purposes as trivial as stealing cryptocurrency, and it happens frequently. I don't think we can pretend we can't connect the dots here.
On a practical level, if we just made 2FA compulsory, it would provide people with a much stronger motivation to contribute and make the experience of it better. So long as we let people not use it, that motivation does not exist. I suspect if we just did it, we'd get a more sophisticated experience with Kerberos and recovery tokens and all the rest of it much faster than we will if we don't.
Adam Williamson wrote:
Do we require 2FA for provenpackager yet?
No. I am a provenpackager and do not have 2FA enabled (nor do I want it to be).
People would say, justifiably so, that it was absolutely unacceptable for us to be allowing single-factor authentication for contributors to a general-purpose operating system in 2024. It is.
This is nonsense propaganda. Most 2FA implementations cannot even guarantee that the second factor is not stored right next to the first factor. Open standards that do not depend on commercial hardware or telecommunication operators, such as TOTP, cannot guarantee it by design. Any 2FA app that works on my PinePhone is also going to work directly on my computer, so you have no way to enforce that I use a different device for the second factor.
2FA is pointless security theater that just makes it a pain to contribute, when we are all this time talking about lowering, not rising, the barrier to entry.
Kevin Kofler
Hi Kevin,
On Sun, Mar 31, 2024, at 7:31 PM, Kevin Kofler via devel wrote:
Adam Williamson wrote:
Do we require 2FA for provenpackager yet?
No. I am a provenpackager and do not have 2FA enabled (nor do I want it to be).
People would say, justifiably so, that it was absolutely unacceptable for us to be allowing single-factor authentication for contributors to a general-purpose operating system in 2024. It is.
This is nonsense propaganda. Most 2FA implementations cannot even guarantee that the second factor is not stored right next to the first factor. Open standards that do not depend on commercial hardware or telecommunication operators, such as TOTP, cannot guarantee it by design. Any 2FA app that works on my PinePhone is also going to work directly on my computer, so you have no way to enforce that I use a different device for the second factor.
2FA is pointless security theater that just makes it a pain to contribute, when we are all this time talking about lowering, not rising, the barrier to entry.
I don't quite agree with you. Two factor authentication whether an actual second factor device or not does prevent credential stuffing which is a common attack method that is easy to perform. It is when people take databases of previously leaked passwords and try them on other accounts that belong to the same person. Since two factors are generally unique per login situation they can't be stuffed in the same way.
Of course there are many things two factor does not protect against.
Regards,
Simon
Am 31.03.24 um 21:19 schrieb Simon de Vlieger:
I don't quite agree with you. Two factor authentication whether an actual second factor device or not does prevent credential stuffing which is a common attack method that is easy to perform. It is when people take databases of previously leaked passwords and try them on other accounts that belong to the same person. Since two factors are generally unique per login situation they can't be stuffed in the same way.
Of course there are many things two factor does not protect against.
2FA in a lot of cases is just access to a different account (e.g. email or even SMS) and these normally aren't unique. Sure, there are other ways like FIDO2, but these are not necessarily used (or liked, quite frankly I know a lot of people who would loose them on a monthly basis, but still are quite smart about other stuff).
This can also lead to a pretty interesting "circle" of 2FA where for example email a is the 2FA address for email b and email b is the 2FA address for email a. If it's the only option it can also lead to a chicken and egg problem for young people who want to create e.g. their first email account. But this paragraph is besides the point.
So, sure, 2FA would prevent people from just trying out leaked passwords. But an attack like this would not be a "spray and pray" attack, but it would be a targeted one. This means that the acceptable effort from the attacker would be quite a bit higher.
2FA would prevent script kiddies and "spray and pray"-style attacks from being successful. But more? Doubtful.
Regards
Kilian Hanich
On Mon, Apr 1, 2024 at 1:10 AM Kilian Hanich via devel devel@lists.fedoraproject.org wrote:
2FA in a lot of cases is just access to a different account (e.g. email or even SMS) and these normally aren't unique. Sure, there are other ways like FIDO2, but these are not necessarily used (or liked, quite frankly I know a lot of people who would loose them on a monthly basis, but still are quite smart about other stuff).
Given that FIDO2 credentials can be stored on your mobile device (and exchanged with other devices), if those people are losing their mobile devices every month they likely have other issues (including a very expensive mobile device replacement budget) for which there is likely no viable solution.
FAS' use of TOTP 2FA is not a great solution compared to FIDO2, and there are well known attacks against TOTP 2FA, but even TOTP 2FA can reduce the doorknob rattling exploits. As TOTP 2FA generators exist for most mobile devices one will tend to have a TOTP 2FA generator with one most of the time.
To the Fedora leadership:
What is the best way to formally propose that 2FA is required for packagers after some date (I suppose we could have different dates for PPs vs others if we wanted to do that in order to get started sooner). Do we need a formal Change Proposal to be voted on by someone? It does not really seem like a FESCo issue to me, but more of a policy issue that might need to go to the Council? I have no doubt that such a proposal will be controversial with some, and all those issues should get a (re-)airing in front of those making the decision.
On Fri, 2024-04-12 at 00:09 +0000, Gary Buhrmaster wrote:
What is the best way to formally propose that 2FA is required for packagers after some date
There is already a FESCo ticket. https://pagure.io/fesco/issue/3186 / Please don't discuss there, discuss here; FESCo will vote in that ticket or a meeting when they feel it appropriate.
On Thu, Apr 11, 2024 at 05:49:27PM -0700, Adam Williamson wrote:
On Fri, 2024-04-12 at 00:09 +0000, Gary Buhrmaster wrote:
What is the best way to formally propose that 2FA is required for packagers after some date
There is already a FESCo ticket. https://pagure.io/fesco/issue/3186 / Please don't discuss there, discuss here; FESCo will vote in that ticket or a meeting when they feel it appropriate.
I was wanting to circle back and add some more info to this thread too.
So, right now as far as I know, IPA doesn't have a way to easily say 'require a otp to be enrolled if you want to be added to this group'.
We do have a script that can check current members of a group(s) for otp and nag them. This is what we do for sysadmin groups, although we haven't done it in a while.
So, if FESCo decided we wanted to enforce 2fa for provenpackagers or whatever, right now that would require some work on some scripting, which I guess would remove people without otp? But then there would still be a window when the user was added and before the script removed them. Or some way for sponsors to check otp status before sponsoring someone, but if thats manually it could be missed.
I think in any case it might be good to find all the {proven}packager members without otp and perhaps email them a note about how to set things up, etc.
kevin
What about simply blocking access to the git repos/koji/bodhi for those without 2fa?
On Fri, Apr 12, 2024 at 12:05 PM Kevin Fenzi kevin@scrye.com wrote:
On Thu, Apr 11, 2024 at 05:49:27PM -0700, Adam Williamson wrote:
On Fri, 2024-04-12 at 00:09 +0000, Gary Buhrmaster wrote:
What is the best way to formally propose that 2FA is required for packagers after some date
There is already a FESCo ticket. https://pagure.io/fesco/issue/3186 / Please don't discuss there, discuss here; FESCo will vote in that ticket or a meeting when they feel it appropriate.
I was wanting to circle back and add some more info to this thread too.
So, right now as far as I know, IPA doesn't have a way to easily say 'require a otp to be enrolled if you want to be added to this group'.
We do have a script that can check current members of a group(s) for otp and nag them. This is what we do for sysadmin groups, although we haven't done it in a while.
So, if FESCo decided we wanted to enforce 2fa for provenpackagers or whatever, right now that would require some work on some scripting, which I guess would remove people without otp? But then there would still be a window when the user was added and before the script removed them. Or some way for sponsors to check otp status before sponsoring someone, but if thats manually it could be missed.
I think in any case it might be good to find all the {proven}packager members without otp and perhaps email them a note about how to set things up, etc.
kevin
devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
On Fri, Apr 12, 2024 at 03:45:40PM -0400, Steve Cossette wrote:
What about simply blocking access to the git repos/koji/bodhi for those without 2fa?
Well, git I suppose could be a hook that checks the status of the user, but koji and bodhi don't really have any place to hook that in directly. They would have to add something in their permissions models to check for specific actions.
Denying access to koji and bodhi entirely for people without 2fa is... way too wide. bodhi updates would never get karma from users who didn't bother to set it up, people just doing scratch builds would be affected, etc.
So, sure, it's possible, but would be a lot of new code needing written.
kevin
On Fri, Apr 12, 2024 at 4:05 PM Kevin Fenzi kevin@scrye.com wrote:
So, if FESCo decided we wanted to enforce 2fa for provenpackagers or whatever, right now that would require some work on some scripting, which I guess would remove people without otp? But then there would still be a window when the user was added and before the script removed them. Or some way for sponsors to check otp status before sponsoring someone, but if thats manually it could be missed.
I think in any case it might be good to find all the {proven}packager members without otp and perhaps email them a note about how to set things up, etc.
Finding the (proven)package members without 2FA might be a useful thing to know in order to influence any decision or the implementation time frame (is it 20% or 80% of (P)Ps?).
That said, I would rather see any such follow up directed email happen after a FESCo decision about 2FA is made in order to avoid possible multiple emails (one sent soonish saying that you *could* add 2FA, should you want to, and another, should the decision be made to require 2FA, to say that you will be required to add 2FA after some date; one email is better).
That does presume that FESCo will make a decision in the near term such that any email text can be appropriately crafted.
While there will always be some window/edge cases, I think we should start with the presumption that most (proven)packagers will wish to do the right thing, and will use 2FA if that is the stated requirement. After the fact cleanup/removals as the community now does for inactive packages may not be ideal but is arguable sufficient as a first step.
On Thu, Apr 11, 2024 at 6:50 PM Adam Williamson adamwill@fedoraproject.org wrote:
On Fri, 2024-04-12 at 00:09 +0000, Gary Buhrmaster wrote:
What is the best way to formally propose that 2FA is required for packagers after some date
There is already a FESCo ticket. https://pagure.io/fesco/issue/3186 / Please don't discuss there, discuss here; FESCo will vote in that ticket or a meeting when they feel it appropriate.
If this is going to end up required in one way or another to a larger group, we should also commit to getting FAS FIDO2 enabled and GOA fixed. The current FAS MFA is less than ideal, though you do get used to it.
I was hesitant to have MFA for a while. Imagine losing a phone with tons of tokens. What a hassle to recover from that. I found it less than ideal for practical reasons.
However, I decided instead to buy two Yubikey (primary and backup), and I add the QRs to both of them with the Yubico App. I also screenshot my QRs, tar them, encrypt them with openssl and gpg, and upload them to two cloud locations also protected by MFA, and remove them from my computer. I repeat that when I add a new QR. I also have a txt together with the encrypted tars documenting the commands used to encrypt/decrypt so I remember the parameters to use. The reason I do that is to be able to load them into a new Yubikey in case I lose one.
There are alternative to Yubikeys if they are too expensive for some. I do find them a good investment in general, though. I found having Yubikeys (at least two), or other similar devices cheaper than phones, to be the most practical way to do MFA. You can even use those same Yubikeys to unlock hard drives (luks), and go passwordless for some applications.
On 4/11/24 17:09, Gary Buhrmaster wrote:
On Mon, Apr 1, 2024 at 1:10 AM Kilian Hanich via devel devel@lists.fedoraproject.org wrote:
2FA in a lot of cases is just access to a different account (e.g. email or even SMS) and these normally aren't unique. Sure, there are other ways like FIDO2, but these are not necessarily used (or liked, quite frankly I know a lot of people who would loose them on a monthly basis, but still are quite smart about other stuff).
Given that FIDO2 credentials can be stored on your mobile device (and exchanged with other devices), if those people are losing their mobile devices every month they likely have other issues (including a very expensive mobile device replacement budget) for which there is likely no viable solution.
FAS' use of TOTP 2FA is not a great solution compared to FIDO2, and there are well known attacks against TOTP 2FA, but even TOTP 2FA can reduce the doorknob rattling exploits. As TOTP 2FA generators exist for most mobile devices one will tend to have a TOTP 2FA generator with one most of the time.
To the Fedora leadership:
What is the best way to formally propose that 2FA is required for packagers after some date (I suppose we could have different dates for PPs vs others if we wanted to do that in order to get started sooner). Do we need a formal Change Proposal to be voted on by someone? It does not really seem like a FESCo issue to me, but more of a policy issue that might need to go to the Council? I have no doubt that such a proposal will be controversial with some, and all those issues should get a (re-)airing in front of those making the decision. -- _______________________________________________ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
On Thu, 2024-04-11 at 19:52 -0700, Carlos Rodriguez-Fernandez wrote:
I was hesitant to have MFA for a while. Imagine losing a phone with tons of tokens. What a hassle to recover from that. I found it less than ideal for practical reasons.
This is one reason most systems provide a sheet of one-time backup codes that you're meant to print out and keep in a safe place, for recovery from exactly that scenario.
Alternatively, if you have an old phone or tablet lying around, just install an MFA app on that and enrol it too, lock it in a cabinet, then if you ever lose your primary phone, use it to recover.
Also, these days, most authenticator apps support some kind of backup mechanism. FreeOTP lets you back up to a file (which you should, of course, keep somewhere safe and ideally encrypted). Google Authenticator can backup To The Cloud.
On Fri, Apr 12, 2024 at 09:47:04AM -0700, Adam Williamson wrote:
On Thu, 2024-04-11 at 19:52 -0700, Carlos Rodriguez-Fernandez wrote:
I was hesitant to have MFA for a while. Imagine losing a phone with tons of tokens. What a hassle to recover from that. I found it less than ideal for practical reasons.
This is one reason most systems provide a sheet of one-time backup codes that you're meant to print out and keep in a safe place, for recovery from exactly that scenario.
Alternatively, if you have an old phone or tablet lying around, just install an MFA app on that and enrol it too, lock it in a cabinet, then if you ever lose your primary phone, use it to recover.
Also, these days, most authenticator apps support some kind of backup mechanism. FreeOTP lets you back up to a file (which you should, of course, keep somewhere safe and ideally encrypted). Google Authenticator can backup To The Cloud.
yeah, I'll put in a plug for the one I use: https://github.com/beemdevelopment/Aegis
It's open source, available on f-droid and play store, can to encrypted backups, pretty active upstream, highly rated in reviews.
kevin
Yes that works too. By the time I was setting up MFA everywhere, and doing the code printing, I recall not all systems giving me that option, and finding the paper thing not very good as recovery mechanism for me, so I went with Yubikeys and my own backup-in-the-cloud mechanism. I was just sharing an option just in case it could serve someone that is hesitating by giving some ideas :).
On 4/12/24 09:47, Adam Williamson wrote:
On Thu, 2024-04-11 at 19:52 -0700, Carlos Rodriguez-Fernandez wrote:
I was hesitant to have MFA for a while. Imagine losing a phone with tons of tokens. What a hassle to recover from that. I found it less than ideal for practical reasons.
This is one reason most systems provide a sheet of one-time backup codes that you're meant to print out and keep in a safe place, for recovery from exactly that scenario.
Alternatively, if you have an old phone or tablet lying around, just install an MFA app on that and enrol it too, lock it in a cabinet, then if you ever lose your primary phone, use it to recover.
Also, these days, most authenticator apps support some kind of backup mechanism. FreeOTP lets you back up to a file (which you should, of course, keep somewhere safe and ideally encrypted). Google Authenticator can backup To The Cloud.
On Fri, 2024-04-12 at 10:10 -0700, Carlos Rodriguez-Fernandez wrote:
Yes that works too. By the time I was setting up MFA everywhere, and doing the code printing, I recall not all systems giving me that option,
Yeah, FreeOTP resisted doing backups for a long time on the basis that it wasn't "secure" enough, which may be what you're remembering. (you could still kinda back it up from a rooted phone, but it was a bit weird.) These days it has a native backup option, though.
and finding the paper thing not very good as recovery mechanism for me, so I went with Yubikeys and my own backup-in-the-cloud mechanism. I was just sharing an option just in case it could serve someone that is hesitating by giving some ideas :).
For sure, whatever works for you is always the best way :)
On Fri, Apr 12, 2024 at 09:47:04AM -0700, Adam Williamson wrote:
On Thu, 2024-04-11 at 19:52 -0700, Carlos Rodriguez-Fernandez wrote:
I was hesitant to have MFA for a while. Imagine losing a phone with tons of tokens. What a hassle to recover from that. I found it less than ideal for practical reasons.
This is one reason most systems provide a sheet of one-time backup codes that you're meant to print out and keep in a safe place, for recovery from exactly that scenario.
Alternatively, if you have an old phone or tablet lying around, just install an MFA app on that and enrol it too, lock it in a cabinet, then if you ever lose your primary phone, use it to recover.
So the problem with github is they don't allow you to have 2FA on a backup device (or rather, it *is* possible, but the process is ludicrous[1]). If you have your phone as second FA and lose it then you have to immediately fall back to the piece of paper.
[1] https://github.com/orgs/community/discussions/78027
I really hope we can avoid that mistake.
Rich.
Also, these days, most authenticator apps support some kind of backup mechanism. FreeOTP lets you back up to a file (which you should, of course, keep somewhere safe and ideally encrypted). Google Authenticator can backup To The Cloud. -- Adam Williamson (he/him/his) Fedora QA Fedora Chat: @adamwill:fedora.im | Mastodon: @adamw@fosstodon.org https://www.happyassassin.net
-- _______________________________________________ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Once upon a time, Richard W.M. Jones rjones@redhat.com said:
So the problem with github is they don't allow you to have 2FA on a backup device (or rather, it *is* possible, but the process is ludicrous[1]). If you have your phone as second FA and lose it then you have to immediately fall back to the piece of paper.
I haven't seen a site with TOTP 2FA allow multiple TOTP codes, they all just store one. It's trivial to scan the TOTP code into multiple devices (depending on the software used, you can sometimes "export" a TOTP code from one device to another by showing a QR code on the first device), so that's hardly a "ludicrous" method.
On Fri, Apr 12, 2024 at 04:50:13PM -0500, Chris Adams wrote:
Once upon a time, Richard W.M. Jones rjones@redhat.com said:
So the problem with github is they don't allow you to have 2FA on a backup device (or rather, it *is* possible, but the process is ludicrous[1]). If you have your phone as second FA and lose it then you have to immediately fall back to the piece of paper.
I haven't seen a site with TOTP 2FA allow multiple TOTP codes, they all just store one. It's trivial to scan the TOTP code into multiple devices (depending on the software used, you can sometimes "export" a TOTP code from one device to another by showing a QR code on the first device), so that's hardly a "ludicrous" method.
I sometimes think how hard it would be to explain all of this to my mother. I don't understand why 2FA needs to be so obscure and clumsy to use.
Rich.
On 4/13/24 01:44, Richard W.M. Jones wrote:
On Fri, Apr 12, 2024 at 04:50:13PM -0500, Chris Adams wrote:
Once upon a time, Richard W.M. Jones rjones@redhat.com said:
So the problem with github is they don't allow you to have 2FA on a backup device (or rather, it *is* possible, but the process is ludicrous[1]). If you have your phone as second FA and lose it then you have to immediately fall back to the piece of paper.
I haven't seen a site with TOTP 2FA allow multiple TOTP codes, they all just store one. It's trivial to scan the TOTP code into multiple devices (depending on the software used, you can sometimes "export" a TOTP code from one device to another by showing a QR code on the first device), so that's hardly a "ludicrous" method.
I sometimes think how hard it would be to explain all of this to my mother. I don't understand why 2FA needs to be so obscure and clumsy to use.
Rich.
You got a really good point there. All MFA implementations the industry has come up with are less than ideal in one way or the other.
On Sat, Apr 13, 2024 at 8:44 AM Richard W.M. Jones rjones@redhat.com wrote:
I sometimes think how hard it would be to explain all of this to my mother. I don't understand why 2FA needs to be so obscure and clumsy to use.
FIDO2 (Apple branded[0] as "passkeys") is not that hard to use, or explain. The problem is that (a) passkeys are not yet universally supported (and, in this case specifically, by FAS[1]), and (b) unlike Apple (macOS, iOS, etc.), Microsoft (Windows), and Android, where the passkey is integrated into the OS inside a protected enclave, there is no trusted integrated support in Linux without an external FIDO2 key[2][3] or using the "scan a QR code" workaround with a mobile device which does support use of passkeys.
Unless your mother is using Linux (and while Mrs. Roberts has been using Linux for a long time, most moms don't), this is likely a time limited issue as more and more sites support passkeys and from the consumer point of view it all mostly just works.
I would like to imagine that FAS' current 2FA will eventually also be reasonably easy with FIDO2/passkeys, which is why I occasionally ask about the FIDO2 support status.
[0] I don't remember if there was any official assignment of the branding, but I heard that Apple was the org that suggested the name.
[1] As I understand it, if/when some of the FAS IdP moves to keycloak, FIDO2 2FA *could* be supported. However, there is no current schedule for that move that I am aware of, and unless Fedora uses the RHBK runtime, building keycloak from source for Fedora can be a real PITA (at least last I looked at it, maybe it has gotten easier).
[2] As I understand it, the issue is the lack of the required trusted environment in generic Linux. There are software implementations that do not have the hardware enclave protections,
[3] External FIDO2 keys are also not free, although I did see a $10 Adafruit FIDO2 key, which is the cheapest I have seen.
Gary Buhrmaster wrote:
[2] As I understand it, the issue is the lack of the required trusted environment in generic Linux. There are software implementations that do not have the hardware enclave protections,
And to be honest, I do not see the problem there. I will use whatever will let me pass the Fedora security theater checks without investing in extra hardware. (This also means that if I am offered the choice, I will pick TOTP over FIDO2 any day, because TOTP does not require me to emulate a fake hardware crypto device like FIDO2 does.)
And in my view, the fact that, in those implementations, there is no Treacherous Computing hardware preventing me from doing what I want with my own private key (e.g., just copying the same key to all my devices, as I can also do with TOTP) is actually a feature, even if it goes against the "security" model.
Kevin Kofler
Am 17.04.24 um 23:34 schrieb Kevin Kofler via devel:
And in my view, the fact that, in those implementations, there is no Treacherous Computing hardware preventing me from doing what I want with my own private key (e.g., just copying the same key to all my devices, as I can also do with TOTP) is actually a feature, even if it goes against the "security" model.
The fact that you can share the keys is actually part of the design and wanted. Apple for exmaple has (or wants to) implement easy sharing of passkeys via AirDrop.
Regards
Kilian Hanich
Kilian Hanich via devel wrote:
The fact that you can share the keys is actually part of the design and wanted. Apple for exmaple has (or wants to) implement easy sharing of passkeys via AirDrop.
So the Apple Cloud can see your private key, but you cannot? Sounds like GREAT "security", LOL…
Kevin Kofler
The idea is rather to scan the same QR twice, for two yubikeys, and then screenshot it and save it securely in case you lose one yubikey and need to load it into a new one.
On Fri, Apr 12, 2024 at 2:39 PM Richard W.M. Jones rjones@redhat.com wrote:
On Fri, Apr 12, 2024 at 09:47:04AM -0700, Adam Williamson wrote:
On Thu, 2024-04-11 at 19:52 -0700, Carlos Rodriguez-Fernandez wrote:
I was hesitant to have MFA for a while. Imagine losing a phone with
tons
of tokens. What a hassle to recover from that. I found it less than ideal for practical reasons.
This is one reason most systems provide a sheet of one-time backup codes that you're meant to print out and keep in a safe place, for recovery from exactly that scenario.
Alternatively, if you have an old phone or tablet lying around, just install an MFA app on that and enrol it too, lock it in a cabinet, then if you ever lose your primary phone, use it to recover.
So the problem with github is they don't allow you to have 2FA on a backup device (or rather, it *is* possible, but the process is ludicrous[1]). If you have your phone as second FA and lose it then you have to immediately fall back to the piece of paper.
[1] https://github.com/orgs/community/discussions/78027
I really hope we can avoid that mistake.
Rich.
Also, these days, most authenticator apps support some kind of backup mechanism. FreeOTP lets you back up to a file (which you should, of course, keep somewhere safe and ideally encrypted). Google Authenticator can backup To The Cloud. -- Adam Williamson (he/him/his) Fedora QA Fedora Chat: @adamwill:fedora.im | Mastodon: @adamw@fosstodon.org https://www.happyassassin.net
-- _______________________________________________ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-leave@lists.fedoraproject.org Fedora Code of Conduct:
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives:
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam, report it:
https://pagure.io/fedora-infrastructure/new_issue
-- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com Fedora Windows cross-compiler. Compile Windows programs, test, and build Windows installers. Over 100 libraries supported. http://fedoraproject.org/wiki/MinGW -- _______________________________________________ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Adam Williamson wrote:
Also, these days, most authenticator apps support some kind of backup mechanism. FreeOTP lets you back up to a file (which you should, of course, keep somewhere safe and ideally encrypted). Google Authenticator can backup To The Cloud.
If you use Keysmith, you can just SFTP your ~/.config/org.kde.keysmith/Keysmith.conf from/to all your GNU/Linux computers including the PinePhone or equivalent, and they will all be able to generate the same TOTP keys with the same master key.
Kevin Kofler
On Sun, Mar 31, 2024 at 5:35 PM Kevin Kofler via devel devel@lists.fedoraproject.org wrote:
Adam Williamson wrote:
Do we require 2FA for provenpackager yet?
No. I am a provenpackager and do not have 2FA enabled (nor do I want it to be).
Whenever 2FA comes up, the requirement for provenpackages to have it enabled has always come out very high on the list of first steps.
I still recommend that. PP is a privilege, not a right. With privileges often comes additional responsibilities, and 2FA should absolutely be one of them.
On Sun, 31 Mar 2024, Neal Gompa wrote:
On Sun, Mar 31, 2024 at 7:36 AM Arthur Bols arthur@bols.dev wrote:
On 31/03/2024 13:03, Kevin Kofler via devel wrote:
This 2FA nonsense needs to stop! GitHub has enforced compulsory 2FA for contributors for a while, starting with "important" projects, then getting stricter and stricter. It has done absolutely nothing to stop this attack. How could it, when the backdoor was apparently introduced by the authorized maintainer? (Or if not, the attacker must have had access to their 2FA secret as well.) So, 2FA DOES NOT SOLVE THIS PROBLEM! STOP FORCING 2FA ON US! And especially DO NOT abuse this incident as an excuse to force 2FA down our throats, since 2FA DOES NOT SOLVE THIS PROBLEM. Sorry for being repetitive, but you were, too. THIS 2FA NONSENSE NEEDS TO STOP!
2FA for Fedora packagers doesn't solve this issue, but that wasn't Adam's point. What Adam is saying is that we're in danger of focusing too much on a specific issue while we should spent our time and energy on the general security aspect of Fedora. 2FA isn't nonsense, it strengthens security by a lot. A compromised (proven)packager account can do a lot of harm and can take a while to be noticed. If this would happen to us, Fedora's reputation would tank immediately. Mint is still regarded as a insecure distro (in my circles) for things that happened before I even entered the linux scene...
Like it or not, this is 2024 and passwords are not as secure as they used to be. Yelling about it isn't going to solve anything. Meanwhile, enabling 2FA helps A LOT even if used incorrectly (e.g. storing it in the same keepassxc database).
At this point, I'm used to MFA for stuff (and I use a password manager that handles 2FA OTPs too), but the Fedora implementation of MFA is uniquely bad because we have to do a lot in the terminal, and our MFA implementation sucks for terminal usage.
If MFA is turned on:
- The Fedora account integration in GNOME breaks
- You need to concatenate password and OTP for getting a krb5 session ticket
- The recovery mechanism involves GPG signed emails
The experience using 2FA for Fedora accounts is sufficiently unpleasant that I really don't want to use it.
We need to fix these problems anyway. For the first two I am working on a potential solution as a part of FreeIPA passwordless authentication support. As you know, FreeIPA supports more than just OTP method that Fedora Project is currently using. The way how that support is implemented through Kerberos makes it uniform for OTP, RADIUS, passkeys, and external IdP pre-authentication methods. Since internal bits of SSSD already implement support for all these methods in a proper way, we might reuse those to improve Fedora user experience.
On Sun, 2024-03-31 at 13:35 +0200, Arthur Bols wrote:
On 31/03/2024 13:03, Kevin Kofler via devel wrote:
This 2FA nonsense needs to stop! GitHub has enforced compulsory 2FA for contributors for a while, starting with "important" projects, then getting stricter and stricter. It has done absolutely nothing to stop this attack. How could it, when the backdoor was apparently introduced by the authorized maintainer? (Or if not, the attacker must have had access to their 2FA secret as well.) So, 2FA DOES NOT SOLVE THIS PROBLEM! STOP FORCING 2FA ON US! And especially DO NOT abuse this incident as an excuse to force 2FA down our throats, since 2FA DOES NOT SOLVE THIS PROBLEM. Sorry for being repetitive, but you were, too. THIS 2FA NONSENSE NEEDS TO STOP!
2FA for Fedora packagers doesn't solve *this* issue, but that wasn't Adam's point. What Adam is saying is that we're in danger of focusing too much on a specific issue while we should spent our time and energy on the general security aspect of Fedora. 2FA isn't nonsense, it strengthens security by a lot.
Thanks Arthur, yes, that was *exactly* the point.
I would argue there's a danger of getting too tied up in very specific technical details of this attack. Yes, it's reasonable to think of ways to mitigate those specific mechanisms, at least at the appropriate levels (arguably, most of this is really directly an issue upstream of us). But it has been - for me - persuasively argued that the specific technical details were decided based on the wider goals of the attack.
I buy the scenario where the starting point was "how can we poison the major distributions?" and everything from there derived from that starting point. xz was picked as the target because of all the specific technical stuff about systemd and ssh links which people are keen to dive deep into the details of, but *if that vector hadn't existed the attacker would just have chosen the next best one*. The specific form of the attack was then customized to the specific properties of xz, very cunningly - the whole thing about hiding the payload in binary test files. But if the attacker had chosen to attack a different project with different properties, they would have customized their attack to *that* environment with equal cunning, and it would probably look quite different.
Worrying *only* about binary blobs in source repos or the specific details of how systemd opens compression libraries feels a bit narrow, to me - and especially so when we do it down at the distribution level where it's not necessarily primarily our responsibility, and I would argue is definitely not the *lowest* hanging fruit if we take a broader view of "what should we as a project be doing to raise the difficulty of supply chain attacks?"
On Sun, Mar 31, 2024 at 09:12:30AM -0700, Adam Williamson wrote:
Thanks Arthur, yes, that was *exactly* the point.
I would argue there's a danger of getting too tied up in very specific technical details of this attack. Yes, it's reasonable to think of ways to mitigate those specific mechanisms, at least at the appropriate levels (arguably, most of this is really directly an issue upstream of us). But it has been - for me - persuasively argued that the specific technical details were decided based on the wider goals of the attack.
I buy the scenario where the starting point was "how can we poison the major distributions?" and everything from there derived from that starting point. xz was picked as the target because of all the specific technical stuff about systemd and ssh links which people are keen to dive deep into the details of, but *if that vector hadn't existed the attacker would just have chosen the next best one*. The specific form of the attack was then customized to the specific properties of xz, very cunningly - the whole thing about hiding the payload in binary test files. But if the attacker had chosen to attack a different project with different properties, they would have customized their attack to *that* environment with equal cunning, and it would probably look quite different.
I think even stepping further back from the technical details is worth considering. I think xz was picked because it had one maintainer who had personal issues and low time/desire to continue work, and they were a good target to be bullied into adding another maintainer. Are there other critical packages we ship in similar situations?
Worrying *only* about binary blobs in source repos or the specific details of how systemd opens compression libraries feels a bit narrow, to me - and especially so when we do it down at the distribution level where it's not necessarily primarily our responsibility, and I would argue is definitely not the *lowest* hanging fruit if we take a broader view of "what should we as a project be doing to raise the difficulty of supply chain attacks?"
Agreed. I think theres lots of places we should improve, and focusing on our own backyard first might be wise. But we can also work on the other parts too!
There are lots of things we _could_ do... we should discuss and consider what ones make sense in what order. We should just do something because we can. ;)
Also, a reminder... nothing is ever 'secure'. security is a process. We consider likelt threats, we try and mitigate them, and we keep doing it. Things change and we should too.
kevin
On Sun, Mar 31, 2024 at 09:39:43AM -0700, Kevin Fenzi wrote:
On Sun, Mar 31, 2024 at 09:12:30AM -0700, Adam Williamson wrote:
Thanks Arthur, yes, that was *exactly* the point.
I would argue there's a danger of getting too tied up in very specific technical details of this attack. Yes, it's reasonable to think of ways to mitigate those specific mechanisms, at least at the appropriate levels (arguably, most of this is really directly an issue upstream of us). But it has been - for me - persuasively argued that the specific technical details were decided based on the wider goals of the attack.
I buy the scenario where the starting point was "how can we poison the major distributions?" and everything from there derived from that starting point. xz was picked as the target because of all the specific technical stuff about systemd and ssh links which people are keen to dive deep into the details of, but *if that vector hadn't existed the attacker would just have chosen the next best one*. The specific form of the attack was then customized to the specific properties of xz, very cunningly - the whole thing about hiding the payload in binary test files. But if the attacker had chosen to attack a different project with different properties, they would have customized their attack to *that* environment with equal cunning, and it would probably look quite different.
I think even stepping further back from the technical details is worth considering. I think xz was picked because it had one maintainer who had personal issues and low time/desire to continue work, and they were a good target to be bullied into adding another maintainer. Are there other critical packages we ship in similar situations?
I can imagine the attacker had some sort of scoring system for projects:
- No upstream patch review process (+10) - Complicated codebase (+20) - Small group of maintainers (+5) - Single, vulnerable maintainer (+50) - Linked to sshd (+1000)
There's something to be said for us doing a similar sort of analysis.
Worrying *only* about binary blobs in source repos or the specific details of how systemd opens compression libraries feels a bit narrow, to me - and especially so when we do it down at the distribution level where it's not necessarily primarily our responsibility, and I would argue is definitely not the *lowest* hanging fruit if we take a broader view of "what should we as a project be doing to raise the difficulty of supply chain attacks?"
Agreed. I think theres lots of places we should improve, and focusing on our own backyard first might be wise. But we can also work on the other parts too!
There are lots of things we _could_ do... we should discuss and consider what ones make sense in what order. We should just do something because we can. ;)
Also, a reminder... nothing is ever 'secure'. security is a process. We consider likelt threats, we try and mitigate them, and we keep doing it. Things change and we should too.
kevin
Rich.
Adam Williamson wrote:
I would argue there's a danger of getting too tied up in very specific technical details of this attack.
But the fact is:
What WOULD have stopped this attack: (one or more of:) * Deleting ALL unit tests in %prep (and then of course not trying to run them later). * Deleting ALL files automatically generated or imported by autotools in %prep, THEN running "autoreconf -i -f". (DO NOT trust autoreconf, it would NOT have done the right thing here. Delete the files, THEN run autoreconf.) * NOT patching OpenSSH downstream to link it against libsystemd against upstream's recommendation.
What WOULD have greatly reduced the impact of this attack: * NOT enabling updates-testing by default for Branched releases.
What WOULD NOT have stopped this attack: (any or all of:) * 2FA. GitHub already enforces 2FA. It did NOT stop this attack. * Any stricter vetting of Fedora contributions. The attack was performed upstream, NOT in Fedora. * More distrust of new Fedora contributors. The offending upgrade was imported by a TRUSTED Fedora contributor. The untrusted new person operated upstream, NOT in Fedora.
Kevin Kofler
On 3/31/24 2:12 PM, Kevin Kofler via devel wrote:
But the fact is:
What WOULD have stopped this attack: (one or more of:)
- Deleting ALL unit tests in %prep (and then of course not trying to run
them later).
While it’s technically correct that deleting tests would have disrupted this specific attack, a policy of deleting and and never running upstream test code would have prevented me from finding and helping upstreams fix dozens and dozens of bugs due to accidentally faulty assumptions that turned out to be violated on different architectures, in different system environments, or with various allegedly-compatible dependency versions. There are even GCC bugs (miscompilations, not only failures to compile) that were discovered and fixed only because packages I maintain were running upstream unit and integration tests. Frankly, “testing the packages we ship, as built in our distribution, is actually bad” seems like a pretty strange and extreme conclusion to draw from all of this.
On Sun, Mar 31, 2024 at 04:09:36PM -0400, Ben Beasley wrote:
On 3/31/24 2:12 PM, Kevin Kofler via devel wrote:
But the fact is:
What WOULD have stopped this attack: (one or more of:)
- Deleting ALL unit tests in %prep (and then of course not trying to run
them later).
While it’s technically correct that deleting tests would have disrupted this specific attack, a policy of deleting and and never running upstream test code would have prevented me from finding and helping upstreams fix dozens and dozens of bugs due to accidentally faulty assumptions that turned out to be violated on different architectures, in different system environments, or with various allegedly-compatible dependency versions. There are even GCC bugs (miscompilations, not only failures to compile) that were discovered and fixed only because packages I maintain were running upstream unit and integration tests. Frankly, “testing the packages we ship, as built in our distribution, is actually bad” seems like a pretty strange and extreme conclusion to draw from all of this.
Deleting the tests makes no sense to me either, but it seems like a mechanism that ensures the test code can't change the build outputs (or a mechanism to detect that it's happened and abort the build) would allow upstream tests to be run without compromising the integrity of the build itself.
Scott Schmit wrote on Sun, Mar 31, 2024 at 05:02:44PM -0400:
Deleting the tests makes no sense to me either, but it seems like a mechanism that ensures the test code can't change the build outputs (or a mechanism to detect that it's happened and abort the build) would allow upstream tests to be run without compromising the integrity of the build itself.
Just to be clear here that wouldn't have been enough: it's not the test step that's modifying the binaries, the actual build step is modified in the right conditions to use data that looks like it belongs to a test (I've read the actual files aren't actually used in any test and just look like test data, I didn't check, it wouldn't be hard to make a test that uses them anyway)
So short of deleting all blobs e.g. all test data this wouldn't have been prevented, just not running tests isn't enough.
In theory it'd be possible to build twice: - one normal build with test data, and run tests at the end - a second build without test data (and confirm we've got an identical binary, builds are reproducible right?!)
But while we might be able to afford the computing cost, I'm not sure it's worth it -- this attack vector happened to use test data, but there are plenty of other ways of doing this, and even just identifying / removing test data in the first place is hard work for packagers (I guess we could try to detect binary files but there is no end to that either, and many builds would just break if we were to automatically remove tests...)
(Anyway, I also think tests bring more benefits than risks in our case)
On Mon, Apr 01, 2024 at 09:06:16AM +0900, Dominique Martinet wrote:
Scott Schmit wrote on Sun, Mar 31, 2024 at 05:02:44PM -0400:
Deleting the tests makes no sense to me either, but it seems like a mechanism that ensures the test code can't change the build outputs (or a mechanism to detect that it's happened and abort the build) would allow upstream tests to be run without compromising the integrity of the build itself.
Just to be clear here that wouldn't have been enough: it's not the test step that's modifying the binaries, the actual build step is modified in the right conditions to use data that looks like it belongs to a test (I've read the actual files aren't actually used in any test and just look like test data, I didn't check, it wouldn't be hard to make a test that uses them anyway)
So short of deleting all blobs e.g. all test data this wouldn't have been prevented, just not running tests isn't enough.
Ugh! I'd missed that detail.
In theory it'd be possible to build twice:
- one normal build with test data, and run tests at the end
- a second build without test data (and confirm we've got an identical
binary, builds are reproducible right?!)
FWIW, I don't think you'd need to build twice:
One approach: 1. do the build 2. do the install 3. generate the RPMs 4. quarantine the RPMs so they're safe from modification - I believe this could be done via SELinux policy - there are probably other mechanisms 5. run the tests - for SELinux, this might be via an `rpmbuild-test` binary that doesn't have rights to touch the output RPMs 6a. if the tests fail, destroy the RPMs and fail out, reproducing the result today 6b. if the tests pass, move/copy the RPMs to the result location and exit cleanly, reproducing the result today
A variation of this would order things as normal but capture hashes of all the files before the tests start and check the hashes after they're done. As long as the hashes can't be changed in between, you can trust that.
Another approach (perhaps more doable in mock, not so easy otherwise): 1. do the build as user1 2. do the tests as user2 3. if the tests pass, resume as user1 as today
Given that the build pollution came from loading data out of the tests, the above might still be useful, but it's not sufficient to prevent a recurrence. (Though if you can identify the test files, you could still segregate them via similar means -- make them unreadable until you're ready to run the tests.)
But while we might be able to afford the computing cost, I'm not sure it's worth it -- this attack vector happened to use test data, but there are plenty of other ways of doing this, and even just identifying / removing test data in the first place is hard work for packagers (I guess we could try to detect binary files but there is no end to that either, and many builds would just break if we were to automatically remove tests...)
Another idea I've seen floated https://chaos.social/@ollibaba/112189619904512612 is to modify the build tools (compilers, etc) to generate SBOM files that log every input file used to produce a given output. Then you have a record to audit to see if stuff is pulled from weird places. That said, that wouldn't help either, because all the machinations were via shell commands that aren't, strictly speaking, build tools. (Except if the linker shows a file as input that was never generated by a compiler…)
But we could adapt that idea -- couldn't we use strace(1)/ptrace(2) to detect open calls to determine the inputs and outputs? And we could protect the resulting records from build/test tampering similarly to the above.
And if the upstream makes it difficult to tell what's test vs not…I think at this point we have this incident as precedent to help explain why we'd prefer less convoluted builds, or at least some explanations.
I dunno how workable that is in practice, though. Though a part of me says that it all boils down to making the build process more transparent than we've previously put up with.
Anyway, just putting that out there in case anyone thinks these are good ideas (or brainstorming toward better ones).
(Anyway, I also think tests bring more benefits than risks in our case)
And it also occurs to me that test failures could just as easily help find a malicious change if some new feature we're adding to Fedora breaks the tests unexpectedly and exposes it (or just something unique about how we've configured things).
+1
Am 01.04.24 um 06:31 schrieb Scott Schmit:
One approach:
- do the build
- do the install
- generate the RPMs
- quarantine the RPMs so they're safe from modification
- I believe this could be done via SELinux policy
- there are probably other mechanisms
- run the tests
- for SELinux, this might be via an `rpmbuild-test` binary that doesn't have rights to touch the output RPMs
6a. if the tests fail, destroy the RPMs and fail out, reproducing the result today 6b. if the tests pass, move/copy the RPMs to the result location and exit cleanly, reproducing the result today
Boils down to separate source and test code/phase
source code: (hopefully not obfuscated to the point where no review is possible) no binaries allowed, best possible review needed to build build phase: source to binary
test code: binaries allowed only needed to test test phase: binary unmodified
Allowing a test file to modify the binary makes it a source file. ?
Christoph
On Mon, Apr 01, 2024 at 09:06:16AM +0900, Dominique Martinet wrote:
Scott Schmit wrote on Sun, Mar 31, 2024 at 05:02:44PM -0400:
Deleting the tests makes no sense to me either, but it seems like a mechanism that ensures the test code can't change the build outputs (or a mechanism to detect that it's happened and abort the build) would allow upstream tests to be run without compromising the integrity of the build itself.
Just to be clear here that wouldn't have been enough: it's not the test step that's modifying the binaries, the actual build step is modified in the right conditions to use data that looks like it belongs to a test (I've read the actual files aren't actually used in any test and just look like test data, I didn't check, it wouldn't be hard to make a test that uses them anyway)
So short of deleting all blobs e.g. all test data this wouldn't have been prevented, just not running tests isn't enough.
Yep.
And since we're talking about xz, note that a second malicious issue has beend found: [1] is a revert of [2] which sabotages CMakeLists.txt to always disable Landlock sandbox.
Clearly, the only reasonable solution is to delete all the CMake cruft ;)
[1] https://git.tukaani.org/?p=xz.git;a=commitdiff;h=f9cf4c05edd14dedfe63833f8cc... [2] https://git.tukaani.org/?p=xz.git;a=commitdiff;h=328c52da8a2bbb81307644efdb5...
Zbyszek
Am 31.03.24 um 23:02 schrieb Scott Schmit:
On Sun, Mar 31, 2024 at 04:09:36PM -0400, Ben Beasley wrote:
On 3/31/24 2:12 PM, Kevin Kofler via devel wrote:
But the fact is:
What WOULD have stopped this attack: (one or more of:)
- Deleting ALL unit tests in %prep (and then of course not trying to run
them later).
While it’s technically correct that deleting tests would have disrupted this specific attack, a policy of deleting and and never running upstream test code would have prevented me from finding and helping upstreams fix dozens and dozens of bugs due to accidentally faulty assumptions that turned out to be violated on different architectures, in different system environments, or with various allegedly-compatible dependency versions. There are even GCC bugs (miscompilations, not only failures to compile) that were discovered and fixed only because packages I maintain were running upstream unit and integration tests. Frankly, “testing the packages we ship, as built in our distribution, is actually bad” seems like a pretty strange and extreme conclusion to draw from all of this.
Deleting the tests makes no sense to me either, but it seems like a mechanism that ensures the test code can't change the build outputs (or a mechanism to detect that it's happened and abort the build) would allow upstream tests to be run without compromising the integrity of the build itself.
It would prevent the attack from being hidden in the tests, but not in any other part of the build.
Also, I have seen build setups which encode the status of tests in the eventual binary and as such info page or integrated bug report generators. Often because some distros sometimes turned them off or ships software even with failed tests and thus pushing that headache to upstream.
Regards
Kilian Hanich
Dne 01. 04. 24 v 3:16 dop. Kilian Hanich via devel napsal(a):
Also, I have seen build setups which encode the status of tests in the eventual binary and as such info page or integrated bug report generators. Often because some distros sometimes turned them off or ships software even with failed tests and thus pushing that headache to upstream.
Can you point me to such case, please? Just being curious.
On Sun, 2024-03-31 at 20:12 +0200, Kevin Kofler via devel wrote:
Adam Williamson wrote:
I would argue there's a danger of getting too tied up in very specific technical details of this attack.
But the fact is:
What WOULD have stopped this attack: (one or more of:)
- Deleting ALL unit tests in %prep (and then of course not trying to run
them later).
This comes with the huge cost that we no longer have any idea if many things actually work in Fedora. There are much better options here, such as ensuring the package build process *isolates* the test data from the compile process without just *removing* it. but again, this is getting far too tied up in the details of this *one* attack. it's the "everyone has to take their shoes off at the airport forever" mistake: just because the *last* attack involved a binary test file, does not mean the *next* one will. This is also known as preparing for the last war, in military strategy.
- Deleting ALL files automatically generated or imported by autotools in
%prep, THEN running "autoreconf -i -f". (DO NOT trust autoreconf, it would NOT have done the right thing here. Delete the files, THEN run autoreconf.)
No. This would not have avoided the attack, because it would not have regenerated the malicious file. We have already established that.
- NOT patching OpenSSH downstream to link it against libsystemd against
upstream's recommendation.
Again, this is far too tied to the specific attack. I'm not interested, as I said, in Monday morning quarterbacking this specific attack. I'm interested in the question of what are the most immediate and effective ways Fedora can lower its likelihood to be directly targeted in a supply chain attack *in future*. Not the last one.
What WOULD have greatly reduced the impact of this attack:
- NOT enabling updates-testing by default for Branched releases.
This would only have helped by coincidence - the coincidence that the compromise was discovered so quickly. Otherwise, we were busy trying to get this update pushed stable as fast as possible, because rwmj wanted that. If the compromise had happened to be caught a few days later, the update would have been pushed stable anyway. Our gating and manual testing processes are not designed to catch security issues, and certainly do not reliably do that job. They did not in this case. So it seems wrong, to me, to suggest we should rely on them to do it in future. Thus I disagree that any possible security benefit gained by doing this would outweigh the substantial disadvantage that we wouldn't get testing of stuff promptly any more.
What WOULD NOT have stopped this attack: (any or all of:)
- 2FA. GitHub already enforces 2FA. It did NOT stop this attack.
Again. I'm not interested in what, with hindsight, might have stopped the attack that already happened. We know for sure that weak authentication processes absolutely are a giant flashing ATTACK HERE neon sign for *future* attackers.
- Any stricter vetting of Fedora contributions. The attack was performed
upstream, NOT in Fedora.
See above.
- More distrust of new Fedora contributors. The offending upgrade was
imported by a TRUSTED Fedora contributor. The untrusted new person operated upstream, NOT in Fedora.
See above again.
Adam,
Is there a way already to achieve test isolation during the rpm build?
Beside doing something ad-hoc with git init or some file system feature, I was wondering if there is something already available and standardized.
Regards, Carlos R.F.
On 3/31/24 20:27, Adam Williamson wrote:
On Sun, 2024-03-31 at 20:12 +0200, Kevin Kofler via devel wrote:
Adam Williamson wrote:
I would argue there's a danger of getting too tied up in very specific technical details of this attack.
But the fact is:
What WOULD have stopped this attack: (one or more of:)
- Deleting ALL unit tests in %prep (and then of course not trying to run
them later).
This comes with the huge cost that we no longer have any idea if many things actually work in Fedora. There are much better options here, such as ensuring the package build process *isolates* the test data from the compile process without just *removing* it. but again, this is getting far too tied up in the details of this *one* attack. it's the "everyone has to take their shoes off at the airport forever" mistake: just because the *last* attack involved a binary test file, does not mean the *next* one will. This is also known as preparing for the last war, in military strategy.
- Deleting ALL files automatically generated or imported by autotools in
%prep, THEN running "autoreconf -i -f". (DO NOT trust autoreconf, it would NOT have done the right thing here. Delete the files, THEN run autoreconf.)
No. This would not have avoided the attack, because it would not have regenerated the malicious file. We have already established that.
- NOT patching OpenSSH downstream to link it against libsystemd against
upstream's recommendation.
Again, this is far too tied to the specific attack. I'm not interested, as I said, in Monday morning quarterbacking this specific attack. I'm interested in the question of what are the most immediate and effective ways Fedora can lower its likelihood to be directly targeted in a supply chain attack *in future*. Not the last one.
What WOULD have greatly reduced the impact of this attack:
- NOT enabling updates-testing by default for Branched releases.
This would only have helped by coincidence - the coincidence that the compromise was discovered so quickly. Otherwise, we were busy trying to get this update pushed stable as fast as possible, because rwmj wanted that. If the compromise had happened to be caught a few days later, the update would have been pushed stable anyway. Our gating and manual testing processes are not designed to catch security issues, and certainly do not reliably do that job. They did not in this case. So it seems wrong, to me, to suggest we should rely on them to do it in future. Thus I disagree that any possible security benefit gained by doing this would outweigh the substantial disadvantage that we wouldn't get testing of stuff promptly any more.
What WOULD NOT have stopped this attack: (any or all of:)
- 2FA. GitHub already enforces 2FA. It did NOT stop this attack.
Again. I'm not interested in what, with hindsight, might have stopped the attack that already happened. We know for sure that weak authentication processes absolutely are a giant flashing ATTACK HERE neon sign for *future* attackers.
- Any stricter vetting of Fedora contributions. The attack was performed
upstream, NOT in Fedora.
See above.
- More distrust of new Fedora contributors. The offending upgrade was
imported by a TRUSTED Fedora contributor. The untrusted new person operated upstream, NOT in Fedora.
See above again.
On Sun, 2024-03-31 at 22:13 -0700, Carlos Rodriguez-Fernandez wrote:
Adam,
Is there a way already to achieve test isolation during the rpm build?
Nothing systematic that I'm aware of, no. It would be tricky because there is no one universal Standard Test System (not even within a single language ecosystem, let alone across all of them). Currently you'd have to design something unique, if you wanted to implement this for your package.
I suppose one approach would be to split the sources-required-for-build and the test suite into Source0 and Source1 (respectively) and only extract Source0 during %prep. Don't extract Source1 until %check (i.e. after %build and %install are already done). I'm just spitballing, though, haven't checked if this is really practical.
Of course, another approach is to really do what Kevin suggests and not ship the test suite in the package source at all, but instead run the tests via Fedora CI, and configure the package to be gated on that CI check (so it wouldn't go to stable without the tests passing). But that's rather a different approach (and would still require 'custom' work to cut the tests out of the source, or at least delete them before running the build).
And I still think at this point we are falling into the trap of thinking too specifically about an attack vector which just *happens* to be the one chosen in *this specific instance*. It's still worthwhile, on some level, for someone to think about that kind of hardening, of course. I am just not convinced it is the most useful thing Fedora could be doing right now in the general area of "supply chain hardening".
On Sun, Mar 31, 2024 at 11:20:17PM -0700, Adam Williamson wrote:
On Sun, 2024-03-31 at 22:13 -0700, Carlos Rodriguez-Fernandez wrote:
Adam,
Is there a way already to achieve test isolation during the rpm build?
Nothing systematic that I'm aware of, no. It would be tricky because there is no one universal Standard Test System (not even within a single language ecosystem, let alone across all of them). Currently you'd have to design something unique, if you wanted to implement this for your package.
If we wanted to pursue that, I'd suggest the following: remount $RPM_BUILD_ROOT read-only for the %check phase (or maybe overmount it with a writable overlayfs that is thrown away after %check finishes, and warn if any modifications were made.) %check is executed after %install, so everything should be in place before %check, and %check may be skipped, so no modifications to installed files should be done in %check.
Considering possible implemention details, machinectl has 'bind' and 'bind --read-only' that might be useful here. But mock uses systemd-nspawn in a way that does register the container with machined. So maybe it'd be more reasonable to just execute a mount command directly from mock.
This is independent of the test system and does not require splitting of the test sources.
Zbyszek
On Mon, Apr 1, 2024 at 7:38 AM Zbigniew Jędrzejewski-Szmek zbyszek@in.waw.pl wrote:
On Sun, Mar 31, 2024 at 11:20:17PM -0700, Adam Williamson wrote:
On Sun, 2024-03-31 at 22:13 -0700, Carlos Rodriguez-Fernandez wrote:
Adam,
Is there a way already to achieve test isolation during the rpm build?
Nothing systematic that I'm aware of, no. It would be tricky because there is no one universal Standard Test System (not even within a single language ecosystem, let alone across all of them). Currently you'd have to design something unique, if you wanted to implement this for your package.
If we wanted to pursue that, I'd suggest the following: remount $RPM_BUILD_ROOT read-only for the %check phase (or maybe overmount it with a writable overlayfs that is thrown away after %check finishes, and warn if any modifications were made.) %check is executed after %install, so everything should be in place before %check, and %check may be skipped, so no modifications to installed files should be done in %check.
Considering possible implemention details, machinectl has 'bind' and 'bind --read-only' that might be useful here. But mock uses systemd-nspawn in a way that does register the container with machined. So maybe it'd be more reasonable to just execute a mount command directly from mock.
This is independent of the test system and does not require splitting of the test sources.
Another thing to consider is making RPM unshare each build phase like it can for scriptlets now at install time[1].
Sandboxing and limiting in rpmbuild itself like we can with rpm can also help with this. I believe moss[2]' boulder tool does this.
[1]: https://github.com/rpm-software-management/rpm/pull/2666 [2]: https://github.com/serpent-os/moss/tree/main/boulder
-- 真実はいつも一つ！/ Always, there's only one truth!
I created a discussion issue for this idea:
https://github.com/rpm-software-management/rpm/discussions/3009
I think it worth pursuing further.
On 4/1/24 04:46, Neal Gompa wrote:
On Mon, Apr 1, 2024 at 7:38 AM Zbigniew Jędrzejewski-Szmek zbyszek@in.waw.pl wrote:
On Sun, Mar 31, 2024 at 11:20:17PM -0700, Adam Williamson wrote:
On Sun, 2024-03-31 at 22:13 -0700, Carlos Rodriguez-Fernandez wrote:
Adam,
Is there a way already to achieve test isolation during the rpm build?
Nothing systematic that I'm aware of, no. It would be tricky because there is no one universal Standard Test System (not even within a single language ecosystem, let alone across all of them). Currently you'd have to design something unique, if you wanted to implement this for your package.
If we wanted to pursue that, I'd suggest the following: remount $RPM_BUILD_ROOT read-only for the %check phase (or maybe overmount it with a writable overlayfs that is thrown away after %check finishes, and warn if any modifications were made.) %check is executed after %install, so everything should be in place before %check, and %check may be skipped, so no modifications to installed files should be done in %check.
Considering possible implemention details, machinectl has 'bind' and 'bind --read-only' that might be useful here. But mock uses systemd-nspawn in a way that does register the container with machined. So maybe it'd be more reasonable to just execute a mount command directly from mock.
This is independent of the test system and does not require splitting of the test sources.
Another thing to consider is making RPM unshare each build phase like it can for scriptlets now at install time[1].
Sandboxing and limiting in rpmbuild itself like we can with rpm can also help with this. I believe moss[2]' boulder tool does this.
-- 真実はいつも一つ！/ Always, there's only one truth! -- _______________________________________________ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
On 4/1/24 14:46, Neal Gompa wrote:
On Mon, Apr 1, 2024 at 7:38 AM Zbigniew Jędrzejewski-Szmek zbyszek@in.waw.pl wrote:
On Sun, Mar 31, 2024 at 11:20:17PM -0700, Adam Williamson wrote:
On Sun, 2024-03-31 at 22:13 -0700, Carlos Rodriguez-Fernandez wrote:
Adam,
Is there a way already to achieve test isolation during the rpm build?
Nothing systematic that I'm aware of, no. It would be tricky because there is no one universal Standard Test System (not even within a single language ecosystem, let alone across all of them). Currently you'd have to design something unique, if you wanted to implement this for your package.
If we wanted to pursue that, I'd suggest the following: remount $RPM_BUILD_ROOT read-only for the %check phase (or maybe overmount it with a writable overlayfs that is thrown away after %check finishes, and warn if any modifications were made.) %check is executed after %install, so everything should be in place before %check, and %check may be skipped, so no modifications to installed files should be done in %check.
Considering possible implemention details, machinectl has 'bind' and 'bind --read-only' that might be useful here. But mock uses systemd-nspawn in a way that does register the container with machined. So maybe it'd be more reasonable to just execute a mount command directly from mock.
This is independent of the test system and does not require splitting of the test sources.
Another thing to consider is making RPM unshare each build phase like it can for scriptlets now at install time[1].
Sandboxing and limiting in rpmbuild itself like we can with rpm can also help with this. I believe moss[2]' boulder tool does this.
Sure, this is an area we've been planning to look into even before this incident. For example https://github.com/rpm-software-management/rpm/issues/2989 and as a result of this thread, https://github.com/rpm-software-management/rpm/issues/3010. And builds with a read-only source: https://github.com/rpm-software-management/rpm/issues/2985
None of this will fix a compromised usptream of course.
- Panu -
Test isolation is still assuming the attack comes in the test phase. The attack can come in the `make`, or in the `make install` too. That's why the idea of other techniques being discussed are still valid, but perhaps not abstracted out enough for a wider defense.
However, the test isolation concept is a good defense. Once the build artifacts is generated, the integrity of the build artifacts must be ensured during any post build phase. Some tests do expect to be executed right after the build, when the build artifacts are still in specific directories within the source root, so that's why I was thinking of something more like at the file system level (e.g., a snapshot that it is then restored, or something like that).
On 3/31/24 23:20, Adam Williamson wrote:
On Sun, 2024-03-31 at 22:13 -0700, Carlos Rodriguez-Fernandez wrote:
Adam,
Is there a way already to achieve test isolation during the rpm build?
Nothing systematic that I'm aware of, no. It would be tricky because there is no one universal Standard Test System (not even within a single language ecosystem, let alone across all of them). Currently you'd have to design something unique, if you wanted to implement this for your package.
I suppose one approach would be to split the sources-required-for-build and the test suite into Source0 and Source1 (respectively) and only extract Source0 during %prep. Don't extract Source1 until %check (i.e. after %build and %install are already done). I'm just spitballing, though, haven't checked if this is really practical.
Of course, another approach is to really do what Kevin suggests and not ship the test suite in the package source at all, but instead run the tests via Fedora CI, and configure the package to be gated on that CI check (so it wouldn't go to stable without the tests passing). But that's rather a different approach (and would still require 'custom' work to cut the tests out of the source, or at least delete them before running the build).
And I still think at this point we are falling into the trap of thinking too specifically about an attack vector which just *happens* to be the one chosen in *this specific instance*. It's still worthwhile, on some level, for someone to think about that kind of hardening, of course. I am just not convinced it is the most useful thing Fedora could be doing right now in the general area of "supply chain hardening".
On Mon, 2024-04-01 at 05:58 -0700, Carlos Rodriguez-Fernandez wrote:
Test isolation is still assuming the attack comes in the test phase.
As I initially suggested it, it does not. My suggestion was that we ensure the test code is not available to the prep / build / install phases *at all*, and is only made available to the test phase.
Understood. However, at least for those unit tests run in the %check, it is going to be almost unfeasible, because of the variability of the way things are done in the different programming ecosystems. In Java, unit tests are nicely separated in a different folder (i.e., `src/test`), but in golang, it is mingled with the source code in `_test.go` files. In C, it depends on the programmers convention.
On 4/1/24 09:29, Adam Williamson wrote:
On Mon, 2024-04-01 at 05:58 -0700, Carlos Rodriguez-Fernandez wrote:
Test isolation is still assuming the attack comes in the test phase.
As I initially suggested it, it does not. My suggestion was that we ensure the test code is not available to the prep / build / install phases *at all*, and is only made available to the test phase.
On Sun, 2024-03-31 at 20:27 -0700, Adam Williamson wrote:
What WOULD have greatly reduced the impact of this attack:
- NOT enabling updates-testing by default for Branched releases.
This would only have helped by coincidence - the coincidence that the compromise was discovered so quickly. Otherwise, we were busy trying to get this update pushed stable as fast as possible, because rwmj wanted that. If the compromise had happened to be caught a few days later, the update would have been pushed stable anyway. Our gating and manual testing processes are not designed to catch security issues, and certainly do not reliably do that job. They did not in this case. So it seems wrong, to me, to suggest we should rely on them to do it in future. Thus I disagree that any possible security benefit gained by doing this would outweigh the substantial disadvantage that we wouldn't get testing of stuff promptly any more.
Having thought this over a bit, I think I should moderate my position here and make it a bit more nuanced.
It occurs to me - maybe you don't agree, but this is how it looks to me - that, ironically, you and I usually argue the exact *opposite* side of this case, no? I argue in *favor* of somewhat-arbitrary delays to packages appearing in 'stable', and you argue *against* them. :D
So, I don't think I can in full conscience justify being so categorical. So let me put more emphasis on the "difference between security and quality" aspect here.
Let's look at the *intent* of the existing policies, here. Practically speaking, what our policies *achieve* is that for stable releases, updates-testing acts to *guard Fedora users' systems*. That is its function there. That's why it is not enabled by default, and we ask only people who want to contribute to testing to enable it, and understand that they are "taking a bullet for quality" by doing so. The idea is that by preventing updates reaching 'ordinary' users' systems before our testers have at least had an opportunity to try the packages out, we will catch at least some quality issues and prevent ordinary users from being affected by them.
The process is specifically designed to achieve this. We have automated *quality* checks on packages in updates-testing (but almost nothing in the way of automated *security* checks). We have a whole process geared towards getting testers to test out the packages and report their findings - in *quality* terms. This is, to me, a job it's reasonable to ask a fairly broad range of folks to help with. It works reasonably well. We have done a reasonable job of implementing a process that lets a person without really specific training install a test update, experience a problem in it, and communicate that they encountered that problem in such a way that we are able to prevent the package reaching a wider swathe of users. It's not *perfect*, but it does provide some demonstrable value. Similarly on the automated testing side, both Fedora CI and openQA provide useful gating of some packages. It is by no means comprehensive, but it *is* useful. We catch bugs and prevent them reaching regular users. We don't catch *all* bugs, but we can reasonably claim to be purposefully and intentionally catching a reasonable number with this system.
OK, that's for stable releases. What about development releases? It's key to realize that for development releases, the process looks very similar, but just that one change - updates-testing defaulting to off - *changes its entire purpose*, and this is by intent. For development releases, the purpose of updates-testing is **NOT** to protect 'ordinary users'. It is to protect **the Fedora development process**.
We don't really intend for there to *be* "ordinary users" of Fedora development releases. We make it fairly clear that, by using a development release, you are taking yourself out of the "regular user" category and putting yourself in the "tester / developer" category. We do not feel such an obligation to provide users of development releases with quality-tested code. On the contrary, to a degree, they are supposed to be *helping* with the process of quality testing for the ultimate goal of making the *final* release good for "ordinary users". So if you're a user of a development release, we do not really feel an obligation to provide the same "safety net" for you that updates- testing provides for "regular users" of stable releases.
Instead, the point of updates-testing in development releases is to let us prevent breakages in *the process of building the distribution*. We expect *all* users of development releases to help with this by flagging up when they see breakage, so we can prevent broken packages appearing in the buildroot or the compose process. *That* is why we have updates-testing for development releases.
So...to return to the initial question, ultimately my argument is that it doesn't make much sense to try and use the existing updates-testing process as a "security gate". Not even for production releases, really, but *especially* not for development releases. That's *not what it's designed to do*. There is very little meaningful automated *security* testing going on for updates-testing. Nor do we have any kind of process to try and get informed *manual* security checks done on packages in updates-testing. Given that, I think the benefit of trying to use our existing updates-testing process to improve security would be very small.
Now, if we were to actually *intentionally design a process* to do some meaningful security checks on 'candidate' packages, that would be an entirely different story. That might be something worth actually doing, and if we were to set out to do that, the question of whether it should be kinda merged into the existing 'updates-testing' process, or should use some different kinda gating approach, would be a reasonable one. My vote might be very different in that case. But I just don't think we would get much security value from *just* enabling updates-testing on Branched, given the background above. Sorry this got much longer and less clear. :D
Adam Williamson wrote:
It occurs to me - maybe you don't agree, but this is how it looks to me
- that, ironically, you and I usually argue the exact *opposite* side
of this case, no? I argue in *favor* of somewhat-arbitrary delays to packages appearing in 'stable', and you argue *against* them. :D
I have never argued against updates-testing existing or that all packages should skip updates-testing. "Please pick up this new upstream version, it has some great new features" as was done here is exactly the kind of changes that SHOULD go through updates-testing. But if the maintainer has something urgent to push out, such as an important regression fix or a critical security fix (e.g., a fix for a backdoor like this one), they should be allowed to decide to skip testing and not be treated as being too incompetent for that (while at the same time allowing any other person, with no other credentials than a FAS account, to +1 the package, allowing it to be autopushed to stable – everyone except the one person most qualified to make that decision). THAT is what I have been arguing for all this time, and I do not see how this contradicts my position here in any way.
Kevin Kofler
On Wed, 2024-04-03 at 00:15 +0200, Kevin Kofler via devel wrote:
Adam Williamson wrote:
It occurs to me - maybe you don't agree, but this is how it looks to me
- that, ironically, you and I usually argue the exact *opposite* side
of this case, no? I argue in *favor* of somewhat-arbitrary delays to packages appearing in 'stable', and you argue *against* them. :D
I have never argued against updates-testing existing or that all packages should skip updates-testing. "Please pick up this new upstream version, it has some great new features" as was done here is exactly the kind of changes that SHOULD go through updates-testing. But if the maintainer has something urgent to push out, such as an important regression fix or a critical security fix (e.g., a fix for a backdoor like this one), they should be allowed to decide to skip testing and not be treated as being too incompetent for that (while at the same time allowing any other person, with no other credentials than a FAS account, to +1 the package, allowing it to be autopushed to stable – everyone except the one person most qualified to make that decision). THAT is what I have been arguing for all this time, and I do not see how this contradicts my position here in any way.
Fair enough. I think the rest of my post stands, though, as it was more about my argument than yours.
Adam Williamson wrote:
- Deleting ALL files automatically generated or imported by autotools in
%prep, THEN running "autoreconf -i -f". (DO NOT trust autoreconf, it would NOT have done the right thing here. Delete the files, THEN run autoreconf.)
No. This would not have avoided the attack, because it would not have regenerated the malicious file. We have already established that.
Just running autoreconf would not. As I wrote: "DO NOT trust autoreconf, it would NOT have done the right thing here." Deleting the file with an explicit rm -f in %prep, and THEN running autoreconf would have regenerated (reimported, actually, this comes from gnulib and is copied unchanged, but in any case it would NOT have contained the malicious additions) the file.
That said, autoreconf needs fixing too, because -f is supposed to regenerate all files that can be regenerated, which is not happening. But if you explicitly delete the files before running autoreconf, then it has to regenerate them no matter what.
Kevin Kofler
On Mon, 2024-04-01 at 23:37 +0200, Kevin Kofler via devel wrote:
Adam Williamson wrote:
- Deleting ALL files automatically generated or imported by autotools in
%prep, THEN running "autoreconf -i -f". (DO NOT trust autoreconf, it would NOT have done the right thing here. Delete the files, THEN run autoreconf.)
No. This would not have avoided the attack, because it would not have regenerated the malicious file. We have already established that.
Just running autoreconf would not. As I wrote: "DO NOT trust autoreconf, it would NOT have done the right thing here." Deleting the file with an explicit rm -f in %prep, and THEN running autoreconf would have regenerated (reimported, actually, this comes from gnulib and is copied unchanged, but in any case it would NOT have contained the malicious additions) the file.
That said, autoreconf needs fixing too, because -f is supposed to regenerate all files that can be regenerated, which is not happening. But if you explicitly delete the files before running autoreconf, then it has to regenerate them no matter what.
Sure, but as others posted upthread, this still doesn't help much. To do this you have to know what m4s are 'standard' and will actually be regenerated, and which are custom and you can't wipe them. And then an attacker could just slip in an extra custom one instead of modifying a 'standard' one.
On Sunday, 31 March 2024 20:12:38 CEST Kevin Kofler via devel wrote:
- Deleting ALL unit tests in %prep (and then of course not trying to run
them later).
I strongly oppose this suggestion. While it would have prevented this particular backdoor as a side-effect, the primary effect of going without unit tests would be an outsize hole in Fedora's QA.
As maintainer of the mold package, I regularly come across failures in %check, mostly in the builds for secondary architectures. This allows me to help upstream - a one-person project with limited resources - fix these bugs before Fedora ships the package.
Moreover, %check has already caught bugs that only manifest themselves with Fedora's compile flags. Here's an example: https://github.com/rui314/mold/pull/877
You used the term 'pointless security theatre' in another email within this thread. *This* would be a prime example.
Best, Christoph
Christoph Erhardt writes:
I strongly oppose this suggestion. While it would have prevented this particular backdoor as a side-effect, the primary effect of going without unit tests would be an outsize hole in Fedora's QA.
There have been several suggestions here for ways that this specific attempt from succeeding.
Any one of them will be very useful as long as it is guaranteed that all backdoor/supply chain attacks in the future are attempted in the exact same technical way.
On Sun, 2024-03-31 at 13:03 +0200, Kevin Kofler via devel wrote:
- Our process for vetting packagers is, let's face it, from a security
perspective almost *comically* patchy. There are 140 sponsors in the packager FAS group. Any one of those people - or someone who compromises any one of those 140 accounts - can grant any other person on earth Fedora packager status. Our policy on how they should do this is https://docs.fedoraproject.org/en-US/package-maintainers/How_to_Sponsor_a_Ne... . The words "trust" and "identity" do not appear in it. There is, AFAIK, no policy or procedure by which inactive sponsors have this power removed. There is no mandatory 2FA policy for sponsors.
We already have a manpower problem, how is removing sponsors going to improve the situation?
Removing inactive sponsors makes it harder for an attacker to compromise an inactive sponsor's account and use it for evil.
However, looking into it it looks like I made a mistake here: we do have an inactive *packager* policy, as well as an inactive *proven packager* policy. I had misremembered that we did not have the former, only the latter. So at *least* a sponsor who becomes completely inactive in the project will eventually get removed as a packager entirely (and hence also as a sponsor). I don't think that's entirely sufficient, but it's better than I thought.
- We have no mechanism to flag when J. Random Packager adds
"Supplements: glibc" to their random leaf node package. As a reminder, *we are a project that allows 1,601 minimally-vetted people to deliver arbitrary code executed as root on hundreds of thousands of systems*, and this mechanism allows any one of those people to cause the package they have complete control over to be automatically pulled in as a dependency on virtually every single one of those systems.
This would get noticed pretty quickly, when that package comes up in update transactions for no reason. I believe this has never happened so far. It is just too obvious.
"Would get noticed pretty quickly" is not really acceptable. It should not be possible to do it at all.
This was just an example really, too. I am not a very good red teamer. I'm not imaginative enough. There are probably all sorts of more subtle ways a malicious person with packager privileges could leverage them to attack more people than they could attack just through 'normal' maintenance of a leaf package. Just for another example, we have had two significant cases recently of "package X incorrectly provides capability Y".
1. sdubby was providing grubby with a higher version than the real grubby package, which seems to have caused it to get installed in preference to grubby on fresh installs and upgrades when it should not have been.
2. some relatively obscure package somehow started including bundled copies of dozens of core system libraries, which made the auto-provides generator generate provides for those libraries. dnf then started preferring it over the 'real' providers of some of those libraries, I think because it had a shorter dep chain.
Neither of these was, AFAICT, flagged as a potential security issue by anyone, but it seems reasonable to me that they should have been. I know about them because we happened to catch them as quality issues, but it is not hard to imagine that a malicious packager may be interested in using the same effect intentionally for evil, and they would obviously be careful to try and set things up such that this would *not* cause anything easily noticeable as a quality defect.
On Sun, Mar 31, 2024 at 01:58:21AM -0700, Adam Williamson wrote:
On Sat, 2024-03-30 at 09:37 +0000, Richard W.M. Jones wrote:
I'm not pretending these will solve everything, but they should make attacks a little harder in future.
I don't disagree with Richard's list. However...more in regards to some of the grandiose ideas in later posts than Richard's list...I think we're in danger of building castles in the sky while not cleaning up the poop in our backyard, here.
Before we start in on the grand fantasies about converting the world off autotools or banning binaries in repos or centralized source depots authenticated by a committee of Top People, can we remember:
- We *still don't have compulsory 2FA for Fedora packagers*. We *still
don't have compulsory 2FA for Fedora packagers*. *WE STILL DON'T HAVE COMPULSORY 2FA FOR FEDORA PACKAGERS*.
- Our process for vetting packagers is, let's face it, from a security
perspective almost *comically* patchy. There are 140 sponsors in the packager FAS group. Any one of those people - or someone who compromises any one of those 140 accounts - can grant any other person on earth Fedora packager status. Our policy on how they should do this is https://docs.fedoraproject.org/en-US/package-maintainers/How_to_Sponsor_a_Ne... . The words "trust" and "identity" do not appear in it. There is, AFAIK, no policy or procedure by which inactive sponsors have this power removed. There is no mandatory 2FA policy for sponsors.
- We have no mechanism to flag when J. Random Packager adds
"Supplements: glibc" to their random leaf node package. As a reminder, *we are a project that allows 1,601 minimally-vetted people to deliver arbitrary code executed as root on hundreds of thousands of systems*, and this mechanism allows any one of those people to cause the package they have complete control over to be automatically pulled in as a dependency on virtually every single one of those systems.
This is as much a distro design problem, as a Fedora process problem. The typical Linux distro model is that everything is installed in the same namespace, and we only avoid interference (whether accidental or intentional) by careful packaging design and review.
This is somewhere where the image based Linux distro model has a potential benefit, with a comparatively slim distro base, and then applications as self contained separated entities, whether server apps in podman containers, or GUI apps in flatpaks.
No easy anwere here though, as the traditional Linux model isn't going away any time in the forseeable future.
- Our main auth system was written years ago by someone who no longer
contributes and nobody is really actively maintaining it[0].
5. We review initial new packages in a moderate level of detail, but after that it is largely a free for all for every rebase. It is up to the maintainer's discretion what to do for new releases, and any oversight is patchy at best. The threat here was in relation to an update to an existing package, and where we have no formal process that was even /interested/ in addressing that threat, let alone capable of stopping it.
Now, in this particular case, it would have been challenging to detect it the problem even with "new package" review process, so I'm sceptical we can easily build a "updated package" process that would have blocked it and yet remain practical for maintainers.
With regards, Daniel
On Sun, 2024-03-31 at 13:28 +0100, Daniel P. Berrangé wrote:
- We have no mechanism to flag when J. Random Packager adds
"Supplements: glibc" to their random leaf node package. As a reminder, *we are a project that allows 1,601 minimally-vetted people to deliver arbitrary code executed as root on hundreds of thousands of systems*, and this mechanism allows any one of those people to cause the package they have complete control over to be automatically pulled in as a dependency on virtually every single one of those systems.
This is as much a distro design problem, as a Fedora process problem. The typical Linux distro model is that everything is installed in the same namespace, and we only avoid interference (whether accidental or intentional) by careful packaging design and review.
This is somewhere where the image based Linux distro model has a potential benefit, with a comparatively slim distro base, and then applications as self contained separated entities, whether server apps in podman containers, or GUI apps in flatpaks.
No easy anwere here though, as the traditional Linux model isn't going away any time in the forseeable future.
Well, definitely no easy answers, but I would argue this is the kind of thing we as a distributor *should* be worrying about and addressing, maybe with more urgency than things that are not primarily the responsibility of the distributor, in the supply chain.
Maybe this needs to go on the growing pile of reasons why the traditional Linux model *does* need to go away. Maybe Fedora, with its foundation of First, should be kind of at the forefront of making that happen.
Adam Williamson wrote:
Maybe this needs to go on the growing pile of reasons why the traditional Linux model *does* need to go away. Maybe Fedora, with its foundation of First, should be kind of at the forefront of making that happen.
Switching to a container-based model is just going to introduce more different library versions (in the worst case, one per container) with a higher probability that one of them is compromised.
Kevin Kofler
On Sun, Mar 31, 2024 at 07:54:08PM +0200, Kevin Kofler via devel wrote:
Adam Williamson wrote:
Maybe this needs to go on the growing pile of reasons why the traditional Linux model *does* need to go away. Maybe Fedora, with its foundation of First, should be kind of at the forefront of making that happen.
Switching to a container-based model is just going to introduce more different library versions (in the worst case, one per container) with a higher probability that one of them is compromised.
Our traditional distro model is not perfect — far from it — and we certainly try to improve it. But I agree with Kevin that in _this particular case_, the other models have smaller chances of catching the issue.
Here the upstream was compromised, so 2FA, upstream signatures, and any other checks don't help at all.
But in our "traditional model" we have one version of the dependency used for everbody, so there is a strong incentive to review and improve this one particular version. The packaging process is also very open: it is absolutely routine for people to change packagaging for packages owned by other maintainers.
The newfangled models are much more about picking particular versions of dependencies and duplicating them in multiple projects. This makes some things easier, and makes things more independent, but I think it'd make the xz bug less likely to be caught. If sshd was packaged as a container or a flatpak, and I saw that it takes .8 instead of .1 seconds to log in, I certainly wouldn't spend the time to figure out why. I'd assume that the authors did something strange and move on to my own things.
We talk a lot about the "new ways", but software must still come from somewhere, and the dependencies need to be maintained… Changing the delivery format is not going to magically makes this unnecessary.
Zbyszek
On Mon, 2024-04-01 at 10:56 +0000, Zbigniew Jędrzejewski-Szmek wrote:
On Sun, Mar 31, 2024 at 07:54:08PM +0200, Kevin Kofler via devel wrote:
Adam Williamson wrote:
Maybe this needs to go on the growing pile of reasons why the traditional Linux model *does* need to go away. Maybe Fedora, with its foundation of First, should be kind of at the forefront of making that happen.
Switching to a container-based model is just going to introduce more different library versions (in the worst case, one per container) with a higher probability that one of them is compromised.
Our traditional distro model is not perfect — far from it — and we certainly try to improve it. But I agree with Kevin that in _this particular case_, the other models have smaller chances of catching the issue.
Here the upstream was compromised, so 2FA, upstream signatures, and any other checks don't help at all.
Yes, to be clear, my "this" was not "the specific technical details of this attack". It was more:
i) the factors I listed in my email about just how many people are trusted to build 'Fedora', when 'Fedora' is essentially a collection of arbitrary scripts executed as root
ii) the fact that this attack reinforces the painful truth that sophisticated attackers *are* extremely interested in attacking the supply chain of which we form a significant component
On Mon, Apr 1, 2024 at 12:22 PM Adam Williamson adamwill@fedoraproject.org wrote:
On Mon, 2024-04-01 at 10:56 +0000, Zbigniew Jędrzejewski-Szmek wrote:
On Sun, Mar 31, 2024 at 07:54:08PM +0200, Kevin Kofler via devel wrote:
Adam Williamson wrote:
Maybe this needs to go on the growing pile of reasons why the traditional Linux model *does* need to go away. Maybe Fedora, with its foundation of First, should be kind of at the forefront of making that happen.
Switching to a container-based model is just going to introduce more different library versions (in the worst case, one per container) with a higher probability that one of them is compromised.
Our traditional distro model is not perfect — far from it — and we certainly try to improve it. But I agree with Kevin that in _this particular case_, the other models have smaller chances of catching the issue.
Here the upstream was compromised, so 2FA, upstream signatures, and any other checks don't help at all.
Yes, to be clear, my "this" was not "the specific technical details of this attack". It was more:
i) the factors I listed in my email about just how many people are trusted to build 'Fedora', when 'Fedora' is essentially a collection of arbitrary scripts executed as root
ii) the fact that this attack reinforces the painful truth that sophisticated attackers *are* extremely interested in attacking the supply chain of which we form a significant component
Can we please reframe it for what it actually is? This is an attack on open source communities. "Supply chain" implies a lot of things that simply don't exist in open source development. Almost the entirety of the sophistication of the attack was social engineering, not technical engineering. There *are* technical things to improve, for sure, but let's not try to make it sound like it's a wholly technical thing that can be solved with technical solutions exclusively. There are people and community problems that need addressing too.
On Mon, 2024-04-01 at 12:27 -0400, Neal Gompa wrote:
ii) the fact that this attack reinforces the painful truth that sophisticated attackers *are* extremely interested in attacking the supply chain of which we form a significant component
Can we please reframe it for what it actually is? This is an attack on open source communities. "Supply chain" implies a lot of things that simply don't exist in open source development. Almost the entirety of the sophistication of the attack was social engineering, not technical engineering. There *are* technical things to improve, for sure, but let's not try to make it sound like it's a wholly technical thing that can be solved with technical solutions exclusively. There are people and community problems that need addressing too.
This feels like a derail, so splitting it into a separate subthread.
Honestly, I don't see how the first part of your paragraph relates to the second. I agree with a lot of the second part, but not the first.
I think we *are* part of a supply chain, regardless of any handwaving about The Open Source Model. If you are part of producing stuff that people use to do Real Life Stuff, you are part of a supply chain. You might want to disclaim various responsibilities for various reasons, but you still are. If you don't want to be part of a supply chain, stop supplying stuff. I get the argument that there's a difference between putting a plank over a stream with a CROSS AT YOUR OWN RISK sign and charging people to cross your toll bridge, but there are *also* some similarities.
I agree that the social engineering aspects of this attack were very significant (though I disagree that was "almost the entirety of the sophistication" - the technical elements were also pretty sophisticated). But I don't see why that leads to bikeshedding about whether this is a "supply chain" attack or not. Why is a "social engineering attack" not a supply chain attack, but a "technical attack" is?
On Mon, Apr 1, 2024 at 4:42 PM Adam Williamson adamwill@fedoraproject.org wrote:
I think we *are* part of a supply chain, regardless of any handwaving about The Open Source Model.
And, more importantly, the industry has agreed to use the term supply chain. Is the term perhaps overloaded, or perhaps too ill-defined/imprecise? Sure. But if one wants to use a different term one would need to work across the industry to change the term, and that is not going to happen.
Gary Buhrmaster wrote:
And, more importantly, the industry has agreed to use the term supply chain. Is the term perhaps overloaded, or perhaps too ill-defined/imprecise? Sure. But if one wants to use a different term one would need to work across the industry to change the term, and that is not going to happen.
Well, one could argue that Free Software is a community, not an industry, so "the industry" cannot agree on anything, and "supply chain" as an industrial term obviously does not apply. And also that it is called "Free Software" and not "Open Source". :-)
Kevin Kofler
Hello,
I have been deleting most of these emails, but I feel like this is a bit myopic.
On Tuesday, April 2, 2024 6:25:56 PM EDT Kevin Kofler via devel wrote:
Gary Buhrmaster wrote:
And, more importantly, the industry has agreed to use the term supply chain. Is the term perhaps overloaded, or perhaps too ill-defined/imprecise? Sure. But if one wants to use a different term one would need to work across the industry to change the term, and that is not going to happen.
Well, one could argue that Free Software is a community, not an industry, so "the industry" cannot agree on anything, and "supply chain" as an industrial term obviously does not apply.
But it does. The term "supply chain" refers to the process of acquiring, organizing, and distributing the necessary resources and components to build and maintain the distribution. This includes acquiring source code from various upstream projects, integrating them into the distribution, and then packaging and distributing the final product to end-users. The supply chain involves a variety of tasks such as software development, testing, quality assurance, responding to bug reports, documentation, and support. IOW, it is a process.
It is the community that carries out the process.
-Steve
On Sun, Mar 31, 2024 at 8:58 AM Adam Williamson adamwill@fedoraproject.org wrote:
- We *still don't have compulsory 2FA for Fedora packagers*. We *still
don't have compulsory 2FA for Fedora packagers*. *WE STILL DON'T HAVE COMPULSORY 2FA FOR FEDORA PACKAGERS*.
What is the status of the FIDO2 implementation in the authentication processes? Whenever 2FA comes up I agree with the principal, but ask for an update on FIDO2. Last I knew, it was still a under-resourced WIP.
Going in that same route, if 2FA becomes mandatory, then we have a stronger defense of the GPG public key in the user profile. The attacker would need not only the credentials, but the 2FA device to change the public GPG.
That then makes one step further possible: enforce commit signing on the downstream project. That makes authorship of changes non repudiateable (that is, important changes like `sources`, all patches, specs, etc...), and also allows the build to check that all commits between the last release and the current build are signed by the committers of the project. This can be done by having the build importing the public GPGs of the committers of the downstream project from pagure/FAS and checking, again, the commits since the last release. The downstream project *source code* would be effectively signed, so the signed rpm then would carry even a higher assurance: "this artifact was built by Fedora Infrastructure", but also "all the downstream sources used to make it are assuredly coming from the maintainers."
It is definitely an additional hassle for the maintainers, but there is value in it, especially in this day and age.
On 3/31/24 01:58, Adam Williamson wrote:
On Sat, 2024-03-30 at 09:37 +0000, Richard W.M. Jones wrote:
I'm not pretending these will solve everything, but they should make attacks a little harder in future.
I don't disagree with Richard's list. However...more in regards to some of the grandiose ideas in later posts than Richard's list...I think we're in danger of building castles in the sky while not cleaning up the poop in our backyard, here.
Before we start in on the grand fantasies about converting the world off autotools or banning binaries in repos or centralized source depots authenticated by a committee of Top People, can we remember:
- We *still don't have compulsory 2FA for Fedora packagers*. We *still
don't have compulsory 2FA for Fedora packagers*. *WE STILL DON'T HAVE COMPULSORY 2FA FOR FEDORA PACKAGERS*.
- Our process for vetting packagers is, let's face it, from a security
perspective almost *comically* patchy. There are 140 sponsors in the packager FAS group. Any one of those people - or someone who compromises any one of those 140 accounts - can grant any other person on earth Fedora packager status. Our policy on how they should do this is https://docs.fedoraproject.org/en-US/package-maintainers/How_to_Sponsor_a_Ne... . The words "trust" and "identity" do not appear in it. There is, AFAIK, no policy or procedure by which inactive sponsors have this power removed. There is no mandatory 2FA policy for sponsors.
- We have no mechanism to flag when J. Random Packager adds
"Supplements: glibc" to their random leaf node package. As a reminder, *we are a project that allows 1,601 minimally-vetted people to deliver arbitrary code executed as root on hundreds of thousands of systems*, and this mechanism allows any one of those people to cause the package they have complete control over to be automatically pulled in as a dependency on virtually every single one of those systems.
- Our main auth system was written years ago by someone who no longer
contributes and nobody is really actively maintaining it[0].
These are just the ones that leap to my tired mind at this moment. I'm sure we can think of many more things we should probably look at before we start pontificating (or, worse, lecturing) about how things should be done upstream of us.
BTW, adding comments to myself, the signed commits and their verification takes care of problems 2FA doesn't. First, ensure authorship information (2FA doesn't leave a mark in the commit), and protects the integrity of the downstream source: the build can ensure that the checked out downstream project doesn't get "dirty" during and after the build, and/or commits are not modified during and after the build.
But this doesn't solve the specific xz problem. That one requires different kinds of defense techniques.
On 3/31/24 08:10, Carlos Rodriguez-Fernandez wrote:
Going in that same route, if 2FA becomes mandatory, then we have a stronger defense of the GPG public key in the user profile. The attacker would need not only the credentials, but the 2FA device to change the public GPG.
That then makes one step further possible: enforce commit signing on the downstream project. That makes authorship of changes non repudiateable (that is, important changes like `sources`, all patches, specs, etc...), and also allows the build to check that all commits between the last release and the current build are signed by the committers of the project. This can be done by having the build importing the public GPGs of the committers of the downstream project from pagure/FAS and checking, again, the commits since the last release. The downstream project *source code* would be effectively signed, so the signed rpm then would carry even a higher assurance: "this artifact was built by Fedora Infrastructure", but also "all the downstream sources used to make it are assuredly coming from the maintainers."
It is definitely an additional hassle for the maintainers, but there is value in it, especially in this day and age.
On 3/31/24 01:58, Adam Williamson wrote:
On Sat, 2024-03-30 at 09:37 +0000, Richard W.M. Jones wrote:
I'm not pretending these will solve everything, but they should make attacks a little harder in future.
I don't disagree with Richard's list. However...more in regards to some of the grandiose ideas in later posts than Richard's list...I think we're in danger of building castles in the sky while not cleaning up the poop in our backyard, here.
Before we start in on the grand fantasies about converting the world off autotools or banning binaries in repos or centralized source depots authenticated by a committee of Top People, can we remember:
- We *still don't have compulsory 2FA for Fedora packagers*. We *still
don't have compulsory 2FA for Fedora packagers*. *WE STILL DON'T HAVE COMPULSORY 2FA FOR FEDORA PACKAGERS*.
- Our process for vetting packagers is, let's face it, from a security
perspective almost *comically* patchy. There are 140 sponsors in the packager FAS group. Any one of those people - or someone who compromises any one of those 140 accounts - can grant any other person on earth Fedora packager status. Our policy on how they should do this is https://docs.fedoraproject.org/en-US/package-maintainers/How_to_Sponsor_a_Ne... . The words "trust" and "identity" do not appear in it. There is, AFAIK, no policy or procedure by which inactive sponsors have this power removed. There is no mandatory 2FA policy for sponsors.
- We have no mechanism to flag when J. Random Packager adds
"Supplements: glibc" to their random leaf node package. As a reminder, *we are a project that allows 1,601 minimally-vetted people to deliver arbitrary code executed as root on hundreds of thousands of systems*, and this mechanism allows any one of those people to cause the package they have complete control over to be automatically pulled in as a dependency on virtually every single one of those systems.
- Our main auth system was written years ago by someone who no longer
contributes and nobody is really actively maintaining it[0].
These are just the ones that leap to my tired mind at this moment. I'm sure we can think of many more things we should probably look at before we start pontificating (or, worse, lecturing) about how things should be done upstream of us.
On Sat, Mar 30, 2024 at 09:37:44AM +0000, Richard W.M. Jones wrote:
I'm not pretending these will solve everything, but they should make attacks a little harder in future.
(1) We should routinely delete autoconf-generated cruft from upstream projects and regenerate it in %prep. It is easier to study the real source rather than dig through the convoluted, generated shell script in an upstream './configure' looking for back doors.
I've noted many times before how inconsistent it is that many of us will happily mock a website saying "curl <url> | sh -", and then with a straight face tell users to download a tarball and run the 1 MB auto-generated 'configure' shell script that is obfuscated to the point of unreadability by autotools. It is the perfect place to hide malicious code in tarballs.
For most projects, just running "autoreconf -fiv" is enough.
Yes, there are some projects that depend on a specific or old version of autoconf. We should fix those. But that doesn't need to delay us from using autoreconf on many projects today.
In the xz case this wouldn't have been enough, it turns out we would also have to delete m4/build-to-host.m4, which then autoreconf regenerates. I don't fully understand why that is.
This is challenging to address too, since often the 'm4/' directory contains a mixture of application provided m4 files, and autotools provided m4 files. You can't blindly purge the m4/ directory. They could have provided a "m4/my-build-to-host.m4" file that interferes or overides the main 'm4/build-to-host.m4'. So even if autoreconf did regenerate that file as desired, you're still not entirely safe.
Still, it would be good practice to have a way to purge any files that are known to be autoconf provided, so at least you know which remaining files to look at for threats / problems.
(2) We should discourage gnulib as much as possible.
In libvirt we took the decision a few years ago to remove gnulib. It's extremely convoluted and almost no one understands how it really works. It's written in obscure m4 macros and shell script.
It's also not necessary for Linux since gnulib is mainly about porting to non-Linux platforms. There are better ways to do this.
Be honest about which platforms you genuinely need portability too in the modern world. Cull obsolete / dead platforms.
While it is a fun intellectual challenge to write portable code against POSIX, most projects are better off not reinventing the wheel. Something like GLib takes out a large part of the portability pain if you really must stick with developing C, and is easy to adopt incrementally. This is what let libvirt cull its use of gnulib, which in turn lets us cull the use of autotools
In the xz case it was a gnulib-derived script which was modified to do the initial injection (original: https://git.savannah.gnu.org/gitweb/?p=gnulib.git;a=blob;f=m4/build-to-host....).
(3) We should have a "security path", like "critical path".
sshd is linked to a lot of libraries:
/lib64/libaudit.so.1 audit-libs /lib64/libc.so.6 glibc /lib64/libcap-ng.so.0 libcap-ng /lib64/libcap.so.2 libcap /lib64/libcom_err.so.2 libcom_err /lib64/libcrypt.so.2 libxcrypt /lib64/libcrypto.so.3 openssl-libs /lib64/libeconf.so.0 libeconf /lib64/libgcc_s.so.1 libgcc /lib64/libgssapi_krb5.so.2 krb5-libs /lib64/libk5crypto.so.3 krb5-libs /lib64/libkeyutils.so.1 keyutils-libs /lib64/libkrb5.so.3 krb5-libs /lib64/libkrb5support.so.0 krb5-libs /lib64/liblz4.so.1 lz4-libs /lib64/liblzma.so.5 xz-libs /lib64/libm.so.6 glibc /lib64/libpam.so.0 pam-libs /lib64/libpcre2-8.so.0 pcre2 /lib64/libresolv.so.2 glibc /lib64/libselinux.so.1 libselinux /lib64/libsystemd.so.0 systemd-libs /lib64/libz.so.1 zlib / zlib-ng /lib64/libzstd.so.1 zstd
Should we have a higher level of attention to these packages? We already have "critical path", but that's a broad category now. These seem like they are "security path" packages, an intentionally small subset associated with very secure services which are enabled by default.
These are just my thoughts on a Saturday morning. Feedback welcome of course.
Rich.
-- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com virt-top is 'top' for virtual machines. Tiny program with many powerful monitoring features, net stats, disk stats, logging, etc. http://people.redhat.com/~rjones/virt-top -- _______________________________________________ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
With regards, Daniel
Regarding downstream defense, prohibiting blobs or differences between tars and git repos may be overkill or difficult at this moment, but a tool to assist maintainers in the identification and analysis of these situations where attacks may be hidden into can be very helpful.
Regarding the latter, where you want to find diff between tars and git repo content, someone may say "let's just use the github API to pull the tar", but not all projects use github, or have that feature. So, I still think there is value in these tools.
I started working on a PoC personal minitool named rpmseclint that utilizes the "VCS" tag to pull the git repo, and the tar, and compares and flags things for further review. Right now, I'm just focused on blobs, and files/directories differences. The intention is to use it at least right before a rebase. I'm just planning to personally use it to get a "feel" for it (and help myself), but open to anyone wanting to explore this idea.
In general, I think the idea of codifying security practices expertise in tools to assist maintainers with detection and analysis has great value; even if it is not enforcing anything at the moment. The maintainer could then explicitly add found problems to an "ignore" list after the analysis, and explain the "why" in the git commit body message, and perpetuate the reasoning in the git history of the downstream project. There is already a lot of best practices codified in the gating pipelines, and I have found them extremely useful myself.
Regards, Carlos R.F.
On 3/30/24 02:37, Richard W.M. Jones wrote:
I'm not pretending these will solve everything, but they should make attacks a little harder in future.
(1) We should routinely delete autoconf-generated cruft from upstream projects and regenerate it in %prep. It is easier to study the real source rather than dig through the convoluted, generated shell script in an upstream './configure' looking for back doors.
For most projects, just running "autoreconf -fiv" is enough.
Yes, there are some projects that depend on a specific or old version of autoconf. We should fix those. But that doesn't need to delay us from using autoreconf on many projects today.
In the xz case this wouldn't have been enough, it turns out we would also have to delete m4/build-to-host.m4, which then autoreconf regenerates. I don't fully understand why that is.
(2) We should discourage gnulib as much as possible.
In libvirt we took the decision a few years ago to remove gnulib. It's extremely convoluted and almost no one understands how it really works. It's written in obscure m4 macros and shell script.
It's also not necessary for Linux since gnulib is mainly about porting to non-Linux platforms. There are better ways to do this.
In the xz case it was a gnulib-derived script which was modified to do the initial injection (original: https://git.savannah.gnu.org/gitweb/?p=gnulib.git;a=blob;f=m4/build-to-host....).
(3) We should have a "security path", like "critical path".
sshd is linked to a lot of libraries:
/lib64/libaudit.so.1 audit-libs /lib64/libc.so.6 glibc /lib64/libcap-ng.so.0 libcap-ng /lib64/libcap.so.2 libcap /lib64/libcom_err.so.2 libcom_err /lib64/libcrypt.so.2 libxcrypt /lib64/libcrypto.so.3 openssl-libs /lib64/libeconf.so.0 libeconf /lib64/libgcc_s.so.1 libgcc /lib64/libgssapi_krb5.so.2 krb5-libs /lib64/libk5crypto.so.3 krb5-libs /lib64/libkeyutils.so.1 keyutils-libs /lib64/libkrb5.so.3 krb5-libs /lib64/libkrb5support.so.0 krb5-libs /lib64/liblz4.so.1 lz4-libs /lib64/liblzma.so.5 xz-libs /lib64/libm.so.6 glibc /lib64/libpam.so.0 pam-libs /lib64/libpcre2-8.so.0 pcre2 /lib64/libresolv.so.2 glibc /lib64/libselinux.so.1 libselinux /lib64/libsystemd.so.0 systemd-libs /lib64/libz.so.1 zlib / zlib-ng /lib64/libzstd.so.1 zstd
Should we have a higher level of attention to these packages? We already have "critical path", but that's a broad category now. These seem like they are "security path" packages, an intentionally small subset associated with very secure services which are enabled by default.
These are just my thoughts on a Saturday morning. Feedback welcome of course.
Rich.
Definitely this attack leveraged places where eyes don't look: distributed tar.gz and blobs.
I put the PoC to flag those two in github[1]
Example output:
$ ./rpmseclint tests/rpmseclint-test.spec -----Diff----- ~ test.txt + additional.txt + blob.txt.gz -----Blobs---- application/gzip blob.txt.gz
Now, against the PoC itself, the diff part can be totally avoided by using generated tars. Not all projects are in github, but those that do... The other way is by using source-git?
Regarding the blobs part, ... an attack could still manage to fool `file` with a crafted binary. However, that would still increase the attack complexity.
[1] https://github.com/carlosrodfern/rpmseclint
On 3/31/24 19:02, Carlos Rodriguez-Fernandez wrote:
Regarding downstream defense, prohibiting blobs or differences between tars and git repos may be overkill or difficult at this moment, but a tool to assist maintainers in the identification and analysis of these situations where attacks may be hidden into can be very helpful.
Regarding the latter, where you want to find diff between tars and git repo content, someone may say "let's just use the github API to pull the tar", but not all projects use github, or have that feature. So, I still think there is value in these tools.
I started working on a PoC personal minitool named rpmseclint that utilizes the "VCS" tag to pull the git repo, and the tar, and compares and flags things for further review. Right now, I'm just focused on blobs, and files/directories differences. The intention is to use it at least right before a rebase. I'm just planning to personally use it to get a "feel" for it (and help myself), but open to anyone wanting to explore this idea.
In general, I think the idea of codifying security practices expertise in tools to assist maintainers with detection and analysis has great value; even if it is not enforcing anything at the moment. The maintainer could then explicitly add found problems to an "ignore" list after the analysis, and explain the "why" in the git commit body message, and perpetuate the reasoning in the git history of the downstream project. There is already a lot of best practices codified in the gating pipelines, and I have found them extremely useful myself.
Regards, Carlos R.F.
On 3/30/24 02:37, Richard W.M. Jones wrote:
I'm not pretending these will solve everything, but they should make attacks a little harder in future.
(1) We should routinely delete autoconf-generated cruft from upstream projects and regenerate it in %prep. It is easier to study the real source rather than dig through the convoluted, generated shell script in an upstream './configure' looking for back doors.
For most projects, just running "autoreconf -fiv" is enough.
Yes, there are some projects that depend on a specific or old version of autoconf. We should fix those. But that doesn't need to delay us from using autoreconf on many projects today.
In the xz case this wouldn't have been enough, it turns out we would also have to delete m4/build-to-host.m4, which then autoreconf regenerates. I don't fully understand why that is.
(2) We should discourage gnulib as much as possible.
In libvirt we took the decision a few years ago to remove gnulib. It's extremely convoluted and almost no one understands how it really works. It's written in obscure m4 macros and shell script.
It's also not necessary for Linux since gnulib is mainly about porting to non-Linux platforms. There are better ways to do this.
In the xz case it was a gnulib-derived script which was modified to do the initial injection (original: https://git.savannah.gnu.org/gitweb/?p=gnulib.git;a=blob;f=m4/build-to-host....).
(3) We should have a "security path", like "critical path".
sshd is linked to a lot of libraries:
/lib64/libaudit.so.1 audit-libs /lib64/libc.so.6 glibc /lib64/libcap-ng.so.0 libcap-ng /lib64/libcap.so.2 libcap /lib64/libcom_err.so.2 libcom_err /lib64/libcrypt.so.2 libxcrypt /lib64/libcrypto.so.3 openssl-libs /lib64/libeconf.so.0 libeconf /lib64/libgcc_s.so.1 libgcc /lib64/libgssapi_krb5.so.2 krb5-libs /lib64/libk5crypto.so.3 krb5-libs /lib64/libkeyutils.so.1 keyutils-libs /lib64/libkrb5.so.3 krb5-libs /lib64/libkrb5support.so.0 krb5-libs /lib64/liblz4.so.1 lz4-libs /lib64/liblzma.so.5 xz-libs /lib64/libm.so.6 glibc /lib64/libpam.so.0 pam-libs /lib64/libpcre2-8.so.0 pcre2 /lib64/libresolv.so.2 glibc /lib64/libselinux.so.1 libselinux /lib64/libsystemd.so.0 systemd-libs /lib64/libz.so.1 zlib / zlib-ng /lib64/libzstd.so.1 zstd
Should we have a higher level of attention to these packages? We already have "critical path", but that's a broad category now. These seem like they are "security path" packages, an intentionally small subset associated with very secure services which are enabled by default.
These are just my thoughts on a Saturday morning. Feedback welcome of course.
Rich.
(3) We should have a "security path", like "critical path".
sshd is linked to a lot of libraries:
/lib64/libaudit.so.1 audit-libs /lib64/libc.so.6 glibc /lib64/libcap-ng.so.0 libcap-ng /lib64/libcap.so.2 libcap /lib64/libcom_err.so.2 libcom_err /lib64/libcrypt.so.2 libxcrypt /lib64/libcrypto.so.3 openssl-libs /lib64/libeconf.so.0 libeconf /lib64/libgcc_s.so.1 libgcc /lib64/libgssapi_krb5.so.2 krb5-libs /lib64/libk5crypto.so.3 krb5-libs /lib64/libkeyutils.so.1 keyutils-libs /lib64/libkrb5.so.3 krb5-libs /lib64/libkrb5support.so.0 krb5-libs /lib64/liblz4.so.1 lz4-libs /lib64/liblzma.so.5 xz-libs /lib64/libm.so.6 glibc /lib64/libpam.so.0 pam-libs /lib64/libpcre2-8.so.0 pcre2 /lib64/libresolv.so.2 glibc /lib64/libselinux.so.1 libselinux /lib64/libsystemd.so.0 systemd-libs /lib64/libz.so.1 zlib / zlib-ng /lib64/libzstd.so.1 zstd
Should we have a higher level of attention to these packages? We already have "critical path", but that's a broad category now. These seem like they are "security path" packages, an intentionally small subset associated with very secure services which are enabled by default.
I agree, but that brings us to the question of what to do about them that's special.
Unrelated to the idea that some packages are special in this way, it's probably worth writing some static analysis tools we could put into rpm-inspect to detect when (a) a binary grows new public keys it didn't have before, and (b) a shared object grows a new ifunc. The latter is dramatically easier, of course, but both of those should be pretty rare events, so they're worth further inspection.
Even if it's just RSA keys that we search for, that would add some benefit, and that's pretty easy if nobody has tried to cover their tracks: scan a binary for a big power of two sized odd number followed by a small prime number, and then filtering that with a more rigorous prime test on the first number will detect RSA keys and probably very little else. Might be worth grepping for "----- BEGIN" as well.
Just some thoughts, I'm sure we'll all have many more where these come from.
On Mon, Apr 01, 2024 at 01:36:48PM -0400, Peter Jones wrote:
Unrelated to the idea that some packages are special in this way, it's probably worth writing some static analysis tools we could put into rpm-inspect to detect when (a) a binary grows new public keys it didn't have before, and (b) a shared object grows a new ifunc. The latter is dramatically easier, of course, but both of those should be pretty rare events, so they're worth further inspection.
I don't see much difference between ifuncs and any other library constructors from the exploit POV, at least with DT_BIND_NOW both is just some extra code run during library initialization. Sure, ifunc handlers affect what ifunc target is later called whenever calling the ifunc function, but harm can be done already when loading the library or the library constructor could overwrite function pointers, vtable pointers or just some data in the library to change its behavior later. So, in addition to watching for new ifuncs (and more importantly, trying to figure out if it is possible to statically prove the set of possible functions it will resolve to and compare to the old set; and if it isn't possible to statically figure out list of possible targets flag it for more careful inspection) we should watch for additions of __attribute__((constructor)) code or C++ namespace scope non-trivial ctors or dynamic initializers of namespace scope variables.
Jakub
On Saturday, 30 March 2024 10:37:44 CEST Richard W.M. Jones wrote:
These are just my thoughts on a Saturday morning. Feedback welcome of course.
I find the use of the ifunc attribute is really uncommon at this place. I would expect it in ffmpeg or some media codecs. In xz it looks like it is only there to hook in the payload. The software I know normally uses target cloning.
I think the use of the ifunc attribute should be a red flag. Can't we check for it with rpmlint and let the security team verify it?
Best regards
Andreas
On Tue, Apr 02, 2024 at 07:40:33AM +0200, Andreas Schneider wrote:
On Saturday, 30 March 2024 10:37:44 CEST Richard W.M. Jones wrote:
These are just my thoughts on a Saturday morning. Feedback welcome of course.
I find the use of the ifunc attribute is really uncommon at this place. I would expect it in ffmpeg or some media codecs. In xz it looks like it is only there to hook in the payload. The software I know normally uses target cloning.
In hindsight it's suspicious, but it's not generally suspicious for a project that needs to generate optimal code for different sub-architectures (eg. something that does fast decompression) to use the mechanism for that purpose, ifunc.
That said, ifunc is a very complicated, fragile but powerful mechanism and I'd like to know from the glibc developers what we should look out for. For example:
- Is it ever valid for ifunc to take control of functions in another library? Can this be detected by ld.so?
- Can some wrappers be developed to make it both easier and safer?
I think the use of the ifunc attribute should be a red flag. Can't we check for it with rpmlint and let the security team verify it?
Rich.
* Richard W. M. Jones:
That said, ifunc is a very complicated, fragile but powerful mechanism and I'd like to know from the glibc developers what we should look out for. For example:
- Is it ever valid for ifunc to take control of functions in another library? Can this be detected by ld.so?
I'm not sure what you mean by “take control”. The IFUNC resolver returns an address that is used as the symbol value, for the specific symbol the IFUNC resolver was associated with at build time. It's not expected to write to ld.so? data structures directly.
I have patches which make key ld.so? data structures read-only, but they still have to be read-write during relocation processing, when IFUNC resolvers run. Under the defend-against-application-bugs model, that's a reasonable trade-off because IFUNC resolvers are unlikely to have vulnerabilities (currently, they cannot even access environment variables reliably). Of course, it doesn't work for IFUNC resolvers written with malicious intent. I don't think there is a reasonable possibility to defend against that because any ld.so? defense we create is out there for everyone to see, so they can work around that.
- Can some wrappers be developed to make it both easier and safer?
GCC already provides function multi-versioning/target clones as a higher-level interface.
Thanks, Florian
Am 02.04.24 um 10:22 schrieb Florian Weimer:
- Can some wrappers be developed to make it both easier and safer?
GCC already provides function multi-versioning/target clones as a higher-level interface.
Also, upstreams should by default properly mark their stuffs with restrictive visibilities.
While we are a few decades to change the defaults, that doesn't mean that one can't choose the better option. So, by default one should choose -fvisibility=hidden and mark the public API with __attribute__((visibility("protected"))) or, if they really want a function to be interpositionable (by e.g. LD_PRELOAD) as __attribute__((visibility("default"))).
As a side effect, if you ever want your library be usable on Windows, you need to do that anyway since hidden is the default there and your public API must be marked explicitly. (Also, Windows doesn't support interposition and also doesn't support cyclic library dependencies without complicated hacks. So yeah, Windows kinda has the better defaults here.)
Some newer languages do that anyway already, but we obviously can't just change it for C and C++ projects.
But depending on the architecture this may not necessarily be possible. So yeah, only upstream can do that, not us.
Regards
Kilian Hanich
On Tue, Apr 02, 2024 at 05:09:18PM +0200, Kilian Hanich via devel wrote:
Am 02.04.24 um 10:22 schrieb Florian Weimer:
- Can some wrappers be developed to make it both easier and safer?
GCC already provides function multi-versioning/target clones as a higher-level interface.
Also, upstreams should by default properly mark their stuffs with restrictive visibilities.
While we are a few decades to change the defaults, that doesn't mean that one can't choose the better option. So, by default one should choose -fvisibility=hidden and mark the public API with __attribute__((visibility("protected"))) or, if they really want a function to be interpositionable (by e.g. LD_PRELOAD) as __attribute__((visibility("default"))).
ISTR this also makes the library faster (faster loading I think?)
Anyway we've done it for all the virt libraries for years.
Rich.
As a side effect, if you ever want your library be usable on Windows, you need to do that anyway since hidden is the default there and your public API must be marked explicitly. (Also, Windows doesn't support interposition and also doesn't support cyclic library dependencies without complicated hacks. So yeah, Windows kinda has the better defaults here.)
Some newer languages do that anyway already, but we obviously can't just change it for C and C++ projects.
But depending on the architecture this may not necessarily be possible. So yeah, only upstream can do that, not us.
Regards
Kilian Hanich
devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
* Kilian Hanich via devel:
Am 02.04.24 um 10:22 schrieb Florian Weimer:
- Can some wrappers be developed to make it both easier and safer?
GCC already provides function multi-versioning/target clones as a higher-level interface.
Also, upstreams should by default properly mark their stuffs with restrictive visibilities.
While we are a few decades to change the defaults, that doesn't mean that one can't choose the better option. So, by default one should choose -fvisibility=hidden and mark the public API with __attribute__((visibility("protected"))) or, if they really want a function to be interpositionable (by e.g. LD_PRELOAD) as __attribute__((visibility("default"))).
I think protected visibility is still broken on x86-64. For function symbols, it's more convenient to use -fno-semantic-interposition and rely on LTO to do the heavy lifting. For data symbols, it may be a better longer term investment to add explicit export lists (maybe even with symbol versioning), and again rely on LTO to make everything else hidden.
Thanks, Florian
On Tuesday, 2 April 2024 09:52:38 CEST Richard W.M. Jones wrote:
On Tue, Apr 02, 2024 at 07:40:33AM +0200, Andreas Schneider wrote:
On Saturday, 30 March 2024 10:37:44 CEST Richard W.M. Jones wrote:
These are just my thoughts on a Saturday morning. Feedback welcome of course.
I find the use of the ifunc attribute is really uncommon at this place. I would expect it in ffmpeg or some media codecs. In xz it looks like it is only there to hook in the payload. The software I know normally uses target cloning.
In hindsight it's suspicious, but it's not generally suspicious for a project that needs to generate optimal code for different sub-architectures (eg. something that does fast decompression) to use the mechanism for that purpose, ifunc.
That said, ifunc is a very complicated, fragile but powerful mechanism and I'd like to know from the glibc developers what we should look out for. For example:
Is it ever valid for ifunc to take control of functions in another library? Can this be detected by ld.so?
Can some wrappers be developed to make it both easier and safer?
Well, if it would do that. I took a quick look at xz and didn't see any specific code for an architecture flavor like x86_64-v3 or avx related. It lacks the implementation for that. All it did was adding the infrastructure without using it. I guess that the use of ifunc would is still be very rare.
Target clones is what you normally see.
* Richard W. M. Jones:
I'm not pretending these will solve everything, but they should make attacks a little harder in future.
(1) We should routinely delete autoconf-generated cruft from upstream projects and regenerate it in %prep. It is easier to study the real source rather than dig through the convoluted, generated shell script in an upstream './configure' looking for back doors.
For most projects, just running "autoreconf -fiv" is enough.
Yes, there are some projects that depend on a specific or old version of autoconf. We should fix those. But that doesn't need to delay us from using autoreconf on many projects today.
Not shipping the m4 files and other artifacts required for regenerating autoconf scripts is not exactly rare, unfortunately. I have filed a bunch of bugs because it's my understanding that this incomplete source code is against Fedora policies, but in the end, there isn't much we can do about it.
But I sympathize with this approach, we should build from sources as much as we can. Maybe not regenerate everything in %prep though, this really belongs into %build. It's invoking a compiler, after all.
In the xz case this wouldn't have been enough, it turns out we would also have to delete m4/build-to-host.m4, which then autoreconf regenerates. I don't fully understand why that is.
I would expect that's what the serial number is for? But that's just a guess.
(2) We should discourage gnulib as much as possible.
Or use Git sources without bundled gnulib, and re-bundled at build time using gnulib-devel. It would make gnulib updates easier. Such updates are sometimes required because gnulib overrides glibc internals without much coordination, and gnulib-using software may fail to build (or run) until gnulib is updated to reflect the internal glibc changes.
However, while I don't particularly like autoconf and gnulib (they are mainly there to support systems other than GNU/Linux, most of them proprietary), I don't expect any steps we take there to stop anyone who targets Fedora specifically. Being different from other distributions doesn't help at all if Fedora is the target.
(3) We should have a "security path", like "critical path".
I think there is a strong overlap with a bootstrap package set that is occasionally discussed. And this leads to the main point I want to make:
Stopping an attack that has already happened does not make much sense. Given that we work in the open, preventing future attacks that target Fedora specifically is likely impossible. Therefore, the focus should be on builder and buildroot integrity, and generic strategies for restoring buildroot integrity after an incident. In this particular case, so far we considered it sufficient to merely downgrade one package. But the package in question was part of the default buildroot. That means that in theory, it could have persisted itself outside the confines of its own package.
sshd is linked to a lot of libraries:
/lib64/libaudit.so.1 audit-libs /lib64/libc.so.6 glibc /lib64/libcap-ng.so.0 libcap-ng /lib64/libcap.so.2 libcap /lib64/libcom_err.so.2 libcom_err /lib64/libcrypt.so.2 libxcrypt /lib64/libcrypto.so.3 openssl-libs /lib64/libeconf.so.0 libeconf /lib64/libgcc_s.so.1 libgcc /lib64/libgssapi_krb5.so.2 krb5-libs /lib64/libk5crypto.so.3 krb5-libs /lib64/libkeyutils.so.1 keyutils-libs /lib64/libkrb5.so.3 krb5-libs /lib64/libkrb5support.so.0 krb5-libs /lib64/liblz4.so.1 lz4-libs /lib64/liblzma.so.5 xz-libs /lib64/libm.so.6 glibc /lib64/libpam.so.0 pam-libs /lib64/libpcre2-8.so.0 pcre2 /lib64/libresolv.so.2 glibc /lib64/libselinux.so.1 libselinux /lib64/libsystemd.so.0 systemd-libs /lib64/libz.so.1 zlib / zlib-ng /lib64/libzstd.so.1 zstd
Should we have a higher level of attention to these packages? We already have "critical path", but that's a broad category now. These seem like they are "security path" packages, an intentionally small subset associated with very secure services which are enabled by default.
The bootstrap set depends on SWIG, and SWIG pulls in language implementations which are not bootstrapped from source using the previous version.
Thanks, Florian
On Tue, Apr 2, 2024 at 4:59 AM Florian Weimer fweimer@redhat.com wrote:
- Richard W. M. Jones:
I'm not pretending these will solve everything, but they should make attacks a little harder in future.
(1) We should routinely delete autoconf-generated cruft from upstream projects and regenerate it in %prep. It is easier to study the real source rather than dig through the convoluted, generated shell script in an upstream './configure' looking for back doors.
For most projects, just running "autoreconf -fiv" is enough.
Yes, there are some projects that depend on a specific or old version of autoconf. We should fix those. But that doesn't need to delay us from using autoreconf on many projects today.
Not shipping the m4 files and other artifacts required for regenerating autoconf scripts is not exactly rare, unfortunately. I have filed a bunch of bugs because it's my understanding that this incomplete source code is against Fedora policies, but in the end, there isn't much we can do about it.
But I sympathize with this approach, we should build from sources as much as we can. Maybe not regenerate everything in %prep though, this really belongs into %build. It's invoking a compiler, after all.
We have a %conf stage for this purpose. We should start using it.
On Tue, Apr 02, 2024 at 10:59:10AM +0200, Florian Weimer wrote:
- Richard W. M. Jones:
In the xz case this wouldn't have been enough, it turns out we would also have to delete m4/build-to-host.m4, which then autoreconf regenerates. I don't fully understand why that is.
I would expect that's what the serial number is for? But that's just a guess.
Yes, in this case the attacker had set the serial number to 30, but the latest upstream serial number was 3. autoreconf won't replace the file in this case unless it is deleted. There really should be an "always replace if you can" option in autoreconf.
Rich.
Richard W.M. Jones wrote:
Yes, in this case the attacker had set the serial number to 30, but the latest upstream serial number was 3. autoreconf won't replace the file in this case unless it is deleted. There really should be an "always replace if you can" option in autoreconf.
Is that not what -f is supposed to do? At least, the documentation claims so, but the implementation does not actually do what is documented.
Kevin Kofler
On Wed, Apr 03, 2024 at 12:47:39AM +0200, Kevin Kofler via devel wrote:
Richard W.M. Jones wrote:
Yes, in this case the attacker had set the serial number to 30, but the latest upstream serial number was 3. autoreconf won't replace the file in this case unless it is deleted. There really should be an "always replace if you can" option in autoreconf.
Is that not what -f is supposed to do? At least, the documentation claims so, but the implementation does not actually do what is documented.
The upstream autoconf discussion says that 'autoreconf -fi' behavior on which 'serial NN' .m4 files to update is determined by automake, not autoconf, in part inspired by semantics desired in gnulib. And the automake and gnulib developers have argued that the upstream behavior is intentional:
https://lists.gnu.org/archive/html/bug-autoconf/2024-04/msg00005.html
So the only sane way to work around the upstream behavior (if we insist that working around it is essential, such as if we insist on running autoreconf in order to pull in modernized config.guess that will allow us to build on more platforms than were present at the time the upstream package was cut), is to do something like removing all .m4 files, then autoreconf -fi to reinstall all of them with known versions of the same .m4 files that we somehow curate as being more reliable than the ones that upstream tested their release tarball with. Lots of busy work, but if we think it will help as another layer of reassurance to keep our customers happy, then we can sign up for it.
However, although it is unlikely that upstream automake will accept a patch to change the existing 'autoreconf -fi' behavior of not overriding an already-present .m4 with a higher serial, getting some other patch in that adds a new option such as 'autoreconf --ignore-serial -fi' might be possible.
Eric Blake wrote:
The upstream autoconf discussion says that 'autoreconf -fi' behavior on which 'serial NN' .m4 files to update is determined by automake, not autoconf, in part inspired by semantics desired in gnulib. And the automake and gnulib developers have argued that the upstream behavior is intentional:
https://lists.gnu.org/archive/html/bug-autoconf/2024-04/msg00005.html
Don't you love intentional bugs? Yet another reason to avoid autotools at all costs.
By the way, the documentation of the serials "feature" actually warns about this (and seems to imply that even without serials, --force does not work as advertised for those files): https://www.gnu.org/software/automake/manual/html_node/Serials.html
| Finally, note that the --force option of aclocal has absolutely no effect | on the files installed by --install. For instance, if you have modified | your local macros, do not expect --install --force to replace the local | macros by their system-wide versions. If you want to do so, simply erase | the local macros you want to revert, and run ‘aclocal --install’.
But the documentation of "autoreconf --force" does not include that warning. And this also makes "--force" pretty much useless as it stands.
We and Debian both need to patch aclocal downstream immediately to make --force actually work. And then of course Fedora needs to actually always run autoreconf -i -f as Debian already does, or the patch will not do much good.
Kevin Kofler
On Wed, Apr 03, 2024 at 07:48:03PM +0200, Kevin Kofler via devel wrote:
Eric Blake wrote:
The upstream autoconf discussion says that 'autoreconf -fi' behavior on which 'serial NN' .m4 files to update is determined by automake, not autoconf, in part inspired by semantics desired in gnulib. And the automake and gnulib developers have argued that the upstream behavior is intentional:
https://lists.gnu.org/archive/html/bug-autoconf/2024-04/msg00005.html
Don't you love intentional bugs? Yet another reason to avoid autotools at all costs.
By the way, the documentation of the serials "feature" actually warns about this (and seems to imply that even without serials, --force does not work as advertised for those files): https://www.gnu.org/software/automake/manual/html_node/Serials.html
Note - that's in the automake manual (it is the actions taken by aclocal...).
| Finally, note that the --force option of aclocal has absolutely no effect | on the files installed by --install. For instance, if you have modified | your local macros, do not expect --install --force to replace the local | macros by their system-wide versions. If you want to do so, simply erase | the local macros you want to revert, and run ‘aclocal --install’.
But the documentation of "autoreconf --force" does not include that warning. And this also makes "--force" pretty much useless as it stands.
...and autoreconf is shipped by autoconf. While the two projects are related, they are independent (you can use autoconf without automake, at which point autoreconf --force does update everything that autoconf touched, since automake's semantics aren't getting in the way; it only invokes aclocal when it is obvious that automake was also in use). But I'm sure (speaking as one of the previous upstream autoconf maintainers, but only as a very infrequent automake patch poster) that a patch to autoconf's docs to mention the automake pitfalls inherited from 'aclocal --force' into 'autoreconf --force' is worthwhile.
We and Debian both need to patch aclocal downstream immediately to make --force actually work. And then of course Fedora needs to actually always run autoreconf -i -f as Debian already does, or the patch will not do much good.
Downstream is certainly entitled to have a different idea of best practices than upstream, especially when downstream is targetting just one OS, rather than the world of machines out there that the autotools originally designed for. That's one of the joys of open source - you don't have to live with upstream decisions.
-
Adam Williamson -
Alexander Bokovoy -
Andreas Schneider -
Artem S. Tashkinov -
Arthur Bols -
Ben Beasley -
Carlos Rodriguez Fernandez -
Carlos Rodriguez-Fernandez
-
Chris Adams -
Christoph Erhardt -
Christoph Karl -
Daniel Alley -
Daniel P. Berrangé -
Dmitry Belyavskiy -
Dominique Martinet -
Eric Blake -
Florian Weimer -
François Rigault -
Gabriel Somlo -
Gary Buhrmaster -
Gordon Messmer -
Jakub Jelinek -
Jonathan Steffan -
Kevin Fenzi -
Kevin Kofler -
Kilian Hanich -
Lennart Poettering -
Marcin Juszkiewicz -
Matthew Miller -
Michael Catanzaro -
Michel Lind -
Miroslav Suchý -
Neal Gompa -
Panu Matilainen -
Peter Jones -
Richard W.M. Jones -
Sam Varshavchik -
Scott Schmit -
Simo Sorce -
Simon de Vlieger -
Stephen Gallagher -
Stephen Smoogen -
Steve Cossette -
Steve Grubb -
Tim Landscheidt -
Todd Zullinger -
Vít Ondruch -
Zbigniew Jędrzejewski-Szmek