Hi,
wow: https://www.openwall.com/lists/oss-security/2024/
I think at this point we clearly cannot trust xz upstream anymore and should probably fork the project.
Kevin Kofler
That would be a bit premature. At this point it looks like one bad actor, and the other maintainer probably wasn't even aware. We should wait and see how this plays out.
On Fri, Mar 29, 2024 at 1:01 PM Kevin Kofler via devel < devel@lists.fedoraproject.org> wrote:
Hi,
wow: https://www.openwall.com/lists/oss-security/2024/
I think at this point we clearly cannot trust xz upstream anymore and should probably fork the project.
Kevin Kofler-- _______________________________________________ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
On Fri, Mar 29, 2024 at 07:00:37PM +0100, Kevin Kofler via devel wrote:
Hi,
wow: https://www.openwall.com/lists/oss-security/2024/
I think at this point we clearly cannot trust xz upstream anymore and should probably fork the project.
I kind of agree here, though it saddens me to say it. Any commit or release by "Jia Tan" or "Hans Jansen" [1] is suspect until proven otherwise, and those go back 2 or more years.
Rich.
[1] Putting quotes here because those are almost certainly not real peoples' names.
On 3/29/24 11:00, Kevin Kofler via devel wrote:
Specifically: https://www.openwall.com/lists/oss-security/2024/03/29/4
I think at this point we clearly cannot trust xz upstream anymore and should probably fork the project.
On Fri, Mar 29, 2024 at 2:08 PM Richard W.M. Jones rjones@redhat.com wrote:
On Fri, Mar 29, 2024 at 07:00:37PM +0100, Kevin Kofler via devel wrote:
Hi,
wow: https://www.openwall.com/lists/oss-security/2024/
I think at this point we clearly cannot trust xz upstream anymore and should probably fork the project.
I kind of agree here, though it saddens me to say it. Any commit or release by "Jia Tan" or "Hans Jansen" [1] is suspect until proven otherwise, and those go back 2 or more years.
I've been rolling in my head for a while now the idea of picking at things where we use gzip or xz to move to zstd where possible, given the benefits of the algorithm. This compromise has kind of raised the profile for me to seriously consider looking into it.
-- 真実はいつも一つ！/ Always, there's only one truth!
On Fri, Mar 29, 2024 at 12:08 PM Richard W.M. Jones rjones@redhat.com wrote:
On Fri, Mar 29, 2024 at 07:00:37PM +0100, Kevin Kofler via devel wrote:
Hi,
wow: https://www.openwall.com/lists/oss-security/2024/
I think at this point we clearly cannot trust xz upstream anymore and should probably fork the project.
I kind of agree here, though it saddens me to say it. Any commit or release by "Jia Tan" or "Hans Jansen" [1] is suspect until proven otherwise, and those go back 2 or more years.
Rich.
[1] Putting quotes here because those are almost certainly not real peoples' names.
That github user has also committed to libarchive, although not since November 2021.
Has this shipped on f40 beta?
Barry
On 29 Mar 2024, at 18:08, Richard W.M. Jones rjones@redhat.com wrote:
On Fri, Mar 29, 2024 at 07:00:37PM +0100, Kevin Kofler via devel wrote: Hi,
wow: https://www.openwall.com/lists/oss-security/2024/
I think at this point we clearly cannot trust xz upstream anymore and should probably fork the project.
I kind of agree here, though it saddens me to say it. Any commit or release by "Jia Tan" or "Hans Jansen" [1] is suspect until proven otherwise, and those go back 2 or more years.
Rich.
[1] Putting quotes here because those are almost certainly not real peoples' names.
-- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com virt-top is 'top' for virtual machines. Tiny program with many powerful monitoring features, net stats, disk stats, logging, etc. http://people.redhat.com/~rjones/virt-top -- _______________________________________________ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Hi,
I'm seeing weird things.
For whatever reason Source for xz was changed 2 months ago[1] to use GH releases instead of tukaani.org site.
The XZ page[2] has a note stating:
"Note: GitHub automatically includes two archives Source code (zip) and Source code (tar.gz) in the releases. These archives cannot be disabled and should be ignored."
And they wayback WayBackMachine[3] doesn't have previous versions.
Do we know if GH release tarballs are safe? @richard, do you remember why you had to change the source for the tarball?
Regards, Mikel
[1] https://src.fedoraproject.org/rpms/xz/c/0c09a6280b4a0c4fd7a9fc742c09469c95ff... [2] https://xz.tukaani.org/ [3] https://web.archive.org/web/20240119212251/https://xz.tukaani.org/
Hau idatzi du Kevin Kofler via devel (devel@lists.fedoraproject.org) erabiltzaileak (2024 mar. 29(a), or. (19:01)):
Hi,
wow: https://www.openwall.com/lists/oss-security/2024/
I think at this point we clearly cannot trust xz upstream anymore and should probably fork the project.
Kevin Kofler-- _______________________________________________ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Yes, F40 beta is affected, along with rawhide, but not F38/F39.
https://discussion.fedoraproject.org/t/warning-malicious-code-in-current-pre...
https://www.redhat.com/en/blog/urgent-security-alert-fedora-41-and-rawhide-u...
https://access.redhat.com/security/cve/CVE-2024-3094
https://www.linkedin.com/posts/fedora-project_urgent-security-alert-for-fedo...
It might be noted that the header of the RH article is wrong and refers to "F41 and rawhide", whereas the RH article content is correct and refers to "F40 and rawhide". Other sources, including the publication of Fedora Project (e.g., on linkedin), also refer to F40 and rawhide. However, the RH CVE article also refers to "F41 and rawhide".
Can someone from RH check and change the RH article header and the RH CVE page content to avoid confusion? I tend to assume that "F41 and rawhide" makes no sense at all since the two are currently equal.
Chris
On 29/03/2024 19.37, Barry wrote:
Has this shipped on f40 beta?
Barry
On 29 Mar 2024, at 18:08, Richard W.M. Jones rjones@redhat.com wrote:
On Fri, Mar 29, 2024 at 07:00:37PM +0100, Kevin Kofler via devel wrote: Hi,
wow: https://www.openwall.com/lists/oss-security/2024/
I think at this point we clearly cannot trust xz upstream anymore and should probably fork the project.
I kind of agree here, though it saddens me to say it. Any commit or release by "Jia Tan" or "Hans Jansen" [1] is suspect until proven otherwise, and those go back 2 or more years.
Rich.
[1] Putting quotes here because those are almost certainly not real peoples' names.
-- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com virt-top is 'top' for virtual machines. Tiny program with many powerful monitoring features, net stats, disk stats, logging, etc. http://people.redhat.com/~rjones/virt-top -- _______________________________________________ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
-- _______________________________________________ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
On Fri, Mar 29 2024 at 06:46:59 PM +00:00:00, Christopher Klooz py0xc3@posteo.net wrote:
Yes, F40 beta is affected, along with rawhide, but not F38/F39.
Unless I'm misunderstanding something, it looks xz-5.6.0-1.fc40 and 5.6.0-2.fc40 are backdoored, yes? Then rjones unknowingly broke the backdoor in two different ways in 5.6.0-3.fc40, (a) by adding the --disable-ifunc configure flag [1], and also (b) by running everything through autoreconf to regenerate the malicious autogoo files [2]. So F40 stable was never affected, but F40 updates-testing looks like it really was backdoored for about one week, between February 27 [3] and March 4 [4].
Hey Richard, if you agree with my quick assessment, then we should ask secalert@redhat.com to update the warning article [5]. (I also don't like the confusing references to "Fedora 41" in that article, since Fedora 41 does not yet exist as something separate from rawhide.)
[1] https://src.fedoraproject.org/rpms/xz/c/c837ae96c716c6d63da2b4a016e9034ade2a... [2] https://src.fedoraproject.org/rpms/xz/c/d2408dde878851ca6350297a738a72496a95... [3] https://bodhi.fedoraproject.org/updates/FEDORA-2024-a7fba89402 [4] https://bodhi.fedoraproject.org/updates/FEDORA-2024-f5033032b8 [5] https://access.redhat.com/security/cve/CVE-2024-3094
On Fri, Mar 29, 2024 at 02:40:48PM -0500, Michael Catanzaro wrote:
On Fri, Mar 29 2024 at 06:46:59 PM +00:00:00, Christopher Klooz py0xc3@posteo.net wrote:
Yes, F40 beta is affected, along with rawhide, but not F38/F39.
Unless I'm misunderstanding something, it looks xz-5.6.0-1.fc40 and 5.6.0-2.fc40 are backdoored, yes? Then rjones unknowingly broke the backdoor in two different ways in 5.6.0-3.fc40, (a) by adding the --disable-ifunc configure flag [1],
Yes.
and also (b) by running everything through autoreconf to regenerate the malicious autogoo files [2].
Sadly this on its own was not sufficient. You also have to delete m4/build-to-host.m4 first. But (a) was sufficient to prevent the backdoor on its own.
So F40 stable was never affected, but F40 updates-testing looks like it really was backdoored for about one week, between February 27 [3] and March 4 [4].
Hey Richard, if you agree with my quick assessment, then we should ask secalert@redhat.com to update the warning article [5]. (I also don't like the confusing references to "Fedora 41" in that article, since Fedora 41 does not yet exist as something separate from rawhide.)
secalert are already well aware and have approved the update. Kevin Fenzi, myself and others were working on it late last night :-(
Rich.
[1] https://src.fedoraproject.org/rpms/xz/c/c837ae96c716c6d63da2b4a016e9034ade2a... [2] https://src.fedoraproject.org/rpms/xz/c/d2408dde878851ca6350297a738a72496a95... [3] https://bodhi.fedoraproject.org/updates/FEDORA-2024-a7fba89402 [4] https://bodhi.fedoraproject.org/updates/FEDORA-2024-f5033032b8 [5] https://access.redhat.com/security/cve/CVE-2024-3094
On Fri, Mar 29, 2024 at 06:46:59PM +0000, Christopher Klooz wrote:
Yes, F40 beta is affected, along with rawhide, but not F38/F39.
https://discussion.fedoraproject.org/t/warning-malicious-code-in-current-pre...
https://www.redhat.com/en/blog/urgent-security-alert-fedora-41-and-rawhide-u...
https://access.redhat.com/security/cve/CVE-2024-3094
https://www.linkedin.com/posts/fedora-project_urgent-security-alert-for-fedo...
It might be noted that the header of the RH article is wrong and refers to "F41 and rawhide", whereas the RH article content is correct and refers to "F40 and rawhide". Other sources, including the publication of Fedora Project (e.g., on linkedin), also refer to F40 and rawhide. However, the RH CVE article also refers to "F41 and rawhide".
Can someone from RH check and change the RH article header and the RH CVE page content to avoid confusion? I tend to assume that "F41 and rawhide" makes no sense at all since the two are currently equal.
There was an F40 change that was vulnerable but it was in testing only briefly. After disabling ifuncs we (accidentally) were not vulnerable in F40. So the RH article is kind of correct.
I still recommend everyone updating to the Epoch: 1 package if they're on F40 or F41.
Also if you're on F41 and/or think you might have installed the vulnerable xz anywhere, note that the exploit has not been fully analyzed and no one really knows what it could do. I'm currently reinstalling a couple of machines from scratch and have regenerated my SSH keys.
Rich.
Chris
On 29/03/2024 19.37, Barry wrote:
Has this shipped on f40 beta?
Barry
On 29 Mar 2024, at 18:08, Richard W.M. Jones rjones@redhat.com wrote:
On Fri, Mar 29, 2024 at 07:00:37PM +0100, Kevin Kofler via devel wrote: Hi,
wow: https://www.openwall.com/lists/oss-security/2024/
I think at this point we clearly cannot trust xz upstream anymore and should probably fork the project.
I kind of agree here, though it saddens me to say it. Any commit or release by "Jia Tan" or "Hans Jansen" [1] is suspect until proven otherwise, and those go back 2 or more years.
Rich.
[1] Putting quotes here because those are almost certainly not real peoples' names.
-- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com virt-top is 'top' for virtual machines. Tiny program with many powerful monitoring features, net stats, disk stats, logging, etc. http://people.redhat.com/~rjones/virt-top -- _______________________________________________ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
-- _______________________________________________ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
-- _______________________________________________ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
On Fri, Mar 29 2024 at 07:56:49 PM +00:00:00, Richard W.M. Jones rjones@redhat.com wrote:
secalert are already well aware and have approved the update. Kevin Fenzi, myself and others were working on it late last night :-(
Sorry, I linked to the wrong article. I meant to link to [1] which says that "At this time the Fedora Linux 40 builds have not been shown to be compromised. We believe the malicious code injection did not take effect in these builds." But this statement contradicts my findings above, and you just replied "yes" to those, implying that my understanding is correct. So I guess either this blog post is wrong and needs to be updated, or you're wrong about me being right. Er, correct? :)
[1] https://www.redhat.com/en/blog/urgent-security-alert-fedora-41-and-rawhide-u...
It would be interesting to study how SELinux would have reacted to such kind of attack against sshd
On Fri, Mar 29 2024 at 07:44:12 PM +01:00:00, Mikel Olasagasti mikel@olasagasti.info wrote:
Do we know if GH release tarballs are safe?
The tarballs generated by GitHub that just include the contents of the git repo should be safe (at least from this particular issue), but the Fedora package is not built from those. It was built from the malicious release tarballs.
On Fri, Mar 29, 2024 at 03:01:34PM -0500, Michael Catanzaro wrote:
On Fri, Mar 29 2024 at 07:56:49 PM +00:00:00, Richard W.M. Jones rjones@redhat.com wrote:
secalert are already well aware and have approved the update. Kevin Fenzi, myself and others were working on it late last night :-(
Sorry, I linked to the wrong article. I meant to link to [1] which says that "At this time the Fedora Linux 40 builds have not been shown to be compromised. We believe the malicious code injection did not take effect in these builds." But this statement contradicts my findings above, and you just replied "yes" to those, implying that my understanding is correct. So I guess either this blog post is wrong and needs to be updated, or you're wrong about me being right. Er, correct? :)
[1] https://www.redhat.com/en/blog/urgent-security-alert-fedora-41-and-rawhide-u...
These are the exact builds which were vulnerable. Note the tags are all empty because Kevin untagged them last night, so you'll probably need to cross-reference these with bodhi updates.
xz-5.6.0-1.fc41 https://koji.fedoraproject.org/koji/buildinfo?buildID=2411083
xz-5.6.0-1.fc40 https://koji.fedoraproject.org/koji/buildinfo?buildID=2411092
xz-5.6.0-2.fc41 https://koji.fedoraproject.org/koji/buildinfo?buildID=2412686
xz-5.6.0-2.fc40 https://koji.fedoraproject.org/koji/buildinfo?buildID=2412698
xz-5.6.0-2.eln136 https://koji.fedoraproject.org/koji/buildinfo?buildID=2412908
xz-5.6.1-1.fc41 https://koji.fedoraproject.org/koji/buildinfo?buildID=2417414
xz-5.6.1-1.eln136 https://koji.fedoraproject.org/koji/buildinfo?buildID=2417425
NOT known to be vulnerable:
* xz-5.6.0-3.fc41 (because --disable-ifunc) * xz-5.6.0-3.fc40 (because --disable-ifunc) * anything < 5.6.0
You can also use the detection script "detect.sh" written by Vegard Nossum (https://www.openwall.com/lists/oss-security/2024/03/29/4)
Rich.
On Fri, Mar 29, 2024 at 07:44:12PM +0100, Mikel Olasagasti wrote:
Hi,
I'm seeing weird things.
For whatever reason Source for xz was changed 2 months ago[1] to use GH releases instead of tukaani.org site.
The XZ page[2] has a note stating:
"Note: GitHub automatically includes two archives Source code (zip) and Source code (tar.gz) in the releases. These archives cannot be disabled and should be ignored."
And they wayback WayBackMachine[3] doesn't have previous versions.
Do we know if GH release tarballs are safe? @richard, do you remember why you had to change the source for the tarball?
Sadly the release tarballs we used *do* contain the vulnerability. I checked myself that the payload is present in the final xz RPMs.
Rich.
Regards, Mikel
[1] https://src.fedoraproject.org/rpms/xz/c/0c09a6280b4a0c4fd7a9fc742c09469c95ff... [2] https://xz.tukaani.org/ [3] https://web.archive.org/web/20240119212251/https://xz.tukaani.org/
Hau idatzi du Kevin Kofler via devel (devel@lists.fedoraproject.org) erabiltzaileak (2024 mar. 29(a), or. (19:01)):
Hi,
wow: https://www.openwall.com/lists/oss-security/2024/
I think at this point we clearly cannot trust xz upstream anymore and should probably fork the project.
Kevin Kofler-- _______________________________________________ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
-- _______________________________________________ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Once upon a time, Richard W.M. Jones rjones@redhat.com said:
On Fri, Mar 29, 2024 at 07:44:12PM +0100, Mikel Olasagasti wrote:
Do we know if GH release tarballs are safe? @richard, do you remember why you had to change the source for the tarball?
Sadly the release tarballs we used *do* contain the vulnerability. I checked myself that the payload is present in the final xz RPMs.
I read that this did not go into the git history, so downloading a Github-generated tarball SHOULD be safe (note SHOULD: I did not personally check).
I guess a new security check when using release tarballs for projects with public git that also supports tarball generation would be to have both sources and compare. Signed sources don't help with the signer is the problem.
Mikel Olasagasti wrote:
For whatever reason Source for xz was changed 2 months ago[1] to use GH releases instead of tukaani.org site.
The public key jia_tan_pubkey.txt did not change at the same time. It was introduced on 2023-05-04 when the package was updated to version 5.4.3. Apparently the current tarballs on github.com and older tarballs on tukaani.org were signed with the same OpenPGP key.
Either the attacker has been preparing this for a long time, and is able to upload files to tukaani.org too, or else the attacker has compromised an honest developer and gained access to their secret OpenPGP key, their Github account, and probably all of their other credentials.
Björn Persson
Michael Catanzaro wrote:
On Fri, Mar 29 2024 at 07:44:12 PM +01:00:00, Mikel Olasagasti mikel@olasagasti.info wrote:
Do we know if GH release tarballs are safe?
The tarballs generated by GitHub that just include the contents of the git repo should be safe (at least from this particular issue),
So it is reported. The bulk of the attack code is in the Git repository, but the line that triggers it is only in the release tarballs, according to the report – but that means that the attacker is or was able to push commits to Github, so at this point it would be foolish to blindly trust the Git repository or the Github-generated tarballs.
Björn Persson
On 29/03/2024 21.01, Richard W.M. Jones wrote:
On Fri, Mar 29, 2024 at 06:46:59PM +0000, Christopher Klooz wrote:
Yes, F40 beta is affected, along with rawhide, but not F38/F39.
https://discussion.fedoraproject.org/t/warning-malicious-code-in-current-pre...
https://www.redhat.com/en/blog/urgent-security-alert-fedora-41-and-rawhide-u...
https://access.redhat.com/security/cve/CVE-2024-3094
https://www.linkedin.com/posts/fedora-project_urgent-security-alert-for-fedo...
It might be noted that the header of the RH article is wrong and refers to "F41 and rawhide", whereas the RH article content is correct and refers to "F40 and rawhide". Other sources, including the publication of Fedora Project (e.g., on linkedin), also refer to F40 and rawhide. However, the RH CVE article also refers to "F41 and rawhide".
Can someone from RH check and change the RH article header and the RH CVE page content to avoid confusion? I tend to assume that "F41 and rawhide" makes no sense at all since the two are currently equal.
There was an F40 change that was vulnerable but it was in testing only briefly. After disabling ifuncs we (accidentally) were not vulnerable in F40. So the RH article is kind of correct.
I still recommend everyone updating to the Epoch: 1 package if they're on F40 or F41.
Also if you're on F41 and/or think you might have installed the vulnerable xz anywhere, note that the exploit has not been fully analyzed and no one really knows what it could do. I'm currently reinstalling a couple of machines from scratch and have regenerated my SSH keys.
Rich.
Thanks for the clarifications and your quick responses to the issue!
However, the article could be still adjusted in some direction to avoid confusion, e.g.:
page header: "Urgent security alert for Fedora Linux 41 and Fedora Rawhide users " (-> 41) page headline: "Urgent security alert for Fedora Linux 40 and Fedora Rawhide users " (-> 40)
Not the most urgent problem at the moment of course, but maybe someone could adjust it at some time. As Michael already mentioned, the term "F41" can be on itself a little confusing at the moment.
Chris
On 29/03/2024 19.37, Barry wrote:
Has this shipped on f40 beta?
Barry
On 29 Mar 2024, at 18:08, Richard W.M. Jones rjones@redhat.com wrote:
On Fri, Mar 29, 2024 at 07:00:37PM +0100, Kevin Kofler via devel wrote: Hi,
wow: https://www.openwall.com/lists/oss-security/2024/
I think at this point we clearly cannot trust xz upstream anymore and should probably fork the project.
I kind of agree here, though it saddens me to say it. Any commit or release by "Jia Tan" or "Hans Jansen" [1] is suspect until proven otherwise, and those go back 2 or more years.
Rich.
[1] Putting quotes here because those are almost certainly not real peoples' names.
-- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com virt-top is 'top' for virtual machines. Tiny program with many powerful monitoring features, net stats, disk stats, logging, etc. http://people.redhat.com/~rjones/virt-top -- _______________________________________________ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
-- _______________________________________________ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
-- _______________________________________________ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
On Fri, Mar 29 2024 at 08:16:55 PM +00:00:00, Richard W.M. Jones rjones@redhat.com wrote:
These are the exact builds which were vulnerable. Note the tags are all empty because Kevin untagged them last night, so you'll probably need to cross-reference these with bodhi updates.
OK, I am going to ask Product Security to edit their blog post to remove the incorrect information. I will CC you on that request.
Thanks,
Michael
Please add “Fedora ELN” as well. We have updates ready that should be composed by midnight tonight, but it should be mentioned in the announcements.
On Fri, Mar 29, 2024 at 5:11 PM Michael Catanzaro mcatanzaro@redhat.com wrote:
On Fri, Mar 29 2024 at 08:16:55 PM +00:00:00, Richard W.M. Jones rjones@redhat.com wrote:
These are the exact builds which were vulnerable. Note the tags are all empty because Kevin untagged them last night, so you'll probably need to cross-reference these with bodhi updates.
OK, I am going to ask Product Security to edit their blog post to remove the incorrect information. I will CC you on that request.
Thanks,
Michael
-- _______________________________________________ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
On Fri, Mar 29 2024 at 04:10:53 PM -05:00:00, Michael Catanzaro mcatanzaro@redhat.com wrote:
OK, I am going to ask Product Security to edit their blog post to remove the incorrect information. I will CC you on that request.
Or maybe I should rephrase this as a "request for clarification," because maybe they know something that we don't. E.g. the Ars article [1] says
"The build environment on Fedora 40, for example, contains incompatibilities that prevent the injection from correctly occurring. Fedora 40 has now reverted to the 5.4.x versions of xz Utils."
[1] https://arstechnica.com/security/2024/03/backdoor-found-in-widely-used-linux...
Now, that's a secondary source, and I'm not confident if it is true, but perhaps Product Security had time to analyze the build logs before publishing and found something that we don't know about. Richard, what do you think?
On Fri, Mar 29, 2024 at 04:46:54PM -0500, Michael Catanzaro wrote:
On Fri, Mar 29 2024 at 04:10:53 PM -05:00:00, Michael Catanzaro mcatanzaro@redhat.com wrote:
OK, I am going to ask Product Security to edit their blog post to remove the incorrect information. I will CC you on that request.
Or maybe I should rephrase this as a "request for clarification," because maybe they know something that we don't. E.g. the Ars article [1] says
"The build environment on Fedora 40, for example, contains incompatibilities that prevent the injection from correctly occurring. Fedora 40 has now reverted to the 5.4.x versions of xz Utils."
[1] https://arstechnica.com/security/2024/03/backdoor-found-in-widely-used-linux...
Yeah that's just a confused report. This is how it actually happened:
(1) We built 5.6.0 for Fedora 40 & 41. Jia Tan was very insistent in emails that we should update.
(2) We got reports later of a valgrind test failure. I also saw it myself in my own projects that use liblzma. We notified Jia Tan of this.
(3) Since the valgrind failure pointed to something with ifuncs, using './configure --disable-ifuncs' was used to fix this in F40 & F41.
(4) xz 5.6.1 was released with a fix for the valgrind failure.
(5) Fedora 40 was now in beta so we kept 5.6.0 + --disable-ifuncs. Fedora 41 was updated to 5.6.1 (enabling ifuncs again).
And now with the benefit of hindsight ...
In step (1) we worked in good faith with upstream. Given how obfuscated the injection is, it's very unlikely we would have found the problem even if we'd spent days inspecting the tarball. (And the initial step of injection is *not* in git, so forget about reviewing git commits.)
The valgrind failure (2) was caused by a bug in the back door.
Disabling ifuncs in (3) disabled the back door, because I think it relies on ifuncs to do its malware, but in any case the obfuscated injection script explicitly skips injection if ifuncs are disabled.
Step (4) fixed the back door valgrind failure.
The Fedora 40 beta freeze in (5) meant we got lucky for F40, not so much for F41.
Rich.
Now, that's a secondary source, and I'm not confident if it is true, but perhaps Product Security had time to analyze the build logs before publishing and found something that we don't know about. Richard, what do you think?
There is a chance Fedora is not affected according to the following analysis:
https://gist.github.com/thesamesam/223949d5a074ebc3dce9ee78baad9e27
Quoting: ===== If those conditions check, the payload is injected into the source tree. We have not analyzed this payload in detail. Here are the main things we know:
The payload only activates if the running program has the process name /usr/bin/sshd. This means that systems that put sshd in /usr/sbin or another folder are not vulnerable. This further suspects targeting systemd systems due to their usrmerge initiative putting all binaries in /usr/bin. =====
We have the patch from https://github.com/openssh/openssh-portable/pull/375 applied, BTW.
On Fri, Mar 29, 2024 at 10:59 PM Richard W.M. Jones rjones@redhat.com wrote:
On Fri, Mar 29, 2024 at 04:46:54PM -0500, Michael Catanzaro wrote:
On Fri, Mar 29 2024 at 04:10:53 PM -05:00:00, Michael Catanzaro mcatanzaro@redhat.com wrote:
OK, I am going to ask Product Security to edit their blog post to remove the incorrect information. I will CC you on that request.
Or maybe I should rephrase this as a "request for clarification," because maybe they know something that we don't. E.g. the Ars article [1] says
"The build environment on Fedora 40, for example, contains incompatibilities that prevent the injection from correctly occurring. Fedora 40 has now reverted to the 5.4.x versions of xz Utils."
[1]
https://arstechnica.com/security/2024/03/backdoor-found-in-widely-used-linux...
Yeah that's just a confused report. This is how it actually happened:
(1) We built 5.6.0 for Fedora 40 & 41. Jia Tan was very insistent in emails that we should update.
(2) We got reports later of a valgrind test failure. I also saw it myself in my own projects that use liblzma. We notified Jia Tan of this.
(3) Since the valgrind failure pointed to something with ifuncs, using './configure --disable-ifuncs' was used to fix this in F40 & F41.
(4) xz 5.6.1 was released with a fix for the valgrind failure.
(5) Fedora 40 was now in beta so we kept 5.6.0 + --disable-ifuncs. Fedora 41 was updated to 5.6.1 (enabling ifuncs again).
And now with the benefit of hindsight ...
In step (1) we worked in good faith with upstream. Given how obfuscated the injection is, it's very unlikely we would have found the problem even if we'd spent days inspecting the tarball. (And the initial step of injection is *not* in git, so forget about reviewing git commits.)
The valgrind failure (2) was caused by a bug in the back door.
Disabling ifuncs in (3) disabled the back door, because I think it relies on ifuncs to do its malware, but in any case the obfuscated injection script explicitly skips injection if ifuncs are disabled.
Step (4) fixed the back door valgrind failure.
The Fedora 40 beta freeze in (5) meant we got lucky for F40, not so much for F41.
Rich.
Now, that's a secondary source, and I'm not confident if it is true, but perhaps Product Security had time to analyze the build logs before publishing and found something that we don't know about. Richard, what do you think?
-- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com nbdkit - Flexible, fast NBD server with plugins https://gitlab.com/nbdkit/nbdkit -- _______________________________________________ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Once upon a time, Richard W.M. Jones rjones@redhat.com said:
(1) We built 5.6.0 for Fedora 40 & 41. Jia Tan was very insistent in emails that we should update.
So this wasn't just a "hey, new upstream version", this was PUSHED on distributions by the culprit. Are they a contributor to any other software in the distribution? I think anything they might have touched has to be considered suspect.
Either (a) their systems have been completely compromised or (b) they did this intentionally. Neither is good.
(2) We got reports later of a valgrind test failure. I also saw it myself in my own projects that use liblzma. We notified Jia Tan of this.
Why does libsystemd pull in libzma (as well as liblz4 and libzstd, because we need three compression libraries in one place)? That seems to be a broad amount of extra code, for a library that's in a number of network-listening services is just linked for socket activation.
Also, while it appears there's more than one developer with Github commit access (one other made commits since the initial "bad" commit), it would seem they aren't doing reviews, so not sure how much xz/liblzma can be trusted to be linked into a whole lot of key programs.
More about this is now published on the Fedora Magazine as well in this statement: https://fedoramagazine.org/cve-2024-3094-security-alert-f40-rawhide/
Thank you to all of our Fedora first responders who stopped something that could have been much worse. We should feel proud here. As far as Fedora and our ecosystem is concerned, the exploit failed.
https://floss.social/@jwildeboer@social.wildeboer.net/112181976426765177
On Fri, Mar 29, 2024 at 2:01 PM Kevin Kofler via devel < devel@lists.fedoraproject.org> wrote:
Hi,
wow: https://www.openwall.com/lists/oss-security/2024/
I think at this point we clearly cannot trust xz upstream anymore and should probably fork the project.
Kevin Kofler-- _______________________________________________ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
This might be a good place to start
On Fri, 2024-03-29 at 15:01 -0500, Michael Catanzaro wrote:
On Fri, Mar 29 2024 at 07:56:49 PM +00:00:00, Richard W.M. Jones rjones@redhat.com wrote:
secalert are already well aware and have approved the update. Kevin Fenzi, myself and others were working on it late last night :-(
Sorry, I linked to the wrong article. I meant to link to [1] which says that "At this time the Fedora Linux 40 builds have not been shown to be compromised. We believe the malicious code injection did not take effect in these builds." But this statement contradicts my findings above, and you just replied "yes" to those, implying that my understanding is correct. So I guess either this blog post is wrong and needs to be updated, or you're wrong about me being right. Er, correct? :)
FWIW, I wrote that text, modified from a slightly different version in the earlier draft that was briefly published, and based on my best understanding at the time (which was that *no* build that reached F40 actually had a working version of the exploit).
If Richard says the exploit potentially worked in 5.6.0-2, then F40 potentially *was* vulnerable for some time, because 5.6.0-2 reached updates-testing. You can use `dnf history info xz` to check if you ever had the vulnerable version installed. I'll see if we can get the post tweaked again; it will be hard to word it with the appropriate level of accuracy and urgency and still be readable, but I'll try...
Oh, and we can't easily fix the URL of the blog post, apparently, because CMSes suck. It seems we're more less stuck with the "41" in that.
Thanks so much for the concerted effort and handling of this, this stuff isn't easy.
Would it be wise to revert to the last version that was signed by Lasse Colin instead or would the impact be too high?
On Fri, Mar 29, 2024 at 07:25:56PM -0500, Chris Adams wrote:
Once upon a time, Richard W.M. Jones rjones@redhat.com said:
(1) We built 5.6.0 for Fedora 40 & 41. Jia Tan was very insistent in emails that we should update.
So this wasn't just a "hey, new upstream version", this was PUSHED on distributions by the culprit.
I have to say that this is not unusual. I myself have even sent messages to other maintainers encouraging them to package or update projects that I've written. A keen upstream maintainer wanting to help or encourage downstream packagers is normally welcome. This is the one case and only case that someone turned out to be malicious (that we know about). They abused our trust.
Are they a contributor to any other software in the distribution? I think anything they might have touched has to be considered suspect.
Yes I agree that anything else touched by this "person" should be considered very suspect.
Either (a) their systems have been completely compromised or (b) they did this intentionally. Neither is good.
The back door is intentional for sure. We don't know the details of if this is an account take-over or a "long con" and what the background to all this is, but I'm sure people are looking into that.
(2) We got reports later of a valgrind test failure. I also saw it myself in my own projects that use liblzma. We notified Jia Tan of this.
Why does libsystemd pull in libzma (as well as liblz4 and libzstd, because we need three compression libraries in one place)? That seems to be a broad amount of extra code, for a library that's in a number of network-listening services is just linked for socket activation.
This is a very real issue. Got another email coming up in a bit about this.
Also, while it appears there's more than one developer with Github commit access (one other made commits since the initial "bad" commit), it would seem they aren't doing reviews, so not sure how much xz/liblzma can be trusted to be linked into a whole lot of key programs.
Rich.
perhaps of add'l interest
https://boehs.org/node/everything-i-know-about-the-xz-backdoor
On Sat, Mar 30, 2024 at 06:13:44AM -0400, pgnd wrote:
perhaps of add'l interest
[speculative link redacted]
I don't think speculation like that is helpful.
The authorities have the ability to subpoena the Google (gmail) account to find out what's really going on. Anything else risks an pile on which might have real world consequences for innocent people.
Rich.
I don't think speculation like that is helpful.
The authorities have the ability to subpoena the Google (gmail) account to find out what's really going on. Anything else risks an pile on which might have real world consequences for innocent people.
It's an additional summary 'data point' -- speculative, or otherwise. It's certainly not the only one popping up in simple search. Ignoring it isn't generally helpful either, especially if it *is* inaccurate.
If there are 'better' resources that summarize in somewhat-human-speak, and put appropriate question marks near/on possible affected packages, (e.g.) liblzma, libarchive, oss-fuzz, etc etc, please do share ...
On 29/03/2024 22.10, Michael Catanzaro wrote:
On Fri, Mar 29 2024 at 08:16:55 PM +00:00:00, Richard W.M. Jones rjones@redhat.com wrote:
These are the exact builds which were vulnerable. Note the tags are all empty because Kevin untagged them last night, so you'll probably need to cross-reference these with bodhi updates.
OK, I am going to ask Product Security to edit their blog post to remove the incorrect information. I will CC you on that request.
Thanks,
Michael
Confusion is increasing a little among different channels, and it would be nice if the RH blog post and the Red Hat CVE page would be updated, and maybe clarified: According to Adam Williamson, F40 is likely to have installed the packages because testing is enabled by default in pre-release. If I got Rich right, the malicious code is likely to be broken on F40, but F40 users still should update to be sure.
At the moment several "versions" and "assumptions" are rising that try to somehow make sense of the different publications (e.g., header of RH article "F41 and rawhide" -> headline in content "F40 and rawhide"). I don't know how the assumption came up that F40 is only affected if users opted in for testing, but that interpretation already ended up in the Fedora Magazine and in the official linkedin post of Fedora (I already asked to correct it).
Creating some clarification and unify our information provision can help to get rid of the current interpretations between "F40 - just don't care" and "F40 - the end of the world is coming" (sorry for the dramatization ;). I think one or two sentences in the RH blog post + RH CVE page should be fine to clarify, to avoid further confusion and to re-unify knowledge towards the facts, of course the same for the Fedora Magazine article but that's already underway.
On Sat, Mar 30 2024 at 12:26:48 PM +00:00:00, Christopher Klooz py0xc3@posteo.net wrote:
If I got Rich right, the malicious code is likely to be broken on F40,
No, that is not correct, as explained by [1] and [2]. We have already asked Red Hat to investigate and fix the blog post. This is still an evolving situation; apologies for the confusion as we sort this out.
[1] https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org/... [2] https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org/...
Then we must have had some communication snafu regarding the Fedora Magazine article, because multiple people including myself flagged the incorrect statement there before the article was published. Hopefully we can get one this fixed, too.
Michael
On Sat, Mar 30 2024 at 09:45:06 AM -05:00:00, Michael Catanzaro mcatanzaro@redhat.com wrote:
No, that is not correct, as explained by [1] and [2].
I pasted the wrong link for [2]. I meant to paste:
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org/...
On 30/03/2024 15.45, Michael Catanzaro wrote:
On Sat, Mar 30 2024 at 12:26:48 PM +00:00:00, Christopher Klooz py0xc3@posteo.net wrote:
If I got Rich right, the malicious code is likely to be broken on F40,
No, that is not correct, as explained by [1] and [2]. We have already asked Red Hat to investigate and fix the blog post. This is still an evolving situation; apologies for the confusion as we sort this out.
[1] https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org/... [2] https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org/...
Then we must have had some communication snafu regarding the Fedora Magazine article, because multiple people including myself flagged the incorrect statement there before the article was published. Hopefully we can get one this fixed, too.
Michael
Thanks for clarifying. I have already deleted the Fedora Magazine link from the related Fedora Discussion topic [1] earlier, and now updated the rest. I re-add the magazine article when it was fixed, I pinged Justin already some hours ago, hopefully it gets updated soon.
[1] https://discussion.fedoraproject.org/t/attention-malicious-code-in-current-p...
It appears that libsystemd links to libraries for lzma/xz, bzip2, gzip and also zstd, because some systemd utilities provide them as options in various different contexts (but not consistently, zstd for instance is seemingly supported by some utilities and not by others, and I see some code such as [0] that doesn't account for it)
I'm sure having all of those different options available is nice in some context or another, but how unrealistic would it be to pare that back to a few slightly more opinionated and consistent choices? Bzip2 for instance isn't particularly good on *any* metric, are there legacy / ecosystem reasons that are sufficiently important for libsystemd to be dragging it around? libsystemd linking 4 different compression libraries does seem a bit excessive (if it can be helped).
[0] https://github.com/systemd/systemd/blob/3799fa803efb04cdd1f1b239c6c64803fe85...
On Sat, Mar 30, 2024 at 05:59:36PM -0000, Daniel Alley wrote:
It appears that libsystemd links to libraries for lzma/xz, bzip2, gzip and also zstd, because some systemd utilities provide them as options in various different contexts (but not consistently, zstd for instance is seemingly supported by some utilities and not by others, and I see some code such as [0] that doesn't account for it)
I'm sure having all of those different options available is nice in some context or another, but how unrealistic would it be to pare that back to a few slightly more opinionated and consistent choices? Bzip2 for instance isn't particularly good on *any* metric, are there legacy / ecosystem reasons that are sufficiently important for libsystemd to be dragging it around? libsystemd linking 4 different compression libraries does seem a bit excessive (if it can be helped).
[0] https://github.com/systemd/systemd/blob/3799fa803efb04cdd1f1b239c6c64803fe85...
libsystemd is linked to compression libraries primarly because those compressors are options for journald files, and we generally want to mainain the property that all journal files from the past, as well as journal files from other systems, can be read by later journal code.
(bzip2 is an exception here. It is only used by systemd-importd and related tools, so libsystemd was never linked to it, IIRC.)
In recent systemd versions, dlopen is used for various compression libraries and other libraries, so they'll be opportunistically loaded if needed, effectively making those dependencies optional at runtime.
Conversions to use dlopen require a bit of boilerplate code and make the code less readable, so we only would to them if there was a reason. Usually, this reason was size and/or the dependency tree. Compression libraries are very small and have no dependencies, so it wasn't considered important to use dlopen for them. But now that there's a new motivation, as mentioned elsewhere in the thread, we'll load pretty much all dependencies of libsystemd via dlopen.
Zbyszek
On 30-03-2024 13:26, Christopher Klooz wrote:
I don't know how the assumption came up that F40 is only affected if users opted in for testing, but that interpretation already ended up in the Fedora Magazine and in the official linkedin post of Fedora (I already asked to correct it).
I believe that statement is correct, since none of the xz-5.6.x packages ever made it to F40 stable. The furthest they've got was updates-testing, which is not enabled in the official Beta releases. However, if you installed F40 before Beta was released, then updates-testing is enabled and users may have installed the vulnerable package with a simple `sudo dnf upgrade`.
I admit the wording could be clearer in that opting in to updates-testing might have been done on your behalf simply by installing F40 sometime between branching and the Beta release. Some users might not be aware of that.
It may also help providing some simple instructions on how users can check if they have any of the vulnerable versions installed in the article itself. I see a comment to that extent.
So, the situation around F40 is somewhat murky since a lot of factors come into play, but the statement that 5.6.x never made to F40 stable is correct[1] and therefore users not having updates-testing enabled could not have installed 5.6.x without expressly enabling it.
[1] https://bodhi.fedoraproject.org/updates/?search=xz-5.6
-- Sandro
regarding sabotaged Landlock sandbox https://mastodon.social/@danderson@hachyderm.io/112185746170709381
On 30/03/2024 20.08, Sandro wrote:
On 30-03-2024 13:26, Christopher Klooz wrote:
I don't know how the assumption came up that F40 is only affected if users opted in for testing, but that interpretation already ended up in the Fedora Magazine and in the official linkedin post of Fedora (I already asked to correct it).
I believe that statement is correct, since none of the xz-5.6.x packages ever made it to F40 stable. The furthest they've got was updates-testing, which is not enabled in the official Beta releases. However, if you installed F40 before Beta was released, then updates-testing is enabled and users may have installed the vulnerable package with a simple `sudo dnf upgrade`.
I admit the wording could be clearer in that opting in to updates-testing might have been done on your behalf simply by installing F40 sometime between branching and the Beta release. Some users might not be aware of that.
It may also help providing some simple instructions on how users can check if they have any of the vulnerable versions installed in the article itself. I see a comment to that extent.
So, the situation around F40 is somewhat murky since a lot of factors come into play, but the statement that 5.6.x never made to F40 stable is correct[1] and therefore users not having updates-testing enabled could not have installed 5.6.x without expressly enabling it.
I don't think this is right. Adam Williamson and Michael Catanzaro already confirmed that F40 has testing enabled by default because it is pre-release. It was also confirmed that some packages could have been installed on F40 variants (see also the points of Michael and Richard here in the devel mailing list). Michael and Adam also wrote some references in the Fedora Discussion topic [1] about this.
It is obviously still an issue that is evolving and what seems clear now might prove different later. But so far I tend to leave the discussion topic as it is and ensure that F40 users expect being compromised and get informed to act correspondingly with the suggested actions. However, I already added a point how users can check if they have a malicious build.
[1] https://discussion.fedoraproject.org/t/attention-malicious-code-in-current-p...
On 30-03-2024 22:10, Christopher Klooz wrote:
On 30/03/2024 20.08, Sandro wrote:
On 30-03-2024 13:26, Christopher Klooz wrote:
I don't know how the assumption came up that F40 is only affected if users opted in for testing, but that interpretation already ended up in the Fedora Magazine and in the official linkedin post of Fedora (I already asked to correct it).
I believe that statement is correct, since none of the xz-5.6.x packages ever made it to F40 stable. The furthest they've got was updates-testing, which is not enabled in the official Beta releases. However, if you installed F40 before Beta was released, then updates-testing is enabled and users may have installed the vulnerable package with a simple `sudo dnf upgrade`.
I admit the wording could be clearer in that opting in to updates-testing might have been done on your behalf simply by installing F40 sometime between branching and the Beta release. Some users might not be aware of that.
It may also help providing some simple instructions on how users can check if they have any of the vulnerable versions installed in the article itself. I see a comment to that extent.
So, the situation around F40 is somewhat murky since a lot of factors come into play, but the statement that 5.6.x never made to F40 stable is correct[1] and therefore users not having updates-testing enabled could not have installed 5.6.x without expressly enabling it.
I don't think this is right. Adam Williamson and Michael Catanzaro already confirmed that F40 has testing enabled by default because it is pre-release. It was also confirmed that some packages could have been installed on F40 variants (see also the points of Michael and Richard here in the devel mailing list). Michael and Adam also wrote some references in the Fedora Discussion topic [1] about this.
From what I understood, F40 Beta, the official Beta release, available from the website as of March 26, has updates-testing disabled by default. That was confirmed by several people in #devel yesterday when the Fedora Magazine article was still being worked on.
It's the RC composes that are made after branching and before Beta is declared GO, that have updates-testing enabled by default. I was one of the persons raising that point. I'm less certain wrt upgrades in the period between branching and Beta release.
If that is incorrect and Beta shipped with updates-testing enabled, deliberately or by accident, then I stand corrected.
It is obviously still an issue that is evolving and what seems clear now might prove different later. But so far I tend to leave the discussion topic as it is and ensure that F40 users expect being compromised and get informed to act correspondingly with the suggested actions. However, I already added a point how users can check if they have a malicious build.
I agree. Once the levees broke, news was traveling fast and, for some, panic may have set in, not helping in determining what information is accurate.
Advise to err on the side of caution, check your system and upgrade if unsure, is certainly what I would tell anyone. Even distros (Arch, Gentoo) where it turned out the payload wasn't injected, acted out of an abundance of caution, put out advisories and updates for their users.
What's written on Discussion looks to be covering the broad spectrum. Maybe the Fedora Magazine article could link to that post for further clarification.
[1] https://discussion.fedoraproject.org/t/attention-malicious-code-in-current-p...
-- Sandro
Il 30/03/24 23:12, Sandro ha scritto:
From what I understood, F40 Beta, the official Beta release, available from the website as of March 26, has updates-testing disabled by default. That was confirmed by several people in #devel yesterday when the Fedora Magazine article was still being worked on.
I updated ~3 days ago from F39 to F40 and I got *-testing repositories enabled
On Sat, Mar 30, 2024 at 11:12:02PM +0100, Sandro wrote:
From what I understood, F40 Beta, the official Beta release, available from the website as of March 26, has updates-testing disabled by default. That
Nope.
was confirmed by several people in #devel yesterday when the Fedora Magazine article was still being worked on.
I am pretty sure I said the opposite...
nirik: Branched enables updates-testing... so if you installed f40 anytime, you will have it enabled and if you then applied updates it would be in them nirik: yes, we disable updates-testing by default right before release.
I guess that could have been read as right before beta release, but thats not the case or what I meant. ;)
It's before _Final_ release that we disable updates-testing. It's enabled by default from when we branch the release off until the time right before release when we switch it (usually with a freeze break/blocker bug)
It's the RC composes that are made after branching and before Beta is declared GO, that have updates-testing enabled by default. I was one of the persons raising that point. I'm less certain wrt upgrades in the period between branching and Beta release.
I think the confusion here is "Beta Release" vs "Final release".
We enable updates-testing at branching time all the way until right before "Final release". :)
If that is incorrect and Beta shipped with updates-testing enabled, deliberately or by accident, then I stand corrected.
Yes, it did/does. :)
The logic is that most people who install betas or pre-releases want to help test updates. If for some reason they don't, they can disable it, but the default option is on.
It is obviously still an issue that is evolving and what seems clear now might prove different later. But so far I tend to leave the discussion topic as it is and ensure that F40 users expect being compromised and get informed to act correspondingly with the suggested actions. However, I already added a point how users can check if they have a malicious build.
I agree. Once the levees broke, news was traveling fast and, for some, panic may have set in, not helping in determining what information is accurate.
Advise to err on the side of caution, check your system and upgrade if unsure, is certainly what I would tell anyone. Even distros (Arch, Gentoo) where it turned out the payload wasn't injected, acted out of an abundance of caution, put out advisories and updates for their users.
What's written on Discussion looks to be covering the broad spectrum. Maybe the Fedora Magazine article could link to that post for further clarification.
Yeah, still lots to know about this...
kevin
I don't know how the assumption came up that F40 is only affected if users opted in for testing, but that interpretation already ended up in the Fedora Magazine and in the official linkedin post of Fedora (I already asked to correct it).
I believe that statement is correct, since none of the xz-5.6.x packages ever made it to F40 stable. The furthest they've got was updates-testing, which is not enabled in the official Beta releases.
The updates-testing repo isn't used in the Beta compose but it is enabled by default so if someone took the beta and then applied updates they would have automatically got the compromised package, there's nuance there.
However, if you installed F40 before Beta was released, then updates-testing is enabled and users may have installed the vulnerable package with a simple `sudo dnf upgrade`.
The same would be the case if they installed Beta.
On Sat, 2024-03-30 at 16:41 -0700, Kevin Fenzi wrote:
On Sat, Mar 30, 2024 at 11:12:02PM +0100, Sandro wrote:
From what I understood, F40 Beta, the official Beta release, available from the website as of March 26, has updates-testing disabled by default. That
Nope.
was confirmed by several people in #devel yesterday when the Fedora Magazine article was still being worked on.
I am pretty sure I said the opposite...
nirik: Branched enables updates-testing... so if you installed f40 anytime, you will have it enabled and if you then applied updates it would be in them nirik: yes, we disable updates-testing by default right before release.
I guess that could have been read as right before beta release, but thats not the case or what I meant. ;)
It's before _Final_ release that we disable updates-testing. It's enabled by default from when we branch the release off until the time right before release when we switch it (usually with a freeze break/blocker bug)
Yes.
https://dl.fedoraproject.org/pub/alt/stage/40_Beta-1.10/Everything/x86_64/os... contains fedora-repos-40-0.4.noarch.rpm . That is from fedora-repos-40- 0.4 . The Koji build for that is https://koji.fedoraproject.org/koji/buildinfo?buildID=2411685 . It records that the dist-git commit was b0be9579a9b45a44bc2e30fbd7d5ef4a268f05a2 . Here is the dist-git repo at that commit: https://src.fedoraproject.org/rpms/fedora-repos/tree/b0be9579a9b45a44bc2e30f... . Note this line: https://src.fedoraproject.org/rpms/fedora-repos/blob/b0be9579a9b45a44bc2e30f... . QED. updates-testing is enabled by default in Fedora 40 Beta. This is normal, intended, and expected.
On 31-03-2024 00:41, Kevin Fenzi wrote:
On Sat, Mar 30, 2024 at 11:12:02PM +0100, Sandro wrote:
From what I understood, F40 Beta, the official Beta release, available from the website as of March 26, has updates-testing disabled by default. That
Nope.
was confirmed by several people in #devel yesterday when the Fedora Magazine article was still being worked on.
I am pretty sure I said the opposite...
nirik: Branched enables updates-testing... so if you installed f40 anytime, you will have it enabled and if you then applied updates it would be in them nirik: yes, we disable updates-testing by default right before release.
I guess that could have been read as right before beta release, but thats not the case or what I meant. ;)
It's before _Final_ release that we disable updates-testing. It's enabled by default from when we branch the release off until the time right before release when we switch it (usually with a freeze break/blocker bug)
Thanks for clarifying that. Context matters and there was a lot going on in the channel at the time. My wording was to the extend covering composes before Beta was released. I used the the term "pre-Beta".
Good to know that the switch is made "pre-Final".
It's the RC composes that are made after branching and before Beta is declared GO, that have updates-testing enabled by default. I was one of the persons raising that point. I'm less certain wrt upgrades in the period between branching and Beta release.
I think the confusion here is "Beta Release" vs "Final release".
Yup! I was thinking "Beta Release".
We enable updates-testing at branching time all the way until right before "Final release". :)
If that is incorrect and Beta shipped with updates-testing enabled, deliberately or by accident, then I stand corrected.
Yes, it did/does. :)
The logic is that most people who install betas or pre-releases want to help test updates. If for some reason they don't, they can disable it, but the default option is on.
That makes sense. I thought along the same lines as to why it is enabled by default right after branching.
Thanks also to @adamwill and @pbrobinson for further clarification/confirmation later on.
The following sentence in the magazine article:
"Fedora Linux 40 Beta users may have received it from the testing repositories, which are enabled by default in branched versions (i.e. pre-Beta) to assist with testing."
should probably be amended in the light of above. Though, most people affected probably took measures already and new installations won't be affected.
-- Sandro
Kevin Fenzi wrote:
Branched enables updates-testing... so if you installed f40 anytime, you will have it enabled and if you then applied updates it would be in them
Yet another thing I always said was a bad idea, and this incident proves it. This would have been filtered before reaching most people if we made people only test what actually ends up in the composed Beta and Final images, i.e., updates that made it out to stable. In addition, having updates-testing enabled makes it unsafe to upgrade a Beta installation to Final because suddenly updates-testing gets disabled, but people still have packages from updates-testing (such as the backdoored xz, but also tons of untested packages or ones that explicitly failed testing) installed.
Kevin Kofler
On Sun, Mar 31, 2024 at 6:50 AM Kevin Kofler via devel devel@lists.fedoraproject.org wrote:
Kevin Fenzi wrote:
Branched enables updates-testing... so if you installed f40 anytime, you will have it enabled and if you then applied updates it would be in them
Yet another thing I always said was a bad idea, and this incident proves it. This would have been filtered before reaching most people if we made people only test what actually ends up in the composed Beta and Final images, i.e., updates that made it out to stable. In addition, having updates-testing enabled makes it unsafe to upgrade a Beta installation to Final because suddenly updates-testing gets disabled, but people still have packages from updates-testing (such as the backdoored xz, but also tons of untested packages or ones that explicitly failed testing) installed.
Well, an easy solution is to make it so "dnf update" is coerced to "dnf distro-sync" for development releases. Then it doesn't matter. We could make that happen for Fedora 41 with the DNF 5 transition (there's already code to make this possible with PackageKit with the current DNF backend, it needs to be migrated into DNF 5).
Disabling updates-testing is a bad plan, because we want updates more aggressively tested during the development cycle.
On 30/03/2024 15.45, Michael Catanzaro wrote:
On Sat, Mar 30 2024 at 12:26:48 PM +00:00:00, Christopher Klooz py0xc3@posteo.net wrote:
If I got Rich right, the malicious code is likely to be broken on F40,
No, that is not correct, as explained by [1] and [2]. We have already asked Red Hat to investigate and fix the blog post. This is still an evolving situation; apologies for the confusion as we sort this out.
[1] https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org/... [2] https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org/...
Then we must have had some communication snafu regarding the Fedora Magazine article, because multiple people including myself flagged the incorrect statement there before the article was published. Hopefully we can get one this fixed, too.
Michael
In case someone from the Fedora Magazine is in the devel mailing list and reads this:
I am not sure if this is intended, but the article on the magazine already spread the false information that "testing" is disabled by default on F40 (this was also spread on LinkedIn - both have been already re-distributed into several channels), and now it says in the first section "Fedora Linux 40 Beta users only using stable repositories are NOT impacted".
I assume that users who already have the false information (which is already widely distributed) in mind do not feel corrected if they now read “Fedora Linux 40 Beta users only using stable repositories are NOT impacted”. They might simply come to the conclusion that they are not affected since they never enabled testing manually. The article does not correct the earlier information but leaves it as potentially valid.
I think you should make clear in the beginning that testing is enabled by default, and unless they changed it themselves, it has to be assumed to be enabled. With the false information spread already through many channels, I assume some people stop reading after the first section.
I just triggered Justin [1] but I am not sure if he is available at the moment. It would be cool if someone with privileges adjusts the article's first section.
[1] https://discussion.fedoraproject.org/t/attention-malicious-code-in-current-p...
On Sun, 31 Mar 2024 at 13:55, Christopher Klooz py0xc3@posteo.net wrote: [..] BTW all that scandal with xz backdoor. Looks like if fedora spec file would be using not
Source0: https://github.com/tukaani-project/%%7Bname%7D/releases/download/v%%7Bversio...
but
Source0: https://github.com/tukaani-project/%%7Bname%7D/archive/v%%7Bversion%7D/%%7Bn...
It would be no issue.
kloczek
On Sun, Mar 31 2024 at 12:55:23 PM +00:00:00, Christopher Klooz py0xc3@posteo.net wrote:
In case someone from the Fedora Magazine is in the devel mailing list and reads this:
I'm really frustrated with our communication regarding this issue. Does anybody know who can fix this?
If we don't know who can fix Fedora Magazine on a Sunday, maybe we should just take down the entire domain until tomorrow. Fedora Magazine is an authoritative source and a lot of people are trusting the wrong information that we're providing.
Thanks for complaining about this, Christopher.
Michael
On Sun, Mar 31 2024 at 07:15:42 AM -04:00:00, Neal Gompa ngompa13@gmail.com wrote:
Well, an easy solution is to make it so "dnf update" is coerced to "dnf distro-sync" for development releases. Then it doesn't matter. We could make that happen for Fedora 41 with the DNF 5 transition (there's already code to make this possible with PackageKit with the current DNF backend, it needs to be migrated into DNF 5).
Disabling updates-testing is a bad plan, because we want updates more aggressively tested during the development cycle.
Agree, people testing beta Fedoras should surely be testing the test updates. But the test updates shouldn't remain installed after updating past the GA release. I don't have a strong opinion on whether 'dnf update' should automatically perform a 'dnf distro-sync', but PackageKit certainly ought to do this. It probably only needs to happen once, though, on or after the final release date.
Michael
On 31/03/2024 16.56, Michael Catanzaro wrote:
On Sun, Mar 31 2024 at 12:55:23 PM +00:00:00, Christopher Klooz py0xc3@posteo.net wrote:
In case someone from the Fedora Magazine is in the devel mailing list and reads this:
I'm really frustrated with our communication regarding this issue. Does anybody know who can fix this?
If we don't know who can fix Fedora Magazine on a Sunday, maybe we should just take down the entire domain until tomorrow. Fedora Magazine is an authoritative source and a lot of people are trusting the wrong information that we're providing.
Thanks for complaining about this, Christopher.
Michael
I don't feel good to implement such means without consensus with other moderators, but given that this article keeps telling affected users that they are not affected while it spreads confusing/misleading information, I have changed the Fedora Discussion topic [1] to make clear about the issue with the Fedora Magazine article in the very beginning, and indirectly warn about it. For average users, Fedora Discussion is the major source next to the magazine afaik. I tried to formulate it accommodating, but it has to be clear that the article's information was not correct and now the update added another incorrect information (Kevin made clear that beta has testing enabled) while the previous incorrect information is not devaluated either.
I think the Fedora Discussion topic's content shall do it for now and satisfy the majority of users. I keep following here, but if there is any change/add I need to do on that page, feel free to send/CC me an email directly, or a message on discourse. I keep available at least until Thursday to update it regularly if necessary.
[1] https://discussion.fedoraproject.org/t/attention-malicious-code-in-current-p...
Chris
On Sun, Mar 31 2024 at 09:56:04 AM -05:00:00, Michael Catanzaro mcatanzaro@redhat.com wrote:
I'm really frustrated with our communication regarding this issue. Does anybody know who can fix this?
The Fedora Magazine article has been fixed (thanks!).
On 31/03/2024 20.21, Michael Catanzaro wrote:
On Sun, Mar 31 2024 at 09:56:04 AM -05:00:00, Michael Catanzaro mcatanzaro@redhat.com wrote:
I'm really frustrated with our communication regarding this issue. Does anybody know who can fix this?
The Fedora Magazine article has been fixed (thanks!).
"*Fedora Linux 40 branched users (i.e. pre-Beta) likely received the potentially vulnerable /5.6.0-2.fc40/ build https://bodhi.fedoraproject.org/updates/FEDORA-2024-4417db3376 if the system updated between March 2nd and March 6th*. Fedora Linux 40 Beta users only using stable repositories are NOT impacted. Fedora Linux 39 and 38 users are also NOT impacted."
-> only pre-beta, not beta, affected -> F40 beta using stable NOT impacted (without challenging the previously distributed assumption that testing is disabled by default)
That's still the same false information, isn't it?
On 31/03/2024 20.52, Christopher Klooz wrote:
On 31/03/2024 20.21, Michael Catanzaro wrote:
On Sun, Mar 31 2024 at 09:56:04 AM -05:00:00, Michael Catanzaro mcatanzaro@redhat.com wrote:
I'm really frustrated with our communication regarding this issue. Does anybody know who can fix this?
The Fedora Magazine article has been fixed (thanks!).
"*Fedora Linux 40 branched users (i.e. pre-Beta) likely received the potentially vulnerable /5.6.0-2.fc40/ build https://bodhi.fedoraproject.org/updates/FEDORA-2024-4417db3376 if the system updated between March 2nd and March 6th*. Fedora Linux 40 Beta users only using stable repositories are NOT impacted. Fedora Linux 39 and 38 users are also NOT impacted."
-> only pre-beta, not beta, affected -> F40 beta using stable NOT impacted (without challenging the previously distributed assumption that testing is disabled by default)
That's still the same false information, isn't it?
Justin just has shown up in discourse. I suggested to get in touch with you, Adam or Kevin since he seemed to be convinced the article is fine as it is. When I refresh the article, it still seems to be unchanged. Is the update you mean already online Michael?
On 31-03-2024 20:54, Christopher Klooz wrote:
On 31/03/2024 20.52, Christopher Klooz wrote:
On 31/03/2024 20.21, Michael Catanzaro wrote:
On Sun, Mar 31 2024 at 09:56:04 AM -05:00:00, Michael Catanzaro mcatanzaro@redhat.com wrote:
I'm really frustrated with our communication regarding this issue. Does anybody know who can fix this?
The Fedora Magazine article has been fixed (thanks!).
"*Fedora Linux 40 branched users (i.e. pre-Beta) likely received the potentially vulnerable /5.6.0-2.fc40/ build https://bodhi.fedoraproject.org/updates/FEDORA-2024-4417db3376 if the system updated between March 2nd and March 6th*. Fedora Linux 40 Beta users only using stable repositories are NOT impacted. Fedora Linux 39 and 38 users are also NOT impacted."
-> only pre-beta, not beta, affected -> F40 beta using stable NOT impacted (without challenging the previously distributed assumption that testing is disabled by default)
That's still the same false information, isn't it?
Justin just has shown up in discourse. I suggested to get in touch with you, Adam or Kevin since he seemed to be convinced the article is fine as it is. When I refresh the article, it still seems to be unchanged. Is the update you mean already online Michael?
I clarified what's wrong with Justin in a DM on Matrix. He was on the same garden path as I was regarding "Beta release" vs. "Final release".
There will be another update to the article.
-- Sandro
Am 31.03.24 um 21:33 schrieb Sandro:
On 31-03-2024 20:54, Christopher Klooz wrote:
On 31/03/2024 20.52, Christopher Klooz wrote:
On 31/03/2024 20.21, Michael Catanzaro wrote:
On Sun, Mar 31 2024 at 09:56:04 AM -05:00:00, Michael Catanzaro mcatanzaro@redhat.com wrote:
I'm really frustrated with our communication regarding this issue. Does anybody know who can fix this?
The Fedora Magazine article has been fixed (thanks!).
"*Fedora Linux 40 branched users (i.e. pre-Beta) likely received the potentially vulnerable /5.6.0-2.fc40/ build https://bodhi.fedoraproject.org/updates/FEDORA-2024-4417db3376 if the system updated between March 2nd and March 6th*. Fedora Linux 40 Beta users only using stable repositories are NOT impacted. Fedora Linux 39 and 38 users are also NOT impacted."
-> only pre-beta, not beta, affected -> F40 beta using stable NOT impacted (without challenging the previously distributed assumption that testing is disabled by default)
That's still the same false information, isn't it?
Justin just has shown up in discourse. I suggested to get in touch with you, Adam or Kevin since he seemed to be convinced the article is fine as it is. When I refresh the article, it still seems to be unchanged. Is the update you mean already online Michael?
I clarified what's wrong with Justin in a DM on Matrix. He was on the same garden path as I was regarding "Beta release" vs. "Final release".
There will be another update to the article.
Not sure, if it was already mentioned -> containers. I had here a toolbox environment with F40. That I had not in my first actions on the screen. The last state had 5.6.0-3 installed but not sure if the previous release was also installed ...
-- Leon
Just to be clear, as I was getting ready to put 40 Beta on a test machine, this has been fixed. I do the install and run updates and the compromised version of xz is never installed?
On 31/03/2024 22.30, Leon Fauster via devel wrote:
Am 31.03.24 um 21:33 schrieb Sandro:
On 31-03-2024 20:54, Christopher Klooz wrote:
On 31/03/2024 20.52, Christopher Klooz wrote:
On 31/03/2024 20.21, Michael Catanzaro wrote:
On Sun, Mar 31 2024 at 09:56:04 AM -05:00:00, Michael Catanzaro mcatanzaro@redhat.com wrote:
I'm really frustrated with our communication regarding this issue. Does anybody know who can fix this?
The Fedora Magazine article has been fixed (thanks!).
"*Fedora Linux 40 branched users (i.e. pre-Beta) likely received the potentially vulnerable /5.6.0-2.fc40/ build https://bodhi.fedoraproject.org/updates/FEDORA-2024-4417db3376 if the system updated between March 2nd and March 6th*. Fedora Linux 40 Beta users only using stable repositories are NOT impacted. Fedora Linux 39 and 38 users are also NOT impacted."
-> only pre-beta, not beta, affected -> F40 beta using stable NOT impacted (without challenging the previously distributed assumption that testing is disabled by default)
That's still the same false information, isn't it?
Justin just has shown up in discourse. I suggested to get in touch with you, Adam or Kevin since he seemed to be convinced the article is fine as it is. When I refresh the article, it still seems to be unchanged. Is the update you mean already online Michael?
I clarified what's wrong with Justin in a DM on Matrix. He was on the same garden path as I was regarding "Beta release" vs. "Final release".
There will be another update to the article.
Not sure, if it was already mentioned -> containers. I had here a toolbox environment with F40. That I had not in my first actions on the screen. The last state had 5.6.0-3 installed but not sure if the previous release was also installed ...
The repo files should be the same on Fedora containers, so if the container is F40 and the testing repo is enabled, it might have installed the malicious build.
Preemptively, I added yesterday to the Fedora Discussion topic that people shall also update their toolbox containers. I am not sure if a container can end up in a condition that is vulnerable (especially since it has no dedicated systemd), but I assume we do not know for sure at this time, and the package was available to toolbox if the testing was enabled on a F40 container (I assume there are already F40 containers available? Didn't verify).
So I suggest to preemptively act with F40 toolboxes in the same way as with F40 if testing was enabled. -> https://discussion.fedoraproject.org/t/attention-malicious-code-in-current-b...
On Sun, Mar 31, 2024 at 10:30:23PM +0200, Leon Fauster via devel wrote:
Not sure, if it was already mentioned -> containers. I had here a toolbox environment with F40. That I had not in my first actions on the screen. The last state had 5.6.0-3 installed but not sure if the previous release was also installed ...
You should pull the latest version and restart any containers you were running.
kevin
On Sun, Mar 31, 2024 at 08:55:37PM +0000, Christopher Klooz wrote:
The repo files should be the same on Fedora containers, so if the container is F40 and the testing repo is enabled, it might have installed the malicious build.
Right, if it was dnf updated during the time that the bad update was in updates-testing.
Folks should pull the latest and restart.
Preemptively, I added yesterday to the Fedora Discussion topic that people shall also update their toolbox containers. I am not sure if a container can end up in a condition that is vulnerable (especially since it has no dedicated systemd), but I assume we do not know for sure at this time, and the package was available to toolbox if the testing was enabled on a F40 container (I assume there are already F40 containers available? Didn't verify).
Yeah, best to be safe and pull the latest that doesn't have the affected build and rerun.
Yes, there are f40 containers available.
kevin
On 31/03/2024 23.11, Kevin Fenzi wrote:
On Sun, Mar 31, 2024 at 08:55:37PM +0000, Christopher Klooz wrote:
The repo files should be the same on Fedora containers, so if the container is F40 and the testing repo is enabled, it might have installed the malicious build.
Right, if it was dnf updated during the time that the bad update was in updates-testing.
Folks should pull the latest and restart.
Preemptively, I added yesterday to the Fedora Discussion topic that people shall also update their toolbox containers. I am not sure if a container can end up in a condition that is vulnerable (especially since it has no dedicated systemd), but I assume we do not know for sure at this time, and the package was available to toolbox if the testing was enabled on a F40 container (I assume there are already F40 containers available? Didn't verify).
Yeah, best to be safe and pull the latest that doesn't have the affected build and rerun.
Yes, there are f40 containers available.
kevin
Great point. I adjusted the Fedora Discussion topic.
On 31/03/2024 21.33, Sandro wrote:
On 31-03-2024 20:54, Christopher Klooz wrote:
On 31/03/2024 20.52, Christopher Klooz wrote:
On 31/03/2024 20.21, Michael Catanzaro wrote:
On Sun, Mar 31 2024 at 09:56:04 AM -05:00:00, Michael Catanzaro mcatanzaro@redhat.com wrote:
I'm really frustrated with our communication regarding this issue. Does anybody know who can fix this?
The Fedora Magazine article has been fixed (thanks!).
"*Fedora Linux 40 branched users (i.e. pre-Beta) likely received the potentially vulnerable /5.6.0-2.fc40/ build https://bodhi.fedoraproject.org/updates/FEDORA-2024-4417db3376 if the system updated between March 2nd and March 6th*. Fedora Linux 40 Beta users only using stable repositories are NOT impacted. Fedora Linux 39 and 38 users are also NOT impacted."
-> only pre-beta, not beta, affected -> F40 beta using stable NOT impacted (without challenging the previously distributed assumption that testing is disabled by default)
That's still the same false information, isn't it?
Justin just has shown up in discourse. I suggested to get in touch with you, Adam or Kevin since he seemed to be convinced the article is fine as it is. When I refresh the article, it still seems to be unchanged. Is the update you mean already online Michael?
I clarified what's wrong with Justin in a DM on Matrix. He was on the same garden path as I was regarding "Beta release" vs. "Final release".
There will be another update to the article.
-- Sandro
Given the issues we had in the recent days in the communications between Fedora Magazine, a discussion if and/or how to use it in CVE handling has evolved in Fedora Discussions. I have just moved this into a dedicated topic, in case you want to add your perspectives: https://discussion.fedoraproject.org/t/xz-lessons-learned-if-how-to-involve-...
On Sun, Mar 31 2024 at 06:52:53 PM +00:00:00, Christopher Klooz py0xc3@posteo.net wrote:
"Fedora Linux 40 branched users (i.e. pre-Beta) likely received the potentially vulnerable 5.6.0-2.fc40 build if the system updated between March 2nd and March 6th. Fedora Linux 40 Beta users only using stable repositories are NOT impacted. Fedora Linux 39 and 38 users are also NOT impacted."
-> only pre-beta, not beta, affected -> F40 beta using stable NOT impacted (without challenging the previously distributed assumption that testing is disabled by default)
That's still the same false information, isn't it?
It looks correct to me. The bug was fixed prior to the final release of F40 beta, so describing it as "pre-beta" makes sense. And people who used only the stable repos were indeed not affected. The article later clarifies that updates-testing is enabled by default (although it would be nicer to do this higher up rather than lower down the page).
On 01/04/2024 16.32, Michael Catanzaro wrote:
On Sun, Mar 31 2024 at 06:52:53 PM +00:00:00, Christopher Klooz py0xc3@posteo.net wrote:
"Fedora Linux 40 branched users (i.e. pre-Beta) likely received the potentially vulnerable 5.6.0-2.fc40 build if the system updated between March 2nd and March 6th. Fedora Linux 40 Beta users only using stable repositories are NOT impacted. Fedora Linux 39 and 38 users are also NOT impacted."
-> only pre-beta, not beta, affected -> F40 beta using stable NOT impacted (without challenging the previously distributed assumption that testing is disabled by default)
That's still the same false information, isn't it?
It looks correct to me. The bug was fixed prior to the final release of F40 beta, so describing it as "pre-beta" makes sense. And people who used only the stable repos were indeed not affected. The article later clarifies that updates-testing is enabled by default (although it would be nicer to do this higher up rather than lower down the page).
Interesting. I thought the below note about "testing = enabled by default" was already mentioned before. I was only worried about the top section. The abstract decides if people keep reading, and with the previously spread information, I had doubt that the sentence motivates people to read further. So I assume the "stable repo not impacted" sentence suggests something false given the previously established context (default = stable, not testing).
Concerning your argument that the bug was fixed prior to the beta release, I answered on the Fedora Discussion topic [1] (to avoid duplication here) since I am not sure if I understand what you mean (and what it implies for the majority of users).
Btw, thanks for your elaborations and clarifications in the recent days ;)
[1] https://discussion.fedoraproject.org/t/xz-lessons-learned-if-how-to-involve-...
On 31/03/2024 23.08, Kevin Fenzi wrote:
On Sun, Mar 31, 2024 at 10:30:23PM +0200, Leon Fauster via devel wrote:
Not sure, if it was already mentioned -> containers. I had here a toolbox environment with F40. That I had not in my first actions on the screen. The last state had 5.6.0-3 installed but not sure if the previous release was also installed ...
You should pull the latest version and restart any containers you were running.
kevin
I assume rawhide has been fixed, too? So that rawhide users also just need to follow the same instructions as F40? Or do they need to reinstall?
If possible, I would like to end the topic updates in Fedora Discussion tomorrow and suggest users that they can consider the issue solved once they conducted all suggested steps and thus stop following the topic anylonger, but for rawhide I still have the warning issued to not use it at this time. Is the rawhide issue still open?
On Mon, 2024-04-01 at 09:32 -0500, Michael Catanzaro wrote:
On Sun, Mar 31 2024 at 06:52:53 PM +00:00:00, Christopher Klooz py0xc3@posteo.net wrote:
"Fedora Linux 40 branched users (i.e. pre-Beta) likely received the potentially vulnerable 5.6.0-2.fc40 build if the system updated between March 2nd and March 6th. Fedora Linux 40 Beta users only using stable repositories are NOT impacted. Fedora Linux 39 and 38 users are also NOT impacted."
-> only pre-beta, not beta, affected -> F40 beta using stable NOT impacted (without challenging the previously distributed assumption that testing is disabled by default)
That's still the same false information, isn't it?
It looks correct to me. The bug was fixed prior to the final release of F40 beta,
This is not really correct, or at least at all relevant. The bug wasn't in F40 Beta simply because the update never made it to 'stable'. Only 'stable' packages go into *composes*. However, saying that is not really useful because anyone who *installed* Beta and then updated it regularly may have got the vulnerable package. We should not say anything to give people the impression that if they installed Beta, they don't need to worry. That is not true or helpful.
so describing it as "pre-beta" makes sense. And people who used only the stable repos were indeed not affected. The article later clarifies that updates-testing is enabled by default (although it would be nicer to do this higher up rather than lower down the page).
For the same reason I think it's dangerous and not useful to try and draw this distinction between notional "people who only use stable repos" and people who use testing. Who would actually install F40 but then manually turn updates-testing off? Very few people. I don't think we should talk about this because it just confuses the issue. It would be like saying a stable release security issue that appeared in a stable update didn't affect people who turned off the updates repo. Technically true, but people don't do that, why would we say it?
We should have a simple and clear message that covers the most common and important case: if you installed Fedora 40 and updated regularly during the vulnerable time frame, you very likely got the vulnerable package and should take appropriate action. We should not confuse this with unnecessary verbiage about stable and testing and pre-Beta and post-Beta.
On Mon, Apr 1 2024 at 10:12:55 AM -07:00:00, Adam Williamson adamwill@fedoraproject.org wrote:
This is not really correct, or at least at all relevant. The bug wasn't in F40 Beta simply because the update never made it to 'stable'. Only 'stable' packages go into *composes*. However, saying that is not really useful because anyone who *installed* Beta and then updated it regularly may have got the vulnerable package. We should not say anything to give people the impression that if they installed Beta, they don't need to worry. That is not true or helpful.
Thing is, the bug was fixed before Fedora 40 Beta was released. If you installed the beta on or after the release date, you never got the builds with ifuncs enabled. This is why it's correct to say that only "pre-beta" builds were backdoored.
Michael
On Mon, 2024-04-01 at 12:16 -0500, Michael Catanzaro wrote:
On Mon, Apr 1 2024 at 10:12:55 AM -07:00:00, Adam Williamson adamwill@fedoraproject.org wrote:
This is not really correct, or at least at all relevant. The bug wasn't in F40 Beta simply because the update never made it to 'stable'. Only 'stable' packages go into *composes*. However, saying that is not really useful because anyone who *installed* Beta and then updated it regularly may have got the vulnerable package. We should not say anything to give people the impression that if they installed Beta, they don't need to worry. That is not true or helpful.
Thing is, the bug was fixed before Fedora 40 Beta was released. If you installed the beta on or after the release date, you never got the builds with ifuncs enabled. This is why it's correct to say that only "pre-beta" builds were backdoored.
Oh, ISWYM. Well, I suppose yes, that does happen to be true. We could communicate that if it's done very carefully and made really clear that it's about the *time frame*, nothing to do with the repositories.
On Mon, Apr 01, 2024 at 05:07:13PM +0000, Christopher Klooz wrote:
On 31/03/2024 23.08, Kevin Fenzi wrote:
On Sun, Mar 31, 2024 at 10:30:23PM +0200, Leon Fauster via devel wrote:
Not sure, if it was already mentioned -> containers. I had here a toolbox environment with F40. That I had not in my first actions on the screen. The last state had 5.6.0-3 installed but not sure if the previous release was also installed ...
You should pull the latest version and restart any containers you were running.
kevin
I assume rawhide has been fixed, too? So that rawhide users also just need to follow the same instructions as F40? Or do they need to reinstall?
Yes. The downgrade was pushed out on friday along with the f40 one.
If possible, I would like to end the topic updates in Fedora Discussion tomorrow and suggest users that they can consider the issue solved once they conducted all suggested steps and thus stop following the topic anylonger, but for rawhide I still have the warning issued to not use it at this time. Is the rawhide issue still open?
No, it should have the same resolution as f40.
kevin
On 01/04/2024 19.27, Kevin Fenzi wrote:
On Mon, Apr 01, 2024 at 05:07:13PM +0000, Christopher Klooz wrote:
On 31/03/2024 23.08, Kevin Fenzi wrote:
On Sun, Mar 31, 2024 at 10:30:23PM +0200, Leon Fauster via devel wrote:
Not sure, if it was already mentioned -> containers. I had here a toolbox environment with F40. That I had not in my first actions on the screen. The last state had 5.6.0-3 installed but not sure if the previous release was also installed ...
You should pull the latest version and restart any containers you were running.
kevin
I assume rawhide has been fixed, too? So that rawhide users also just need to follow the same instructions as F40? Or do they need to reinstall?
Yes. The downgrade was pushed out on friday along with the f40 one.
If possible, I would like to end the topic updates in Fedora Discussion tomorrow and suggest users that they can consider the issue solved once they conducted all suggested steps and thus stop following the topic anylonger, but for rawhide I still have the warning issued to not use it at this time. Is the rawhide issue still open?
No, it should have the same resolution as f40.
kevin
Thanks! I update it.
On Mon, Apr 1, 2024 at 5:27 PM Kevin Fenzi kevin@scrye.com wrote:
Yes. The downgrade was pushed out on friday along with the f40 one.
Of course, your mirror may vary as to availability (as I recall, in my particular case, my test VM for rawhide did not get the update for a day or so).
It does bring up a potential point that perhaps Fedora should have an additional repo (let's call it "emergency fixes") that is not community mirrored (so any mirrors for load sharing would be fully controlled by the project), with a short refresh time, and containing only packages that need to get out immediately. If a critical fix needs to get out to the community it could be (almost) immediately available. After a few days (when public mirrors would be expected to have updated) those packages could be removed (reducing the load on this repo). For the next time, of course (and there will be a next time, there is always a next time).
On 01-04-2024 19:12, Adam Williamson wrote:
On Mon, 2024-04-01 at 09:32 -0500, Michael Catanzaro wrote:
On Sun, Mar 31 2024 at 06:52:53 PM +00:00:00, Christopher Klooz py0xc3@posteo.net wrote:
"Fedora Linux 40 branched users (i.e. pre-Beta) likely received the potentially vulnerable 5.6.0-2.fc40 build if the system updated between March 2nd and March 6th. Fedora Linux 40 Beta users only using stable repositories are NOT impacted. Fedora Linux 39 and 38 users are also NOT impacted."
-> only pre-beta, not beta, affected -> F40 beta using stable NOT impacted (without challenging the previously distributed assumption that testing is disabled by default)
That's still the same false information, isn't it?
It looks correct to me. The bug was fixed prior to the final release of F40 beta,
This is not really correct, or at least at all relevant. The bug wasn't in F40 Beta simply because the update never made it to 'stable'. Only 'stable' packages go into *composes*. However, saying that is not really useful because anyone who *installed* Beta and then updated it regularly may have got the vulnerable package. We should not say anything to give people the impression that if they installed Beta, they don't need to worry. That is not true or helpful.
so describing it as "pre-beta" makes sense. And people who used only the stable repos were indeed not affected. The article later clarifies that updates-testing is enabled by default (although it would be nicer to do this higher up rather than lower down the page).
For the same reason I think it's dangerous and not useful to try and draw this distinction between notional "people who only use stable repos" and people who use testing. Who would actually install F40 but then manually turn updates-testing off? Very few people. I don't think we should talk about this because it just confuses the issue. It would be like saying a stable release security issue that appeared in a stable update didn't affect people who turned off the updates repo. Technically true, but people don't do that, why would we say it?
This boils down to the initial confusion as to when `updates-testing` is switched off. Both Justin and I thought it would be turned off again as soon as Beta is officially released.
If you take that confusion into account, making that distinction makes perfect sense. Unfortunately, it turned out to be the wrong assumption.
We should have a simple and clear message that covers the most common and important case: if you installed Fedora 40 and updated regularly during the vulnerable time frame, you very likely got the vulnerable package and should take appropriate action. We should not confuse this with unnecessary verbiage about stable and testing and pre-Beta and post-Beta.
Agreed. I'm sure the text would have been different if the confusion (see above) hadn't happened.
OTOH, I also expect users, even inexperienced users, to bring some common sense. I oppose having to put "contents may be hot" on a coffee cup ...
-- Sandro
On Mon, Apr 1 2024 at 10:25:16 AM -07:00:00, Adam Williamson adamwill@fedoraproject.org wrote:
Oh, ISWYM. Well, I suppose yes, that does happen to be true. We could communicate that if it's done very carefully and made really clear that it's about the *time frame*, nothing to do with the repositories.
It's been brought to my attention that the Fedora Magazine article [1] has been updated yet again, and now it warns that the 5.6.0-3.fc40 build was "tainted." This build is not affected by the backdoor because ifuncs were disabled.
I'm quite frustrated now. The message from Richard and other engineers has been very consistent. The bad builds are 5.6.0-1.fc40 and 5.6.0-2.fc40. We have been saying as much since Friday. Please fix the article once again.
(There is no strong reason to believe 5.6.0 is otherwise worse than 5.4.6, since both versions were released by the same attacker. It doesn't make sense to refer to the -3 build as "tainted.")
Michael
[1] https://fedoramagazine.org/cve-2024-3094-security-alert-f40-rawhide/
On Mon, Apr 01, 2024 at 05:47:10PM +0000, Gary Buhrmaster wrote:
It does bring up a potential point that perhaps Fedora should have an additional repo (let's call it "emergency fixes") that is not community mirrored (so any mirrors for load sharing would be fully controlled by the project), with a short refresh time, and containing only packages that need to get out immediately. If a critical fix needs to get out to the community it could be (almost) immediately available. After a few days (when public mirrors would be expected to have updated) those packages could be removed (reducing the load on this repo). For the next time, of course (and there will be a next time, there is always a next time).
https://pagure.io/releng/issue/5886
(This was after Heartbleed, FWIW.)
-- _______________________________________________ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
On Mon, Apr 1, 2024 at 9:17 PM Matthew Miller mattdm@fedoraproject.org wrote:
On Mon, Apr 01, 2024 at 05:47:10PM +0000, Gary Buhrmaster wrote:
It does bring up a potential point that perhaps Fedora should have an additional repo (let's call it "emergency fixes") that is not community mirrored (so any mirrors for load sharing would be fully controlled by the project), with a short refresh time, and containing only packages that need to get out immediately. If a critical fix needs to get out to the community it could be (almost) immediately available. After a few days (when public mirrors would be expected to have updated) those packages could be removed (reducing the load on this repo). For the next time, of course (and there will be a next time, there is always a next time).
https://pagure.io/releng/issue/5886
(This was after Heartbleed, FWIW.)
I am glad to know "it's all good" :-(
I am not convinced dlopen will it make secure in the end. I am not sure this is a good solution. dlopen makes those dependencies non-obvious from packaging side and non-visible from ldd or similar checking programs.
I think it should be considered to offer more than one dynamic library. For example most services I maintain links to libsystemd just because it wants sd_notify calls in their services. Without any proof I would expect quite many services would have similar problem. Could be perhaps libsystemd broken into few more dynamic linked libraries? I somehow doubt any kind of compression is needed for sd_notify calls.
Could be even smaller library libsystemd-notify linked from libsystemd, which would allow end applications to explicitly declare they need more limited set of functions?
Is there now relatively simple way to analyze packages depending on libsystemd, how rich functionality they need to use? I somehow doubt many packages need explicit work with journal. If they do, okay, use more heavy-weight library. But quite alot of programs needs just sd_notify* and sd_listen* calls, maybe with addition to sd_booted.
The best example is sshd itself, which plays important role in this story. For a very good reason.
$ objdump -T /usr/sbin/sshd | grep sd_ 0000000000000000 DF *UND* 0000000000000000 (LIBSYSTEMD_209) sd_notify
Less code linked into libraries means less code needs to be analysed for malware or security scans. It also allows user to explicitly declare just limited functions should be available.
I have made crude script getting calls used on my own system. While there is larger group with also journal involved, I think number of packages using just sd_notify, sd_listen or similar is not unimportant.
$ dnf repoquery --whatdepends systemd-libs --alldeps | sort -u | while read PKG; do rpm -q "$PKG" >/dev/null && echo "$PKG"; done | tee installed $ cat installed | while read PKG; do rpm -ql "$PKG" | grep -E '^(/usr/s?bin/|/usr/lib64/|/usr/libexec)' | while read BIN; do objdump -T "$BIN" 2>/dev/null | grep -E ' (sd|udev)_' | sed -e 's/[0-9a-f]+\s*DF\s*(*UND*|.text)\s*[0-9a-f]+\s*//' -e "s|^|$PKG ${BIN}: |" ; done; done | tee libsystemd-calls.txt
Attached the results in libsystemd-calls.txt attachment.
I would prefer ability to explicitly declare I want just limited functionality for cooperating with systemd, without any obscure dlopen calls. They are even more suspicious in low-level library and I don't think they should be used to solve resource limitations. They make it more difficult to discover actually used dependencies by static analysis tools. udev has already own library, perhaps notify and bus should be able to use with a lightweight library too?
Regards, Petr
On 30. 03. 24 19:43, Zbigniew Jędrzejewski-Szmek wrote:
libsystemd is linked to compression libraries primarly because those compressors are options for journald files, and we generally want to mainain the property that all journal files from the past, as well as journal files from other systems, can be read by later journal code.
(bzip2 is an exception here. It is only used by systemd-importd and related tools, so libsystemd was never linked to it, IIRC.)
In recent systemd versions, dlopen is used for various compression libraries and other libraries, so they'll be opportunistically loaded if needed, effectively making those dependencies optional at runtime.
Conversions to use dlopen require a bit of boilerplate code and make the code less readable, so we only would to them if there was a reason. Usually, this reason was size and/or the dependency tree. Compression libraries are very small and have no dependencies, so it wasn't considered important to use dlopen for them. But now that there's a new motivation, as mentioned elsewhere in the thread, we'll load pretty much all dependencies of libsystemd via dlopen.
Zbyszek
I am not convinced dlopen will it make secure in the end. I am not sure this is a good solution. dlopen makes those dependencies non-obvious from packaging side and non-visible from ldd or similar checking programs.
I think it should be considered to offer more than one dynamic library. For example most services I maintain links to libsystemd just because it wants sd_notify calls in their services. Without any proof I would expect quite many services would have similar problem. Could be perhaps libsystemd broken into few more dynamic linked libraries? I somehow doubt any kind of compression is needed for sd_notify calls.
Could be even smaller library libsystemd-notify linked from libsystemd, which would allow end applications to explicitly declare they need more limited set of functions?
Is there now relatively simple way to analyze packages depending on libsystemd, how rich functionality they need to use? I somehow doubt many packages need explicit work with journal. If they do, okay, use more heavy-weight library. But quite alot of programs needs just sd_notify* and sd_listen* calls, maybe with addition to sd_booted.
The best example is sshd itself, which plays important role in this story. For a very good reason.
$ objdump -T /usr/sbin/sshd | grep sd_ 0000000000000000 DF *UND* 0000000000000000 (LIBSYSTEMD_209) sd_notify
Less code linked into libraries means less code needs to be analysed for malware or security scans. It also allows an user to explicitly declare just limited functions should be available.
I have made crude script getting calls used on my own system. While there is larger group with also journal involved, I think number of packages using just sd_notify, sd_listen or similar is not unimportant.
$ dnf repoquery --whatdepends systemd-libs --alldeps | sort -u | while read PKG; do rpm -q "$PKG" >/dev/null && echo "$PKG"; done | tee installed $ cat installed | while read PKG; do rpm -ql "$PKG" | grep -E '^(/usr/s?bin/|/usr/lib64/|/usr/libexec)' | while read BIN; do objdump -T "$BIN" 2>/dev/null | grep -E ' (sd|udev)_' | sed -e 's/[0-9a-f]+\s*DF\s*(*UND*|.text)\s*[0-9a-f]+\s*//' -e "s|^|$PKG ${BIN}: |" ; done; done | tee libsystemd-calls.txt
Attached the results in libsystemd-calls.txt attachment.
I would prefer ability to explicitly declare I want just limited functionality for cooperating with systemd, without any obscure dlopen calls. They are even more suspicious in low-level library and I don't think they should be used to solve resource limitations. They make it more difficult to discover actually used dependencies by static analysis tools. udev has already own library, perhaps notify and bus should be able to use with a more lightweight library too?
Regards, Petr
(edit: attached compressed list.)
On 30. 03. 24 19:43, Zbigniew Jędrzejewski-Szmek wrote:
libsystemd is linked to compression libraries primarly because those compressors are options for journald files, and we generally want to mainain the property that all journal files from the past, as well as journal files from other systems, can be read by later journal code.
(bzip2 is an exception here. It is only used by systemd-importd and related tools, so libsystemd was never linked to it, IIRC.)
In recent systemd versions, dlopen is used for various compression libraries and other libraries, so they'll be opportunistically loaded if needed, effectively making those dependencies optional at runtime.
Conversions to use dlopen require a bit of boilerplate code and make the code less readable, so we only would to them if there was a reason. Usually, this reason was size and/or the dependency tree. Compression libraries are very small and have no dependencies, so it wasn't considered important to use dlopen for them. But now that there's a new motivation, as mentioned elsewhere in the thread, we'll load pretty much all dependencies of libsystemd via dlopen.
Zbyszek
Petr Menšík Software Engineer, RHEL Red Hat,http://www.redhat.com/ PGP: DFCF908DB7C87E8E529925BC4931CA5B6C9FC5CB
On Di, 02.04.24 14:04, Petr Menšík (pemensik@redhat.com) wrote:
I am not convinced dlopen will it make secure in the end. I am not sure this is a good solution. dlopen makes those dependencies non-obvious from packaging side and non-visible from ldd or similar checking programs.
I think it should be considered to offer more than one dynamic library.
It was considered. To the point that a decade ago, libsystemd originally was split up. But it was awful to maintain.
But let's not repeat this discussion here. Read up here: https://github.com/systemd/systemd/issues/32028
For example most services I maintain links to libsystemd just because it wants sd_notify calls in their services. Without any proof I would expect quite many services would have similar problem. Could be perhaps libsystemd broken into few more dynamic linked libraries? I somehow doubt any kind of compression is needed for sd_notify calls.
Could be even smaller library libsystemd-notify linked from libsystemd, which would allow end applications to explicitly declare they need more limited set of functions?
Why link? I'd really just suggest people to implement the trivial client on their own. It's a trivial protocol for a reason: so that people can implement it natively in their language and framwork without bothering with our upstream libsystemd.
use libsystemd if you link against it anways and hack in C, or if you really don't are about the extra dep for a single function, but otherwise, why would you use the lib?
It *literally* is just sending a text string "READY=1" in an AF_UNIX datagram to a socket whose path is provided to you in the $NOTIFY_SOCKET env var. Anyone with a moderate understanding of UNIX APIs should be able to hack that up in a a few lines of code. It's a protocol I can summarize and explain in *one* frickin' sentence.
sshd upstream understood this btw:
https://bugzilla.mindrot.org/show_bug.cgi?id=2641
Lennart
-- Lennart Poettering, Berlin
On Tue, 2 Apr 2024 at 08:18, Lennart Poettering mzerqung@0pointer.de wrote:
On Di, 02.04.24 14:04, Petr Menšík (pemensik@redhat.com) wrote:
Could be even smaller library libsystemd-notify linked from libsystemd, which would allow end applications to explicitly declare they need more limited set of functions?
Why link? I'd really just suggest people to implement the trivial client on their own. It's a trivial protocol for a reason: so that people can implement it natively in their language and framwork without bothering with our upstream libsystemd.
I think because enough people have learned that anything which is listed as a 'trivial protocol' tends to get messed up if you don't really understand it. Many developers have also learned that if you mess up a 'trivial protocol' you tend to get blamed for being an idiot or worse so it is better to just find something which already implements it, is tested and known to work and move on with your day. I think the two combined with some other social reasons are why various modern languages (go, python, java, rust, etc) have grown giant ecosystems where packages focus on implementing a trivial thing and you are to include just that versus a library of other things. [Of course you then end up with 30 ways to implement the same trivial thing which pull in various other trivial things...]
I am not saying this is good behaviour or what should be done, but it seems expected these days.
use libsystemd if you link against it anways and hack in C, or if you really don't are about the extra dep for a single function, but otherwise, why would you use the lib?
It *literally* is just sending a text string "READY=1" in an AF_UNIX datagram to a socket whose path is provided to you in the $NOTIFY_SOCKET env var. Anyone with a moderate understanding of UNIX APIs should be able to hack that up in a a few lines of code. It's a protocol I can summarize and explain in *one* frickin' sentence.
sshd upstream understood this btw:
https://bugzilla.mindrot.org/show_bug.cgi?id=2641
Lennart
-- Lennart Poettering, Berlin -- _______________________________________________ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
On 02. 04. 24 14:17, Lennart Poettering wrote:
On Di, 02.04.24 14:04, Petr Menšík (pemensik@redhat.com) wrote:
I am not convinced dlopen will it make secure in the end. I am not sure this is a good solution. dlopen makes those dependencies non-obvious from packaging side and non-visible from ldd or similar checking programs.
I think it should be considered to offer more than one dynamic library.
It was considered. To the point that a decade ago, libsystemd originally was split up. But it was awful to maintain.
But let's not repeat this discussion here. Read up here: https://github.com/systemd/systemd/issues/32028
Oh, I hate the way discussions are moderated at systemd upstream. I would prefer to discuss here.
It appears you do not want to make it simple to use just notify lib. But luckily it does not have to be maintained by systemd team. Yes, the protocol is trivial. But it still needs to handle conditions when it does not run under systemd. When it does, but error occurs. While number of lines required are very low, it would have need to be duplicated many times in many services. More or less the same code.
It is apparent either everyone will implement it themselves or systemd-independent project needs to provide just limited sd_notify shared library. At least it needs documentation. Indication it implements sd_notify in some way is nice to have bonus.
For example most services I maintain links to libsystemd just because it wants sd_notify calls in their services. Without any proof I would expect quite many services would have similar problem. Could be perhaps libsystemd broken into few more dynamic linked libraries? I somehow doubt any kind of compression is needed for sd_notify calls.
Could be even smaller library libsystemd-notify linked from libsystemd, which would allow end applications to explicitly declare they need more limited set of functions?
Why link? I'd really just suggest people to implement the trivial client on their own. It's a trivial protocol for a reason: so that people can implement it natively in their language and framwork without bothering with our upstream libsystemd.
Because the number of applications solving the same problem is rather high. It is not issue to be solved in about 3 projects. There are dozens of daemons, which should implement the same functionality. Common base even with a tiny library still makes a sense to me. Especially if most of them linked unnecessarily to libsystemd and thought that was the best option.
use libsystemd if you link against it anways and hack in C, or if you really don't are about the extra dep for a single function, but otherwise, why would you use the lib?
It *literally* is just sending a text string "READY=1" in an AF_UNIX datagram to a socket whose path is provided to you in the $NOTIFY_SOCKET env var. Anyone with a moderate understanding of UNIX APIs should be able to hack that up in a a few lines of code. It's a protocol I can summarize and explain in *one* frickin' sentence.
Yes, I admit, it is simple line oriented protocol. But you need to get notify socket path, handle failures when opening it. You need to know what to write there, so also documentation is needed. Man page for sd_notify is nice. Implementation of sd_notifyf() function would not be trivial. While it is not difficult to get right, it does not make sense to have over 15 different implementations, when it can share the same code. When you want to include also error messages, it certainly becomes more than few lines.
Function pid_notify_with_fds_internal from ./src/libsystemd/sd-daemon/sd-daemon.c certainly is not just few lines, as suggested. Should I ask why, if not necessary? It seems whole sd-daemon.c has 775 lines, which would be nice to have separate library. Things missing are some helpers. Socket activation is not as needed as core sd_notify, but might be a good addition. Still needs just environment and pointer to ints. No unwanted dependencies, I think that would make a nice standalone shared library. It provides everything I would need in common service.
sshd upstream understood this btw:
https://bugzilla.mindrot.org/show_bug.cgi?id=2641
Lennart
-- Lennart Poettering, Berlin
I think they preferred having fast fix over having long-term sustainable solution. I haven't seen offer to provide libsystemd-notify library with minimal dependencies, which they kindly refused. They preferred bundled code over systemd bloat in comment #13. I haven't seen bloat-less library considered in the bug, but might have missed it?
The question is, how should be the separate library called. libsystem_notify? Ideas for a better name?
Compat include header to make it otherwise unmodified with libsystemd would be nice as well.
Lennart Poettering wrote:
It *literally* is just sending a text string "READY=1" in an AF_UNIX datagram to a socket whose path is provided to you in the $NOTIFY_SOCKET env var.
I see so many ways one can get this wrong. E.g., using some abstraction for the socket write that can also write to regular files, without checking that "$NOTIFY_SOCKET" is really a socket (or checking it with a TOCTOU vulnerability), introducing an arbitrary file overwrite vulnerability.
Kevin Kofler
On Di, 02.04.24 19:44, Petr Menšík (pemensik@redhat.com) wrote:
Function pid_notify_with_fds_internal from ./src/libsystemd/sd-daemon/sd-daemon.c certainly is not just few lines, as suggested. Should I ask why, if not necessary?
The version in sd-daemon.c is a bit more complex because it supports a few things typically not necessary:
- In systemd we support sd_notify() also via AF_VSOCK. i.e. PID 1 will send sd_notify() messages itself to a supervisor VMM or similar, to inform it when the system is booted up. It uses the very same protocol as we usually use via AF_UNIX, but over AF_VSOCK. Support for AF_VSOCK should be completely unnecessary for anyone but systemd (i.e. PID1) itself, hence we didn't add it to the copy/paste version version. The AF_VSOCK thing is awesome btw, because it allows you to properly schedule activation of a bunch of VMs in order, with correct waiting until they are ready to respond.
- The code allows overriding the sender struct ucred (i.e. the SCM_CREDS control message te service manager will see). This exists to implement the "systemd-notify" tool, which is always going to the be a child process of the shell script it is invoked off, and which wants to patch the sender PID hence to its own PPID. Nothing should need that except of that tool.
- The code allows sending along fds to the service manager, to make use of the "fdstore" concept in systemd (i.e. the service manager can keep copies of a service's active fds in escrow, so that the service can implement seamless restrats without losing any resources).
So yes, most folks won#t need any of these three things, hence the copy version of the code doesn't mention it.
I think they preferred having fast fix over having long-term sustainable solution. I haven't seen offer to provide libsystemd-notify library with minimal dependencies, which they kindly refused. They preferred bundled code over systemd bloat in comment #13. I haven't seen bloat-less library considered in the bug, but might have missed it?
uff, "bloat", "bloat", "bloat".
I mean, libsystemd in git main doesn't pull in the compression libs anymore, it doesnt pull in gcrypt either. It pulls in libc and libcap only. Do you have a problem with these libraries?
Or are you concerned about the extra space in memory? Well, libsystemd is going to be in memory anyway, you might as well use it. After all, if Linux is smart enough to load it into memory only once, and just give each consumer their own immutable memory map. Or in other words, having yet another shared lib in the mix just makes things worse: more of the same code, not less...
The way I see it, there are two options from my systemd perspective:
1. Use libsystemd.so, it doesn't pull in any other deps anymore (except for libc and libcap), and is installed everywhere anyway.
Use this when: - Your project is in C - You don't care about an extra dep - You use the other stuff libsystemd offers anyway (i.e. sd-bus, sd-event or so)
1. Roll your own sd_notify(). [In case you hack is a C project: you can start with the copylib the man pages now include (i.e. https://www.freedesktop.org/software/systemd/man/devel/sd_notify.html#Notes) and adjust it to your framework of choice, coding style and so on. i.e. adjust the error handling, logging to your choice.]
Use this when: - Your project is not in C or - You really don't want an extra dep
Lennart
-- Lennart Poettering, Berlin
Good point. Check testing it is actually expected unix socket would be quite nice. Especially when the file sd-daemon.c implements sd_is_socket_unix function, but never uses it itself. libsystemd verifies this using socket_address_parse_unix or socket_address_parse_vsock in pid_notify_with_fds_internal.
All in all, I don't see any good reason, why shared functionality should be copy&pasted to every project. That is the worst long-term idea hindering maintenance of shared functionality. I have been part of project having a lot of copy&paste. It was quick to write, but hard to maintain afterwards. Shared libraries make sense even when they are small, especially when that small part is needed often. Which is exactly the case of sd_notify variants.
Reusing easy to read and with no dependencies except libc is still much better idea.
I think projects having bundled implementation should mark it somehow in their packages. Something like Provides: bundled(systemd/sd_notify) ? Or bundled(systemd-libs) ? I think such functionality should be searchable somehow in the repository. I think we should define how it should be marked in case of inline copy. bundled(systemd) seems a bit too strong, but according to Bundling specification in packaging guidelines [1].
I like idea in marked comment [2]. It might get even launcher-independent way of daemon to signal its readiness. We do not support alternatives to systemd, but I see no reason to prevent upstream project to reuse the same way for any alternative, which may use readiness signal from the service. Other distributions still allow other combinations. It would be much better to support alternative process managers from shared library than in each project itself.
PS: once again comments are marked as spam, when they simply disagree with systemd team stance, but are otherwise to the point. That unfortunately makes no constructive discussion possible on systemd upstream issues.
1. https://docs.fedoraproject.org/en-US/packaging-guidelines/#bundling 2. https://github.com/systemd/systemd/issues/32028#issuecomment-2034099384
On 02. 04. 24 20:05, Kevin Kofler via devel wrote:
Lennart Poettering wrote:
It *literally* is just sending a text string "READY=1" in an AF_UNIX datagram to a socket whose path is provided to you in the $NOTIFY_SOCKET env var.
I see so many ways one can get this wrong. E.g., using some abstraction for the socket write that can also write to regular files, without checking that "$NOTIFY_SOCKET" is really a socket (or checking it with a TOCTOU vulnerability), introducing an arbitrary file overwrite vulnerability.
Kevin Kofler
-
Adam Williamson -
Alex Thomas -
Barry -
Björn Persson -
Chris Adams -
Christopher Klooz -
Daniel Alley -
Dmitry Belyavskiy -
Gary Buhrmaster -
Germano Massullo -
Jerry James -
Jonathan Wright -
Justin W. Flory (he/him) -
Kevin Fenzi -
Kevin Kofler -
Lennart Poettering -
Leon Fauster -
Matthew Miller -
Michael Catanzaro -
Mikel Olasagasti -
Neal Gompa -
Peter Robinson -
Petr Menšík -
pgnd -
Richard W.M. Jones -
Samuel Sieb -
Sandro -
Simon de Vlieger -
Stephen Gallagher -
Stephen Smoogen -
Tomasz Kłoczko -
Zbigniew Jędrzejewski-Szmek