Hello, community, I need your wisdom for planning a disruptive change.
Fedora 28 had https://fedoraproject.org/wiki/Changes/StrongCryptoSettings Fedora 33 had https://fedoraproject.org/wiki/Changes/StrongCryptoSettings2 I believe we should start planning for the next cryptographic defaults tightening. And next time it's gonna be even more disruptive because of SHA-1 (again).
SHA-1 is a hash function from 1995, which collision resistance is no longer to be relied upon for security [1]. At the same time, it's not like software has successfully migrated off it, not even close. It's not a question of "if" the world should migrate from it, sooner or later we must part ways with it. (Technically, some acute energy crisis or a collapse of civilization forever raising the costs of computations thousandfold would also do, but let's agree that migrating to a more modern hash is the way =)
We've been disabling it in TLS, but its usage is much wider than TLS. The next agonizing step is to restrict its usage for signatures on the cryptographic libraries level, with openssl being the scariest one.
Good news is, RHEL-9 is gonna lead the way and thus will take a lot of the hits first. Fedora doesn't have to pioneer it. Bad news is, Fedora has to follow suit someday anyway, and this brings me to how does one land such a change.
---
Fedora is a large distribution with short release cycles, and the only realistic way to weed out its reliance on SHA-1 signatures from all of its numerous dark corners is to break them. Make creation and verification fail in default configuration. But it's unreasonable to just wait for, say, Fedora 37 branch-off and break it in Rawhide for Fedora 38. The fallout will just be too big.
Maintainers need time to get bugs, look into them, think, analyze, react and test --- and that's just if it fails correctly! Unfortunately, it's not just that the error paths are as dusty as they get because the program counter has never set foot on them before. Some maintainers might even find that picking a different hash function renders their code non-interoperable, or even that protocols they implement have SHA-1 hardcoded in the spec. Or that everything is ready, but real world deployments need another decade. Or that on-disk formats are just hard to change and migrate. Took git years to migrate from SHA-1, and some others haven't even started. There are gonna be investigations, planning, exceptions, upstream changes, opt-out mechanisms, arguing, compromises, waiting out, all kinds of things. It's gonna be big. Too big for a single release cycle.
---
But how does one land something and give the distribution the extra cycles needed to react? That's not really clear to me.
An obvious thing is to announce it in one cycle and land in another one. The downsides are well-documented in "The Hitchhiker's Guide to the Galaxy": announcements are one weak measure, and then it's too late.
A second scheme I can come up with is a "jump scare". Break the functionality in Fedora 37 Rawhide, make most of the affected people realize the depth of the problem, then unbreak it. Break again for Fedora 38 and never fix.
This could also be extended into "let one stable release slide'. Break in 37 Rawhide, unbreak on branched off 37, but never in Rawhide.
But these are all rather... crude? Sure there should be better ways, preferably something explored before. I'm all for pulling this tooth out smoothly, but I need hints on how to do it. I hope that together we can devise a better plan than these.
So, how does one land a change that's bigger than a release cycle?
On Tue, Mar 08, 2022 at 07:40:15PM +0100, Alexander Sosedkin wrote:
the only realistic way to weed out its reliance on SHA-1 signatures from all of its numerous dark corners is to break them. Make creation and verification fail in default configuration.
That sounds like a terrible plan. We should make newer hashes the default, and we can make tools print a warning if sha1 is used where it shouldn't, but please don't break things on purpose.
For many many things sha1 is just fine. Just like md5 or even crc32. Not everything is about cryptographic security. Also, users will want to verify old signatures essentially forever. This should be always possible. And finally, the world is huge, and other users will provide sha1 signatures no matter what we do, and it is better to check those than to completely ignore them.
Zbyszek
(This is a bit like browsers disliking self-signed certs and http:// — it certainly is OK to make users aware of the issue, but actually disallowing those is terribly annoying and will only result in users jumping to different tools.)
On Tue, 2022-03-08 at 20:51 +0100, Zbigniew Jędrzejewski-Szmek wrote:
On Tue, Mar 08, 2022 at 07:40:15PM +0100, Alexander Sosedkin wrote:
the only realistic way to weed out its reliance on SHA-1 signatures from all of its numerous dark corners is to break them. Make creation and verification fail in default configuration.
That sounds like a terrible plan. We should make newer hashes the default, and we can make tools print a warning if sha1 is used where it shouldn't, but please don't break things on purpose.
For many many things sha1 is just fine. Just like md5 or even crc32. Not everything is about cryptographic security.
I would like to make it clear that this is just for *Cryptographic Signatures*, this is not a plan to block SHA-1 for all uses.
And for Cryptographic Signatures everything is about cryptography and SHA-1 is not safe anymore.
And to be extra clear this means: Certificates, TLS session setup, DNSSEC (although a lot of signatures are still SHA-1 based there ...), VPNs session establishment, PGP, etc...
Also, users will want to verify old signatures essentially forever.
This is only reasonable for stuff like emails, where you may have a reasonable expectation that the archived messages have not been tampered with after the fact. Allowing verification of signatures with SHA-1 for any "online" communication would be pointless.
This should be always possible. And finally, the world is huge, and other users will provide sha1 signatures no matter what we do, and it is better to check those than to completely ignore them.
We need to move the needle at some point. We will be able to set LEGACY crypto policies to allow SHA-1 verification, but we need to be First and Secure here as well.
Simo.
Zbyszek
(This is a bit like browsers disliking self-signed certs and http:// — it certainly is OK to make users aware of the issue, but actually disallowing those is terribly annoying and will only result in users jumping to different tools.) _______________________________________________ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
On Tue, Mar 8, 2022 at 3:11 PM Simo Sorce simo@redhat.com wrote:
On Tue, 2022-03-08 at 20:51 +0100, Zbigniew Jędrzejewski-Szmek wrote:
On Tue, Mar 08, 2022 at 07:40:15PM +0100, Alexander Sosedkin wrote:
the only realistic way to weed out its reliance on SHA-1 signatures from all of its numerous dark corners is to break them. Make creation and verification fail in default configuration.
That sounds like a terrible plan. We should make newer hashes the default, and we can make tools print a warning if sha1 is used where it shouldn't, but please don't break things on purpose.
For many many things sha1 is just fine. Just like md5 or even crc32. Not everything is about cryptographic security.
I would like to make it clear that this is just for *Cryptographic Signatures*, this is not a plan to block SHA-1 for all uses.
And for Cryptographic Signatures everything is about cryptography and SHA-1 is not safe anymore.
And to be extra clear this means: Certificates, TLS session setup, DNSSEC (although a lot of signatures are still SHA-1 based there ...), VPNs session establishment, PGP, etc...
Also, users will want to verify old signatures essentially forever.
This is only reasonable for stuff like emails, where you may have a reasonable expectation that the archived messages have not been tampered with after the fact. Allowing verification of signatures with SHA-1 for any "online" communication would be pointless.
This should be always possible. And finally, the world is huge, and other users will provide sha1 signatures no matter what we do, and it is better to check those than to completely ignore them.
We need to move the needle at some point. We will be able to set LEGACY crypto policies to allow SHA-1 verification, but we need to be First and Secure here as well.
Did we check to make sure such a change won't break Fedora's *own* GPG keys and the GPG keys of preferred third party repositories? It was a very unpleasant surprise to have all of CentOS' *second-party* keys break with that change.
On 3/8/22 15:23, Neal Gompa wrote:
On Tue, Mar 8, 2022 at 3:11 PM Simo Sorce simo@redhat.com wrote:
On Tue, 2022-03-08 at 20:51 +0100, Zbigniew Jędrzejewski-Szmek wrote:
On Tue, Mar 08, 2022 at 07:40:15PM +0100, Alexander Sosedkin wrote:
the only realistic way to weed out its reliance on SHA-1 signatures from all of its numerous dark corners is to break them. Make creation and verification fail in default configuration.
That sounds like a terrible plan. We should make newer hashes the default, and we can make tools print a warning if sha1 is used where it shouldn't, but please don't break things on purpose.
For many many things sha1 is just fine. Just like md5 or even crc32. Not everything is about cryptographic security.
I would like to make it clear that this is just for *Cryptographic Signatures*, this is not a plan to block SHA-1 for all uses.
And for Cryptographic Signatures everything is about cryptography and SHA-1 is not safe anymore.
And to be extra clear this means: Certificates, TLS session setup, DNSSEC (although a lot of signatures are still SHA-1 based there ...), VPNs session establishment, PGP, etc...
Also, users will want to verify old signatures essentially forever.
This is only reasonable for stuff like emails, where you may have a reasonable expectation that the archived messages have not been tampered with after the fact. Allowing verification of signatures with SHA-1 for any "online" communication would be pointless.
This should be always possible. And finally, the world is huge, and other users will provide sha1 signatures no matter what we do, and it is better to check those than to completely ignore them.
We need to move the needle at some point. We will be able to set LEGACY crypto policies to allow SHA-1 verification, but we need to be First and Secure here as well.
Did we check to make sure such a change won't break Fedora's *own* GPG keys and the GPG keys of preferred third party repositories? It was a very unpleasant surprise to have all of CentOS' *second-party* keys break with that change.
Fedora’s RPM package signatures are safe, and have been since at least Fedora 25 if not earlier. I can confirm that every single package in the Fedora 25 and Fedora 32 archives are signed with SHA-256 or later, and presumably the same holds for all releases since Fedora 25.
Qubes OS’s rpmcanon tool can be used to check if a package is signed with SHA-1: it will return an `InsecureAlgorithm` error for such packages unless the `--allow-weak-hashes` flag is passed. It does so by parsing the signatures itself before passing them to RPM.
On Tue, Mar 8, 2022 at 3:34 PM Demi Marie Obenour demiobenour@gmail.com wrote:
On 3/8/22 15:23, Neal Gompa wrote:
On Tue, Mar 8, 2022 at 3:11 PM Simo Sorce simo@redhat.com wrote:
On Tue, 2022-03-08 at 20:51 +0100, Zbigniew Jędrzejewski-Szmek wrote:
On Tue, Mar 08, 2022 at 07:40:15PM +0100, Alexander Sosedkin wrote:
the only realistic way to weed out its reliance on SHA-1 signatures from all of its numerous dark corners is to break them. Make creation and verification fail in default configuration.
That sounds like a terrible plan. We should make newer hashes the default, and we can make tools print a warning if sha1 is used where it shouldn't, but please don't break things on purpose.
For many many things sha1 is just fine. Just like md5 or even crc32. Not everything is about cryptographic security.
I would like to make it clear that this is just for *Cryptographic Signatures*, this is not a plan to block SHA-1 for all uses.
And for Cryptographic Signatures everything is about cryptography and SHA-1 is not safe anymore.
And to be extra clear this means: Certificates, TLS session setup, DNSSEC (although a lot of signatures are still SHA-1 based there ...), VPNs session establishment, PGP, etc...
Also, users will want to verify old signatures essentially forever.
This is only reasonable for stuff like emails, where you may have a reasonable expectation that the archived messages have not been tampered with after the fact. Allowing verification of signatures with SHA-1 for any "online" communication would be pointless.
This should be always possible. And finally, the world is huge, and other users will provide sha1 signatures no matter what we do, and it is better to check those than to completely ignore them.
We need to move the needle at some point. We will be able to set LEGACY crypto policies to allow SHA-1 verification, but we need to be First and Secure here as well.
Did we check to make sure such a change won't break Fedora's *own* GPG keys and the GPG keys of preferred third party repositories? It was a very unpleasant surprise to have all of CentOS' *second-party* keys break with that change.
Fedora’s RPM package signatures are safe, and have been since at least Fedora 25 if not earlier. I can confirm that every single package in the Fedora 25 and Fedora 32 archives are signed with SHA-256 or later, and presumably the same holds for all releases since Fedora 25.
Qubes OS’s rpmcanon tool can be used to check if a package is signed with SHA-1: it will return an `InsecureAlgorithm` error for such packages unless the `--allow-weak-hashes` flag is passed. It does so by parsing the signatures itself before passing them to RPM.
Did you check our preferred third-parties? From memory, we have shipped, are shipping, or will be shipping repository definitions for repositories from the following providers:
* RPM Fusion * Google * COPR * Microsoft
On Tue, Mar 8, 2022 at 8:52 PM Zbigniew Jędrzejewski-Szmek zbyszek@in.waw.pl wrote:
On Tue, Mar 08, 2022 at 07:40:15PM +0100, Alexander Sosedkin wrote:
the only realistic way to weed out its reliance on SHA-1 signatures from all of its numerous dark corners is to break them. Make creation and verification fail in default configuration.
That sounds like a terrible plan. We should make newer hashes the default, and we can make tools print a warning if sha1 is used where it shouldn't,
Yeah, that'd never gonna work. Cryptographic libraries aren't in position to do any meaningful logging, and even if they did, it'd be gleefully ignored until we break things.
but please don't break things on purpose.
If you manage to come out with the way we can move forward to this century crypto that doesn't involve breaking things on purpose, definitely share one with us, you'd be our saviour. =(
Yes, backwards compatibility is very important. Yes, you can still make a modern Fedora do RC4 or DES, with just a little bit of extra configuration. No, we must phase things out of defaults, there's no way around this.
For many many things sha1 is just fine. Just like md5 or even crc32. Not everything is about cryptographic security. Also, users will want to verify old signatures essentially forever. This should be always possible. And finally, the world is huge, and other users will provide sha1 signatures no matter what we do, and it is better to check those than to completely ignore them.
Zbyszek
(This is a bit like browsers disliking self-signed certs and http:// — it certainly is OK to make users aware of the issue, but actually disallowing those is terribly annoying and will only result in users jumping to different tools.) _______________________________________________ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
On Tue, Mar 8, 2022, at 1:40 PM, Alexander Sosedkin wrote:
But these are all rather... crude? Sure there should be better ways, preferably something explored before.
One general technique I like is the "warn and sleep" approach; example: https://github.com/coreos/rpm-ostree/pull/2098
Of course, printing to e.g. stderr or even more strongly adding a sleep(5) call in the middle of cryptographic libraries has a huge blast radius on its own. But it's smaller than removing it entirely.
(And, remember I am a big fan of the mental model of rolling out changes to the OS core first, not all packages, so some sort of opt-in warn-and-sleep approach could even be prototyped out in e.g. Fedora CoreOS first, where we have CI that covers most things we care about)
On Tue, Mar 8, 2022 at 6:40 PM Alexander Sosedkin asosedkin@redhat.com wrote:
Good news is, RHEL-9 is gonna lead the way and thus will take a lot of the hits first. Fedora doesn't have to pioneer it. Bad news is, Fedora has to follow suit someday anyway, and this brings me to how does one land such a change.
Yes, and the change has already "broken" COPR with c9s/epel9
Related bugzillas (and discussions):
https://bugzilla.redhat.com/show_bug.cgi?id=2059101 https://bugzilla.redhat.com/show_bug.cgi?id=2059424
and the COPR (in progress) adjustment:
https://pagure.io/copr/copr/issue/2106
Gary
On Tue, Mar 08, 2022 at 07:40:15PM +0100, Alexander Sosedkin wrote:
We've been disabling it in TLS, but its usage is much wider than TLS. The next agonizing step is to restrict its usage for signatures on the cryptographic libraries level, with openssl being the scariest one.
Good news is, RHEL-9 is gonna lead the way and thus will take a lot of the hits first. Fedora doesn't have to pioneer it. Bad news is, Fedora has to follow suit someday anyway, and this brings me to how does one land such a change.
Fedora is a large distribution with short release cycles, and the only realistic way to weed out its reliance on SHA-1 signatures from all of its numerous dark corners is to break them. Make creation and verification fail in default configuration. But it's unreasonable to just wait for, say, Fedora 37 branch-off and break it in Rawhide for Fedora 38. The fallout will just be too big.
If RHEL-9 has lead the way, what are the stats for real world RHEL impact ?
What is/was the absolute number of packages and % number of packages from the RHEL distro that saw breakage ?
Such figures can give us a better idea of impact on Fedora beyond "too big".
Assuming RHEL-9 has dealt with the problems, Fedora should inherit those fixes, which gives us a good base for the most commonly used / important packages in Fedora.
If the breakage % from RHEL was single digits, and those were the most important packages to fix from Fedora's POV too, then maybe the fall is not in fact "too big". It might be sufficient to identify a few important remaining packages to validate, and just accept the fallout for the remaining less important packages in Fedora can be fixed after the fact ?
IIUC we have a simple workaround of letting someone set the crypto policies on their machine back to LEGACY still
Regards, Daniel
On Wed, Mar 9, 2022 at 10:20 AM Daniel P. Berrangé berrange@redhat.com wrote:
On Tue, Mar 08, 2022 at 07:40:15PM +0100, Alexander Sosedkin wrote:
We've been disabling it in TLS, but its usage is much wider than TLS. The next agonizing step is to restrict its usage for signatures on the cryptographic libraries level, with openssl being the scariest one.
Good news is, RHEL-9 is gonna lead the way and thus will take a lot of the hits first. Fedora doesn't have to pioneer it. Bad news is, Fedora has to follow suit someday anyway, and this brings me to how does one land such a change.
Fedora is a large distribution with short release cycles, and the only realistic way to weed out its reliance on SHA-1 signatures from all of its numerous dark corners is to break them. Make creation and verification fail in default configuration. But it's unreasonable to just wait for, say, Fedora 37 branch-off and break it in Rawhide for Fedora 38. The fallout will just be too big.
If RHEL-9 has lead the way, what are the stats for real world RHEL impact ?
We'll know when the real world starts using RHEL-9 en masse?
What is/was the absolute number of packages and % number of packages from the RHEL distro that saw breakage ?
Does preventing the distro from installing altogether count as 100%? If yes, 100%. =) Jokes aside, I can't give you an accurate estimate yet.
Such figures can give us a better idea of impact on Fedora beyond "too big".
Assuming RHEL-9 has dealt with the problems, Fedora should inherit those fixes, which gives us a good base for the most commonly used / important packages in Fedora.
Yeah, that's what I meant by the good news. But that won't solve all Fedora problems.
If the breakage % from RHEL was single digits, and those were the most important packages to fix from Fedora's POV too, then maybe the fall is not in fact "too big". It might be sufficient to identify a few important remaining packages to validate, and just accept the fallout for the remaining less important packages in Fedora can be fixed after the fact ?
At a quick glance, I eyeball RHEL at ~2k packages and Fedora at ~22k packages. I think that limited analysis 's enough to safely claim that leaving the 90% of the packages you've labelled "less important" to be "fixed after the fact" is gonna be a disaster. One cycle doesn't sound enough.
IIUC we have a simple workaround of letting someone set the crypto policies on their machine back to LEGACY still
Yes, I'll sure leave SHA-1 signatures allowed in LEGACY for some more years. Similarly, I could break it in FUTURE rather early, but nearly nobody would notice until it hits DEFAULT.
Regards, Daniel -- |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :| _______________________________________________ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
On Wed, Mar 09, 2022 at 10:46:21AM +0100, Alexander Sosedkin wrote:
On Wed, Mar 9, 2022 at 10:20 AM Daniel P. Berrangé berrange@redhat.com wrote:
On Tue, Mar 08, 2022 at 07:40:15PM +0100, Alexander Sosedkin wrote:
We've been disabling it in TLS, but its usage is much wider than TLS. The next agonizing step is to restrict its usage for signatures on the cryptographic libraries level, with openssl being the scariest one.
Good news is, RHEL-9 is gonna lead the way and thus will take a lot of the hits first. Fedora doesn't have to pioneer it. Bad news is, Fedora has to follow suit someday anyway, and this brings me to how does one land such a change.
Fedora is a large distribution with short release cycles, and the only realistic way to weed out its reliance on SHA-1 signatures from all of its numerous dark corners is to break them. Make creation and verification fail in default configuration. But it's unreasonable to just wait for, say, Fedora 37 branch-off and break it in Rawhide for Fedora 38. The fallout will just be too big.
If RHEL-9 has lead the way, what are the stats for real world RHEL impact ?
We'll know when the real world starts using RHEL-9 en masse?
What is/was the absolute number of packages and % number of packages from the RHEL distro that saw breakage ?
Does preventing the distro from installing altogether count as 100%? If yes, 100%. =) Jokes aside, I can't give you an accurate estimate yet.
Perhaps a useful first step is to just modify the three main crypto libs (gnutls, openssl, and nss) to send a scary warnihg message to stderr/syslog any time they get use of SHA1 in a signature. Leave that active for a release cycle and see how many bug reports we get.
Such figures can give us a better idea of impact on Fedora beyond "too big".
Assuming RHEL-9 has dealt with the problems, Fedora should inherit those fixes, which gives us a good base for the most commonly used / important packages in Fedora.
Yeah, that's what I meant by the good news. But that won't solve all Fedora problems.
If the breakage % from RHEL was single digits, and those were the most important packages to fix from Fedora's POV too, then maybe the fall is not in fact "too big". It might be sufficient to identify a few important remaining packages to validate, and just accept the fallout for the remaining less important packages in Fedora can be fixed after the fact ?
At a quick glance, I eyeball RHEL at ~2k packages and Fedora at ~22k packages. I think that limited analysis 's enough to safely claim that leaving the 90% of the packages you've labelled "less important" to be "fixed after the fact" is gonna be a disaster. One cycle doesn't sound enough.
That 90% of packages are not all going to be using cryptographic signatures though. Only a relatively small subset will do anything crypto related, and most of that just be HTTPS and completely outsourced to a crypto library.
IOW of that 90% of packages not in RHEL, it could conceivably be single digits that will be using cryptographic signatures with SHA-1. An even smaller percentage of those will be considered important enough to be blockers for this change.
With regards, Daniel
On Wed, Mar 9, 2022 at 10:57 AM Daniel P. Berrangé berrange@redhat.com wrote:
On Wed, Mar 09, 2022 at 10:46:21AM +0100, Alexander Sosedkin wrote:
On Wed, Mar 9, 2022 at 10:20 AM Daniel P. Berrangé berrange@redhat.com wrote:
On Tue, Mar 08, 2022 at 07:40:15PM +0100, Alexander Sosedkin wrote:
We've been disabling it in TLS, but its usage is much wider than TLS. The next agonizing step is to restrict its usage for signatures on the cryptographic libraries level, with openssl being the scariest one.
Good news is, RHEL-9 is gonna lead the way and thus will take a lot of the hits first. Fedora doesn't have to pioneer it. Bad news is, Fedora has to follow suit someday anyway, and this brings me to how does one land such a change.
Fedora is a large distribution with short release cycles, and the only realistic way to weed out its reliance on SHA-1 signatures from all of its numerous dark corners is to break them. Make creation and verification fail in default configuration. But it's unreasonable to just wait for, say, Fedora 37 branch-off and break it in Rawhide for Fedora 38. The fallout will just be too big.
If RHEL-9 has lead the way, what are the stats for real world RHEL impact ?
We'll know when the real world starts using RHEL-9 en masse?
What is/was the absolute number of packages and % number of packages from the RHEL distro that saw breakage ?
Does preventing the distro from installing altogether count as 100%? If yes, 100%. =) Jokes aside, I can't give you an accurate estimate yet.
Perhaps a useful first step is to just modify the three main crypto libs (gnutls, openssl, and nss) to send a scary warnihg message to stderr/syslog any time they get use of SHA1 in a signature. Leave that active for a release cycle and see how many bug reports we get.
I left my crystal ball at home today, but I don't need it to say it'd be ~0 bugs filed if we log to syslog and ~3 if we log to stderr/stdout, all named "$CRYPTOLIB has no business messing up my stderr/stdout", which we'll promptly close by reverting the changes.
Such figures can give us a better idea of impact on Fedora beyond "too big".
Assuming RHEL-9 has dealt with the problems, Fedora should inherit those fixes, which gives us a good base for the most commonly used / important packages in Fedora.
Yeah, that's what I meant by the good news. But that won't solve all Fedora problems.
If the breakage % from RHEL was single digits, and those were the most important packages to fix from Fedora's POV too, then maybe the fall is not in fact "too big". It might be sufficient to identify a few important remaining packages to validate, and just accept the fallout for the remaining less important packages in Fedora can be fixed after the fact ?
At a quick glance, I eyeball RHEL at ~2k packages and Fedora at ~22k packages. I think that limited analysis 's enough to safely claim that leaving the 90% of the packages you've labelled "less important" to be "fixed after the fact" is gonna be a disaster. One cycle doesn't sound enough.
That 90% of packages are not all going to be using cryptographic signatures though. Only a relatively small subset will do anything crypto related, and most of that just be HTTPS and completely outsourced to a crypto library.
IOW of that 90% of packages not in RHEL, it could conceivably be single digits that will be using cryptographic signatures with SHA-1. An even smaller percentage of those will be considered important enough to be blockers for this change.
With regards, Daniel -- |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
Alexander Sosedkin asosedkin@redhat.com writes:
Daniel P. Berrangé berrange@redhat.com wrote:
Perhaps a useful first step is to just modify the three main crypto libs (gnutls, openssl, and nss) to send a scary warnihg message to stderr/syslog any time they get use of SHA1 in a signature. Leave that active for a release cycle and see how many bug reports we get.
I left my crystal ball at home today, but I don't need it to say it'd be ~0 bugs filed if we log to syslog and ~3 if we log to stderr/stdout, all named "$CRYPTOLIB has no business messing up my stderr/stdout",
It's clear you want SHA-1 gone, but the way you've written this maybe isn't conveying what you wan, as it sounds like you're also unwilling to process the bugs that result requesting its removal. (If you, who want it gone, aren't willing to participate in that, why should maintainers care?)
As I understood the proposal, it would be for the crypto lib to log a message like:
[timestamp] /usr/bin/firefox used DEPRECATED SHA-1 invocation
This is similar to what happened for /var/run: sure, it was annoying to basically everyone involved, but the bugs also went to the relevant packages.
which we'll promptly close by reverting the changes.
I don't see why you'd do that instead of reassigning to the appropriate packages or (better) helping them migrate.
Be well, --Robbie
On Wed, Mar 9, 2022 at 4:40 PM Robbie Harwood rharwood@redhat.com wrote:
Alexander Sosedkin asosedkin@redhat.com writes:
Daniel P. Berrangé berrange@redhat.com wrote:
Perhaps a useful first step is to just modify the three main crypto libs (gnutls, openssl, and nss) to send a scary warnihg message to stderr/syslog any time they get use of SHA1 in a signature. Leave that active for a release cycle and see how many bug reports we get.
I left my crystal ball at home today, but I don't need it to say it'd be ~0 bugs filed if we log to syslog and ~3 if we log to stderr/stdout, all named "$CRYPTOLIB has no business messing up my stderr/stdout",
It's clear you want SHA-1 gone, but the way you've written this maybe isn't conveying what you wan, as it sounds like you're also unwilling to process the bugs that result requesting its removal. (If you, who want it gone, aren't willing to participate in that, why should maintainers care?)
Oh, nobody wants to go to the dentist, myself included. But it's as inevitable as it's not delegatable.
Deprecating SHA-1 is no small thing, and one me could realistically do that for just a single small random thing. I can't realistically fix something big like git. I can't make DNSSEC deployments worldwide to move away from SHA-1. Besides small consultations, what I'm sure I can do is: keep postponing, raise discussions that don't convey the impact accurately, and then disable and collect my negative karma.
I want to know, ultimately, how can I break it for devs but not the users so that 1. the need for it is communicated early, 2. all the relevant places are timely identified, 3. maintainers have a plenty of time to resolve it for good or opt out, 4. and the users are affected as late and as smoothly as possible. Hope these are all reasonable things to wish for.
---
As I understood the proposal, it would be for the crypto lib to log a message like:
[timestamp] /usr/bin/firefox used DEPRECATED SHA-1 invocation
This is similar to what happened for /var/run: sure, it was annoying to basically everyone involved, but the bugs also went to the relevant packages.
which we'll promptly close by reverting the changes.
I don't see why you'd do that instead of reassigning to the appropriate packages or (better) helping them migrate.
Because a cryptographic library polluting stdout/stderr is a bug! It's a bad idea on so many fronts: * for logging to be seen, it should go where the app logging goes * stderr/stdout is not always wired up to something at all * and when it is, *we break the output of the application*
I think I've seen a bug when a library just opening a descriptor for its own private usage made it all go sideways, and you suggest actively interfering.
---
Just for example, imagine we get a bug report: "none of my git scripts work anymore", telling us that users can no longer reliably script git because there's `/usr/libexec/git-core/git-clone used DEPRECATED SHA-1 invocation` all over the output they parse.
We reassign it to git, git maintainers take their time and then decide: "we're gonna default to SHA-256 next month and disallow SHA-1 by default in 3 years because $VALID_REASONS, and for now we want $THIS_BEHAVIOUR instead". Or some other intricate, complicated and unique answer.
Then what? Output of dozens of tools is still broken, users are rightfully upset, and, to add insult to injury, SHA-1 signatures still work. So, I guess, we're supposed to add an API to suppress the warning? Could be OK to use'em because it's gated behind an app opt-in or something.
And then next release we disable it and then a different hundred of tools, those whose output nobody monitors or parses or even plumbs anywhere, break and we've effectively given them no forewarning. The <1% of those launching libreoffice from the terminal will have to intersect with those importing some specific format of documents to even get a slim chance of knowing why they should beware of the leopard.
And it'll also break git again, giving a second round of headache to git maintainers, forcing them to roll-back interim solutions and deploy long-term ones.
I think this approach manages to do it wrong twice, and that's not counting the very idea of libraries messing with stderr/stdout in the first place, which is, IMO, unthinkable.
Breaking things and providing overrides to use them if really safe or necessary or opted into means interrupting maintainers just once to decide whether they override and keep using it or change to something else. Sounds much better to me. Less distractions, more consistency. Git breaks, git gets just a single prod to do the right long-term thing. Problem is, how does one stretch that in time and shield users from it.
---
That being said, I'm not bent on just breaking it Michael Bay-style. On the contrary, I've started this thread precisely because I don't want to. I am open to other solutions. It's just that we shouldn't overdo them either. Like, I think Fedora SELinux denial UX is good. These don't just go to syslog or pollute stderr, they, IIRC, give desktop users instantaneous notifications that are easy to link to some specific thing that just failed on you. Doesn't mean we should write a daemon and applets for all DEs to dbus around requests to allow SHA-1 per executable path. But, IDK, if y'all tell me that waiting for another release and then disabling SHA-1 only on Tuesdays and Sundays only in rawhide for another 16 months is just what we need to soften the blow, then that's what I'm gonna roll with. As long as it's eventually done.
On Wed, Mar 09, 2022 at 12:14:28PM +0100, Alexander Sosedkin wrote:
I left my crystal ball at home today, but I don't need it to say it'd be ~0 bugs filed if we log to syslog and ~3 if we log to stderr/stdout, all named "$CRYPTOLIB has no business messing up my stderr/stdout", which we'll promptly close by reverting the changes.
Yeah, that's a little more cynically-phrased than I'd put it, but I had similar thoughts.
But: maybe if we logged it _and_ had a tool people could run to look specifically for those log entries, we could do something like a Test Day where people could send in reports?
On Wed, Mar 09, 2022 at 12:21:18PM -0500, Matthew Miller wrote:
On Wed, Mar 09, 2022 at 12:14:28PM +0100, Alexander Sosedkin wrote:
I left my crystal ball at home today, but I don't need it to say it'd be ~0 bugs filed if we log to syslog and ~3 if we log to stderr/stdout, all named "$CRYPTOLIB has no business messing up my stderr/stdout", which we'll promptly close by reverting the changes.
Yeah, that's a little more cynically-phrased than I'd put it, but I had similar thoughts.
But: maybe if we logged it _and_ had a tool people could run to look specifically for those log entries, we could do something like a Test Day where people could send in reports?
Or just have it logging in rawhide, not in the final release.
Regards, Daniel
On Wed, Mar 09, 2022 at 05:40:50PM +0000, Daniel P. Berrangé wrote:
But: maybe if we logged it _and_ had a tool people could run to look specifically for those log entries, we could do something like a Test Day where people could send in reports?
Or just have it logging in rawhide, not in the final release.
Yes, but we still need someone to look at those logs.
On Wed, Mar 09, 2022 at 01:05:38PM -0500, Matthew Miller wrote:
On Wed, Mar 09, 2022 at 05:40:50PM +0000, Daniel P. Berrangé wrote:
But: maybe if we logged it _and_ had a tool people could run to look specifically for those log entries, we could do something like a Test Day where people could send in reports?
Or just have it logging in rawhide, not in the final release.
Yes, but we still need someone to look at those logs.
Don't hate me for suggesting this, but the warning message on stderr could suggest/request reporting the problem to bugzilla to bring it to maintainer's attention.
*** NOTICE: pid 1234 (firefox) used cryptographic signature *** NOTICE: with deprecated SHA-1 algorithm. This usage *** NOTICE: will be prevented by future policy. Please report *** NOTICE: this problem to https://bugzilla.redhat.com
Make it a one time message per process to avoid producing pages of repeated text, and allow 'touch /etc/crypto-policies/dont-warn-me' to hide it system wide.
It would be irritating enough to make at least some subset of rawhide users followup with reporting bugs, and thus let maintainers understand how badly they are impacted.
With regards, Daniel
On Wed, Mar 9, 2022 at 7:19 PM Daniel P. Berrangé berrange@redhat.com wrote:
On Wed, Mar 09, 2022 at 01:05:38PM -0500, Matthew Miller wrote:
On Wed, Mar 09, 2022 at 05:40:50PM +0000, Daniel P. Berrangé wrote:
But: maybe if we logged it _and_ had a tool people could run to look specifically for those log entries, we could do something like a Test Day where people could send in reports?
Or just have it logging in rawhide, not in the final release.
Yes, but we still need someone to look at those logs.
Don't hate me for suggesting this, but the warning message on stderr could suggest/request reporting the problem to bugzilla to bring it to maintainer's attention.
*** NOTICE: pid 1234 (firefox) used cryptographic signature *** NOTICE: with deprecated SHA-1 algorithm. This usage *** NOTICE: will be prevented by future policy. Please report *** NOTICE: this problem to https://bugzilla.redhat.com
Make it a one time message per process to avoid producing pages of repeated text, and allow 'touch /etc/crypto-policies/dont-warn-me' to hide it system wide.
It would be irritating enough
I've just checked and it's not just that my firefox already prints ~20 variations on "FAIL" on startup but I'm also pretty sure I see these for the first time, meaning I've never cared to check before.
to make at least some subset of rawhide users followup with reporting bugs, and thus let maintainers understand how badly they are impacted.
With regards, Daniel -- |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :| _______________________________________________ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
On Wed, Mar 9, 2022 at 6:22 PM Matthew Miller mattdm@fedoraproject.org wrote:
On Wed, Mar 09, 2022 at 12:14:28PM +0100, Alexander Sosedkin wrote:
I left my crystal ball at home today, but I don't need it to say it'd be ~0 bugs filed if we log to syslog and ~3 if we log to stderr/stdout, all named "$CRYPTOLIB has no business messing up my stderr/stdout", which we'll promptly close by reverting the changes.
Yeah, that's a little more cynically-phrased than I'd put it, but I had similar thoughts.
But: maybe if we logged it _and_ had a tool people could run to look specifically for those log entries, we could do something like a Test Day where people could send in reports?
On one hand, this still is a risky territory of libraries leaning heavily into things that libraries shouldn't do because they can never guess the weird ways they're being used when they have existing interfaces to deny operations with some algorithms.
On the other hand, this makes much more sense to me than the stdout/stderr proposal. Especially if it accompanies a drawn-out multicycle change, it could be a noticeable impact dampener.
-- Matthew Miller mattdm@fedoraproject.org Fedora Project Leader _______________________________________________ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
On Wed, Mar 09, 2022 at 06:45:49PM +0100, Alexander Sosedkin wrote:
On Wed, Mar 9, 2022 at 6:22 PM Matthew Miller mattdm@fedoraproject.org wrote:
On Wed, Mar 09, 2022 at 12:14:28PM +0100, Alexander Sosedkin wrote:
I left my crystal ball at home today, but I don't need it to say it'd be ~0 bugs filed if we log to syslog and ~3 if we log to stderr/stdout, all named "$CRYPTOLIB has no business messing up my stderr/stdout", which we'll promptly close by reverting the changes.
Yeah, that's a little more cynically-phrased than I'd put it, but I had similar thoughts.
But: maybe if we logged it _and_ had a tool people could run to look specifically for those log entries, we could do something like a Test Day where people could send in reports?
On one hand, this still is a risky territory of libraries leaning heavily into things that libraries shouldn't do because they can never guess the weird ways they're being used when they have existing interfaces to deny operations with some algorithms.
On the other hand, this makes much more sense to me than the stdout/stderr proposal.
The reason I suggested stderr was because it is about the only channel you can have a reasonable guarantee of being able to use from a library that has no knowledge about its execution environment.
Security policies applied to procesess (SELinux, seccomp, namespaces or containerization in general) can easily prevent ability to use journald, syslog, or opening of log files, leading to messages not being visible. At worst, with seccomp an attempt to use the blocked feature may lead to an abort of the application.
With regards, Daniel
On Wed, Mar 9, 2022 at 7:05 PM Daniel P. Berrangé berrange@redhat.com wrote:
On Wed, Mar 09, 2022 at 06:45:49PM +0100, Alexander Sosedkin wrote:
On Wed, Mar 9, 2022 at 6:22 PM Matthew Miller mattdm@fedoraproject.org wrote:
On Wed, Mar 09, 2022 at 12:14:28PM +0100, Alexander Sosedkin wrote:
I left my crystal ball at home today, but I don't need it to say it'd be ~0 bugs filed if we log to syslog and ~3 if we log to stderr/stdout, all named "$CRYPTOLIB has no business messing up my stderr/stdout", which we'll promptly close by reverting the changes.
Yeah, that's a little more cynically-phrased than I'd put it, but I had similar thoughts.
But: maybe if we logged it _and_ had a tool people could run to look specifically for those log entries, we could do something like a Test Day where people could send in reports?
On one hand, this still is a risky territory of libraries leaning heavily into things that libraries shouldn't do because they can never guess the weird ways they're being used when they have existing interfaces to deny operations with some algorithms.
On the other hand, this makes much more sense to me than the stdout/stderr proposal.
The reason I suggested stderr was because it is about the only channel you can have a reasonable guarantee of being able to use from a library that has no knowledge about its execution environment.
I don't think there is a reasonable guarantee you can use it. It's normal for a daemon to close inherited file descriptors and open something radically different as FD 3, right? And then we're writing our scary messages who knows where.
Security policies applied to procesess (SELinux, seccomp, namespaces or containerization in general) can easily prevent ability to use journald, syslog, or opening of log files, leading to messages not being visible.
If the goal is some visibility, then it might...
At worst, with seccomp an attempt to use the blocked feature may lead to an abort of the application.
sigh.
With regards, Daniel -- |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
On Wed, Mar 09, 2022 at 07:13:14PM +0100, Alexander Sosedkin wrote:
On Wed, Mar 9, 2022 at 7:05 PM Daniel P. Berrangé berrange@redhat.com wrote:
On Wed, Mar 09, 2022 at 06:45:49PM +0100, Alexander Sosedkin wrote:
On Wed, Mar 9, 2022 at 6:22 PM Matthew Miller mattdm@fedoraproject.org wrote:
On Wed, Mar 09, 2022 at 12:14:28PM +0100, Alexander Sosedkin wrote:
I left my crystal ball at home today, but I don't need it to say it'd be ~0 bugs filed if we log to syslog and ~3 if we log to stderr/stdout, all named "$CRYPTOLIB has no business messing up my stderr/stdout", which we'll promptly close by reverting the changes.
Yeah, that's a little more cynically-phrased than I'd put it, but I had similar thoughts.
But: maybe if we logged it _and_ had a tool people could run to look specifically for those log entries, we could do something like a Test Day where people could send in reports?
On one hand, this still is a risky territory of libraries leaning heavily into things that libraries shouldn't do because they can never guess the weird ways they're being used when they have existing interfaces to deny operations with some algorithms.
On the other hand, this makes much more sense to me than the stdout/stderr proposal.
The reason I suggested stderr was because it is about the only channel you can have a reasonable guarantee of being able to use from a library that has no knowledge about its execution environment.
I don't think there is a reasonable guarantee you can use it. It's normal for a daemon to close inherited file descriptors and open something radically different as FD 3, right? And then we're writing our scary messages who knows where.
I presume you mean FD 2, not 3, as stderr==2.
Anything is possible, but I don't think it is common for apps to connect something strange to FD 2, precisely because that is designated as stderr.
There exist way too many libraries that will print error messages and warnings to stderr, for any non-trivial application to consider FD==2 safe to use for any functional purposes other than adhoc text output. Any application that wants to disconnect from its inherited stderr stream, will re-open it on /dev/null, or a text logfile, if they don't want to risk being broken by random code using stderr.
Even glibc will print to stderr when it sees malloc corruption, and stuff in stderr will end up in the systemd journal for any system services.
With regards, Daniel
On ke, 09 maalis 2022, Daniel P. Berrangé wrote:
On Wed, Mar 09, 2022 at 10:46:21AM +0100, Alexander Sosedkin wrote:
On Wed, Mar 9, 2022 at 10:20 AM Daniel P. Berrangé berrange@redhat.com wrote:
On Tue, Mar 08, 2022 at 07:40:15PM +0100, Alexander Sosedkin wrote:
We've been disabling it in TLS, but its usage is much wider than TLS. The next agonizing step is to restrict its usage for signatures on the cryptographic libraries level, with openssl being the scariest one.
Good news is, RHEL-9 is gonna lead the way and thus will take a lot of the hits first. Fedora doesn't have to pioneer it. Bad news is, Fedora has to follow suit someday anyway, and this brings me to how does one land such a change.
Fedora is a large distribution with short release cycles, and the only realistic way to weed out its reliance on SHA-1 signatures from all of its numerous dark corners is to break them. Make creation and verification fail in default configuration. But it's unreasonable to just wait for, say, Fedora 37 branch-off and break it in Rawhide for Fedora 38. The fallout will just be too big.
If RHEL-9 has lead the way, what are the stats for real world RHEL impact ?
We'll know when the real world starts using RHEL-9 en masse?
What is/was the absolute number of packages and % number of packages from the RHEL distro that saw breakage ?
Does preventing the distro from installing altogether count as 100%? If yes, 100%. =) Jokes aside, I can't give you an accurate estimate yet.
Perhaps a useful first step is to just modify the three main crypto libs (gnutls, openssl, and nss) to send a scary warnihg message to stderr/syslog any time they get use of SHA1 in a signature. Leave that active for a release cycle and see how many bug reports we get.
Such figures can give us a better idea of impact on Fedora beyond "too big".
Assuming RHEL-9 has dealt with the problems, Fedora should inherit those fixes, which gives us a good base for the most commonly used / important packages in Fedora.
Yeah, that's what I meant by the good news. But that won't solve all Fedora problems.
If the breakage % from RHEL was single digits, and those were the most important packages to fix from Fedora's POV too, then maybe the fall is not in fact "too big". It might be sufficient to identify a few important remaining packages to validate, and just accept the fallout for the remaining less important packages in Fedora can be fixed after the fact ?
At a quick glance, I eyeball RHEL at ~2k packages and Fedora at ~22k packages. I think that limited analysis 's enough to safely claim that leaving the 90% of the packages you've labelled "less important" to be "fixed after the fact" is gonna be a disaster. One cycle doesn't sound enough.
That 90% of packages are not all going to be using cryptographic signatures though. Only a relatively small subset will do anything crypto related, and most of that just be HTTPS and completely outsourced to a crypto library.
IOW of that 90% of packages not in RHEL, it could conceivably be single digits that will be using cryptographic signatures with SHA-1. An even smaller percentage of those will be considered important enough to be blockers for this change.
Specifically, Kerberos is one of those protocols and MIT Kerberos is one of those implementations which currently forced to use SHA-1 due to interoperability issues and also due to compliance with RFCs.
In the context of Fedora development, for example, use of Anonymous PKINIT requires usage of SHA-1 in PKINIT pre-authentication module in MIT Kerberos. We have to support that or Fedora developers would not be able to obtain their Kerberos tickets.
For more details see https://bugzilla.redhat.com/show_bug.cgi?id=2060798
On Wed, Mar 09, 2022 at 02:32:48PM +0200, Alexander Bokovoy wrote:
On ke, 09 maalis 2022, Daniel P. Berrangé wrote:
On Wed, Mar 09, 2022 at 10:46:21AM +0100, Alexander Sosedkin wrote:
On Wed, Mar 9, 2022 at 10:20 AM Daniel P. Berrangé berrange@redhat.com wrote:
On Tue, Mar 08, 2022 at 07:40:15PM +0100, Alexander Sosedkin wrote:
We've been disabling it in TLS, but its usage is much wider than TLS. The next agonizing step is to restrict its usage for signatures on the cryptographic libraries level, with openssl being the scariest one.
Good news is, RHEL-9 is gonna lead the way and thus will take a lot of the hits first. Fedora doesn't have to pioneer it. Bad news is, Fedora has to follow suit someday anyway, and this brings me to how does one land such a change.
Fedora is a large distribution with short release cycles, and the only realistic way to weed out its reliance on SHA-1 signatures from all of its numerous dark corners is to break them. Make creation and verification fail in default configuration. But it's unreasonable to just wait for, say, Fedora 37 branch-off and break it in Rawhide for Fedora 38. The fallout will just be too big.
If RHEL-9 has lead the way, what are the stats for real world RHEL impact ?
We'll know when the real world starts using RHEL-9 en masse?
What is/was the absolute number of packages and % number of packages from the RHEL distro that saw breakage ?
Does preventing the distro from installing altogether count as 100%? If yes, 100%. =) Jokes aside, I can't give you an accurate estimate yet.
Perhaps a useful first step is to just modify the three main crypto libs (gnutls, openssl, and nss) to send a scary warnihg message to stderr/syslog any time they get use of SHA1 in a signature. Leave that active for a release cycle and see how many bug reports we get.
Such figures can give us a better idea of impact on Fedora beyond "too big".
Assuming RHEL-9 has dealt with the problems, Fedora should inherit those fixes, which gives us a good base for the most commonly used / important packages in Fedora.
Yeah, that's what I meant by the good news. But that won't solve all Fedora problems.
If the breakage % from RHEL was single digits, and those were the most important packages to fix from Fedora's POV too, then maybe the fall is not in fact "too big". It might be sufficient to identify a few important remaining packages to validate, and just accept the fallout for the remaining less important packages in Fedora can be fixed after the fact ?
At a quick glance, I eyeball RHEL at ~2k packages and Fedora at ~22k packages. I think that limited analysis 's enough to safely claim that leaving the 90% of the packages you've labelled "less important" to be "fixed after the fact" is gonna be a disaster. One cycle doesn't sound enough.
That 90% of packages are not all going to be using cryptographic signatures though. Only a relatively small subset will do anything crypto related, and most of that just be HTTPS and completely outsourced to a crypto library.
IOW of that 90% of packages not in RHEL, it could conceivably be single digits that will be using cryptographic signatures with SHA-1. An even smaller percentage of those will be considered important enough to be blockers for this change.
Specifically, Kerberos is one of those protocols and MIT Kerberos is one of those implementations which currently forced to use SHA-1 due to interoperability issues and also due to compliance with RFCs.
In the context of Fedora development, for example, use of Anonymous PKINIT requires usage of SHA-1 in PKINIT pre-authentication module in MIT Kerberos. We have to support that or Fedora developers would not be able to obtain their Kerberos tickets.
For more details see https://bugzilla.redhat.com/show_bug.cgi?id=2060798
I assume RHEL is going to have to solve that problem with Krb because it wouldn't be viable to break Krb in RHEL, any more than it would in Fedora.
With regards, Daniel
On ke, 09 maalis 2022, Daniel P. Berrangé wrote:
On Wed, Mar 09, 2022 at 02:32:48PM +0200, Alexander Bokovoy wrote:
On ke, 09 maalis 2022, Daniel P. Berrangé wrote:
On Wed, Mar 09, 2022 at 10:46:21AM +0100, Alexander Sosedkin wrote:
On Wed, Mar 9, 2022 at 10:20 AM Daniel P. Berrangé berrange@redhat.com wrote:
On Tue, Mar 08, 2022 at 07:40:15PM +0100, Alexander Sosedkin wrote:
We've been disabling it in TLS, but its usage is much wider than TLS. The next agonizing step is to restrict its usage for signatures on the cryptographic libraries level, with openssl being the scariest one.
Good news is, RHEL-9 is gonna lead the way and thus will take a lot of the hits first. Fedora doesn't have to pioneer it. Bad news is, Fedora has to follow suit someday anyway, and this brings me to how does one land such a change.
Fedora is a large distribution with short release cycles, and the only realistic way to weed out its reliance on SHA-1 signatures from all of its numerous dark corners is to break them. Make creation and verification fail in default configuration. But it's unreasonable to just wait for, say, Fedora 37 branch-off and break it in Rawhide for Fedora 38. The fallout will just be too big.
If RHEL-9 has lead the way, what are the stats for real world RHEL impact ?
We'll know when the real world starts using RHEL-9 en masse?
What is/was the absolute number of packages and % number of packages from the RHEL distro that saw breakage ?
Does preventing the distro from installing altogether count as 100%? If yes, 100%. =) Jokes aside, I can't give you an accurate estimate yet.
Perhaps a useful first step is to just modify the three main crypto libs (gnutls, openssl, and nss) to send a scary warnihg message to stderr/syslog any time they get use of SHA1 in a signature. Leave that active for a release cycle and see how many bug reports we get.
Such figures can give us a better idea of impact on Fedora beyond "too big".
Assuming RHEL-9 has dealt with the problems, Fedora should inherit those fixes, which gives us a good base for the most commonly used / important packages in Fedora.
Yeah, that's what I meant by the good news. But that won't solve all Fedora problems.
If the breakage % from RHEL was single digits, and those were the most important packages to fix from Fedora's POV too, then maybe the fall is not in fact "too big". It might be sufficient to identify a few important remaining packages to validate, and just accept the fallout for the remaining less important packages in Fedora can be fixed after the fact ?
At a quick glance, I eyeball RHEL at ~2k packages and Fedora at ~22k packages. I think that limited analysis 's enough to safely claim that leaving the 90% of the packages you've labelled "less important" to be "fixed after the fact" is gonna be a disaster. One cycle doesn't sound enough.
That 90% of packages are not all going to be using cryptographic signatures though. Only a relatively small subset will do anything crypto related, and most of that just be HTTPS and completely outsourced to a crypto library.
IOW of that 90% of packages not in RHEL, it could conceivably be single digits that will be using cryptographic signatures with SHA-1. An even smaller percentage of those will be considered important enough to be blockers for this change.
Specifically, Kerberos is one of those protocols and MIT Kerberos is one of those implementations which currently forced to use SHA-1 due to interoperability issues and also due to compliance with RFCs.
In the context of Fedora development, for example, use of Anonymous PKINIT requires usage of SHA-1 in PKINIT pre-authentication module in MIT Kerberos. We have to support that or Fedora developers would not be able to obtain their Kerberos tickets.
For more details see https://bugzilla.redhat.com/show_bug.cgi?id=2060798
I assume RHEL is going to have to solve that problem with Krb because it wouldn't be viable to break Krb in RHEL, any more than it would in Fedora.
Yes, the approach is described in the comment 6 of that bug.
On 3/9/22 1:56 AM, Daniel P. Berrangé wrote:
On Wed, Mar 09, 2022 at 10:46:21AM +0100, Alexander Sosedkin wrote:
On Wed, Mar 9, 2022 at 10:20 AM Daniel P. Berrangé berrange@redhat.com wrote:
On Tue, Mar 08, 2022 at 07:40:15PM +0100, Alexander Sosedkin wrote:
We've been disabling it in TLS, but its usage is much wider than TLS. The next agonizing step is to restrict its usage for signatures on the cryptographic libraries level, with openssl being the scariest one.
Good news is, RHEL-9 is gonna lead the way and thus will take a lot of the hits first. Fedora doesn't have to pioneer it. Bad news is, Fedora has to follow suit someday anyway, and this brings me to how does one land such a change.
Fedora is a large distribution with short release cycles, and the only realistic way to weed out its reliance on SHA-1 signatures from all of its numerous dark corners is to break them. Make creation and verification fail in default configuration. But it's unreasonable to just wait for, say, Fedora 37 branch-off and break it in Rawhide for Fedora 38. The fallout will just be too big.
If RHEL-9 has lead the way, what are the stats for real world RHEL impact ?
We'll know when the real world starts using RHEL-9 en masse?
What is/was the absolute number of packages and % number of packages from the RHEL distro that saw breakage ?
Does preventing the distro from installing altogether count as 100%? If yes, 100%. =) Jokes aside, I can't give you an accurate estimate yet.
Perhaps a useful first step is to just modify the three main crypto libs (gnutls, openssl, and nss) to send a scary warnihg message to stderr/syslog any time they get use of SHA1 in a signature. Leave that active for a release cycle and see how many bug reports we get.
To be clear, the actual mechanism to turn off SHA1 for signatures doesn't involve any changes to any of our crypto libraries, it involves changing the crypto policies file. With crypto policies, you can actually turn off almost any algorithm involved in SSL or signatures in all of our libraries. There is really no good way to 'log' from the crypto libraries.
Actually I think that provides a way forward that might work.
1) in fedora 37, provide a policy that turns SHA-1 off. in our testing, we encourage people to run with that policy and write bugs against components.
2) in fedora 38, SHA-1 gets turned of in the default policy and ships that way. Things that still fail would only work once in the legacy policy.
3) some day in the future (fedora 39?) it gets turned off legacy as well.
bob
On 15/03/2022 22:45, Robert Relyea wrote:
- in fedora 37, provide a policy that turns SHA-1 off. in our testing,
we encourage people to run with that policy and write bugs against components.
That policy already exists in Fedora 34 and 35 where the FUTURE policy does not allow SHA1 in signature algorithms.
Tom
On 16. Mar 2022, at 00:04, Tom Hughes via devel devel@lists.fedoraproject.org wrote:
On 15/03/2022 22:45, Robert Relyea wrote:
- in fedora 37, provide a policy that turns SHA-1 off. in our testing, we encourage people to run with that policy and write bugs against components.
That policy already exists in Fedora 34 and 35 where the FUTURE policy does not allow SHA1 in signature algorithms.
In the case of OpenSSL, that only affects use of SHA1 as signature algorithms in TLS. It does not cover arbitrary signatures with a SHA1 digest, which is what we are proposing.
HTH, Clemens
Robert Relyea wrote:
- in fedora 38, SHA-1 gets turned of in the default policy and ships
that way.
Isn't that the default already? I use the default crypto policy, and I had a case last year where Seamonkey and Firefox refused to talk to a certain web server, which I worked around by temporarily adding "SHA1" to /etc/crypto-policies/back-ends/nss.config.
Björn Persson
On Tue, Mar 08, 2022 at 07:40:15PM +0100, Alexander Sosedkin wrote:
Took git years to migrate from SHA-1, and some others haven't even started.
git is a good example showing that this won't be easy. The SHA-256 object format is still marked as experimental and not the default.
Is there a plan to migrate the Fedora repositories?
On Wed, Mar 09, 2022 at 10:44:28AM +0100, Miroslav Lichvar wrote:
On Tue, Mar 08, 2022 at 07:40:15PM +0100, Alexander Sosedkin wrote:
Took git years to migrate from SHA-1, and some others haven't even started.
git is a good example showing that this won't be easy. The SHA-256 object format is still marked as experimental and not the default.
Unless I'm mistaken Git isn't using SHA1 in cryptographic signatures, it is using it directly for hashing. The proposal isn't banning all use of SHA1, just SHA-1 based signatures.
With regards, Daniel
On 3/9/22 04:48, Daniel P. Berrangé wrote:
On Wed, Mar 09, 2022 at 10:44:28AM +0100, Miroslav Lichvar wrote:
On Tue, Mar 08, 2022 at 07:40:15PM +0100, Alexander Sosedkin wrote:
Took git years to migrate from SHA-1, and some others haven't even started.
git is a good example showing that this won't be easy. The SHA-256 object format is still marked as experimental and not the default.
Unless I'm mistaken Git isn't using SHA1 in cryptographic signatures, it is using it directly for hashing. The proposal isn't banning all use of SHA1, just SHA-1 based signatures.
Git *does* use SHA-1 in signatures, via signed Git tags and commits. However, Git also includes code that can detect all known relatively fast collision attacks on SHA-1. A brute force attack (with 2^80 time complexity) is still possible, but the amount of resources required is such that the attack is not considered likely. Furthermore, in many cases, one would not only need to find a collision in SHA-1, but also get one of the colliding objects past code review. Finding a collision that will pass code review is likely far, *far* more difficult than just finding a collision.
In short, Git *does* need to move away from SHA-1, but it isn’t a crisis — yet.
Previous tightening of crypto defaults caused problems with us connecting to older ssh servers.
I am particularly interested / worried about sshd from RHEL 5, 6 & 7 for virt-p2v and virt-v2v conversions. This broke before, requiring us to advise users to set the global policy for the machine to LEGACY (thus ironically weakening crypto for everything).
Also I have some ancient network equipment that cannot be upgraded but needs older ssh protocols. I can't connect to it from Fedora unless I set the crypto policy to LEGACY.
Anyway I'm wondering if the SHA-1 change will impact ssh further?
Rich.
On Wed, Mar 9, 2022 at 2:47 PM Richard W.M. Jones rjones@redhat.com wrote:
Previous tightening of crypto defaults caused problems with us connecting to older ssh servers.
I am particularly interested / worried about sshd from RHEL 5, 6 & 7 for virt-p2v and virt-v2v conversions. This broke before, requiring us to advise users to set the global policy for the machine to LEGACY (thus ironically weakening crypto for everything).
Also I have some ancient network equipment that cannot be upgraded but needs older ssh protocols. I can't connect to it from Fedora unless I set the crypto policy to LEGACY.
Anyway I'm wondering if the SHA-1 change will impact ssh further?
IIRC, the only SHA-1 thing that should be left in DEFAULT for SSH is SHA-1 as HMAC, which doesn't rely on collision resistance. So, not this round.
Rich.
-- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com virt-df lists disk usage of guests without needing to install any software inside the virtual machine. Supports Linux and Windows. http://people.redhat.com/~rjones/virt-df/ _______________________________________________ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
On 3/9/22 08:52, Alexander Sosedkin wrote:
On Wed, Mar 9, 2022 at 2:47 PM Richard W.M. Jones rjones@redhat.com wrote:
Previous tightening of crypto defaults caused problems with us connecting to older ssh servers.
I am particularly interested / worried about sshd from RHEL 5, 6 & 7 for virt-p2v and virt-v2v conversions. This broke before, requiring us to advise users to set the global policy for the machine to LEGACY (thus ironically weakening crypto for everything).
Also I have some ancient network equipment that cannot be upgraded but needs older ssh protocols. I can't connect to it from Fedora unless I set the crypto policy to LEGACY.
Anyway I'm wondering if the SHA-1 change will impact ssh further?
IIRC, the only SHA-1 thing that should be left in DEFAULT for SSH is SHA-1 as HMAC, which doesn't rely on collision resistance. So, not this round.
SHA-1 HMAC is still considered to be a perfectly good MAC. For new protocols, I recommend using Blake2b instead, but that is purely for performance reasons, not because HMAC-SHA-1 is broken. There are no known attacks on HMAC-SHA-1, and it is actually stronger than AES-128 in CBC-MAC (160 bits of security vs. 128).
At least RHEL 6 issues can be fixed server-side generating an ecdsa key...
On Wed, Mar 9, 2022 at 2:47 PM Richard W.M. Jones rjones@redhat.com wrote:
Previous tightening of crypto defaults caused problems with us connecting to older ssh servers.
I am particularly interested / worried about sshd from RHEL 5, 6 & 7 for virt-p2v and virt-v2v conversions. This broke before, requiring us to advise users to set the global policy for the machine to LEGACY (thus ironically weakening crypto for everything).
Also I have some ancient network equipment that cannot be upgraded but needs older ssh protocols. I can't connect to it from Fedora unless I set the crypto policy to LEGACY.
Anyway I'm wondering if the SHA-1 change will impact ssh further?
Rich.
-- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com virt-df lists disk usage of guests without needing to install any software inside the virtual machine. Supports Linux and Windows. http://people.redhat.com/~rjones/virt-df/ _______________________________________________ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
On Wed, Mar 09, 2022 at 02:54:40PM +0100, Dmitry Belyavskiy wrote:
At least RHEL 6 issues can be fixed server-side generating an ecdsa key...
If you can log in ...
This doesn't really work for us because of the way virt-v2v works -- it wants to ssh to the RHEL 5/6/7 server to fetch some files with as little pre-configuration as possible.
Rich.
On Wed, Mar 9, 2022 at 2:47 PM Richard W.M. Jones rjones@redhat.com wrote:
Previous tightening of crypto defaults caused problems with us connecting to older ssh servers. I am particularly interested / worried about sshd from RHEL 5, 6 & 7 for virt-p2v and virt-v2v conversions. This broke before, requiring us to advise users to set the global policy for the machine to LEGACY (thus ironically weakening crypto for everything). Also I have some ancient network equipment that cannot be upgraded but needs older ssh protocols. I can't connect to it from Fedora unless I set the crypto policy to LEGACY. Anyway I'm wondering if the SHA-1 change will impact ssh further? Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/ ~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com virt-df lists disk usage of guests without needing to install any software inside the virtual machine. Supports Linux and Windows. http://people.redhat.com/~rjones/virt-df/ _______________________________________________ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/ code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/ devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/ fedora-infrastructure-- Dmitry Belyavskiy
devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Once upon a time, Richard W.M. Jones rjones@redhat.com said:
Previous tightening of crypto defaults caused problems with us connecting to older ssh servers.
I also have had trouble connecting to major vendor websites. The vendor response is just "works in Chrome and Firefox on Windows, must be your problem".
I am particularly interested / worried about sshd from RHEL 5, 6 & 7 for virt-p2v and virt-v2v conversions. This broke before, requiring us to advise users to set the global policy for the machine to LEGACY (thus ironically weakening crypto for everything).
Also I have some ancient network equipment that cannot be upgraded but needs older ssh protocols. I can't connect to it from Fedora unless I set the crypto policy to LEGACY.
Yeah, the model in general seems a little broken to me, especially as I found the policies are implemented unevenly (IIRC my problem was OpenSSL couldn't connect but GnuTLS could for example), which just leads to confusion.
I understand and approve of having good system-wide defaults, but there needs to be a way to connect to a specific site/device/whatever without having to lower the system-wide policy. For SSH, you can usually do that by adjusting the settings on a per-device basis on the command line or in ~/.ssh/config (setting PublickeyAcceptedKeyTypes, KexAlgorithms, HostKeyAlgorithms, and/or Ciphers as needed).
I had to SSH to a FreeBSD 4.x server last year! So many SSH config options required... it had been up without a reboot since 2007 IIRC.
I am very much not a UI/UX person, but Firefox and other browsers really could use a good way to override system crypto policy on a per-site basis.
On Wednesday, 09 March 2022 at 15:59, Chris Adams wrote: [...]
I am very much not a UI/UX person, but Firefox and other browsers really could use a good way to override system crypto policy on a per-site basis.
Have you opened a ticket in Firefox bugzilla?
Regards, Dominik
On Tue, Mar 8, 2022 at 1:40 PM Alexander Sosedkin asosedkin@redhat.com wrote:
Hello, community, I need your wisdom for planning a disruptive change.
Fedora 28 had https://fedoraproject.org/wiki/Changes/StrongCryptoSettings Fedora 33 had https://fedoraproject.org/wiki/Changes/StrongCryptoSettings2 I believe we should start planning for the next cryptographic defaults tightening. And next time it's gonna be even more disruptive because of SHA-1 (again).
SHA-1 is a hash function from 1995, which collision resistance is no longer to be relied upon for security [1]. At the same time, it's not like software has successfully migrated off it, not even close. It's not a question of "if" the world should migrate from it, sooner or later we must part ways with it. (Technically, some acute energy crisis or a collapse of civilization forever raising the costs of computations thousandfold would also do, but let's agree that migrating to a more modern hash is the way =)
We've been disabling it in TLS, but its usage is much wider than TLS. The next agonizing step is to restrict its usage for signatures on the cryptographic libraries level, with openssl being the scariest one.
Good news is, RHEL-9 is gonna lead the way and thus will take a lot of the hits first. Fedora doesn't have to pioneer it. Bad news is, Fedora has to follow suit someday anyway, and this brings me to how does one land such a change.
Given that RHEL 9 already switched the crypto-policy defaults, and we presume that future RHEL releases will not deviate from that, is this change already made in ELN? I honestly can't tell because it looks like it is, but I don't see any conditionalization in the spec file that would lead me to believe the same default isn't already flipped in Rawhide. That leaves me wondering what needs to be changed and where at this point.
josh
Fedora is a large distribution with short release cycles, and the only realistic way to weed out its reliance on SHA-1 signatures from all of its numerous dark corners is to break them. Make creation and verification fail in default configuration. But it's unreasonable to just wait for, say, Fedora 37 branch-off and break it in Rawhide for Fedora 38. The fallout will just be too big.
Maintainers need time to get bugs, look into them, think, analyze, react and test --- and that's just if it fails correctly! Unfortunately, it's not just that the error paths are as dusty as they get because the program counter has never set foot on them before. Some maintainers might even find that picking a different hash function renders their code non-interoperable, or even that protocols they implement have SHA-1 hardcoded in the spec. Or that everything is ready, but real world deployments need another decade. Or that on-disk formats are just hard to change and migrate. Took git years to migrate from SHA-1, and some others haven't even started. There are gonna be investigations, planning, exceptions, upstream changes, opt-out mechanisms, arguing, compromises, waiting out, all kinds of things. It's gonna be big. Too big for a single release cycle.
But how does one land something and give the distribution the extra cycles needed to react? That's not really clear to me.
An obvious thing is to announce it in one cycle and land in another one. The downsides are well-documented in "The Hitchhiker's Guide to the Galaxy": announcements are one weak measure, and then it's too late.
A second scheme I can come up with is a "jump scare". Break the functionality in Fedora 37 Rawhide, make most of the affected people realize the depth of the problem, then unbreak it. Break again for Fedora 38 and never fix.
This could also be extended into "let one stable release slide'. Break in 37 Rawhide, unbreak on branched off 37, but never in Rawhide.
But these are all rather... crude? Sure there should be better ways, preferably something explored before. I'm all for pulling this tooth out smoothly, but I need hints on how to do it. I hope that together we can devise a better plan than these.
So, how does one land a change that's bigger than a release cycle?
[1] https://eprint.iacr.org/2020/014 _______________________________________________ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
On Wed, Mar 23, 2022 at 12:51 AM Josh Boyer jwboyer@fedoraproject.org wrote:
On Tue, Mar 8, 2022 at 1:40 PM Alexander Sosedkin asosedkin@redhat.com wrote:
Hello, community, I need your wisdom for planning a disruptive change.
Fedora 28 had https://fedoraproject.org/wiki/Changes/StrongCryptoSettings Fedora 33 had https://fedoraproject.org/wiki/Changes/StrongCryptoSettings2 I believe we should start planning for the next cryptographic defaults tightening. And next time it's gonna be even more disruptive because of SHA-1 (again).
SHA-1 is a hash function from 1995, which collision resistance is no longer to be relied upon for security [1]. At the same time, it's not like software has successfully migrated off it, not even close. It's not a question of "if" the world should migrate from it, sooner or later we must part ways with it. (Technically, some acute energy crisis or a collapse of civilization forever raising the costs of computations thousandfold would also do, but let's agree that migrating to a more modern hash is the way =)
We've been disabling it in TLS, but its usage is much wider than TLS. The next agonizing step is to restrict its usage for signatures on the cryptographic libraries level, with openssl being the scariest one.
Good news is, RHEL-9 is gonna lead the way and thus will take a lot of the hits first. Fedora doesn't have to pioneer it. Bad news is, Fedora has to follow suit someday anyway, and this brings me to how does one land such a change.
Given that RHEL 9 already switched the crypto-policy defaults, and we presume that future RHEL releases will not deviate from that, is this change already made in ELN? I honestly can't tell because it looks like it is, but I don't see any conditionalization in the spec file that would lead me to believe the same default isn't already flipped in Rawhide. That leaves me wondering what needs to be changed and where at this point.
josh
No, it's neither in ELN nor in Rawhide yet.
Fedora is a large distribution with short release cycles, and the only realistic way to weed out its reliance on SHA-1 signatures from all of its numerous dark corners is to break them. Make creation and verification fail in default configuration. But it's unreasonable to just wait for, say, Fedora 37 branch-off and break it in Rawhide for Fedora 38. The fallout will just be too big.
Maintainers need time to get bugs, look into them, think, analyze, react and test --- and that's just if it fails correctly! Unfortunately, it's not just that the error paths are as dusty as they get because the program counter has never set foot on them before. Some maintainers might even find that picking a different hash function renders their code non-interoperable, or even that protocols they implement have SHA-1 hardcoded in the spec. Or that everything is ready, but real world deployments need another decade. Or that on-disk formats are just hard to change and migrate. Took git years to migrate from SHA-1, and some others haven't even started. There are gonna be investigations, planning, exceptions, upstream changes, opt-out mechanisms, arguing, compromises, waiting out, all kinds of things. It's gonna be big. Too big for a single release cycle.
But how does one land something and give the distribution the extra cycles needed to react? That's not really clear to me.
An obvious thing is to announce it in one cycle and land in another one. The downsides are well-documented in "The Hitchhiker's Guide to the Galaxy": announcements are one weak measure, and then it's too late.
A second scheme I can come up with is a "jump scare". Break the functionality in Fedora 37 Rawhide, make most of the affected people realize the depth of the problem, then unbreak it. Break again for Fedora 38 and never fix.
This could also be extended into "let one stable release slide'. Break in 37 Rawhide, unbreak on branched off 37, but never in Rawhide.
But these are all rather... crude? Sure there should be better ways, preferably something explored before. I'm all for pulling this tooth out smoothly, but I need hints on how to do it. I hope that together we can devise a better plan than these.
So, how does one land a change that's bigger than a release cycle?
[1] https://eprint.iacr.org/2020/014 _______________________________________________ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
On Wed, Mar 23, 2022 at 7:05 AM Alexander Sosedkin asosedkin@redhat.com wrote:
On Wed, Mar 23, 2022 at 12:51 AM Josh Boyer jwboyer@fedoraproject.org wrote:
On Tue, Mar 8, 2022 at 1:40 PM Alexander Sosedkin asosedkin@redhat.com wrote:
Hello, community, I need your wisdom for planning a disruptive change.
Fedora 28 had https://fedoraproject.org/wiki/Changes/StrongCryptoSettings Fedora 33 had https://fedoraproject.org/wiki/Changes/StrongCryptoSettings2 I believe we should start planning for the next cryptographic defaults tightening. And next time it's gonna be even more disruptive because of SHA-1 (again).
SHA-1 is a hash function from 1995, which collision resistance is no longer to be relied upon for security [1]. At the same time, it's not like software has successfully migrated off it, not even close. It's not a question of "if" the world should migrate from it, sooner or later we must part ways with it. (Technically, some acute energy crisis or a collapse of civilization forever raising the costs of computations thousandfold would also do, but let's agree that migrating to a more modern hash is the way =)
We've been disabling it in TLS, but its usage is much wider than TLS. The next agonizing step is to restrict its usage for signatures on the cryptographic libraries level, with openssl being the scariest one.
Good news is, RHEL-9 is gonna lead the way and thus will take a lot of the hits first. Fedora doesn't have to pioneer it. Bad news is, Fedora has to follow suit someday anyway, and this brings me to how does one land such a change.
Given that RHEL 9 already switched the crypto-policy defaults, and we presume that future RHEL releases will not deviate from that, is this change already made in ELN? I honestly can't tell because it looks like it is, but I don't see any conditionalization in the spec file that would lead me to believe the same default isn't already flipped in Rawhide. That leaves me wondering what needs to be changed and where at this point.
josh
No, it's neither in ELN nor in Rawhide yet.
Could you please make this change in ELN? It might actually prove fruitful for the broader Fedora activity because it gives you an isolated base to see what happens.
If we already know there are other similar changes for future RHEL releases, making those in ELN now would go a long way towards sorting those out as well. That can be a follow-on after SHA-1 though, which is certainly more pressing.
josh
On Tue, Mar 8, 2022 at 7:40 PM Alexander Sosedkin asosedkin@redhat.com wrote:
Hello, community, I need your wisdom for planning a disruptive change.
Fedora 28 had https://fedoraproject.org/wiki/Changes/StrongCryptoSettings Fedora 33 had https://fedoraproject.org/wiki/Changes/StrongCryptoSettings2 I believe we should start planning for the next cryptographic defaults tightening. And next time it's gonna be even more disruptive because of SHA-1 (again).
SHA-1 is a hash function from 1995, which collision resistance is no longer to be relied upon for security [1]. At the same time, it's not like software has successfully migrated off it, not even close. It's not a question of "if" the world should migrate from it, sooner or later we must part ways with it. (Technically, some acute energy crisis or a collapse of civilization forever raising the costs of computations thousandfold would also do, but let's agree that migrating to a more modern hash is the way =)
We've been disabling it in TLS, but its usage is much wider than TLS. The next agonizing step is to restrict its usage for signatures on the cryptographic libraries level, with openssl being the scariest one.
Good news is, RHEL-9 is gonna lead the way and thus will take a lot of the hits first. Fedora doesn't have to pioneer it. Bad news is, Fedora has to follow suit someday anyway, and this brings me to how does one land such a change.
Fedora is a large distribution with short release cycles, and the only realistic way to weed out its reliance on SHA-1 signatures from all of its numerous dark corners is to break them. Make creation and verification fail in default configuration. But it's unreasonable to just wait for, say, Fedora 37 branch-off and break it in Rawhide for Fedora 38. The fallout will just be too big.
Maintainers need time to get bugs, look into them, think, analyze, react and test --- and that's just if it fails correctly! Unfortunately, it's not just that the error paths are as dusty as they get because the program counter has never set foot on them before. Some maintainers might even find that picking a different hash function renders their code non-interoperable, or even that protocols they implement have SHA-1 hardcoded in the spec. Or that everything is ready, but real world deployments need another decade. Or that on-disk formats are just hard to change and migrate. Took git years to migrate from SHA-1, and some others haven't even started. There are gonna be investigations, planning, exceptions, upstream changes, opt-out mechanisms, arguing, compromises, waiting out, all kinds of things. It's gonna be big. Too big for a single release cycle.
But how does one land something and give the distribution the extra cycles needed to react? That's not really clear to me.
An obvious thing is to announce it in one cycle and land in another one. The downsides are well-documented in "The Hitchhiker's Guide to the Galaxy": announcements are one weak measure, and then it's too late.
A second scheme I can come up with is a "jump scare". Break the functionality in Fedora 37 Rawhide, make most of the affected people realize the depth of the problem, then unbreak it. Break again for Fedora 38 and never fix.
This could also be extended into "let one stable release slide'. Break in 37 Rawhide, unbreak on branched off 37, but never in Rawhide.
But these are all rather... crude? Sure there should be better ways, preferably something explored before. I'm all for pulling this tooth out smoothly, but I need hints on how to do it. I hope that together we can devise a better plan than these.
So, how does one land a change that's bigger than a release cycle?
"You know these lights in the theaters that go out gradually? When the guy ve-ery slo-o-owly pulls the plug out?" - a joke from my childhood.
Hello, it's been quiet for a while, and I've been busy but kept thinking about all the useful feedback you folks gave me. Not that it made me flesh out a perfect plan, but hopefully at least a less terrible one.
Regarding smudging the change in time, how does the following three-phaser sound?
Phase 1 ("Wake-up Call"): In Fedora 37, disable SHA-1 signatures verification/creation in FUTURE policy, i.e. opt-in only. Come up with some logging solution; I'd prefer something non-invasive like eBPF USDT probes [2], but maybe even stderr could work, you've been moderately convincing. (FUTURE change is *maybe* doable in F36, but not logging.) Announce it as a system-wide change anyway for visibility, call for Test Days to report which apps/workflows rely on SHA-1 signatures either from the logs or from opting into blocking operations and seeing what starts failing hard. That'd have to be very actively called for to make an impact, impact that'd mostly be just maintainers thinking what will they do in
Phase 2 ("Jump Scare"): As soon as f37 branch-off happens, disable signature verification in DEFAULT in *38 rawhide*. Cue an influx of bugreports because things get broken for all testers and not just the ones who opt in. I anticipate this to be the most eye-opening step even if we test a lot in the previous phase, so to smooth it out more we then *revert* the change in 38 before the release, so the released Fedora behaves just like in 37 and whatever wasn't sorted out in time gets an extra cycle. A second Fedora change should be filed for visibility, but clearly stating this will not affect f38 released.
Phase 3 ("Return of the Panik"): And then Fedora 39 comes, where the revert hasn't happened, goes through the whole release cycle, but this time the change goes through and reaches stable. Again, a system-wide change, a third one for the same thing.
With the 37-38-39 numbers, that'd mean the change reaching the users in autumn 2023, with lead times of: ~ 3.5 cycles for the most proactive developers to see this thread and panic ~ 3 cycles for the testers to proactively report bugs (logging/opting in) ~ 2 cycles to address everything else coming from rawhide testing before it reaches stable by either switching to some other algorithm, making the users explicitly opt into trusting SHA-1 signatures somehow, or, in the most high-profile cases, having a widely publicised exception (and some plan for the future).
Questions: * Do you find this smudging reasonable? * The usual tightening of the other less controversial algorithms, should it follow the same smudging/reverting plan since we're going through all that hassle anyway? * Does the 37-38-39 timeframe feel right? * Do I need to first run this contraption of a plan by FESCo or some other smart folks? * Is there a better signalling mechanism than filing 3 system-wide changes? * What'd be the right mechanism for others to take the wheel if everything goes sideways and the need arises to revise the plan mid-execution?
Other kinds of input are, of course, also appreciated. Even the calls to magically attain the mutually exclusive goals of offering secure defaults while not breaking insecure workflows that don't offer actual solutions might serve as a useful mood check. I know it ain't the best plan. Let's figure out the right thing to do.
On Tue, Mar 29, 2022 at 08:12:47PM +0200, Alexander Sosedkin wrote:
"You know these lights in the theaters that go out gradually? When the guy ve-ery slo-o-owly pulls the plug out?"
- a joke from my childhood.
Hello, it's been quiet for a while, and I've been busy but kept thinking about all the useful feedback you folks gave me. Not that it made me flesh out a perfect plan, but hopefully at least a less terrible one.
Regarding smudging the change in time, how does the following three-phaser sound?
Might work. :) A few comments inline...
Phase 1 ("Wake-up Call"): In Fedora 37, disable SHA-1 signatures verification/creation in FUTURE policy, i.e. opt-in only. Come up with some logging solution; I'd prefer something non-invasive like eBPF USDT probes [2], but maybe even stderr could work, you've been moderately convincing. (FUTURE change is *maybe* doable in F36, but not logging.)
Well, we just shipped beta today, so I think it's too late to land any f36 changes at this point.
Announce it as a system-wide change anyway for visibility, call for Test Days to report which apps/workflows rely on SHA-1 signatures either from the logs or from opting into blocking operations and seeing what starts failing hard. That'd have to be very actively called for to make an impact, impact that'd mostly be just maintainers thinking what will they do in
Just a related note here, FUTURE also breaks installing anything via dnf with the default metalink setup. This is because digicert (where we get our *.fedoraproject.org wildcard cert) seems to always issue certs from it's 2048bit CA. :( (If anyone knows how to get them to issue from a newer CA that works please let me know)
Phase 2 ("Jump Scare"): As soon as f37 branch-off happens, disable signature verification in DEFAULT in *38 rawhide*. Cue an influx of bugreports because things get broken for all testers and not just the ones who opt in. I anticipate this to be the most eye-opening step even if we test a lot in the previous phase, so to smooth it out more we then *revert* the change in 38 before the release, so the released Fedora behaves just like in 37 and whatever wasn't sorted out in time gets an extra cycle.
Right before the release? Or right before Beta? or ?
People kind of expect the beta will be something they can test and will behave as the final, so changing things after beta seems like a bad idea in general. That said, shipping beta with it would get a lot more exposure.
A second Fedora change should be filed for visibility, but clearly stating this will not affect f38 released.
Phase 3 ("Return of the Panik"): And then Fedora 39 comes, where the revert hasn't happened, goes through the whole release cycle, but this time the change goes through and reaches stable. Again, a system-wide change, a third one for the same thing.
With the 37-38-39 numbers, that'd mean the change reaching the users in autumn 2023, with lead times of: ~ 3.5 cycles for the most proactive developers to see this thread and panic ~ 3 cycles for the testers to proactively report bugs (logging/opting in) ~ 2 cycles to address everything else coming from rawhide testing before it reaches stable by either switching to some other algorithm, making the users explicitly opt into trusting SHA-1 signatures somehow, or, in the most high-profile cases, having a widely publicised exception (and some plan for the future).
Questions:
- Do you find this smudging reasonable?
I think it's probibly the best we can do.
- The usual tightening of the other less controversial algorithms, should it follow the same smudging/reverting plan since we're going through all that hassle anyway?
I don't think they would need to have this long a runway.
- Does the 37-38-39 timeframe feel right?
- Do I need to first run this contraption of a plan by FESCo or some other smart folks?
Well, I think run it by everyone on this list. I don't think people will hold back. ;)
- Is there a better signalling mechanism than filing 3 system-wide changes?
- What'd be the right mechanism for others to take the wheel if everything goes sideways and the need arises to revise the plan mid-execution?
I wonder if it would make sense to have a checkpoint before the revert in f38, and if things look substantually less broken than we fear, we just don't revert it and let it go out in f38. But that might muddy up the communications.
Other kinds of input are, of course, also appreciated. Even the calls to magically attain the mutually exclusive goals of offering secure defaults while not breaking insecure workflows that don't offer actual solutions might serve as a useful mood check. I know it ain't the best plan. Let's figure out the right thing to do.
Thanks for taking input on this and being willing to work out the best way forward.
kevin
FWIW Alexander's plan sounds reasonable to me.
On Tue, Mar 29 2022 at 03:34:49 PM -0700, Kevin Fenzi kevin@scrye.com wrote:
Well, we just shipped beta today, so I think it's too late to land any f36 changes at this point.
This is a non-default configuration that I strongly suspect nobody or almost nobody uses, except for testing. Changing this prior to F36 stable release seems like no big deal.
Michael
On Tue, Mar 29, 2022 at 03:34:49PM -0700, Kevin Fenzi wrote:
On Tue, Mar 29, 2022 at 08:12:47PM +0200, Alexander Sosedkin wrote:
"You know these lights in the theaters that go out gradually? When the guy ve-ery slo-o-owly pulls the plug out?"
- a joke from my childhood.
Hello, it's been quiet for a while, and I've been busy but kept thinking about all the useful feedback you folks gave me. Not that it made me flesh out a perfect plan, but hopefully at least a less terrible one.
Regarding smudging the change in time, how does the following three-phaser sound?
Might work. :) A few comments inline...
Phase 1 ("Wake-up Call"): In Fedora 37, disable SHA-1 signatures verification/creation in FUTURE policy, i.e. opt-in only. Come up with some logging solution; I'd prefer something non-invasive like eBPF USDT probes [2], but maybe even stderr could work, you've been moderately convincing. (FUTURE change is *maybe* doable in F36, but not logging.)
Well, we just shipped beta today, so I think it's too late to land any f36 changes at this point.
Announce it as a system-wide change anyway for visibility, call for Test Days to report which apps/workflows rely on SHA-1 signatures either from the logs or from opting into blocking operations and seeing what starts failing hard. That'd have to be very actively called for to make an impact, impact that'd mostly be just maintainers thinking what will they do in
Just a related note here, FUTURE also breaks installing anything via dnf with the default metalink setup. This is because digicert (where we get our *.fedoraproject.org wildcard cert) seems to always issue certs from it's 2048bit CA. :( (If anyone knows how to get them to issue from a newer CA that works please let me know)
This means that probably almost nobody uses FUTURE :(
On Tue, Mar 29, 2022 at 08:05:00PM -0500, Michael Catanzaro wrote:
On Tue, Mar 29 2022 at 03:34:49 PM -0700, Kevin Fenzi kevin@scrye.com wrote:
Well, we just shipped beta today, so I think it's too late to land any f36 changes at this point.
This is a non-default configuration that I strongly suspect nobody or almost nobody uses, except for testing. Changing this prior to F36 stable release seems like no big deal.
Considering that nobody uses FUTURE, I agree that we can do it for F36. This effectively moves everything one release forward, which is probably good. I don't think we want to drag this out *too* much.
Phase 2 ("Jump Scare"): As soon as f37 branch-off happens, disable signature verification in DEFAULT in *38 rawhide*. Cue an influx of bugreports because things get broken for all testers and not just the ones who opt in. I anticipate this to be the most eye-opening step even if we test a lot in the previous phase, so to smooth it out more we then *revert* the change in 38 before the release, so the released Fedora behaves just like in 37 and whatever wasn't sorted out in time gets an extra cycle.
Right before the release? Or right before Beta? or ?
People kind of expect the beta will be something they can test and will behave as the final, so changing things after beta seems like a bad idea in general. That said, shipping beta with it would get a lot more exposure.
Agreed. Any user-visible changes should be done before Beta.
A second Fedora change should be filed for visibility, but clearly stating this will not affect f38 released.
Phase 3 ("Return of the Panik"): And then Fedora 39 comes, where the revert hasn't happened, goes through the whole release cycle, but this time the change goes through and reaches stable. Again, a system-wide change, a third one for the same thing.
With the 37-38-39 numbers, that'd mean the change reaching the users in autumn 2023, with lead times of: ~ 3.5 cycles for the most proactive developers to see this thread and panic ~ 3 cycles for the testers to proactively report bugs (logging/opting in) ~ 2 cycles to address everything else coming from rawhide testing before it reaches stable by either switching to some other algorithm, making the users explicitly opt into trusting SHA-1 signatures somehow, or, in the most high-profile cases, having a widely publicised exception (and some plan for the future).
Questions:
- Do you find this smudging reasonable?
I think it's probibly the best we can do.
- The usual tightening of the other less controversial algorithms, should it follow the same smudging/reverting plan since we're going through all that hassle anyway?
I don't think they would need to have this long a runway.
- Does the 37-38-39 timeframe feel right?
- Do I need to first run this contraption of a plan by FESCo or some other smart folks?
Well, I think run it by everyone on this list. I don't think people will hold back. ;)
- Is there a better signalling mechanism than filing 3 system-wide changes?
- What'd be the right mechanism for others to take the wheel if everything goes sideways and the need arises to revise the plan mid-execution?
Please just file a late change for F36. We can then manually link the same page into F37 and F38 (or even F39 if appropriate).
I wonder if it would make sense to have a checkpoint before the revert in f38, and if things look substantually less broken than we fear, we just don't revert it and let it go out in f38. But that might muddy up the communications.
Other kinds of input are, of course, also appreciated. Even the calls to magically attain the mutually exclusive goals of offering secure defaults while not breaking insecure workflows that don't offer actual solutions might serve as a useful mood check. I know it ain't the best plan. Let's figure out the right thing to do.
Thanks for taking input on this and being willing to work out the best way forward.
Seconded. Thank you.
Zbyszek
On Thu, Apr 7, 2022 at 9:06 PM Zbigniew Jędrzejewski-Szmek zbyszek@in.waw.pl wrote:
On Tue, Mar 29, 2022 at 03:34:49PM -0700, Kevin Fenzi wrote:
On Tue, Mar 29, 2022 at 08:12:47PM +0200, Alexander Sosedkin wrote:
"You know these lights in the theaters that go out gradually? When the guy ve-ery slo-o-owly pulls the plug out?"
- a joke from my childhood.
Hello, it's been quiet for a while, and I've been busy but kept thinking about all the useful feedback you folks gave me. Not that it made me flesh out a perfect plan, but hopefully at least a less terrible one.
Regarding smudging the change in time, how does the following three-phaser sound?
Might work. :) A few comments inline...
Phase 1 ("Wake-up Call"): In Fedora 37, disable SHA-1 signatures verification/creation in FUTURE policy, i.e. opt-in only. Come up with some logging solution; I'd prefer something non-invasive like eBPF USDT probes [2], but maybe even stderr could work, you've been moderately convincing. (FUTURE change is *maybe* doable in F36, but not logging.)
Well, we just shipped beta today, so I think it's too late to land any f36 changes at this point.
Announce it as a system-wide change anyway for visibility, call for Test Days to report which apps/workflows rely on SHA-1 signatures either from the logs or from opting into blocking operations and seeing what starts failing hard. That'd have to be very actively called for to make an impact, impact that'd mostly be just maintainers thinking what will they do in
Just a related note here, FUTURE also breaks installing anything via dnf with the default metalink setup. This is because digicert (where we get our *.fedoraproject.org wildcard cert) seems to always issue certs from it's 2048bit CA. :( (If anyone knows how to get them to issue from a newer CA that works please let me know)
This means that probably almost nobody uses FUTURE :(
On Tue, Mar 29, 2022 at 08:05:00PM -0500, Michael Catanzaro wrote:
On Tue, Mar 29 2022 at 03:34:49 PM -0700, Kevin Fenzi kevin@scrye.com wrote:
Well, we just shipped beta today, so I think it's too late to land any f36 changes at this point.
This is a non-default configuration that I strongly suspect nobody or almost nobody uses, except for testing. Changing this prior to F36 stable release seems like no big deal.
Considering that nobody uses FUTURE, I agree that we can do it for F36. This effectively moves everything one release forward, which is probably good.
I'm afraid not, because there's also a logging phase proposed, one that shouldn't go to F36 and keeps the plan from shifting forward. Meaning: F36: FUTURE update F37: FUTURE update + logging F38: DEFAULT update in rawhide only F39: DEFAULT update that reaches release hasn't really shortened.
I don't think we want to drag this out *too* much.
And neither do we want to rush it. The more ahead of the world we would be, the more work it entails for us, so it's a balancing act.
Phase 2 ("Jump Scare"): As soon as f37 branch-off happens, disable signature verification in DEFAULT in *38 rawhide*. Cue an influx of bugreports because things get broken for all testers and not just the ones who opt in. I anticipate this to be the most eye-opening step even if we test a lot in the previous phase, so to smooth it out more we then *revert* the change in 38 before the release, so the released Fedora behaves just like in 37 and whatever wasn't sorted out in time gets an extra cycle.
Right before the release? Or right before Beta? or ?
People kind of expect the beta will be something they can test and will behave as the final, so changing things after beta seems like a bad idea in general. That said, shipping beta with it would get a lot more exposure.
Agreed. Any user-visible changes should be done before Beta.
A second Fedora change should be filed for visibility, but clearly stating this will not affect f38 released.
Phase 3 ("Return of the Panik"): And then Fedora 39 comes, where the revert hasn't happened, goes through the whole release cycle, but this time the change goes through and reaches stable. Again, a system-wide change, a third one for the same thing.
With the 37-38-39 numbers, that'd mean the change reaching the users in autumn 2023, with lead times of: ~ 3.5 cycles for the most proactive developers to see this thread and panic ~ 3 cycles for the testers to proactively report bugs (logging/opting in) ~ 2 cycles to address everything else coming from rawhide testing before it reaches stable by either switching to some other algorithm, making the users explicitly opt into trusting SHA-1 signatures somehow, or, in the most high-profile cases, having a widely publicised exception (and some plan for the future).
Questions:
- Do you find this smudging reasonable?
I think it's probibly the best we can do.
- The usual tightening of the other less controversial algorithms, should it follow the same smudging/reverting plan since we're going through all that hassle anyway?
I don't think they would need to have this long a runway.
- Does the 37-38-39 timeframe feel right?
- Do I need to first run this contraption of a plan by FESCo or some other smart folks?
Well, I think run it by everyone on this list. I don't think people will hold back. ;)
- Is there a better signalling mechanism than filing 3 system-wide changes?
- What'd be the right mechanism for others to take the wheel if everything goes sideways and the need arises to revise the plan mid-execution?
Please just file a late change for F36. We can then manually link the same page into F37 and F38 (or even F39 if appropriate).
Noted, 4 system-wide changes =)
I wonder if it would make sense to have a checkpoint before the revert in f38, and if things look substantually less broken than we fear, we just don't revert it and let it go out in f38. But that might muddy up the communications.
Other kinds of input are, of course, also appreciated. Even the calls to magically attain the mutually exclusive goals of offering secure defaults while not breaking insecure workflows that don't offer actual solutions might serve as a useful mood check. I know it ain't the best plan. Let's figure out the right thing to do.
Thanks for taking input on this and being willing to work out the best way forward.
Seconded. Thank you.
Zbyszek
Dne 08. 03. 22 v 19:40 Alexander Sosedkin napsal(a):
Hello, community, I need your wisdom for planning a disruptive change.
Fedora 28 had https://fedoraproject.org/wiki/Changes/StrongCryptoSettings Fedora 33 had https://fedoraproject.org/wiki/Changes/StrongCryptoSettings2 I believe we should start planning for the next cryptographic defaults tightening. And next time it's gonna be even more disruptive because of SHA-1 (again).
SHA-1 is a hash function from 1995, which collision resistance is no longer to be relied upon for security [1]. At the same time, it's not like software has successfully migrated off it, not even close. It's not a question of "if" the world should migrate from it, sooner or later we must part ways with it. (Technically, some acute energy crisis or a collapse of civilization forever raising the costs of computations thousandfold would also do, but let's agree that migrating to a more modern hash is the way =)
We've been disabling it in TLS, but its usage is much wider than TLS. The next agonizing step is to restrict its usage for signatures on the cryptographic libraries level, with openssl being the scariest one.
Good news is, RHEL-9 is gonna lead the way and thus will take a lot of the hits first.
Alexander,
Could this be enabled in ELN? This is not really question but suggestion. It is unfortunate, that ELN, while intermediate step for c9s does not have this enabled yet.
For example, we are working on Ruby 3.1 for c9s [1] just to figure that while everything just works in Fedora/ELN, it does not work in c9s. Maybe working with ELN would make the migrations easier also for Fedora.
Vít
[1] https://gitlab.com/redhat/centos-stream/rpms/ruby/-/merge_requests/10
Fedora doesn't have to pioneer it. Bad news is, Fedora has to follow suit someday anyway, and this brings me to how does one land such a change.
Fedora is a large distribution with short release cycles, and the only realistic way to weed out its reliance on SHA-1 signatures from all of its numerous dark corners is to break them. Make creation and verification fail in default configuration. But it's unreasonable to just wait for, say, Fedora 37 branch-off and break it in Rawhide for Fedora 38. The fallout will just be too big.
Maintainers need time to get bugs, look into them, think, analyze, react and test --- and that's just if it fails correctly! Unfortunately, it's not just that the error paths are as dusty as they get because the program counter has never set foot on them before. Some maintainers might even find that picking a different hash function renders their code non-interoperable, or even that protocols they implement have SHA-1 hardcoded in the spec. Or that everything is ready, but real world deployments need another decade. Or that on-disk formats are just hard to change and migrate. Took git years to migrate from SHA-1, and some others haven't even started. There are gonna be investigations, planning, exceptions, upstream changes, opt-out mechanisms, arguing, compromises, waiting out, all kinds of things. It's gonna be big. Too big for a single release cycle.
But how does one land something and give the distribution the extra cycles needed to react? That's not really clear to me.
An obvious thing is to announce it in one cycle and land in another one. The downsides are well-documented in "The Hitchhiker's Guide to the Galaxy": announcements are one weak measure, and then it's too late.
A second scheme I can come up with is a "jump scare". Break the functionality in Fedora 37 Rawhide, make most of the affected people realize the depth of the problem, then unbreak it. Break again for Fedora 38 and never fix.
This could also be extended into "let one stable release slide'. Break in 37 Rawhide, unbreak on branched off 37, but never in Rawhide.
But these are all rather... crude? Sure there should be better ways, preferably something explored before. I'm all for pulling this tooth out smoothly, but I need hints on how to do it. I hope that together we can devise a better plan than these.
So, how does one land a change that's bigger than a release cycle?
[1] https://eprint.iacr.org/2020/014 _______________________________________________ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
On Wed, Apr 6, 2022 at 1:00 PM Vít Ondruch vondruch@redhat.com wrote:
Dne 08. 03. 22 v 19:40 Alexander Sosedkin napsal(a):
Hello, community, I need your wisdom for planning a disruptive change.
Fedora 28 had https://fedoraproject.org/wiki/Changes/StrongCryptoSettings Fedora 33 had https://fedoraproject.org/wiki/Changes/StrongCryptoSettings2 I believe we should start planning for the next cryptographic defaults tightening. And next time it's gonna be even more disruptive because of SHA-1 (again).
SHA-1 is a hash function from 1995, which collision resistance is no longer to be relied upon for security [1]. At the same time, it's not like software has successfully migrated off it, not even close. It's not a question of "if" the world should migrate from it, sooner or later we must part ways with it. (Technically, some acute energy crisis or a collapse of civilization forever raising the costs of computations thousandfold would also do, but let's agree that migrating to a more modern hash is the way =)
We've been disabling it in TLS, but its usage is much wider than TLS. The next agonizing step is to restrict its usage for signatures on the cryptographic libraries level, with openssl being the scariest one.
Good news is, RHEL-9 is gonna lead the way and thus will take a lot of the hits first.
Alexander,
Could this be enabled in ELN? This is not really question but suggestion. It is unfortunate, that ELN, while intermediate step for c9s does not have this enabled yet.
For example, we are working on Ruby 3.1 for c9s [1] just to figure that while everything just works in Fedora/ELN, it does not work in c9s. Maybe working with ELN would make the migrations easier also for Fedora.
Vít
It's on my plans, but it'd take some time as it's not trivial.
[1] https://gitlab.com/redhat/centos-stream/rpms/ruby/-/merge_requests/10
Fedora doesn't have to pioneer it. Bad news is, Fedora has to follow suit someday anyway, and this brings me to how does one land such a change.
Fedora is a large distribution with short release cycles, and the only realistic way to weed out its reliance on SHA-1 signatures from all of its numerous dark corners is to break them. Make creation and verification fail in default configuration. But it's unreasonable to just wait for, say, Fedora 37 branch-off and break it in Rawhide for Fedora 38. The fallout will just be too big.
Maintainers need time to get bugs, look into them, think, analyze, react and test --- and that's just if it fails correctly! Unfortunately, it's not just that the error paths are as dusty as they get because the program counter has never set foot on them before. Some maintainers might even find that picking a different hash function renders their code non-interoperable, or even that protocols they implement have SHA-1 hardcoded in the spec. Or that everything is ready, but real world deployments need another decade. Or that on-disk formats are just hard to change and migrate. Took git years to migrate from SHA-1, and some others haven't even started. There are gonna be investigations, planning, exceptions, upstream changes, opt-out mechanisms, arguing, compromises, waiting out, all kinds of things. It's gonna be big. Too big for a single release cycle.
But how does one land something and give the distribution the extra cycles needed to react? That's not really clear to me.
An obvious thing is to announce it in one cycle and land in another one. The downsides are well-documented in "The Hitchhiker's Guide to the Galaxy": announcements are one weak measure, and then it's too late.
A second scheme I can come up with is a "jump scare". Break the functionality in Fedora 37 Rawhide, make most of the affected people realize the depth of the problem, then unbreak it. Break again for Fedora 38 and never fix.
This could also be extended into "let one stable release slide'. Break in 37 Rawhide, unbreak on branched off 37, but never in Rawhide.
But these are all rather... crude? Sure there should be better ways, preferably something explored before. I'm all for pulling this tooth out smoothly, but I need hints on how to do it. I hope that together we can devise a better plan than these.
So, how does one land a change that's bigger than a release cycle?
[1] https://eprint.iacr.org/2020/014 _______________________________________________ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
On Wed, Apr 6, 2022 at 7:09 AM Alexander Sosedkin asosedkin@redhat.com wrote:
On Wed, Apr 6, 2022 at 1:00 PM Vít Ondruch vondruch@redhat.com wrote:
...
Alexander,
Could this be enabled in ELN? This is not really question but suggestion. It is unfortunate, that ELN, while intermediate step for c9s does not have this enabled yet.
For example, we are working on Ruby 3.1 for c9s [1] just to figure that while everything just works in Fedora/ELN, it does not work in c9s. Maybe working with ELN would make the migrations easier also for Fedora.
Vít
It's on my plans, but it'd take some time as it's not trivial.
Alexander, I can try to help you with this if you need it. Just let me know.
On Wed, Apr 6, 2022 at 10:43 AM Stephen Gallagher sgallagh@redhat.com wrote:
On Wed, Apr 6, 2022 at 7:09 AM Alexander Sosedkin asosedkin@redhat.com wrote:
On Wed, Apr 6, 2022 at 1:00 PM Vít Ondruch vondruch@redhat.com wrote:
...
Alexander,
Could this be enabled in ELN? This is not really question but suggestion. It is unfortunate, that ELN, while intermediate step for c9s does not have this enabled yet.
For example, we are working on Ruby 3.1 for c9s [1] just to figure that while everything just works in Fedora/ELN, it does not work in c9s. Maybe working with ELN would make the migrations easier also for Fedora.
Vít
It's on my plans, but it'd take some time as it's not trivial.
Alexander, I can try to help you with this if you need it. Just let me know.
I put together a potential solution for testing this in ELN and submitted an MR: https://src.fedoraproject.org/rpms/crypto-policies/pull-request/10
It's a little bit heavy-handed of an approach, but it should work.
Dne 06. 04. 22 v 22:29 Stephen Gallagher napsal(a):
On Wed, Apr 6, 2022 at 10:43 AM Stephen Gallagher sgallagh@redhat.com wrote:
On Wed, Apr 6, 2022 at 7:09 AM Alexander Sosedkin asosedkin@redhat.com wrote:
On Wed, Apr 6, 2022 at 1:00 PM Vít Ondruch vondruch@redhat.com wrote:
...
Alexander,
Could this be enabled in ELN? This is not really question but suggestion. It is unfortunate, that ELN, while intermediate step for c9s does not have this enabled yet.
For example, we are working on Ruby 3.1 for c9s [1] just to figure that while everything just works in Fedora/ELN, it does not work in c9s. Maybe working with ELN would make the migrations easier also for Fedora.
Vít
It's on my plans, but it'd take some time as it's not trivial.
Alexander, I can try to help you with this if you need it. Just let me know.
I put together a potential solution for testing this in ELN and submitted an MR: https://src.fedoraproject.org/rpms/crypto-policies/pull-request/10
Nice!
It's a little bit heavy-handed of an approach, but it should work.
Sorry for keeping asking naive questions, but is there a chance to do some mass rebuild prior this lands, to asses the impact? Better to avoid (complete?) breakage of ELN and give change to fix the (most outstanding) issues prior it lands.
Thx
Vít
devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
On Thu, Apr 7, 2022 at 4:53 AM Vít Ondruch vondruch@redhat.com wrote:
Dne 06. 04. 22 v 22:29 Stephen Gallagher napsal(a):
On Wed, Apr 6, 2022 at 10:43 AM Stephen Gallagher sgallagh@redhat.com wrote:
On Wed, Apr 6, 2022 at 7:09 AM Alexander Sosedkin asosedkin@redhat.com wrote:
On Wed, Apr 6, 2022 at 1:00 PM Vít Ondruch vondruch@redhat.com wrote:
...
Alexander,
Could this be enabled in ELN? This is not really question but suggestion. It is unfortunate, that ELN, while intermediate step for c9s does not have this enabled yet.
For example, we are working on Ruby 3.1 for c9s [1] just to figure that while everything just works in Fedora/ELN, it does not work in c9s. Maybe working with ELN would make the migrations easier also for Fedora.
Vít
It's on my plans, but it'd take some time as it's not trivial.
Alexander, I can try to help you with this if you need it. Just let me know.
I put together a potential solution for testing this in ELN and submitted an MR: https://src.fedoraproject.org/rpms/crypto-policies/pull-request/10
Nice!
It's a little bit heavy-handed of an approach, but it should work.
Sorry for keeping asking naive questions, but is there a chance to do some mass rebuild prior this lands, to asses the impact? Better to avoid (complete?) breakage of ELN and give change to fix the (most outstanding) issues prior it lands.
I'm not sure what benefit that would have. The crypto policies are generally applied at runtime, not build-time.
On Thursday, April 7, 2022 3:45:45 PM CEST Stephen Gallagher wrote:
Sorry for keeping asking naive questions, but is there a chance to do some mass rebuild prior this lands, to asses the impact? Better to avoid (complete?) breakage of ELN and give change to fix the (most outstanding) issues prior it lands.
I'm not sure what benefit that would have. The crypto policies are generally applied at runtime, not build-time.
The policies applied at run time can break builds though, especially when they include regression tests that use crypto.
Kamil
On Thu, Apr 7, 2022 at 10:12 AM Kamil Dudka kdudka@redhat.com wrote:
On Thursday, April 7, 2022 3:45:45 PM CEST Stephen Gallagher wrote:
Sorry for keeping asking naive questions, but is there a chance to do some mass rebuild prior this lands, to asses the impact? Better to avoid (complete?) breakage of ELN and give change to fix the (most outstanding) issues prior it lands.
I'm not sure what benefit that would have. The crypto policies are generally applied at runtime, not build-time.
The policies applied at run time can break builds though, especially when they include regression tests that use crypto.
Ah, that hadn't occurred to me. I'm still not sure it's worth running a mass-rebuild specifically for this, though. I think it might be better to clean things up as they come up and then piggy-back on the next Fedora mass-rebuild.
Dne 07. 04. 22 v 16:13 Stephen Gallagher napsal(a):
On Thu, Apr 7, 2022 at 10:12 AM Kamil Dudka kdudka@redhat.com wrote:
On Thursday, April 7, 2022 3:45:45 PM CEST Stephen Gallagher wrote:
Sorry for keeping asking naive questions, but is there a chance to do some mass rebuild prior this lands, to asses the impact? Better to avoid (complete?) breakage of ELN and give change to fix the (most outstanding) issues prior it lands.
I'm not sure what benefit that would have. The crypto policies are generally applied at runtime, not build-time.
The policies applied at run time can break builds though, especially when they include regression tests that use crypto.
Yep, that is precisely the case. Actually it is higher chance that the testsuites are problem then the runtime, because the test suites trying to assert every code path when during runtime likely just subset of functionality is used (not mentioning that test suites tends to use weaker cryptography, because it is faster and it won't cause security concerns).
Ah, that hadn't occurred to me. I'm still not sure it's worth running a mass-rebuild specifically for this, though. I think it might be better to clean things up as they come up and then piggy-back on the next Fedora mass-rebuild.
At minimum, you count with broken Ruby :)
Vít
Hi Stephen,
Stephen Gallagher sgallagh@redhat.com wrote:
On Wed, Apr 6, 2022 at 10:43 AM Stephen Gallagher sgallagh@redhat.com wrote:
I put together a potential solution for testing this in ELN and submitted an MR: https://src.fedoraproject.org/rpms/crypto-policies/pull-request/10
It's a little bit heavy-handed of an approach, but it should work.
Please keep in mind that Fedora’s openssl package currently does not have the patches that introduce the rh-allow-sha1-signatures option that would be required to do this. Merging your pull request without this change will cause all binaries that use OpenSSL 3.x (1.1 will ignore it) to fail to start.
See https://bugzilla.redhat.com/show_bug.cgi?id=2070977 which tracks implementing this. If you want to disable SHA-1 by default in ELN, we should probably also have an option to re-enable it using an environment variable for package build and test execution purposes. I’ll start working on this now.
Another status update for transparency purposes:
1. openssl-3.0.2-3 and crypto-policies-20220412-1.git97fe449 now distrust SHA-1 signatures in FUTURE policy or NO-SHA1 subpolicy. Meaning that updating the packages and issuing `update-crypto-policies --set FUTURE` / `update-crypto-policies --set DEFAULT:NO-SHA1` can be used to preview the impact on Fedora 36 / 37.
2. A decision has been made for Fedora ELN to track CentOS Stream 9 crypto-policies. A side-tag rebuild has been triggered prior to the switch, and has found ELN to be in a pretty broken shape in general. Work has been temporarily stalled on that side, but I hope to get back to it.
3. I've drafted the following wiki pages so far: https://fedoraproject.org/wiki/Changes/StrongCryptoSettings3Forewarning1 https://fedoraproject.org/wiki/SHA1SignaturesGuidance https://fedoraproject.org/wiki/WeakCryptographyException feedback is welcome as usual.
Another status update:
1. USDT/eBPF tracing turned out to be a fruitful logging approach. Clemens Lang has kindly added USDT probes to the latest openssl builds, traceable with a small tool [1] available from a copr [2]. Yes, this way it doesn't log into your face unpromptedly, like stderr logging would, but, in turn, it's so non-invasive that we're comfortable with bringing it to Fedora 36 as well. 2. crypto-policies has added FEDORA38 and TEST-FEDORA39 policies for not following the change and testing it early correspondingly. 3. change wiki pages have been drafted for all 4 proposed rollout phases, the Fedora 36 one has been marked as ChangeReadyForWrangler.
What I'd like some community help with beyond regular guidance: 1. reviews for the relevant wiki pages: https://fedoraproject.org/wiki/Changes/StrongCryptoSettings3Forewarning1 https://fedoraproject.org/wiki/Changes/StrongCryptoSettings3Forewarning2 https://fedoraproject.org/wiki/Changes/StrongCryptoSettings3Forewarning3 https://fedoraproject.org/wiki/Changes/StrongCryptoSettings3 https://fedoraproject.org/wiki/SHA1SignaturesGuidance https://fedoraproject.org/wiki/WeakCryptographyException 2. proposing the testing activities outlined there for Test Days. how is it done?
[1] https://gitlab.com/redhat-crypto/fedora-sha1sig-tracer [2] https://copr.fedorainfracloud.org/coprs/asosedkin/sha1sig-tracer
-
Alexander Bokovoy -
Alexander Sosedkin -
Björn Persson -
Chris Adams -
Clemens Lang -
Colin Walters -
Daniel P. Berrangé -
Demi Marie Obenour -
Dmitry Belyavskiy -
Dominik 'Rathann' Mierzejewski -
Gary Buhrmaster -
Josh Boyer -
Kamil Dudka -
Kevin Fenzi -
Matthew Miller -
Michael Catanzaro -
Miroslav Lichvar -
Neal Gompa -
Richard W.M. Jones -
Robbie Harwood -
Robert Relyea -
Simo Sorce -
Stephen Gallagher -
Tom Hughes -
Vít Ondruch -
Zbigniew Jędrzejewski-Szmek