Landing a larger-than-release change (distrusting SHA-1 signatures)

[Alexander Sosedkin]

Hello, community, I need your wisdom for planning a disruptive change.

Fedora 28 had https://fedoraproject.org/wiki/Changes/StrongCryptoSettings
Fedora 33 had https://fedoraproject.org/wiki/Changes/StrongCryptoSettings2
I believe we should start planning
for the next cryptographic defaults tightening.
And next time it's gonna be even more disruptive because of SHA-1 (again).

SHA-1 is a hash function from 1995,
which collision resistance is no longer to be relied upon for security [1].
At the same time, it's not like software has successfully migrated off it,
not even close.
It's not a question of "if" the world should migrate from it,
sooner or later we must part ways with it.
(Technically, some acute energy crisis or a collapse of civilization
 forever raising the costs of computations thousandfold would also do,
 but let's agree that migrating to a more modern hash is the way =)

We've been disabling it in TLS, but its usage is much wider than TLS.
The next agonizing step is to restrict its usage for signatures
on the cryptographic libraries level, with openssl being the scariest one.

Good news is, RHEL-9 is gonna lead the way
and thus will take a lot of the hits first.
Fedora doesn't have to pioneer it.
Bad news is, Fedora has to follow suit someday anyway,
and this brings me to how does one land such a change.

---

Fedora is a large distribution with short release cycles, and
the only realistic way to weed out its reliance on SHA-1 signatures
from all of its numerous dark corners is to break them.
Make creation and verification fail in default configuration.
But it's unreasonable to just wait for, say, Fedora 37 branch-off
and break it in Rawhide for Fedora 38.
The fallout will just be too big.

Maintainers need time to get bugs, look into them, think,
analyze, react and test --- and that's just if it fails correctly!
Unfortunately, it's not just that the error paths are as dusty as they get
because the program counter has never set foot on them before.
Some maintainers might even find that
picking a different hash function renders their code non-interoperable,
or even that protocols they implement have SHA-1 hardcoded in the spec.
Or that everything is ready, but real world deployments need another decade.
Or that on-disk formats are just hard to change and migrate.
Took git years to migrate from SHA-1, and some others haven't even started.
There are gonna be investigations, planning, exceptions, upstream changes,
opt-out mechanisms, arguing, compromises, waiting out, all kinds of things.
It's gonna be big. Too big for a single release cycle.

---

But how does one land something and give the distribution
the extra cycles needed to react? That's not really clear to me.

An obvious thing is to announce it in one cycle and land in another one.
The downsides are well-documented
in "The Hitchhiker's Guide to the Galaxy":
announcements are one weak measure, and then it's too late.

A second scheme I can come up with is a "jump scare".
Break the functionality in Fedora 37 Rawhide,
make most of the affected people realize the depth of the problem,
then unbreak it. Break again for Fedora 38 and never fix.

This could also be extended into "let one stable release slide'.
Break in 37 Rawhide, unbreak on branched off 37,
but never in Rawhide.

But these are all rather... crude?
Sure there should be better ways,
preferably something explored before.
I'm all for pulling this tooth out smoothly,
but I need hints on how to do it.
I hope that together we can devise a better plan than these.

So, how does one land a change that's bigger than a release cycle?

[1] https://eprint.iacr.org/2020/014

[Zbigniew Jędrzejewski-Szmek]

On Tue, Mar 08, 2022 at 07:40:15PM +0100, Alexander Sosedkin wrote:
>  the only realistic way to weed out its reliance on SHA-1 signatures
>  from all of its numerous dark corners is to break them.
>  Make creation and verification fail in default configuration. 
That sounds like a terrible plan. We should make newer hashes
the default, and we can make tools print a warning if sha1 is used
where it shouldn't, but please don't break things on purpose.

For many many things sha1 is just fine. Just like md5 or even
crc32. Not everything is about cryptographic security.
Also, users will want to verify old signatures essentially forever.
This should be always possible. And finally, the world is huge,
and other users will provide sha1 signatures no matter what we do,
and it is better to check those than to completely ignore them.

Zbyszek

(This is a bit like browsers disliking self-signed certs and http://
— it certainly is OK to make users aware of the issue, but actually
disallowing those is terribly annoying and will only result in users
jumping to different tools.)

[Colin Walters]

On Tue, Mar 8, 2022, at 1:40 PM, Alexander Sosedkin wrote:
>  But these are all rather... crude?
>  Sure there should be better ways,
>  preferably something explored before. 
One general technique I like is the "warn and sleep" approach; example:
https://github.com/coreos/rpm-ostree/pull/2098

Of course, printing to e.g. stderr or even more strongly adding a sleep(5) call in the
middle of cryptographic libraries has a huge blast radius on its own.  But it's
smaller than removing it entirely.  

(And, remember I am a big fan of the mental model of rolling out changes to the OS core
first, not all packages, so some sort of opt-in warn-and-sleep approach could even be
prototyped out in e.g. Fedora CoreOS first, where we have CI that covers most things we
care about)

[Simo Sorce]

On Tue, 2022-03-08 at 20:51 +0100, Zbigniew Jędrzejewski-Szmek wrote:
>  On Tue, Mar 08, 2022 at 07:40:15PM +0100, Alexander Sosedkin wrote:
>  > the only realistic way to weed out its reliance on SHA-1 signatures
>  > from all of its numerous dark corners is to break them.
>  > Make creation and verification fail in default configuration.
>  
>  That sounds like a terrible plan. We should make newer hashes
>  the default, and we can make tools print a warning if sha1 is used
>  where it shouldn't, but please don't break things on purpose.
>  
>  For many many things sha1 is just fine. Just like md5 or even
>  crc32. Not everything is about cryptographic security. 
I would like to make it clear that this is just for *Cryptographic
Signatures*, this is not a plan to block SHA-1 for all uses.

And for Cryptographic Signatures everything is about cryptography and
SHA-1 is not safe anymore.

And to be extra clear this means: Certificates, TLS session setup,
DNSSEC (although a lot of signatures are still SHA-1 based there ...),
VPNs session establishment, PGP, etc...

>  Also, users will want to verify old signatures essentially forever.
This is only reasonable for stuff like emails, where you may have a
reasonable expectation that the archived messages have not been
tampered with after the fact. Allowing verification of signatures with
SHA-1 for any "online" communication would be pointless.

>  This should be always possible. And finally, the world is huge,
>  and other users will provide sha1 signatures no matter what we do,
>  and it is better to check those than to completely ignore them. 
We need to move the needle at some point. We will be able to set LEGACY
crypto policies to allow SHA-1 verification, but we need to be First
and Secure here as well.

Simo.

>  
>  Zbyszek
>  
>  (This is a bit like browsers disliking self-signed certs and http://
>  — it certainly is OK to make users aware of the issue, but actually
>  disallowing those is terribly annoying and will only result in users
>  jumping to different tools.)
>  _______________________________________________
>  devel mailing list -- devel(a)lists.fedoraproject.org
>  To unsubscribe send an email to devel-leave(a)lists.fedoraproject.org
>  Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
>  List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
>  List Archives:
> https://lists.fedoraproject.org/archives/list/[e-mail]
>  Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
-- 
Simo Sorce
RHEL Crypto Team
Red Hat, Inc

[Neal Gompa]

On Tue, Mar 8, 2022 at 3:11 PM Simo Sorce <simo(a)redhat.com&gt; wrote:
>  On Tue, 2022-03-08 at 20:51 +0100, Zbigniew Jędrzejewski-Szmek wrote:
>  > On Tue, Mar 08, 2022 at 07:40:15PM +0100, Alexander Sosedkin wrote:
>  > > the only realistic way to weed out its reliance on SHA-1 signatures
>  > > from all of its numerous dark corners is to break them.
>  > > Make creation and verification fail in default configuration.
>  >
>  > That sounds like a terrible plan. We should make newer hashes
>  > the default, and we can make tools print a warning if sha1 is used
>  > where it shouldn't, but please don't break things on purpose.
>  >
>  > For many many things sha1 is just fine. Just like md5 or even
>  > crc32. Not everything is about cryptographic security.
>
>  I would like to make it clear that this is just for *Cryptographic
>  Signatures*, this is not a plan to block SHA-1 for all uses.
>
>  And for Cryptographic Signatures everything is about cryptography and
>  SHA-1 is not safe anymore.
>
>  And to be extra clear this means: Certificates, TLS session setup,
>  DNSSEC (although a lot of signatures are still SHA-1 based there ...),
>  VPNs session establishment, PGP, etc...
>
>  > Also, users will want to verify old signatures essentially forever.
>
>  This is only reasonable for stuff like emails, where you may have a
>  reasonable expectation that the archived messages have not been
>  tampered with after the fact. Allowing verification of signatures with
>  SHA-1 for any "online" communication would be pointless.
>
>  > This should be always possible. And finally, the world is huge,
>  > and other users will provide sha1 signatures no matter what we do,
>  > and it is better to check those than to completely ignore them.
>
>  We need to move the needle at some point. We will be able to set LEGACY
>  crypto policies to allow SHA-1 verification, but we need to be First
>  and Secure here as well.
>  
Did we check to make sure such a change won't break Fedora's *own* GPG
keys and the GPG keys of preferred third party repositories? It was a
very unpleasant surprise to have all of CentOS' *second-party* keys
break with that change.



-- 
真実はいつも一つ！/ Always, there's only one truth!

[Demi Marie Obenour]

On 3/8/22 15:23, Neal Gompa wrote:
>  On Tue, Mar 8, 2022 at 3:11 PM Simo Sorce <simo(a)redhat.com&gt;
> wrote:
> >
> > On Tue, 2022-03-08 at 20:51 +0100, Zbigniew Jędrzejewski-Szmek wrote:
> >> On Tue, Mar 08, 2022 at 07:40:15PM +0100, Alexander Sosedkin wrote:
> >>> the only realistic way to weed out its reliance on SHA-1 signatures
> >>> from all of its numerous dark corners is to break them.
> >>> Make creation and verification fail in default configuration.
> >>
> >> That sounds like a terrible plan. We should make newer hashes
> >> the default, and we can make tools print a warning if sha1 is used
> >> where it shouldn't, but please don't break things on purpose.
> >>
> >> For many many things sha1 is just fine. Just like md5 or even
> >> crc32. Not everything is about cryptographic security.
> >
> > I would like to make it clear that this is just for *Cryptographic
> > Signatures*, this is not a plan to block SHA-1 for all uses.
> >
> > And for Cryptographic Signatures everything is about cryptography and
> > SHA-1 is not safe anymore.
> >
> > And to be extra clear this means: Certificates, TLS session setup,
> > DNSSEC (although a lot of signatures are still SHA-1 based there ...),
> > VPNs session establishment, PGP, etc...
> >
> >> Also, users will want to verify old signatures essentially forever.
> >
> > This is only reasonable for stuff like emails, where you may have a
> > reasonable expectation that the archived messages have not been
> > tampered with after the fact. Allowing verification of signatures with
> > SHA-1 for any "online" communication would be pointless.
> >
> >> This should be always possible. And finally, the world is huge,
> >> and other users will provide sha1 signatures no matter what we do,
> >> and it is better to check those than to completely ignore them.
> >
> > We need to move the needle at some point. We will be able to set LEGACY
> > crypto policies to allow SHA-1 verification, but we need to be First
> > and Secure here as well.
> >
>  
>  Did we check to make sure such a change won't break Fedora's *own* GPG
>  keys and the GPG keys of preferred third party repositories? It was a
>  very unpleasant surprise to have all of CentOS' *second-party* keys
>  break with that change. 
Fedora’s RPM package signatures are safe, and have been since
at least Fedora 25 if not earlier.  I can confirm that every single
package in the Fedora 25 and Fedora 32 archives are signed with SHA-256
or later, and presumably the same holds for all releases since Fedora
25.

Qubes OS’s rpmcanon tool can be used to check if a package is signed
with SHA-1: it will return an `InsecureAlgorithm` error for such
packages unless the `--allow-weak-hashes` flag is passed.  It does so
by parsing the signatures itself before passing them to RPM.
-- 
Sincerely,
Demi Marie Obenour (she/her/hers)

[Neal Gompa]

On Tue, Mar 8, 2022 at 3:34 PM Demi Marie Obenour <demiobenour(a)gmail.com&gt; wrote:
>  On 3/8/22 15:23, Neal Gompa wrote:
>  > On Tue, Mar 8, 2022 at 3:11 PM Simo Sorce <simo(a)redhat.com&gt; wrote:
>  >>
>  >> On Tue, 2022-03-08 at 20:51 +0100, Zbigniew Jędrzejewski-Szmek wrote:
>  >>> On Tue, Mar 08, 2022 at 07:40:15PM +0100, Alexander Sosedkin wrote:
>  >>>> the only realistic way to weed out its reliance on SHA-1 signatures
>  >>>> from all of its numerous dark corners is to break them.
>  >>>> Make creation and verification fail in default configuration.
>  >>>
>  >>> That sounds like a terrible plan. We should make newer hashes
>  >>> the default, and we can make tools print a warning if sha1 is used
>  >>> where it shouldn't, but please don't break things on purpose.
>  >>>
>  >>> For many many things sha1 is just fine. Just like md5 or even
>  >>> crc32. Not everything is about cryptographic security.
>  >>
>  >> I would like to make it clear that this is just for *Cryptographic
>  >> Signatures*, this is not a plan to block SHA-1 for all uses.
>  >>
>  >> And for Cryptographic Signatures everything is about cryptography and
>  >> SHA-1 is not safe anymore.
>  >>
>  >> And to be extra clear this means: Certificates, TLS session setup,
>  >> DNSSEC (although a lot of signatures are still SHA-1 based there ...),
>  >> VPNs session establishment, PGP, etc...
>  >>
>  >>> Also, users will want to verify old signatures essentially forever.
>  >>
>  >> This is only reasonable for stuff like emails, where you may have a
>  >> reasonable expectation that the archived messages have not been
>  >> tampered with after the fact. Allowing verification of signatures with
>  >> SHA-1 for any "online" communication would be pointless.
>  >>
>  >>> This should be always possible. And finally, the world is huge,
>  >>> and other users will provide sha1 signatures no matter what we do,
>  >>> and it is better to check those than to completely ignore them.
>  >>
>  >> We need to move the needle at some point. We will be able to set LEGACY
>  >> crypto policies to allow SHA-1 verification, but we need to be First
>  >> and Secure here as well.
>  >>
>  >
>  > Did we check to make sure such a change won't break Fedora's *own* GPG
>  > keys and the GPG keys of preferred third party repositories? It was a
>  > very unpleasant surprise to have all of CentOS' *second-party* keys
>  > break with that change.
>
>  Fedora’s RPM package signatures are safe, and have been since
>  at least Fedora 25 if not earlier.  I can confirm that every single
>  package in the Fedora 25 and Fedora 32 archives are signed with SHA-256
>  or later, and presumably the same holds for all releases since Fedora
>  25.
>
>  Qubes OS’s rpmcanon tool can be used to check if a package is signed
>  with SHA-1: it will return an `InsecureAlgorithm` error for such
>  packages unless the `--allow-weak-hashes` flag is passed.  It does so
>  by parsing the signatures itself before passing them to RPM. 
Did you check our preferred third-parties? From memory, we have
shipped, are shipping, or will be shipping repository definitions for
repositories from the following providers:

* RPM Fusion
* Google
* COPR
* Microsoft


-- 
真実はいつも一つ！/ Always, there's only one truth!

[Gary Buhrmaster]

On Tue, Mar 8, 2022 at 6:40 PM Alexander Sosedkin <asosedkin(a)redhat.com&gt; wrote:

>  Good news is, RHEL-9 is gonna lead the way
>  and thus will take a lot of the hits first.
>  Fedora doesn't have to pioneer it.
>  Bad news is, Fedora has to follow suit someday anyway,
>  and this brings me to how does one land such a change. 
Yes, and the change has already "broken"
COPR with c9s/epel9

Related bugzillas (and discussions):

   https://bugzilla.redhat.com/show_bug.cgi?id=2059101
   https://bugzilla.redhat.com/show_bug.cgi?id=2059424

and the COPR (in progress) adjustment:

  https://pagure.io/copr/copr/issue/2106


Gary

[Daniel P. Berrangé]

On Tue, Mar 08, 2022 at 07:40:15PM +0100, Alexander Sosedkin wrote:
>  We've been disabling it in TLS, but its usage is much wider than
> TLS.
>  The next agonizing step is to restrict its usage for signatures
>  on the cryptographic libraries level, with openssl being the scariest one.
>  
>  Good news is, RHEL-9 is gonna lead the way
>  and thus will take a lot of the hits first.
>  Fedora doesn't have to pioneer it.
>  Bad news is, Fedora has to follow suit someday anyway,
>  and this brings me to how does one land such a change.
>  
>  ---
>  
>  Fedora is a large distribution with short release cycles, and
>  the only realistic way to weed out its reliance on SHA-1 signatures
>  from all of its numerous dark corners is to break them.
>  Make creation and verification fail in default configuration.
>  But it's unreasonable to just wait for, say, Fedora 37 branch-off
>  and break it in Rawhide for Fedora 38.
>  The fallout will just be too big. 
If RHEL-9 has lead the way, what are the stats for real world
RHEL impact ?

What is/was the absolute number of packages and % number of
packages from the RHEL distro  that saw breakage ?

Such figures can give us a better idea of impact on Fedora
beyond "too big".

Assuming RHEL-9 has dealt with the problems, Fedora should
inherit those fixes, which gives us a good base for the most
commonly used / important packages in Fedora.

If the breakage % from RHEL was single digits, and those
were the most important packages to fix from Fedora's POV
too, then maybe the fall is not in fact "too big". It might
be sufficient to identify a few important remaining packages
to validate, and just accept the fallout for the remaining
less important packages in Fedora can be fixed after the
fact ?

IIUC we have a simple workaround of letting someone set the
crypto policies on their machine back to LEGACY still

Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

[Alexander Sosedkin]

On Tue, Mar 8, 2022 at 8:52 PM Zbigniew Jędrzejewski-Szmek
<zbyszek(a)in.waw.pl&gt; wrote:
>  On Tue, Mar 08, 2022 at 07:40:15PM +0100, Alexander Sosedkin wrote:
>  > the only realistic way to weed out its reliance on SHA-1 signatures
>  > from all of its numerous dark corners is to break them.
>  > Make creation and verification fail in default configuration.
>
>  That sounds like a terrible plan. We should make newer hashes
>  the default, and we can make tools print a warning if sha1 is used
>  where it shouldn't, 
Yeah, that'd never gonna work.
Cryptographic libraries aren't in position to do any meaningful logging,
and even if they did, it'd be gleefully ignored until we break things.

>  but please don't break things on purpose. 
If you manage to come out with the way
we can move forward to this century crypto
that doesn't involve breaking things on purpose,
definitely share one with us, you'd be our saviour. =(

Yes, backwards compatibility is very important.
Yes, you can still make a modern Fedora do RC4 or DES,
with just a little bit of extra configuration.
No, we must phase things out of defaults, there's no way around this.

>  For many many things sha1 is just fine. Just like md5 or even
>  crc32. Not everything is about cryptographic security.
>  Also, users will want to verify old signatures essentially forever.
>  This should be always possible. And finally, the world is huge,
>  and other users will provide sha1 signatures no matter what we do,
>  and it is better to check those than to completely ignore them.
>
>  Zbyszek
>
>  (This is a bit like browsers disliking self-signed certs and http://
>  — it certainly is OK to make users aware of the issue, but actually
>  disallowing those is terribly annoying and will only result in users
>  jumping to different tools.)
>  _______________________________________________
>  devel mailing list -- devel(a)lists.fedoraproject.org
>  To unsubscribe send an email to devel-leave(a)lists.fedoraproject.org
>  Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
>  List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
>  List Archives:
> https://lists.fedoraproject.org/archives/list/[e-mail]
>  Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure

[Miroslav Lichvar]

On Tue, Mar 08, 2022 at 07:40:15PM +0100, Alexander Sosedkin wrote:
>  Took git years to migrate from SHA-1, and some others haven't
> even started. 
git is a good example showing that this won't be easy. The SHA-256
object format is still marked as experimental and not the default.

Is there a plan to migrate the Fedora repositories?

-- 
Miroslav Lichvar

[Alexander Sosedkin]

On Wed, Mar 9, 2022 at 10:20 AM Daniel P. Berrangé <berrange(a)redhat.com&gt; wrote:
>  On Tue, Mar 08, 2022 at 07:40:15PM +0100, Alexander Sosedkin wrote:
>  > We've been disabling it in TLS, but its usage is much wider than TLS.
>  > The next agonizing step is to restrict its usage for signatures
>  > on the cryptographic libraries level, with openssl being the scariest one.
>  >
>  > Good news is, RHEL-9 is gonna lead the way
>  > and thus will take a lot of the hits first.
>  > Fedora doesn't have to pioneer it.
>  > Bad news is, Fedora has to follow suit someday anyway,
>  > and this brings me to how does one land such a change.
>  >
>  > ---
>  >
>  > Fedora is a large distribution with short release cycles, and
>  > the only realistic way to weed out its reliance on SHA-1 signatures
>  > from all of its numerous dark corners is to break them.
>  > Make creation and verification fail in default configuration.
>  > But it's unreasonable to just wait for, say, Fedora 37 branch-off
>  > and break it in Rawhide for Fedora 38.
>  > The fallout will just be too big.
>
>  If RHEL-9 has lead the way, what are the stats for real world
>  RHEL impact ? 
We'll know when the real world starts using RHEL-9 en masse?

>  What is/was the absolute number of packages and % number of
>  packages from the RHEL distro  that saw breakage ? 
Does preventing the distro from installing altogether count as 100%?
If yes, 100%. =)
Jokes aside, I can't give you an accurate estimate yet.

>  Such figures can give us a better idea of impact on Fedora
>  beyond "too big".
>
>  Assuming RHEL-9 has dealt with the problems, Fedora should
>  inherit those fixes, which gives us a good base for the most
>  commonly used / important packages in Fedora. 
Yeah, that's what I meant by the good news.
But that won't solve all Fedora problems.

>  If the breakage % from RHEL was single digits, and those
>  were the most important packages to fix from Fedora's POV
>  too, then maybe the fall is not in fact "too big". It might
>  be sufficient to identify a few important remaining packages
>  to validate, and just accept the fallout for the remaining
>  less important packages in Fedora can be fixed after the
>  fact ? 
At a quick glance,
I eyeball RHEL at ~2k packages and Fedora at ~22k packages.
I think that limited analysis 's enough to safely claim
that leaving the 90% of the packages you've labelled "less important"
to be "fixed after the fact" is gonna be a disaster.
One cycle doesn't sound enough.

>  IIUC we have a simple workaround of letting someone set the
>  crypto policies on their machine back to LEGACY still 
Yes, I'll sure leave SHA-1 signatures allowed in LEGACY for some more years.
Similarly, I could break it in FUTURE rather early,
but nearly nobody would notice until it hits DEFAULT.

>  Regards,
>  Daniel
>  --
>  |: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
>  |: https://libvirt.org         -o-            https://fstop138.berrange.com :|
>  |: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|
>  _______________________________________________
>  devel mailing list -- devel(a)lists.fedoraproject.org
>  To unsubscribe send an email to devel-leave(a)lists.fedoraproject.org
>  Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
>  List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
>  List Archives:
> https://lists.fedoraproject.org/archives/list/[e-mail]
>  Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure

[Daniel P. Berrangé]

On Wed, Mar 09, 2022 at 10:44:28AM +0100, Miroslav Lichvar wrote:
>  On Tue, Mar 08, 2022 at 07:40:15PM +0100, Alexander Sosedkin wrote:
>  > Took git years to migrate from SHA-1, and some others haven't even started.
>  
>  git is a good example showing that this won't be easy. The SHA-256
>  object format is still marked as experimental and not the default. 
Unless I'm mistaken Git isn't using SHA1 in cryptographic signatures,
it is using it directly for hashing. The proposal isn't banning all
use of SHA1, just SHA-1 based signatures.

With regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

[Daniel P. Berrangé]

On Wed, Mar 09, 2022 at 10:46:21AM +0100, Alexander Sosedkin wrote:
>  On Wed, Mar 9, 2022 at 10:20 AM Daniel P. Berrangé
> <berrange(a)redhat.com&gt; wrote:
>  >
>  > On Tue, Mar 08, 2022 at 07:40:15PM +0100, Alexander Sosedkin wrote:
>  > > We've been disabling it in TLS, but its usage is much wider than TLS.
>  > > The next agonizing step is to restrict its usage for signatures
>  > > on the cryptographic libraries level, with openssl being the scariest one.
>  > >
>  > > Good news is, RHEL-9 is gonna lead the way
>  > > and thus will take a lot of the hits first.
>  > > Fedora doesn't have to pioneer it.
>  > > Bad news is, Fedora has to follow suit someday anyway,
>  > > and this brings me to how does one land such a change.
>  > >
>  > > ---
>  > >
>  > > Fedora is a large distribution with short release cycles, and
>  > > the only realistic way to weed out its reliance on SHA-1 signatures
>  > > from all of its numerous dark corners is to break them.
>  > > Make creation and verification fail in default configuration.
>  > > But it's unreasonable to just wait for, say, Fedora 37 branch-off
>  > > and break it in Rawhide for Fedora 38.
>  > > The fallout will just be too big.
>  >
>  > If RHEL-9 has lead the way, what are the stats for real world
>  > RHEL impact ?
>  
>  We'll know when the real world starts using RHEL-9 en masse?
>  
>  > What is/was the absolute number of packages and % number of
>  > packages from the RHEL distro  that saw breakage ?
>  
>  Does preventing the distro from installing altogether count as 100%?
>  If yes, 100%. =)
>  Jokes aside, I can't give you an accurate estimate yet. 
Perhaps a useful first step is to just modify the three main
crypto libs (gnutls, openssl, and nss) to send a scary warnihg
message to stderr/syslog any time they get use of SHA1 in a
signature. Leave that active for a release cycle and see how
many bug reports we get. 

>  > Such figures can give us a better idea of impact on Fedora
>  > beyond "too big".
>  >
>  > Assuming RHEL-9 has dealt with the problems, Fedora should
>  > inherit those fixes, which gives us a good base for the most
>  > commonly used / important packages in Fedora.
>  
>  Yeah, that's what I meant by the good news.
>  But that won't solve all Fedora problems.
>  
>  > If the breakage % from RHEL was single digits, and those
>  > were the most important packages to fix from Fedora's POV
>  > too, then maybe the fall is not in fact "too big". It might
>  > be sufficient to identify a few important remaining packages
>  > to validate, and just accept the fallout for the remaining
>  > less important packages in Fedora can be fixed after the
>  > fact ?
>  
>  At a quick glance,
>  I eyeball RHEL at ~2k packages and Fedora at ~22k packages.
>  I think that limited analysis 's enough to safely claim
>  that leaving the 90% of the packages you've labelled "less important"
>  to be "fixed after the fact" is gonna be a disaster.
>  One cycle doesn't sound enough. 
That 90% of packages are not all going to be using cryptographic
signatures though. Only a relatively small subset will do anything
crypto related, and most of that just be HTTPS and completely
outsourced to a crypto library.

IOW of that 90% of packages not in RHEL, it could conceivably be
single digits that will be using cryptographic signatures with
SHA-1. An even smaller percentage of those will be considered 
important enough to be blockers for this change. 


With regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

[Demi Marie Obenour]

On 3/9/22 04:48, Daniel P. Berrangé wrote:
>  On Wed, Mar 09, 2022 at 10:44:28AM +0100, Miroslav Lichvar wrote:
> > On Tue, Mar 08, 2022 at 07:40:15PM +0100, Alexander Sosedkin wrote:
> >> Took git years to migrate from SHA-1, and some others haven't even started.
> >
> > git is a good example showing that this won't be easy. The SHA-256
> > object format is still marked as experimental and not the default.
>  
>  Unless I'm mistaken Git isn't using SHA1 in cryptographic signatures,
>  it is using it directly for hashing. The proposal isn't banning all
>  use of SHA1, just SHA-1 based signatures. 
Git *does* use SHA-1 in signatures, via signed Git tags and commits.
However, Git also includes code that can detect all known relatively
fast collision attacks on SHA-1.  A brute force attack (with 2^80 time
complexity) is still possible, but the amount of resources required is
such that the attack is not considered likely.  Furthermore, in many
cases, one would not only need to find a collision in SHA-1, but also
get one of the colliding objects past code review.  Finding a collision
that will pass code review is likely far, *far* more difficult than
just finding a collision.

In short, Git *does* need to move away from SHA-1, but it isn’t a
crisis — yet.

-- 
Sincerely,
Demi Marie Obenour (she/her/hers)

[Alexander Sosedkin]

On Wed, Mar 9, 2022 at 10:57 AM Daniel P. Berrangé <berrange(a)redhat.com&gt; wrote:
>  On Wed, Mar 09, 2022 at 10:46:21AM +0100, Alexander Sosedkin wrote:
>  > On Wed, Mar 9, 2022 at 10:20 AM Daniel P. Berrangé <berrange(a)redhat.com&gt;
> wrote:
>  > >
>  > > On Tue, Mar 08, 2022 at 07:40:15PM +0100, Alexander Sosedkin wrote:
>  > > > We've been disabling it in TLS, but its usage is much wider than TLS.
>  > > > The next agonizing step is to restrict its usage for signatures
>  > > > on the cryptographic libraries level, with openssl being the scariest
> one.
>  > > >
>  > > > Good news is, RHEL-9 is gonna lead the way
>  > > > and thus will take a lot of the hits first.
>  > > > Fedora doesn't have to pioneer it.
>  > > > Bad news is, Fedora has to follow suit someday anyway,
>  > > > and this brings me to how does one land such a change.
>  > > >
>  > > > ---
>  > > >
>  > > > Fedora is a large distribution with short release cycles, and
>  > > > the only realistic way to weed out its reliance on SHA-1 signatures
>  > > > from all of its numerous dark corners is to break them.
>  > > > Make creation and verification fail in default configuration.
>  > > > But it's unreasonable to just wait for, say, Fedora 37 branch-off
>  > > > and break it in Rawhide for Fedora 38.
>  > > > The fallout will just be too big.
>  > >
>  > > If RHEL-9 has lead the way, what are the stats for real world
>  > > RHEL impact ?
>  >
>  > We'll know when the real world starts using RHEL-9 en masse?
>  >
>  > > What is/was the absolute number of packages and % number of
>  > > packages from the RHEL distro  that saw breakage ?
>  >
>  > Does preventing the distro from installing altogether count as 100%?
>  > If yes, 100%. =)
>  > Jokes aside, I can't give you an accurate estimate yet.
>
>  Perhaps a useful first step is to just modify the three main
>  crypto libs (gnutls, openssl, and nss) to send a scary warnihg
>  message to stderr/syslog any time they get use of SHA1 in a
>  signature. Leave that active for a release cycle and see how
>  many bug reports we get. 
I left my crystal ball at home today,
but I don't need it to say it'd be ~0 bugs filed if we log to syslog
and ~3 if we log to stderr/stdout, all named
"$CRYPTOLIB has no business messing up my stderr/stdout",
which we'll promptly close by reverting the changes.

>  > > Such figures can give us a better idea of impact on Fedora
>  > > beyond "too big".
>  > >
>  > > Assuming RHEL-9 has dealt with the problems, Fedora should
>  > > inherit those fixes, which gives us a good base for the most
>  > > commonly used / important packages in Fedora.
>  >
>  > Yeah, that's what I meant by the good news.
>  > But that won't solve all Fedora problems.
>  >
>  > > If the breakage % from RHEL was single digits, and those
>  > > were the most important packages to fix from Fedora's POV
>  > > too, then maybe the fall is not in fact "too big". It might
>  > > be sufficient to identify a few important remaining packages
>  > > to validate, and just accept the fallout for the remaining
>  > > less important packages in Fedora can be fixed after the
>  > > fact ?
>  >
>  > At a quick glance,
>  > I eyeball RHEL at ~2k packages and Fedora at ~22k packages.
>  > I think that limited analysis 's enough to safely claim
>  > that leaving the 90% of the packages you've labelled "less important"
>  > to be "fixed after the fact" is gonna be a disaster.
>  > One cycle doesn't sound enough.
>
>  That 90% of packages are not all going to be using cryptographic
>  signatures though. Only a relatively small subset will do anything
>  crypto related, and most of that just be HTTPS and completely
>  outsourced to a crypto library.
>
>  IOW of that 90% of packages not in RHEL, it could conceivably be
>  single digits that will be using cryptographic signatures with
>  SHA-1. An even smaller percentage of those will be considered
>  important enough to be blockers for this change.
>
>
>  With regards,
>  Daniel
>  --
>  |: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
>  |: https://libvirt.org         -o-            https://fstop138.berrange.com :|
>  |: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|
>

[Alexander Bokovoy]

On ke, 09 maalis 2022, Daniel P. Berrangé wrote:
> On Wed, Mar 09, 2022 at 10:46:21AM +0100, Alexander Sosedkin wrote:
> > On Wed, Mar 9, 2022 at 10:20 AM Daniel P. Berrangé <berrange(a)redhat.com&gt;
> wrote:
> > >
> > > On Tue, Mar 08, 2022 at 07:40:15PM +0100, Alexander Sosedkin wrote:
> > > > We've been disabling it in TLS, but its usage is much wider than TLS.
> > > > The next agonizing step is to restrict its usage for signatures
> > > > on the cryptographic libraries level, with openssl being the scariest one.
> > > >
> > > > Good news is, RHEL-9 is gonna lead the way
> > > > and thus will take a lot of the hits first.
> > > > Fedora doesn't have to pioneer it.
> > > > Bad news is, Fedora has to follow suit someday anyway,
> > > > and this brings me to how does one land such a change.
> > > >
> > > > ---
> > > >
> > > > Fedora is a large distribution with short release cycles, and
> > > > the only realistic way to weed out its reliance on SHA-1 signatures
> > > > from all of its numerous dark corners is to break them.
> > > > Make creation and verification fail in default configuration.
> > > > But it's unreasonable to just wait for, say, Fedora 37 branch-off
> > > > and break it in Rawhide for Fedora 38.
> > > > The fallout will just be too big.
> > >
> > > If RHEL-9 has lead the way, what are the stats for real world
> > > RHEL impact ?
> >
> > We'll know when the real world starts using RHEL-9 en masse?
> >
> > > What is/was the absolute number of packages and % number of
> > > packages from the RHEL distro  that saw breakage ?
> >
> > Does preventing the distro from installing altogether count as 100%?
> > If yes, 100%. =)
> > Jokes aside, I can't give you an accurate estimate yet.
>
> Perhaps a useful first step is to just modify the three main
> crypto libs (gnutls, openssl, and nss) to send a scary warnihg
> message to stderr/syslog any time they get use of SHA1 in a
> signature. Leave that active for a release cycle and see how
> many bug reports we get.
>
> > > Such figures can give us a better idea of impact on Fedora
> > > beyond "too big".
> > >
> > > Assuming RHEL-9 has dealt with the problems, Fedora should
> > > inherit those fixes, which gives us a good base for the most
> > > commonly used / important packages in Fedora.
> >
> > Yeah, that's what I meant by the good news.
> > But that won't solve all Fedora problems.
> >
> > > If the breakage % from RHEL was single digits, and those
> > > were the most important packages to fix from Fedora's POV
> > > too, then maybe the fall is not in fact "too big". It might
> > > be sufficient to identify a few important remaining packages
> > > to validate, and just accept the fallout for the remaining
> > > less important packages in Fedora can be fixed after the
> > > fact ?
> >
> > At a quick glance,
> > I eyeball RHEL at ~2k packages and Fedora at ~22k packages.
> > I think that limited analysis 's enough to safely claim
> > that leaving the 90% of the packages you've labelled "less important"
> > to be "fixed after the fact" is gonna be a disaster.
> > One cycle doesn't sound enough.
>
> That 90% of packages are not all going to be using cryptographic
> signatures though. Only a relatively small subset will do anything
> crypto related, and most of that just be HTTPS and completely
> outsourced to a crypto library.
>
> IOW of that 90% of packages not in RHEL, it could conceivably be
> single digits that will be using cryptographic signatures with
> SHA-1. An even smaller percentage of those will be considered
> important enough to be blockers for this change. 
Specifically, Kerberos is one of those protocols and MIT Kerberos is one
of those implementations which currently forced to use SHA-1 due to
interoperability issues and also due to compliance with RFCs.

In the context of Fedora development, for example, use of Anonymous
PKINIT requires usage of SHA-1 in PKINIT pre-authentication module in
MIT Kerberos. We have to support that or Fedora developers would not be
able to obtain their Kerberos tickets.

For more details see https://bugzilla.redhat.com/show_bug.cgi?id=2060798


-- 
/ Alexander Bokovoy
Sr. Principal Software Engineer
Security / Identity Management Engineering
Red Hat Limited, Finland

[Daniel P. Berrangé]

On Wed, Mar 09, 2022 at 02:32:48PM +0200, Alexander Bokovoy wrote:
>  On ke, 09 maalis 2022, Daniel P. Berrangé wrote:
>  > On Wed, Mar 09, 2022 at 10:46:21AM +0100, Alexander Sosedkin wrote:
>  > > On Wed, Mar 9, 2022 at 10:20 AM Daniel P. Berrangé <berrange(a)redhat.com&gt;
> wrote:
>  > > >
>  > > > On Tue, Mar 08, 2022 at 07:40:15PM +0100, Alexander Sosedkin wrote:
>  > > > > We've been disabling it in TLS, but its usage is much wider than
> TLS.
>  > > > > The next agonizing step is to restrict its usage for signatures
>  > > > > on the cryptographic libraries level, with openssl being the scariest
> one.
>  > > > >
>  > > > > Good news is, RHEL-9 is gonna lead the way
>  > > > > and thus will take a lot of the hits first.
>  > > > > Fedora doesn't have to pioneer it.
>  > > > > Bad news is, Fedora has to follow suit someday anyway,
>  > > > > and this brings me to how does one land such a change.
>  > > > >
>  > > > > ---
>  > > > >
>  > > > > Fedora is a large distribution with short release cycles, and
>  > > > > the only realistic way to weed out its reliance on SHA-1 signatures
>  > > > > from all of its numerous dark corners is to break them.
>  > > > > Make creation and verification fail in default configuration.
>  > > > > But it's unreasonable to just wait for, say, Fedora 37
> branch-off
>  > > > > and break it in Rawhide for Fedora 38.
>  > > > > The fallout will just be too big.
>  > > >
>  > > > If RHEL-9 has lead the way, what are the stats for real world
>  > > > RHEL impact ?
>  > > 
>  > > We'll know when the real world starts using RHEL-9 en masse?
>  > > 
>  > > > What is/was the absolute number of packages and % number of
>  > > > packages from the RHEL distro  that saw breakage ?
>  > > 
>  > > Does preventing the distro from installing altogether count as 100%?
>  > > If yes, 100%. =)
>  > > Jokes aside, I can't give you an accurate estimate yet.
>  > 
>  > Perhaps a useful first step is to just modify the three main
>  > crypto libs (gnutls, openssl, and nss) to send a scary warnihg
>  > message to stderr/syslog any time they get use of SHA1 in a
>  > signature. Leave that active for a release cycle and see how
>  > many bug reports we get.
>  > 
>  > > > Such figures can give us a better idea of impact on Fedora
>  > > > beyond "too big".
>  > > >
>  > > > Assuming RHEL-9 has dealt with the problems, Fedora should
>  > > > inherit those fixes, which gives us a good base for the most
>  > > > commonly used / important packages in Fedora.
>  > > 
>  > > Yeah, that's what I meant by the good news.
>  > > But that won't solve all Fedora problems.
>  > > 
>  > > > If the breakage % from RHEL was single digits, and those
>  > > > were the most important packages to fix from Fedora's POV
>  > > > too, then maybe the fall is not in fact "too big". It might
>  > > > be sufficient to identify a few important remaining packages
>  > > > to validate, and just accept the fallout for the remaining
>  > > > less important packages in Fedora can be fixed after the
>  > > > fact ?
>  > > 
>  > > At a quick glance,
>  > > I eyeball RHEL at ~2k packages and Fedora at ~22k packages.
>  > > I think that limited analysis 's enough to safely claim
>  > > that leaving the 90% of the packages you've labelled "less
> important"
>  > > to be "fixed after the fact" is gonna be a disaster.
>  > > One cycle doesn't sound enough.
>  > 
>  > That 90% of packages are not all going to be using cryptographic
>  > signatures though. Only a relatively small subset will do anything
>  > crypto related, and most of that just be HTTPS and completely
>  > outsourced to a crypto library.
>  > 
>  > IOW of that 90% of packages not in RHEL, it could conceivably be
>  > single digits that will be using cryptographic signatures with
>  > SHA-1. An even smaller percentage of those will be considered
>  > important enough to be blockers for this change.
>  
>  Specifically, Kerberos is one of those protocols and MIT Kerberos is one
>  of those implementations which currently forced to use SHA-1 due to
>  interoperability issues and also due to compliance with RFCs.
>  
>  In the context of Fedora development, for example, use of Anonymous
>  PKINIT requires usage of SHA-1 in PKINIT pre-authentication module in
>  MIT Kerberos. We have to support that or Fedora developers would not be
>  able to obtain their Kerberos tickets.
>  
>  For more details see https://bugzilla.redhat.com/show_bug.cgi?id=2060798 
I assume RHEL is going to have to solve that problem with Krb
because it wouldn't be viable to break Krb in RHEL, any more
than it would in Fedora.

With regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

[Alexander Bokovoy]

On ke, 09 maalis 2022, Daniel P. Berrangé wrote:
> On Wed, Mar 09, 2022 at 02:32:48PM +0200, Alexander Bokovoy wrote:
> > On ke, 09 maalis 2022, Daniel P. Berrangé wrote:
> > > On Wed, Mar 09, 2022 at 10:46:21AM +0100, Alexander Sosedkin wrote:
> > > > On Wed, Mar 9, 2022 at 10:20 AM Daniel P. Berrangé
> <berrange(a)redhat.com&gt; wrote:
> > > > >
> > > > > On Tue, Mar 08, 2022 at 07:40:15PM +0100, Alexander Sosedkin wrote:
> > > > > > We've been disabling it in TLS, but its usage is much wider
> than TLS.
> > > > > > The next agonizing step is to restrict its usage for signatures
> > > > > > on the cryptographic libraries level, with openssl being the
> scariest one.
> > > > > >
> > > > > > Good news is, RHEL-9 is gonna lead the way
> > > > > > and thus will take a lot of the hits first.
> > > > > > Fedora doesn't have to pioneer it.
> > > > > > Bad news is, Fedora has to follow suit someday anyway,
> > > > > > and this brings me to how does one land such a change.
> > > > > >
> > > > > > ---
> > > > > >
> > > > > > Fedora is a large distribution with short release cycles, and
> > > > > > the only realistic way to weed out its reliance on SHA-1
> signatures
> > > > > > from all of its numerous dark corners is to break them.
> > > > > > Make creation and verification fail in default configuration.
> > > > > > But it's unreasonable to just wait for, say, Fedora 37
> branch-off
> > > > > > and break it in Rawhide for Fedora 38.
> > > > > > The fallout will just be too big.
> > > > >
> > > > > If RHEL-9 has lead the way, what are the stats for real world
> > > > > RHEL impact ?
> > > >
> > > > We'll know when the real world starts using RHEL-9 en masse?
> > > >
> > > > > What is/was the absolute number of packages and % number of
> > > > > packages from the RHEL distro  that saw breakage ?
> > > >
> > > > Does preventing the distro from installing altogether count as 100%?
> > > > If yes, 100%. =)
> > > > Jokes aside, I can't give you an accurate estimate yet.
> > >
> > > Perhaps a useful first step is to just modify the three main
> > > crypto libs (gnutls, openssl, and nss) to send a scary warnihg
> > > message to stderr/syslog any time they get use of SHA1 in a
> > > signature. Leave that active for a release cycle and see how
> > > many bug reports we get.
> > >
> > > > > Such figures can give us a better idea of impact on Fedora
> > > > > beyond "too big".
> > > > >
> > > > > Assuming RHEL-9 has dealt with the problems, Fedora should
> > > > > inherit those fixes, which gives us a good base for the most
> > > > > commonly used / important packages in Fedora.
> > > >
> > > > Yeah, that's what I meant by the good news.
> > > > But that won't solve all Fedora problems.
> > > >
> > > > > If the breakage % from RHEL was single digits, and those
> > > > > were the most important packages to fix from Fedora's POV
> > > > > too, then maybe the fall is not in fact "too big". It might
> > > > > be sufficient to identify a few important remaining packages
> > > > > to validate, and just accept the fallout for the remaining
> > > > > less important packages in Fedora can be fixed after the
> > > > > fact ?
> > > >
> > > > At a quick glance,
> > > > I eyeball RHEL at ~2k packages and Fedora at ~22k packages.
> > > > I think that limited analysis 's enough to safely claim
> > > > that leaving the 90% of the packages you've labelled "less
> important"
> > > > to be "fixed after the fact" is gonna be a disaster.
> > > > One cycle doesn't sound enough.
> > >
> > > That 90% of packages are not all going to be using cryptographic
> > > signatures though. Only a relatively small subset will do anything
> > > crypto related, and most of that just be HTTPS and completely
> > > outsourced to a crypto library.
> > >
> > > IOW of that 90% of packages not in RHEL, it could conceivably be
> > > single digits that will be using cryptographic signatures with
> > > SHA-1. An even smaller percentage of those will be considered
> > > important enough to be blockers for this change.
> >
> > Specifically, Kerberos is one of those protocols and MIT Kerberos is one
> > of those implementations which currently forced to use SHA-1 due to
> > interoperability issues and also due to compliance with RFCs.
> >
> > In the context of Fedora development, for example, use of Anonymous
> > PKINIT requires usage of SHA-1 in PKINIT pre-authentication module in
> > MIT Kerberos. We have to support that or Fedora developers would not be
> > able to obtain their Kerberos tickets.
> >
> > For more details see https://bugzilla.redhat.com/show_bug.cgi?id=2060798
>
> I assume RHEL is going to have to solve that problem with Krb
> because it wouldn't be viable to break Krb in RHEL, any more
> than it would in Fedora. 
Yes, the approach is described in the comment 6 of that bug.


-- 
/ Alexander Bokovoy
Sr. Principal Software Engineer
Security / Identity Management Engineering
Red Hat Limited, Finland

[Richard W.M. Jones]

Previous tightening of crypto defaults caused problems with us
connecting to older ssh servers.

I am particularly interested / worried about sshd from RHEL 5, 6 & 7
for virt-p2v and virt-v2v conversions.  This broke before, requiring
us to advise users to set the global policy for the machine to LEGACY
(thus ironically weakening crypto for everything).

Also I have some ancient network equipment that cannot be upgraded but
needs older ssh protocols.  I can't connect to it from Fedora unless I
set the crypto policy to LEGACY.

Anyway I'm wondering if the SHA-1 change will impact ssh further?

Rich.

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
virt-df lists disk usage of guests without needing to install any
software inside the virtual machine.  Supports Linux and Windows.
http://people.redhat.com/~rjones/virt-df/

[Alexander Sosedkin]

On Wed, Mar 9, 2022 at 2:47 PM Richard W.M. Jones <rjones(a)redhat.com&gt; wrote:
>  Previous tightening of crypto defaults caused problems with us
>  connecting to older ssh servers.
>
>  I am particularly interested / worried about sshd from RHEL 5, 6 & 7
>  for virt-p2v and virt-v2v conversions.  This broke before, requiring
>  us to advise users to set the global policy for the machine to LEGACY
>  (thus ironically weakening crypto for everything).
>
>  Also I have some ancient network equipment that cannot be upgraded but
>  needs older ssh protocols.  I can't connect to it from Fedora unless I
>  set the crypto policy to LEGACY.
>
>  Anyway I'm wondering if the SHA-1 change will impact ssh further? 
IIRC, the only SHA-1 thing that should be left in DEFAULT for SSH
is SHA-1 as HMAC, which doesn't rely on collision resistance.
So, not this round.

>  Rich.
>
>  --
>  Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
>  Read my programming and virtualization blog: http://rwmj.wordpress.com
>  virt-df lists disk usage of guests without needing to install any
>  software inside the virtual machine.  Supports Linux and Windows.
>  http://people.redhat.com/~rjones/virt-df/
>  _______________________________________________
>  devel mailing list -- devel(a)lists.fedoraproject.org
>  To unsubscribe send an email to devel-leave(a)lists.fedoraproject.org
>  Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
>  List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
>  List Archives:
> https://lists.fedoraproject.org/archives/list/[e-mail]
>  Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure

[Dmitry Belyavskiy]

At least RHEL 6 issues can be fixed server-side generating an ecdsa key...

On Wed, Mar 9, 2022 at 2:47 PM Richard W.M. Jones <rjones(a)redhat.com&gt; wrote:

>  Previous tightening of crypto defaults caused problems with us
>  connecting to older ssh servers.
>
>  I am particularly interested / worried about sshd from RHEL 5, 6 & 7
>  for virt-p2v and virt-v2v conversions.  This broke before, requiring
>  us to advise users to set the global policy for the machine to LEGACY
>  (thus ironically weakening crypto for everything).
>
>  Also I have some ancient network equipment that cannot be upgraded but
>  needs older ssh protocols.  I can't connect to it from Fedora unless I
>  set the crypto policy to LEGACY.
>
>  Anyway I'm wondering if the SHA-1 change will impact ssh further?
>
>  Rich.
>
>  --
>  Richard Jones, Virtualization Group, Red Hat
>  http://people.redhat.com/~rjones
>  Read my programming and virtualization blog: http://rwmj.wordpress.com
>  virt-df lists disk usage of guests without needing to install any
>  software inside the virtual machine.  Supports Linux and Windows.
>  http://people.redhat.com/~rjones/virt-df/
>  _______________________________________________
>  devel mailing list -- devel(a)lists.fedoraproject.org
>  To unsubscribe send an email to devel-leave(a)lists.fedoraproject.org
>  Fedora Code of Conduct:
>  https://docs.fedoraproject.org/en-US/project/code-of-conduct/
>  List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
>  List Archives:
>  https://lists.fedoraproject.org/archives/list/[e-mail]
>  Do not reply to spam on the list, report it:
>  https://pagure.io/fedora-infrastructure
>  

-- 
Dmitry Belyavskiy

[Richard W.M. Jones]

On Wed, Mar 09, 2022 at 02:54:40PM +0100, Dmitry Belyavskiy wrote:
>  At least RHEL 6 issues can be fixed server-side generating an ecdsa
> key... 
If you can log in ...

This doesn't really work for us because of the way virt-v2v works --
it wants to ssh to the RHEL 5/6/7 server to fetch some files with as
little pre-configuration as possible.

Rich.

>  On Wed, Mar 9, 2022 at 2:47 PM Richard W.M. Jones
> <rjones(a)redhat.com&gt; wrote:
>  
>      Previous tightening of crypto defaults caused problems with us
>      connecting to older ssh servers.
>  
>      I am particularly interested / worried about sshd from RHEL 5, 6 & 7
>      for virt-p2v and virt-v2v conversions.  This broke before, requiring
>      us to advise users to set the global policy for the machine to LEGACY
>      (thus ironically weakening crypto for everything).
>  
>      Also I have some ancient network equipment that cannot be upgraded but
>      needs older ssh protocols.  I can't connect to it from Fedora unless I
>      set the crypto policy to LEGACY.
>  
>      Anyway I'm wondering if the SHA-1 change will impact ssh further?
>  
>      Rich.
>  
>      --
>      Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/
>      ~rjones
>      Read my programming and virtualization blog: http://rwmj.wordpress.com
>      virt-df lists disk usage of guests without needing to install any
>      software inside the virtual machine.  Supports Linux and Windows.
>      http://people.redhat.com/~rjones/virt-df/
>      _______________________________________________
>      devel mailing list -- devel(a)lists.fedoraproject.org
>      To unsubscribe send an email to devel-leave(a)lists.fedoraproject.org
>      Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/
>      code-of-conduct/
>      List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
>      List Archives: https://lists.fedoraproject.org/archives/list/
>      devel(a)lists.fedoraproject.org
>      Do not reply to spam on the list, report it: https://pagure.io/
>      fedora-infrastructure
>  
>  
>  
>  --
>  Dmitry Belyavskiy 
>  _______________________________________________
>  devel mailing list -- devel(a)lists.fedoraproject.org
>  To unsubscribe send an email to devel-leave(a)lists.fedoraproject.org
>  Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
>  List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
>  List Archives:
> https://lists.fedoraproject.org/archives/list/[e-mail]
>  Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
virt-builder quickly builds VMs from scratch
http://libguestfs.org/virt-builder.1.html

[Chris Adams]

Once upon a time, Richard W.M. Jones <rjones(a)redhat.com&gt; said:
>  Previous tightening of crypto defaults caused problems with us
>  connecting to older ssh servers. 
I also have had trouble connecting to major vendor websites.  The vendor
response is just "works in Chrome and Firefox on Windows, must be your
problem".

>  I am particularly interested / worried about sshd from RHEL 5, 6
> & 7
>  for virt-p2v and virt-v2v conversions.  This broke before, requiring
>  us to advise users to set the global policy for the machine to LEGACY
>  (thus ironically weakening crypto for everything).
>  
>  Also I have some ancient network equipment that cannot be upgraded but
>  needs older ssh protocols.  I can't connect to it from Fedora unless I
>  set the crypto policy to LEGACY. 
Yeah, the model in general seems a little broken to me, especially as I
found the policies are implemented unevenly (IIRC my problem was OpenSSL
couldn't connect but GnuTLS could for example), which just leads to
confusion.

I understand and approve of having good system-wide defaults, but there
needs to be a way to connect to a specific site/device/whatever without
having to lower the system-wide policy.  For SSH, you can usually do
that by adjusting the settings on a per-device basis on the command line
or in ~/.ssh/config (setting PublickeyAcceptedKeyTypes, KexAlgorithms,
HostKeyAlgorithms, and/or Ciphers as needed).

I had to SSH to a FreeBSD 4.x server last year!  So many SSH config
options required... it had been up without a reboot since 2007 IIRC.

I am very much not a UI/UX person, but Firefox and other browsers really
could use a good way to override system crypto policy on a per-site
basis.

-- 
Chris Adams <linux(a)cmadams.net&gt;

[Dominik 'Rathann' Mierzejewski]

On Wednesday, 09 March 2022 at 15:59, Chris Adams wrote:
[...]
>  I am very much not a UI/UX person, but Firefox and other browsers
> really
>  could use a good way to override system crypto policy on a per-site
>  basis. 
Have you opened a ticket in Firefox bugzilla?

Regards,
Dominik
-- 
Fedora   https://getfedora.org  |  RPM Fusion  http://rpmfusion.org
There should be a science of discontent. People need hard times and
oppression to develop psychic muscles.
        -- from "Collected Sayings of Muad'Dib" by the Princess Irulan

[Robbie Harwood]

Alexander Sosedkin <asosedkin(a)redhat.com&gt; writes:

>  Daniel P. Berrangé <berrange(a)redhat.com&gt; wrote:
>
> > Perhaps a useful first step is to just modify the three main
> > crypto libs (gnutls, openssl, and nss) to send a scary warnihg
> > message to stderr/syslog any time they get use of SHA1 in a
> > signature. Leave that active for a release cycle and see how
> > many bug reports we get.
>
>  I left my crystal ball at home today,
>  but I don't need it to say it'd be ~0 bugs filed if we log to syslog
>  and ~3 if we log to stderr/stdout, all named
>  "$CRYPTOLIB has no business messing up my stderr/stdout", 
It's clear you want SHA-1 gone, but the way you've written this maybe
isn't conveying what you wan, as it sounds like you're also unwilling to
process the bugs that result requesting its removal.  (If you, who want
it gone, aren't willing to participate in that, why should maintainers
care?)

As I understood the proposal, it would be for the crypto lib to log a
message like:

   [timestamp] /usr/bin/firefox used DEPRECATED SHA-1 invocation

This is similar to what happened for /var/run: sure, it was annoying to
basically everyone involved, but the bugs also went to the relevant
packages.

>  which we'll promptly close by reverting the changes. 
I don't see why you'd do that instead of reassigning to the appropriate
packages or (better) helping them migrate.

Be well,
--Robbie

[Matthew Miller]

On Wed, Mar 09, 2022 at 12:14:28PM +0100, Alexander Sosedkin wrote:
>  I left my crystal ball at home today,
>  but I don't need it to say it'd be ~0 bugs filed if we log to syslog
>  and ~3 if we log to stderr/stdout, all named
>  "$CRYPTOLIB has no business messing up my stderr/stdout",
>  which we'll promptly close by reverting the changes. 
Yeah, that's a little more cynically-phrased than I'd put it, but I had
similar thoughts.

But: maybe if we logged it _and_ had a tool people could run to
look specifically for those log entries, we could do something like a Test
Day where people could send in reports?

-- 
Matthew Miller
<mattdm(a)fedoraproject.org&gt;
Fedora Project Leader

[Alexander Sosedkin]

On Wed, Mar 9, 2022 at 4:40 PM Robbie Harwood <rharwood(a)redhat.com&gt; wrote:
>  Alexander Sosedkin <asosedkin(a)redhat.com&gt; writes:
>
>  > Daniel P. Berrangé <berrange(a)redhat.com&gt; wrote:
>  >
>  >> Perhaps a useful first step is to just modify the three main
>  >> crypto libs (gnutls, openssl, and nss) to send a scary warnihg
>  >> message to stderr/syslog any time they get use of SHA1 in a
>  >> signature. Leave that active for a release cycle and see how
>  >> many bug reports we get.
>  >
>  > I left my crystal ball at home today,
>  > but I don't need it to say it'd be ~0 bugs filed if we log to syslog
>  > and ~3 if we log to stderr/stdout, all named
>  > "$CRYPTOLIB has no business messing up my stderr/stdout",
>
>  It's clear you want SHA-1 gone, but the way you've written this maybe
>  isn't conveying what you wan, as it sounds like you're also unwilling to
>  process the bugs that result requesting its removal.  (If you, who want
>  it gone, aren't willing to participate in that, why should maintainers
>  care?) 
Oh, nobody wants to go to the dentist, myself included.
But it's as inevitable as it's not delegatable.

Deprecating SHA-1 is no small thing,
and one me could realistically do that for just a single small random thing.
I can't realistically fix something big like git.
I can't make DNSSEC deployments worldwide to move away from SHA-1.
Besides small consultations, what I'm sure I can do is: keep postponing,
raise discussions that don't convey the impact accurately,
and then disable and collect my negative karma.

I want to know, ultimately, how can I break it for devs but not the users
so that 1. the need for it is communicated early,
2. all the relevant places are timely identified,
3. maintainers have a plenty of time to resolve it for good or opt out,
4. and the users are affected as late and as smoothly as possible.
Hope these are all reasonable things to wish for.

---

>  As I understood the proposal, it would be for the crypto lib to log
> a
>  message like:
>
>     [timestamp] /usr/bin/firefox used DEPRECATED SHA-1 invocation
>
>  This is similar to what happened for /var/run: sure, it was annoying to
>  basically everyone involved, but the bugs also went to the relevant
>  packages.
>
>  > which we'll promptly close by reverting the changes.
>
>  I don't see why you'd do that instead of reassigning to the appropriate
>  packages or (better) helping them migrate. 
Because a cryptographic library polluting stdout/stderr is a bug!
It's a bad idea on so many fronts:
* for logging to be seen, it should go where the app logging goes
* stderr/stdout is not always wired up to something at all
* and when it is, *we break the output of the application*

I think I've seen a bug when a library
just opening a descriptor for its own private usage
made it all go sideways, and you suggest actively interfering.

---

Just for example, imagine we get a bug report:
"none of my git scripts work anymore",
telling us that users can no longer reliably script git because there's
`/usr/libexec/git-core/git-clone used DEPRECATED SHA-1 invocation`
all over the output they parse.

We reassign it to git, git maintainers take their time and then decide:
"we're gonna default to SHA-256 next month
 and disallow SHA-1 by default in 3 years
 because $VALID_REASONS, and for now we want $THIS_BEHAVIOUR instead".
Or some other intricate, complicated and unique answer.

Then what? Output of dozens of tools is still broken, users are
rightfully upset,
and, to add insult to injury, SHA-1 signatures still work.
So, I guess, we're supposed to add an API to suppress the warning?
Could be OK to use'em because it's gated behind an app opt-in or something.

And then next release we disable it and then a different hundred of tools,
those whose output nobody monitors or parses or even plumbs anywhere,
break and we've effectively given them no forewarning.
The <1% of those launching libreoffice from the terminal
will have to intersect with those importing some specific format of documents
to even get a slim chance of knowing why they should beware of the leopard.

And it'll also break git again,
giving a second round of headache to git maintainers,
forcing them to roll-back interim solutions and deploy long-term ones.

I think this approach manages to do it wrong twice,
and that's not counting the very idea of libraries messing with
stderr/stdout in the first place, which is, IMO, unthinkable.

Breaking things and providing overrides to use them
if really safe or necessary or opted into
means interrupting maintainers just once
to decide whether they override and keep using it or change to something else.
Sounds much better to me. Less distractions, more consistency.
Git breaks, git gets just a single prod to do the right long-term thing.
Problem is, how does one stretch that in time and shield users from it.

---

That being said, I'm not bent on just breaking it Michael Bay-style.
On the contrary, I've started this thread precisely because I don't want to.
I am open to other solutions. It's just that we shouldn't overdo them either.
Like, I think Fedora SELinux denial UX is good.
These don't just go to syslog or pollute stderr,
they, IIRC, give desktop users instantaneous notifications
that are easy to link to some specific thing that just failed on you.
Doesn't mean we should write a daemon and applets for all DEs
to dbus around requests to allow SHA-1 per executable path.
But, IDK, if y'all tell me that waiting for another release
and then disabling SHA-1 only on Tuesdays and Sundays
only in rawhide for another 16 months is just what we need to soften the blow,
then that's what I'm gonna roll with. As long as it's eventually done.

[Daniel P. Berrangé]

On Wed, Mar 09, 2022 at 12:21:18PM -0500, Matthew Miller wrote:
>  On Wed, Mar 09, 2022 at 12:14:28PM +0100, Alexander Sosedkin wrote:
>  > I left my crystal ball at home today,
>  > but I don't need it to say it'd be ~0 bugs filed if we log to syslog
>  > and ~3 if we log to stderr/stdout, all named
>  > "$CRYPTOLIB has no business messing up my stderr/stdout",
>  > which we'll promptly close by reverting the changes.
>  
>  Yeah, that's a little more cynically-phrased than I'd put it, but I had
>  similar thoughts.
>  
>  But: maybe if we logged it _and_ had a tool people could run to
>  look specifically for those log entries, we could do something like a Test
>  Day where people could send in reports? 
Or just have it logging in rawhide, not in the final release. 

Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

[Alexander Sosedkin]

On Wed, Mar 9, 2022 at 6:22 PM Matthew Miller <mattdm(a)fedoraproject.org&gt; wrote:
>  On Wed, Mar 09, 2022 at 12:14:28PM +0100, Alexander Sosedkin wrote:
>  > I left my crystal ball at home today,
>  > but I don't need it to say it'd be ~0 bugs filed if we log to syslog
>  > and ~3 if we log to stderr/stdout, all named
>  > "$CRYPTOLIB has no business messing up my stderr/stdout",
>  > which we'll promptly close by reverting the changes.
>
>  Yeah, that's a little more cynically-phrased than I'd put it, but I had
>  similar thoughts.
>
>  But: maybe if we logged it _and_ had a tool people could run to
>  look specifically for those log entries, we could do something like a Test
>  Day where people could send in reports? 
On one hand, this still is a risky territory
of libraries leaning heavily into things that libraries shouldn't do
because they can never guess the weird ways they're being used
when they have existing interfaces to deny operations with some algorithms.

On the other hand, this makes much more sense to me
than the stdout/stderr proposal.
Especially if it accompanies a drawn-out multicycle change,
it could be a noticeable impact dampener.

>  --
>  Matthew Miller
>  <mattdm(a)fedoraproject.org&gt;
>  Fedora Project Leader
>  _______________________________________________
>  devel mailing list -- devel(a)lists.fedoraproject.org
>  To unsubscribe send an email to devel-leave(a)lists.fedoraproject.org
>  Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
>  List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
>  List Archives:
> https://lists.fedoraproject.org/archives/list/[e-mail]
>  Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure

[Daniel P. Berrangé]

On Wed, Mar 09, 2022 at 06:45:49PM +0100, Alexander Sosedkin wrote:
>  On Wed, Mar 9, 2022 at 6:22 PM Matthew Miller
> <mattdm(a)fedoraproject.org&gt; wrote:
>  >
>  > On Wed, Mar 09, 2022 at 12:14:28PM +0100, Alexander Sosedkin wrote:
>  > > I left my crystal ball at home today,
>  > > but I don't need it to say it'd be ~0 bugs filed if we log to syslog
>  > > and ~3 if we log to stderr/stdout, all named
>  > > "$CRYPTOLIB has no business messing up my stderr/stdout",
>  > > which we'll promptly close by reverting the changes.
>  >
>  > Yeah, that's a little more cynically-phrased than I'd put it, but I had
>  > similar thoughts.
>  >
>  > But: maybe if we logged it _and_ had a tool people could run to
>  > look specifically for those log entries, we could do something like a Test
>  > Day where people could send in reports?
>  
>  On one hand, this still is a risky territory
>  of libraries leaning heavily into things that libraries shouldn't do
>  because they can never guess the weird ways they're being used
>  when they have existing interfaces to deny operations with some algorithms.
>  
>  On the other hand, this makes much more sense to me
>  than the stdout/stderr proposal. 
The reason I suggested stderr was because it is about the only channel
you can have a reasonable guarantee of being able to use from a library
that has no knowledge about its execution environment.

Security policies applied to procesess (SELinux, seccomp, namespaces or
containerization in general) can easily prevent ability to use journald,
syslog, or opening of log files, leading to messages not being visible.
At worst, with seccomp an attempt to use the blocked feature may lead
to an abort of the application.

With regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

[Matthew Miller]

On Wed, Mar 09, 2022 at 05:40:50PM +0000, Daniel P. Berrangé wrote:
>  > But: maybe if we logged it _and_ had a tool people could run to
>  > look specifically for those log entries, we could do something like a Test
>  > Day where people could send in reports?
>  
>  Or just have it logging in rawhide, not in the final release.  
Yes, but we still need someone to look at those logs.

-- 
Matthew Miller
<mattdm(a)fedoraproject.org&gt;
Fedora Project Leader

[Alexander Sosedkin]

On Wed, Mar 9, 2022 at 7:05 PM Daniel P. Berrangé <berrange(a)redhat.com&gt; wrote:
>  On Wed, Mar 09, 2022 at 06:45:49PM +0100, Alexander Sosedkin wrote:
>  > On Wed, Mar 9, 2022 at 6:22 PM Matthew Miller <mattdm(a)fedoraproject.org&gt;
> wrote:
>  > >
>  > > On Wed, Mar 09, 2022 at 12:14:28PM +0100, Alexander Sosedkin wrote:
>  > > > I left my crystal ball at home today,
>  > > > but I don't need it to say it'd be ~0 bugs filed if we log to
> syslog
>  > > > and ~3 if we log to stderr/stdout, all named
>  > > > "$CRYPTOLIB has no business messing up my stderr/stdout",
>  > > > which we'll promptly close by reverting the changes.
>  > >
>  > > Yeah, that's a little more cynically-phrased than I'd put it, but I
> had
>  > > similar thoughts.
>  > >
>  > > But: maybe if we logged it _and_ had a tool people could run to
>  > > look specifically for those log entries, we could do something like a Test
>  > > Day where people could send in reports?
>  >
>  > On one hand, this still is a risky territory
>  > of libraries leaning heavily into things that libraries shouldn't do
>  > because they can never guess the weird ways they're being used
>  > when they have existing interfaces to deny operations with some algorithms.
>  >
>  > On the other hand, this makes much more sense to me
>  > than the stdout/stderr proposal.
>
>  The reason I suggested stderr was because it is about the only channel
>  you can have a reasonable guarantee of being able to use from a library
>  that has no knowledge about its execution environment. 
I don't think there is a reasonable guarantee you can use it.
It's normal for a daemon to close inherited file descriptors
and open something radically different as FD 3, right?
And then we're writing our scary messages who knows where.

>  Security policies applied to procesess (SELinux, seccomp, namespaces
> or
>  containerization in general) can easily prevent ability to use journald,
>  syslog, or opening of log files, leading to messages not being visible. 
If the goal is some visibility, then it might...

>  At worst, with seccomp an attempt to use the blocked feature may
> lead
>  to an abort of the application. 
sigh.

>  With regards,
>  Daniel
>  --
>  |: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
>  |: https://libvirt.org         -o-            https://fstop138.berrange.com :|
>  |: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|
>

[Daniel P. Berrangé]

On Wed, Mar 09, 2022 at 01:05:38PM -0500, Matthew Miller wrote:
>  On Wed, Mar 09, 2022 at 05:40:50PM +0000, Daniel P. Berrangé wrote:
>  > > But: maybe if we logged it _and_ had a tool people could run to
>  > > look specifically for those log entries, we could do something like a Test
>  > > Day where people could send in reports?
>  > 
>  > Or just have it logging in rawhide, not in the final release. 
>  
>  Yes, but we still need someone to look at those logs. 
Don't hate me for suggesting this, but the warning message on stderr
could suggest/request reporting the problem to bugzilla to bring it
to maintainer's attention. 

   *** NOTICE: pid 1234 (firefox) used cryptographic signature
   *** NOTICE: with deprecated SHA-1 algorithm. This usage
   *** NOTICE: will be prevented by future policy. Please report
   *** NOTICE: this problem to https://bugzilla.redhat.com

Make it a one time message per process to avoid producing
pages of repeated text, and allow 'touch /etc/crypto-policies/dont-warn-me'
to hide it system wide.

It would be irritating enough to make at least some subset of rawhide
users followup with reporting bugs, and thus let maintainers understand
how badly they are impacted. 

With regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

[Daniel P. Berrangé]

On Wed, Mar 09, 2022 at 07:13:14PM +0100, Alexander Sosedkin wrote:
>  On Wed, Mar 9, 2022 at 7:05 PM Daniel P. Berrangé
> <berrange(a)redhat.com&gt; wrote:
>  >
>  > On Wed, Mar 09, 2022 at 06:45:49PM +0100, Alexander Sosedkin wrote:
>  > > On Wed, Mar 9, 2022 at 6:22 PM Matthew Miller <mattdm(a)fedoraproject.org&gt;
> wrote:
>  > > >
>  > > > On Wed, Mar 09, 2022 at 12:14:28PM +0100, Alexander Sosedkin wrote:
>  > > > > I left my crystal ball at home today,
>  > > > > but I don't need it to say it'd be ~0 bugs filed if we log to
> syslog
>  > > > > and ~3 if we log to stderr/stdout, all named
>  > > > > "$CRYPTOLIB has no business messing up my stderr/stdout",
>  > > > > which we'll promptly close by reverting the changes.
>  > > >
>  > > > Yeah, that's a little more cynically-phrased than I'd put it, but
> I had
>  > > > similar thoughts.
>  > > >
>  > > > But: maybe if we logged it _and_ had a tool people could run to
>  > > > look specifically for those log entries, we could do something like a
> Test
>  > > > Day where people could send in reports?
>  > >
>  > > On one hand, this still is a risky territory
>  > > of libraries leaning heavily into things that libraries shouldn't do
>  > > because they can never guess the weird ways they're being used
>  > > when they have existing interfaces to deny operations with some algorithms.
>  > >
>  > > On the other hand, this makes much more sense to me
>  > > than the stdout/stderr proposal.
>  >
>  > The reason I suggested stderr was because it is about the only channel
>  > you can have a reasonable guarantee of being able to use from a library
>  > that has no knowledge about its execution environment.
>  
>  I don't think there is a reasonable guarantee you can use it.
>  It's normal for a daemon to close inherited file descriptors
>  and open something radically different as FD 3, right?
>  And then we're writing our scary messages who knows where. 
I presume you mean FD 2, not 3, as stderr==2.

Anything is possible, but I don't think it is common for apps
to connect something strange to FD 2, precisely because that is
designated as stderr.

There exist way too many libraries that will print error messages
and warnings to stderr, for any non-trivial application to consider
FD==2 safe to use for any functional purposes other than adhoc text
output. Any application that wants to disconnect from its inherited
stderr stream, will re-open it on /dev/null, or a text logfile, if
they don't want to risk being broken by random code using stderr.

Even glibc will print to stderr when it sees malloc corruption,
and stuff in stderr will end up in the systemd journal for any
system services.

With regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

[Alexander Sosedkin]

On Wed, Mar 9, 2022 at 7:19 PM Daniel P. Berrangé <berrange(a)redhat.com&gt; wrote:
>  On Wed, Mar 09, 2022 at 01:05:38PM -0500, Matthew Miller wrote:
>  > On Wed, Mar 09, 2022 at 05:40:50PM +0000, Daniel P. Berrangé wrote:
>  > > > But: maybe if we logged it _and_ had a tool people could run to
>  > > > look specifically for those log entries, we could do something like a
> Test
>  > > > Day where people could send in reports?
>  > >
>  > > Or just have it logging in rawhide, not in the final release.
>  >
>  > Yes, but we still need someone to look at those logs.
>
>  Don't hate me for suggesting this, but the warning message on stderr
>  could suggest/request reporting the problem to bugzilla to bring it
>  to maintainer's attention.
>
>     *** NOTICE: pid 1234 (firefox) used cryptographic signature
>     *** NOTICE: with deprecated SHA-1 algorithm. This usage
>     *** NOTICE: will be prevented by future policy. Please report
>     *** NOTICE: this problem to https://bugzilla.redhat.com
>
>  Make it a one time message per process to avoid producing
>  pages of repeated text, and allow 'touch /etc/crypto-policies/dont-warn-me'
>  to hide it system wide.
>
>  It would be irritating enough 
I've just checked and it's not just that
my firefox already prints ~20 variations on "FAIL" on startup
but I'm also pretty sure I see these for the first time,
meaning I've never cared to check before.

>  to make at least some subset of rawhide
>  users followup with reporting bugs, and thus let maintainers understand
>  how badly they are impacted.
>
>  With regards,
>  Daniel
>  --
>  |: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
>  |: https://libvirt.org         -o-            https://fstop138.berrange.com :|
>  |: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|
>  _______________________________________________
>  devel mailing list -- devel(a)lists.fedoraproject.org
>  To unsubscribe send an email to devel-leave(a)lists.fedoraproject.org
>  Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
>  List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
>  List Archives:
> https://lists.fedoraproject.org/archives/list/[e-mail]
>  Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure

[Demi Marie Obenour]

On 3/9/22 08:52, Alexander Sosedkin wrote:
>  On Wed, Mar 9, 2022 at 2:47 PM Richard W.M. Jones
> <rjones(a)redhat.com&gt; wrote:
> >
> > Previous tightening of crypto defaults caused problems with us
> > connecting to older ssh servers.
> >
> > I am particularly interested / worried about sshd from RHEL 5, 6 & 7
> > for virt-p2v and virt-v2v conversions.  This broke before, requiring
> > us to advise users to set the global policy for the machine to LEGACY
> > (thus ironically weakening crypto for everything).
> >
> > Also I have some ancient network equipment that cannot be upgraded but
> > needs older ssh protocols.  I can't connect to it from Fedora unless I
> > set the crypto policy to LEGACY.
> >
> > Anyway I'm wondering if the SHA-1 change will impact ssh further?
>  
>  IIRC, the only SHA-1 thing that should be left in DEFAULT for SSH
>  is SHA-1 as HMAC, which doesn't rely on collision resistance.
>  So, not this round. 
SHA-1 HMAC is still considered to be a perfectly good MAC.  For new
protocols, I recommend using Blake2b instead, but that is purely for
performance reasons, not because HMAC-SHA-1 is broken.  There are no
known attacks on HMAC-SHA-1, and it is actually stronger than AES-128
in CBC-MAC (160 bits of security vs. 128).

-- 
Sincerely,
Demi Marie Obenour (she/her/hers)

[Robert Relyea]

On 3/9/22 1:56 AM, Daniel P. Berrangé wrote:
>  On Wed, Mar 09, 2022 at 10:46:21AM +0100, Alexander Sosedkin wrote:
> > On Wed, Mar 9, 2022 at 10:20 AM Daniel P. Berrangé <berrange(a)redhat.com&gt;
> wrote:
> >> On Tue, Mar 08, 2022 at 07:40:15PM +0100, Alexander Sosedkin wrote:
> >>> We've been disabling it in TLS, but its usage is much wider than TLS.
> >>> The next agonizing step is to restrict its usage for signatures
> >>> on the cryptographic libraries level, with openssl being the scariest one.
> >>>
> >>> Good news is, RHEL-9 is gonna lead the way
> >>> and thus will take a lot of the hits first.
> >>> Fedora doesn't have to pioneer it.
> >>> Bad news is, Fedora has to follow suit someday anyway,
> >>> and this brings me to how does one land such a change.
> >>>
> >>> ---
> >>>
> >>> Fedora is a large distribution with short release cycles, and
> >>> the only realistic way to weed out its reliance on SHA-1 signatures
> >>> from all of its numerous dark corners is to break them.
> >>> Make creation and verification fail in default configuration.
> >>> But it's unreasonable to just wait for, say, Fedora 37 branch-off
> >>> and break it in Rawhide for Fedora 38.
> >>> The fallout will just be too big.
> >> If RHEL-9 has lead the way, what are the stats for real world
> >> RHEL impact ?
> > We'll know when the real world starts using RHEL-9 en masse?
> >
> >> What is/was the absolute number of packages and % number of
> >> packages from the RHEL distro  that saw breakage ?
> > Does preventing the distro from installing altogether count as 100%?
> > If yes, 100%. =)
> > Jokes aside, I can't give you an accurate estimate yet.
>  Perhaps a useful first step is to just modify the three main
>  crypto libs (gnutls, openssl, and nss) to send a scary warnihg
>  message to stderr/syslog any time they get use of SHA1 in a
>  signature. Leave that active for a release cycle and see how
>  many bug reports we get. 
To be clear, the actual mechanism to turn off SHA1 for signatures 
doesn't involve any changes to any of our crypto libraries, it involves 
changing the crypto policies file. With crypto policies, you can 
actually turn off almost any algorithm involved in SSL or signatures in 
all of our libraries. There is really no good way to 'log' from the 
crypto libraries.

Actually I think that provides a way forward that might work.

1) in fedora 37, provide a policy that turns SHA-1 off. in our testing, 
we encourage people to run with that policy and write bugs against 
components.

2) in fedora 38, SHA-1 gets turned of in the default policy and ships 
that way. Things that still fail would only work once in the legacy policy.

3) some day in the future (fedora 39?) it gets turned off legacy as well.

bob

[Tom Hughes]

On 15/03/2022 22:45, Robert Relyea wrote:

>  1) in fedora 37, provide a policy that turns SHA-1 off. in our
> testing, 
>  we encourage people to run with that policy and write bugs against 
>  components. 
That policy already exists in Fedora 34 and 35 where the FUTURE policy
does not allow SHA1 in signature algorithms.

Tom

-- 
Tom Hughes (tom(a)compton.nu)
http://compton.nu/

[Björn Persson]

Robert Relyea wrote:
>  2) in fedora 38, SHA-1 gets turned of in the default policy and ships
>
>  that way. 
Isn't that the default already? I use the default crypto policy, and I
had a case last year where Seamonkey and Firefox refused to talk to a
certain web server, which I worked around by temporarily adding "SHA1"
to /etc/crypto-policies/back-ends/nss.config.

Björn Persson

[Clemens Lang]

>  On 16. Mar 2022, at 00:04, Tom Hughes via devel
> <devel(a)lists.fedoraproject.org&gt; wrote:
>  
>  On 15/03/2022 22:45, Robert Relyea wrote:
>  
> > 1) in fedora 37, provide a policy that turns SHA-1 off. in our testing, we encourage
> people to run with that policy and write bugs against components.
>  
>  That policy already exists in Fedora 34 and 35 where the FUTURE policy
>  does not allow SHA1 in signature algorithms. 
In the case of OpenSSL, that only affects use of SHA1 as signature algorithms in TLS.
It does not cover arbitrary signatures with a SHA1 digest, which is what we are
proposing.


HTH,
Clemens

-- 
Clemens Lang
RHEL Crypto Team
Red Hat

[Josh Boyer]

On Tue, Mar 8, 2022 at 1:40 PM Alexander Sosedkin <asosedkin(a)redhat.com&gt; wrote:
>  Hello, community, I need your wisdom for planning a disruptive change.
>
>  Fedora 28 had https://fedoraproject.org/wiki/Changes/StrongCryptoSettings
>  Fedora 33 had https://fedoraproject.org/wiki/Changes/StrongCryptoSettings2
>  I believe we should start planning
>  for the next cryptographic defaults tightening.
>  And next time it's gonna be even more disruptive because of SHA-1 (again).
>
>  SHA-1 is a hash function from 1995,
>  which collision resistance is no longer to be relied upon for security [1].
>  At the same time, it's not like software has successfully migrated off it,
>  not even close.
>  It's not a question of "if" the world should migrate from it,
>  sooner or later we must part ways with it.
>  (Technically, some acute energy crisis or a collapse of civilization
>   forever raising the costs of computations thousandfold would also do,
>   but let's agree that migrating to a more modern hash is the way =)
>
>  We've been disabling it in TLS, but its usage is much wider than TLS.
>  The next agonizing step is to restrict its usage for signatures
>  on the cryptographic libraries level, with openssl being the scariest one.
>
>  Good news is, RHEL-9 is gonna lead the way
>  and thus will take a lot of the hits first.
>  Fedora doesn't have to pioneer it.
>  Bad news is, Fedora has to follow suit someday anyway,
>  and this brings me to how does one land such a change. 
Given that RHEL 9 already switched the crypto-policy defaults, and we
presume that future RHEL releases will not deviate from that, is this
change already made in ELN?  I honestly can't tell because it looks
like it is, but I don't see any conditionalization in the spec file
that would lead me to believe the same default isn't already flipped
in Rawhide.  That leaves me wondering what needs to be changed and
where at this point.

josh

>
> ---
>
> Fedora is a large distribution with short release cycles, and
> the only realistic way to weed out its reliance on SHA-1 signatures
> from all of its numerous dark corners is to break them.
> Make creation and verification fail in default configuration.
> But it's unreasonable to just wait for, say, Fedora 37 branch-off
> and break it in Rawhide for Fedora 38.
> The fallout will just be too big.
>
> Maintainers need time to get bugs, look into them, think,
> analyze, react and test --- and that's just if it fails correctly!
> Unfortunately, it's not just that the error paths are as dusty as they get
> because the program counter has never set foot on them before.
> Some maintainers might even find that
> picking a different hash function renders their code non-interoperable,
> or even that protocols they implement have SHA-1 hardcoded in the spec.
> Or that everything is ready, but real world deployments need another decade.
> Or that on-disk formats are just hard to change and migrate.
> Took git years to migrate from SHA-1, and some others haven't even started.
> There are gonna be investigations, planning, exceptions, upstream changes,
> opt-out mechanisms, arguing, compromises, waiting out, all kinds of things.
> It's gonna be big. Too big for a single release cycle.
>
> ---
>
> But how does one land something and give the distribution
> the extra cycles needed to react? That's not really clear to me.
>
> An obvious thing is to announce it in one cycle and land in another one.
> The downsides are well-documented
> in "The Hitchhiker's Guide to the Galaxy":
> announcements are one weak measure, and then it's too late.
>
> A second scheme I can come up with is a "jump scare".
> Break the functionality in Fedora 37 Rawhide,
> make most of the affected people realize the depth of the problem,
> then unbreak it. Break again for Fedora 38 and never fix.
>
> This could also be extended into "let one stable release slide'.
> Break in 37 Rawhide, unbreak on branched off 37,
> but never in Rawhide.
>
> But these are all rather... crude?
> Sure there should be better ways,
> preferably something explored before.
> I'm all for pulling this tooth out smoothly,
> but I need hints on how to do it.
> I hope that together we can devise a better plan than these.
>
> So, how does one land a change that's bigger than a release cycle?
>
> [1] https://eprint.iacr.org/2020/014
> _______________________________________________
> devel mailing list -- devel(a)lists.fedoraproject.org
> To unsubscribe send an email to devel-leave(a)lists.fedoraproject.org
> Fedora Code of Conduct:
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives:
https://lists.fedoraproject.org/archives/list/[e-mail]
> Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure

[Alexander Sosedkin]

On Wed, Mar 23, 2022 at 12:51 AM Josh Boyer <jwboyer(a)fedoraproject.org&gt; wrote:
>  On Tue, Mar 8, 2022 at 1:40 PM Alexander Sosedkin <asosedkin(a)redhat.com&gt; wrote:
>  >
>  > Hello, community, I need your wisdom for planning a disruptive change.
>  >
>  > Fedora 28 had https://fedoraproject.org/wiki/Changes/StrongCryptoSettings
>  > Fedora 33 had https://fedoraproject.org/wiki/Changes/StrongCryptoSettings2
>  > I believe we should start planning
>  > for the next cryptographic defaults tightening.
>  > And next time it's gonna be even more disruptive because of SHA-1 (again).
>  >
>  > SHA-1 is a hash function from 1995,
>  > which collision resistance is no longer to be relied upon for security [1].
>  > At the same time, it's not like software has successfully migrated off it,
>  > not even close.
>  > It's not a question of "if" the world should migrate from it,
>  > sooner or later we must part ways with it.
>  > (Technically, some acute energy crisis or a collapse of civilization
>  >  forever raising the costs of computations thousandfold would also do,
>  >  but let's agree that migrating to a more modern hash is the way =)
>  >
>  > We've been disabling it in TLS, but its usage is much wider than TLS.
>  > The next agonizing step is to restrict its usage for signatures
>  > on the cryptographic libraries level, with openssl being the scariest one.
>  >
>  > Good news is, RHEL-9 is gonna lead the way
>  > and thus will take a lot of the hits first.
>  > Fedora doesn't have to pioneer it.
>  > Bad news is, Fedora has to follow suit someday anyway,
>  > and this brings me to how does one land such a change.
>
>  Given that RHEL 9 already switched the crypto-policy defaults, and we
>  presume that future RHEL releases will not deviate from that, is this
>  change already made in ELN?  I honestly can't tell because it looks
>  like it is, but I don't see any conditionalization in the spec file
>  that would lead me to believe the same default isn't already flipped
>  in Rawhide.  That leaves me wondering what needs to be changed and
>  where at this point.
>
>  josh 
No, it's neither in ELN nor in Rawhide yet.

>  > Fedora is a large distribution with short release cycles, and
>  > the only realistic way to weed out its reliance on SHA-1 signatures
>  > from all of its numerous dark corners is to break them.
>  > Make creation and verification fail in default configuration.
>  > But it's unreasonable to just wait for, say, Fedora 37 branch-off
>  > and break it in Rawhide for Fedora 38.
>  > The fallout will just be too big.
>  >
>  > Maintainers need time to get bugs, look into them, think,
>  > analyze, react and test --- and that's just if it fails correctly!
>  > Unfortunately, it's not just that the error paths are as dusty as they get
>  > because the program counter has never set foot on them before.
>  > Some maintainers might even find that
>  > picking a different hash function renders their code non-interoperable,
>  > or even that protocols they implement have SHA-1 hardcoded in the spec.
>  > Or that everything is ready, but real world deployments need another decade.
>  > Or that on-disk formats are just hard to change and migrate.
>  > Took git years to migrate from SHA-1, and some others haven't even started.
>  > There are gonna be investigations, planning, exceptions, upstream changes,
>  > opt-out mechanisms, arguing, compromises, waiting out, all kinds of things.
>  > It's gonna be big. Too big for a single release cycle.
>  >
>  > ---
>  >
>  > But how does one land something and give the distribution
>  > the extra cycles needed to react? That's not really clear to me.
>  >
>  > An obvious thing is to announce it in one cycle and land in another one.
>  > The downsides are well-documented
>  > in "The Hitchhiker's Guide to the Galaxy":
>  > announcements are one weak measure, and then it's too late.
>  >
>  > A second scheme I can come up with is a "jump scare".
>  > Break the functionality in Fedora 37 Rawhide,
>  > make most of the affected people realize the depth of the problem,
>  > then unbreak it. Break again for Fedora 38 and never fix.
>  >
>  > This could also be extended into "let one stable release slide'.
>  > Break in 37 Rawhide, unbreak on branched off 37,
>  > but never in Rawhide.
>  >
>  > But these are all rather... crude?
>  > Sure there should be better ways,
>  > preferably something explored before.
>  > I'm all for pulling this tooth out smoothly,
>  > but I need hints on how to do it.
>  > I hope that together we can devise a better plan than these.
>  >
>  > So, how does one land a change that's bigger than a release cycle?
>  >
>  > [1] https://eprint.iacr.org/2020/014
>  > _______________________________________________
>  > devel mailing list -- devel(a)lists.fedoraproject.org
>  > To unsubscribe send an email to devel-leave(a)lists.fedoraproject.org
>  > Fedora Code of Conduct:
> https://docs.fedoraproject.org/en-US/project/code-of-conduct/
>  > List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
>  > List Archives:
> https://lists.fedoraproject.org/archives/list/[e-mail]
>  > Do not reply to spam on the list, report it:
> https://pagure.io/fedora-infrastructure
>  _______________________________________________
>  devel mailing list -- devel(a)lists.fedoraproject.org
>  To unsubscribe send an email to devel-leave(a)lists.fedoraproject.org
>  Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
>  List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
>  List Archives:
> https://lists.fedoraproject.org/archives/list/[e-mail]
>  Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure

[Josh Boyer]

On Wed, Mar 23, 2022 at 7:05 AM Alexander Sosedkin <asosedkin(a)redhat.com&gt; wrote:
>  On Wed, Mar 23, 2022 at 12:51 AM Josh Boyer <jwboyer(a)fedoraproject.org&gt; wrote:
>  >
>  > On Tue, Mar 8, 2022 at 1:40 PM Alexander Sosedkin <asosedkin(a)redhat.com&gt;
> wrote:
>  > >
>  > > Hello, community, I need your wisdom for planning a disruptive change.
>  > >
>  > > Fedora 28 had https://fedoraproject.org/wiki/Changes/StrongCryptoSettings
>  > > Fedora 33 had https://fedoraproject.org/wiki/Changes/StrongCryptoSettings2
>  > > I believe we should start planning
>  > > for the next cryptographic defaults tightening.
>  > > And next time it's gonna be even more disruptive because of SHA-1 (again).
>  > >
>  > > SHA-1 is a hash function from 1995,
>  > > which collision resistance is no longer to be relied upon for security [1].
>  > > At the same time, it's not like software has successfully migrated off it,
>  > > not even close.
>  > > It's not a question of "if" the world should migrate from it,
>  > > sooner or later we must part ways with it.
>  > > (Technically, some acute energy crisis or a collapse of civilization
>  > >  forever raising the costs of computations thousandfold would also do,
>  > >  but let's agree that migrating to a more modern hash is the way =)
>  > >
>  > > We've been disabling it in TLS, but its usage is much wider than TLS.
>  > > The next agonizing step is to restrict its usage for signatures
>  > > on the cryptographic libraries level, with openssl being the scariest one.
>  > >
>  > > Good news is, RHEL-9 is gonna lead the way
>  > > and thus will take a lot of the hits first.
>  > > Fedora doesn't have to pioneer it.
>  > > Bad news is, Fedora has to follow suit someday anyway,
>  > > and this brings me to how does one land such a change.
>  >
>  > Given that RHEL 9 already switched the crypto-policy defaults, and we
>  > presume that future RHEL releases will not deviate from that, is this
>  > change already made in ELN?  I honestly can't tell because it looks
>  > like it is, but I don't see any conditionalization in the spec file
>  > that would lead me to believe the same default isn't already flipped
>  > in Rawhide.  That leaves me wondering what needs to be changed and
>  > where at this point.
>  >
>  > josh
>
>  No, it's neither in ELN nor in Rawhide yet. 
Could you please make this change in ELN?  It might actually prove
fruitful for the broader Fedora activity because it gives you an
isolated base to see what happens.

If we already know there are other similar changes for future RHEL
releases, making those in ELN now would go a long way towards sorting
those out as well.  That can be a follow-on after SHA-1 though, which
is certainly more pressing.

josh

[Alexander Sosedkin]

On Tue, Mar 8, 2022 at 7:40 PM Alexander Sosedkin <asosedkin(a)redhat.com&gt; wrote:
>  Hello, community, I need your wisdom for planning a disruptive change.
>
>  Fedora 28 had https://fedoraproject.org/wiki/Changes/StrongCryptoSettings
>  Fedora 33 had https://fedoraproject.org/wiki/Changes/StrongCryptoSettings2
>  I believe we should start planning
>  for the next cryptographic defaults tightening.
>  And next time it's gonna be even more disruptive because of SHA-1 (again).
>
>  SHA-1 is a hash function from 1995,
>  which collision resistance is no longer to be relied upon for security [1].
>  At the same time, it's not like software has successfully migrated off it,
>  not even close.
>  It's not a question of "if" the world should migrate from it,
>  sooner or later we must part ways with it.
>  (Technically, some acute energy crisis or a collapse of civilization
>   forever raising the costs of computations thousandfold would also do,
>   but let's agree that migrating to a more modern hash is the way =)
>
>  We've been disabling it in TLS, but its usage is much wider than TLS.
>  The next agonizing step is to restrict its usage for signatures
>  on the cryptographic libraries level, with openssl being the scariest one.
>
>  Good news is, RHEL-9 is gonna lead the way
>  and thus will take a lot of the hits first.
>  Fedora doesn't have to pioneer it.
>  Bad news is, Fedora has to follow suit someday anyway,
>  and this brings me to how does one land such a change.
>
>  ---
>
>  Fedora is a large distribution with short release cycles, and
>  the only realistic way to weed out its reliance on SHA-1 signatures
>  from all of its numerous dark corners is to break them.
>  Make creation and verification fail in default configuration.
>  But it's unreasonable to just wait for, say, Fedora 37 branch-off
>  and break it in Rawhide for Fedora 38.
>  The fallout will just be too big.
>
>  Maintainers need time to get bugs, look into them, think,
>  analyze, react and test --- and that's just if it fails correctly!
>  Unfortunately, it's not just that the error paths are as dusty as they get
>  because the program counter has never set foot on them before.
>  Some maintainers might even find that
>  picking a different hash function renders their code non-interoperable,
>  or even that protocols they implement have SHA-1 hardcoded in the spec.
>  Or that everything is ready, but real world deployments need another decade.
>  Or that on-disk formats are just hard to change and migrate.
>  Took git years to migrate from SHA-1, and some others haven't even started.
>  There are gonna be investigations, planning, exceptions, upstream changes,
>  opt-out mechanisms, arguing, compromises, waiting out, all kinds of things.
>  It's gonna be big. Too big for a single release cycle.
>
>  ---
>
>  But how does one land something and give the distribution
>  the extra cycles needed to react? That's not really clear to me.
>
>  An obvious thing is to announce it in one cycle and land in another one.
>  The downsides are well-documented
>  in "The Hitchhiker's Guide to the Galaxy":
>  announcements are one weak measure, and then it's too late.
>
>  A second scheme I can come up with is a "jump scare".
>  Break the functionality in Fedora 37 Rawhide,
>  make most of the affected people realize the depth of the problem,
>  then unbreak it. Break again for Fedora 38 and never fix.
>
>  This could also be extended into "let one stable release slide'.
>  Break in 37 Rawhide, unbreak on branched off 37,
>  but never in Rawhide.
>
>  But these are all rather... crude?
>  Sure there should be better ways,
>  preferably something explored before.
>  I'm all for pulling this tooth out smoothly,
>  but I need hints on how to do it.
>  I hope that together we can devise a better plan than these.
>
>  So, how does one land a change that's bigger than a release cycle?
>
>  [1] https://eprint.iacr.org/2020/014 

"You know these lights in the theaters that go out gradually?
 When the guy ve-ery slo-o-owly pulls the plug out?"
 - a joke from my childhood.


Hello, it's been quiet for a while, and I've been busy
but kept thinking about all the useful feedback you folks gave me.
Not that it made me flesh out a perfect plan,
but hopefully at least a less terrible one.

Regarding smudging the change in time,
how does the following three-phaser sound?

Phase 1 ("Wake-up Call"):
  In Fedora 37, disable SHA-1 signatures verification/creation
  in FUTURE policy, i.e. opt-in only.
  Come up with some logging solution;
  I'd prefer something non-invasive like eBPF USDT probes [2],
  but maybe even stderr could work, you've been moderately convincing.
  (FUTURE change is *maybe* doable in F36, but not logging.)
  Announce it as a system-wide change anyway for visibility,
  call for Test Days to report which apps/workflows rely on SHA-1 signatures
  either from the logs
  or from opting into blocking operations and seeing what starts failing hard.
  That'd have to be very actively called for to make an impact,
  impact that'd mostly be just maintainers thinking what will they do in

Phase 2 ("Jump Scare"):
  As soon as f37 branch-off happens,
  disable signature verification in DEFAULT in *38 rawhide*.
  Cue an influx of bugreports because things get broken for all testers
  and not just the ones who opt in.
  I anticipate this to be the most eye-opening step
  even if we test a lot in the previous phase, so to smooth it out more
  we then *revert* the change in 38 before the release,
  so the released Fedora behaves just like in 37
  and whatever wasn't sorted out in time gets an extra cycle.
  A second Fedora change should be filed for visibility,
  but clearly stating this will not affect f38 released.

Phase 3 ("Return of the Panik"):
  And then Fedora 39 comes, where the revert hasn't happened,
  goes through the whole release cycle,
  but this time the change goes through and reaches stable.
  Again, a system-wide change, a third one for the same thing.

With the 37-38-39 numbers, that'd mean the change
reaching the users in autumn 2023, with lead times of:
  ~ 3.5 cycles for the most proactive developers to see this thread and panic
  ~ 3 cycles for the testers to proactively report bugs (logging/opting in)
  ~ 2 cycles to address everything else coming from rawhide testing
    before it reaches stable by either
    switching to some other algorithm,
    making the users explicitly opt into trusting SHA-1 signatures somehow,
    or, in the most high-profile cases,
    having a widely publicised exception (and some plan for the future).

Questions:
  * Do you find this smudging reasonable?
  * The usual tightening of the other less controversial algorithms,
    should it follow the same smudging/reverting plan
    since we're going through all that hassle anyway?
  * Does the 37-38-39 timeframe feel right?
  * Do I need to first run this contraption of a plan
    by FESCo or some other smart folks?
  * Is there a better signalling mechanism than filing 3 system-wide changes?
  * What'd be the right mechanism for others to take the wheel if everything
    goes sideways and the need arises to revise the plan mid-execution?

Other kinds of input are, of course, also appreciated.
Even the calls to magically attain the mutually exclusive goals
of offering secure defaults while not breaking insecure workflows
that don't offer actual solutions might serve as a useful mood check.
I know it ain't the best plan. Let's figure out the right thing to do.

[2] https://www.youtube.com/watch?v=CLlxl7OCPfs

[Kevin Fenzi]

On Tue, Mar 29, 2022 at 08:12:47PM +0200, Alexander Sosedkin wrote:
>  
>  "You know these lights in the theaters that go out gradually?
>   When the guy ve-ery slo-o-owly pulls the plug out?"
>   - a joke from my childhood.
>  
>  
>  Hello, it's been quiet for a while, and I've been busy
>  but kept thinking about all the useful feedback you folks gave me.
>  Not that it made me flesh out a perfect plan,
>  but hopefully at least a less terrible one.
>  
>  Regarding smudging the change in time,
>  how does the following three-phaser sound? 
Might work. :) A few comments inline...

>  Phase 1 ("Wake-up Call"):
>    In Fedora 37, disable SHA-1 signatures verification/creation
>    in FUTURE policy, i.e. opt-in only.
>    Come up with some logging solution;
>    I'd prefer something non-invasive like eBPF USDT probes [2],
>    but maybe even stderr could work, you've been moderately convincing.
>    (FUTURE change is *maybe* doable in F36, but not logging.) 
Well, we just shipped beta today, so I think it's too late to land any
f36 changes at this point. 

>    Announce it as a system-wide change anyway for visibility,
>    call for Test Days to report which apps/workflows rely on SHA-1 signatures
>    either from the logs
>    or from opting into blocking operations and seeing what starts failing hard.
>    That'd have to be very actively called for to make an impact,
>    impact that'd mostly be just maintainers thinking what will they do in 
Just a related note here, FUTURE also breaks installing anything via dnf
with the default metalink setup. This is because digicert (where we get
our *.fedoraproject.org wildcard cert) seems to always issue certs from
it's 2048bit CA. :( (If anyone knows how to get them to issue from a
newer CA that works please let me know)

>  Phase 2 ("Jump Scare"):
>    As soon as f37 branch-off happens,
>    disable signature verification in DEFAULT in *38 rawhide*.
>    Cue an influx of bugreports because things get broken for all testers
>    and not just the ones who opt in.
>    I anticipate this to be the most eye-opening step
>    even if we test a lot in the previous phase, so to smooth it out more
>    we then *revert* the change in 38 before the release,
>    so the released Fedora behaves just like in 37
>    and whatever wasn't sorted out in time gets an extra cycle. 
Right before the release? 
Or right before Beta? 
or ?

People kind of expect the beta will be something they can test and will
behave as the final, so changing things after beta seems like a bad idea
in general. That said, shipping beta with it would get a lot more
exposure.

>    A second Fedora change should be filed for visibility,
>    but clearly stating this will not affect f38 released.
>  
>  Phase 3 ("Return of the Panik"):
>    And then Fedora 39 comes, where the revert hasn't happened,
>    goes through the whole release cycle,
>    but this time the change goes through and reaches stable.
>    Again, a system-wide change, a third one for the same thing.
>  
>  With the 37-38-39 numbers, that'd mean the change
>  reaching the users in autumn 2023, with lead times of:
>    ~ 3.5 cycles for the most proactive developers to see this thread and panic
>    ~ 3 cycles for the testers to proactively report bugs (logging/opting in)
>    ~ 2 cycles to address everything else coming from rawhide testing
>      before it reaches stable by either
>      switching to some other algorithm,
>      making the users explicitly opt into trusting SHA-1 signatures somehow,
>      or, in the most high-profile cases,
>      having a widely publicised exception (and some plan for the future).
>  
>  Questions:
>    * Do you find this smudging reasonable? 
I think it's probibly the best we can do. 

>    * The usual tightening of the other less controversial algorithms,
>      should it follow the same smudging/reverting plan
>      since we're going through all that hassle anyway? 
I don't think they would need to have this long a runway.

>    * Does the 37-38-39 timeframe feel right?
>    * Do I need to first run this contraption of a plan
>      by FESCo or some other smart folks? 
Well, I think run it by everyone on this list. 
I don't think people will hold back. ;) 

>    * Is there a better signalling mechanism than filing 3 system-wide
> changes?
>    * What'd be the right mechanism for others to take the wheel if everything
>      goes sideways and the need arises to revise the plan mid-execution? 
I wonder if it would make sense to have a checkpoint before the revert
in f38, and if things look substantually less broken than we fear, we
just don't revert it and let it go out in f38. But that might muddy up
the communications.

>  Other kinds of input are, of course, also appreciated.
>  Even the calls to magically attain the mutually exclusive goals
>  of offering secure defaults while not breaking insecure workflows
>  that don't offer actual solutions might serve as a useful mood check.
>  I know it ain't the best plan. Let's figure out the right thing to do. 
Thanks for taking input on this and being willing to work out the best
way forward. 

kevin

[Michael Catanzaro]

FWIW Alexander's plan sounds reasonable to me.

On Tue, Mar 29 2022 at 03:34:49 PM -0700, Kevin Fenzi <kevin(a)scrye.com&gt; 
wrote:
>  Well, we just shipped beta today, so I think it's too late to
> land any
>  f36 changes at this point. 
This is a non-default configuration that I strongly suspect nobody or 
almost nobody uses, except for testing. Changing this prior to F36 
stable release seems like no big deal.

Michael

[Vít Ondruch]

Dne 08. 03. 22 v 19:40 Alexander Sosedkin napsal(a):
>  Hello, community, I need your wisdom for planning a disruptive
> change.
>
>  Fedora 28 had https://fedoraproject.org/wiki/Changes/StrongCryptoSettings
>  Fedora 33 had https://fedoraproject.org/wiki/Changes/StrongCryptoSettings2
>  I believe we should start planning
>  for the next cryptographic defaults tightening.
>  And next time it's gonna be even more disruptive because of SHA-1 (again).
>
>  SHA-1 is a hash function from 1995,
>  which collision resistance is no longer to be relied upon for security [1].
>  At the same time, it's not like software has successfully migrated off it,
>  not even close.
>  It's not a question of "if" the world should migrate from it,
>  sooner or later we must part ways with it.
>  (Technically, some acute energy crisis or a collapse of civilization
>    forever raising the costs of computations thousandfold would also do,
>    but let's agree that migrating to a more modern hash is the way =)
>
>  We've been disabling it in TLS, but its usage is much wider than TLS.
>  The next agonizing step is to restrict its usage for signatures
>  on the cryptographic libraries level, with openssl being the scariest one.
>
>  Good news is, RHEL-9 is gonna lead the way
>  and thus will take a lot of the hits first.
>  

Alexander,

Could this be enabled in ELN? This is not really question but 
suggestion. It is unfortunate, that ELN, while intermediate step for c9s 
does not have this enabled yet.

For example, we are working on Ruby 3.1 for c9s [1] just to figure that 
while everything just works in Fedora/ELN, it does not work in c9s. 
Maybe working with ELN would make the migrations easier also for Fedora.


Vít


[1] https://gitlab.com/redhat/centos-stream/rpms/ruby/-/merge_requests/10


>  Fedora doesn't have to pioneer it.
>  Bad news is, Fedora has to follow suit someday anyway,
>  and this brings me to how does one land such a change.
>
>  ---
>
>  Fedora is a large distribution with short release cycles, and
>  the only realistic way to weed out its reliance on SHA-1 signatures
>  from all of its numerous dark corners is to break them.
>  Make creation and verification fail in default configuration.
>  But it's unreasonable to just wait for, say, Fedora 37 branch-off
>  and break it in Rawhide for Fedora 38.
>  The fallout will just be too big.
>
>  Maintainers need time to get bugs, look into them, think,
>  analyze, react and test --- and that's just if it fails correctly!
>  Unfortunately, it's not just that the error paths are as dusty as they get
>  because the program counter has never set foot on them before.
>  Some maintainers might even find that
>  picking a different hash function renders their code non-interoperable,
>  or even that protocols they implement have SHA-1 hardcoded in the spec.
>  Or that everything is ready, but real world deployments need another decade.
>  Or that on-disk formats are just hard to change and migrate.
>  Took git years to migrate from SHA-1, and some others haven't even started.
>  There are gonna be investigations, planning, exceptions, upstream changes,
>  opt-out mechanisms, arguing, compromises, waiting out, all kinds of things.
>  It's gonna be big. Too big for a single release cycle.
>
>  ---
>
>  But how does one land something and give the distribution
>  the extra cycles needed to react? That's not really clear to me.
>
>  An obvious thing is to announce it in one cycle and land in another one.
>  The downsides are well-documented
>  in "The Hitchhiker's Guide to the Galaxy":
>  announcements are one weak measure, and then it's too late.
>
>  A second scheme I can come up with is a "jump scare".
>  Break the functionality in Fedora 37 Rawhide,
>  make most of the affected people realize the depth of the problem,
>  then unbreak it. Break again for Fedora 38 and never fix.
>
>  This could also be extended into "let one stable release slide'.
>  Break in 37 Rawhide, unbreak on branched off 37,
>  but never in Rawhide.
>
>  But these are all rather... crude?
>  Sure there should be better ways,
>  preferably something explored before.
>  I'm all for pulling this tooth out smoothly,
>  but I need hints on how to do it.
>  I hope that together we can devise a better plan than these.
>
>  So, how does one land a change that's bigger than a release cycle?
>
>  [1] https://eprint.iacr.org/2020/014
>  _______________________________________________
>  devel mailing list -- devel(a)lists.fedoraproject.org
>  To unsubscribe send an email to devel-leave(a)lists.fedoraproject.org
>  Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
>  List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
>  List Archives:
> https://lists.fedoraproject.org/archives/list/[e-mail]
>  Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
>

[Alexander Sosedkin]

On Wed, Apr 6, 2022 at 1:00 PM Vít Ondruch <vondruch(a)redhat.com&gt; wrote:
>  Dne 08. 03. 22 v 19:40 Alexander Sosedkin napsal(a):
>  > Hello, community, I need your wisdom for planning a disruptive change.
>  >
>  > Fedora 28 had https://fedoraproject.org/wiki/Changes/StrongCryptoSettings
>  > Fedora 33 had https://fedoraproject.org/wiki/Changes/StrongCryptoSettings2
>  > I believe we should start planning
>  > for the next cryptographic defaults tightening.
>  > And next time it's gonna be even more disruptive because of SHA-1 (again).
>  >
>  > SHA-1 is a hash function from 1995,
>  > which collision resistance is no longer to be relied upon for security [1].
>  > At the same time, it's not like software has successfully migrated off it,
>  > not even close.
>  > It's not a question of "if" the world should migrate from it,
>  > sooner or later we must part ways with it.
>  > (Technically, some acute energy crisis or a collapse of civilization
>  >   forever raising the costs of computations thousandfold would also do,
>  >   but let's agree that migrating to a more modern hash is the way =)
>  >
>  > We've been disabling it in TLS, but its usage is much wider than TLS.
>  > The next agonizing step is to restrict its usage for signatures
>  > on the cryptographic libraries level, with openssl being the scariest one.
>  >
>  > Good news is, RHEL-9 is gonna lead the way
>  > and thus will take a lot of the hits first.
>
>
>  Alexander,
>
>  Could this be enabled in ELN? This is not really question but
>  suggestion. It is unfortunate, that ELN, while intermediate step for c9s
>  does not have this enabled yet.
>
>  For example, we are working on Ruby 3.1 for c9s [1] just to figure that
>  while everything just works in Fedora/ELN, it does not work in c9s.
>  Maybe working with ELN would make the migrations easier also for Fedora.
>
>
>  Vít 
It's on my plans, but it'd take some time as it's not trivial.

>  [1]
> https://gitlab.com/redhat/centos-stream/rpms/ruby/-/merge_requests/10
>
>
>  > Fedora doesn't have to pioneer it.
>  > Bad news is, Fedora has to follow suit someday anyway,
>  > and this brings me to how does one land such a change.
>  >
>  > ---
>  >
>  > Fedora is a large distribution with short release cycles, and
>  > the only realistic way to weed out its reliance on SHA-1 signatures
>  > from all of its numerous dark corners is to break them.
>  > Make creation and verification fail in default configuration.
>  > But it's unreasonable to just wait for, say, Fedora 37 branch-off
>  > and break it in Rawhide for Fedora 38.
>  > The fallout will just be too big.
>  >
>  > Maintainers need time to get bugs, look into them, think,
>  > analyze, react and test --- and that's just if it fails correctly!
>  > Unfortunately, it's not just that the error paths are as dusty as they get
>  > because the program counter has never set foot on them before.
>  > Some maintainers might even find that
>  > picking a different hash function renders their code non-interoperable,
>  > or even that protocols they implement have SHA-1 hardcoded in the spec.
>  > Or that everything is ready, but real world deployments need another decade.
>  > Or that on-disk formats are just hard to change and migrate.
>  > Took git years to migrate from SHA-1, and some others haven't even started.
>  > There are gonna be investigations, planning, exceptions, upstream changes,
>  > opt-out mechanisms, arguing, compromises, waiting out, all kinds of things.
>  > It's gonna be big. Too big for a single release cycle.
>  >
>  > ---
>  >
>  > But how does one land something and give the distribution
>  > the extra cycles needed to react? That's not really clear to me.
>  >
>  > An obvious thing is to announce it in one cycle and land in another one.
>  > The downsides are well-documented
>  > in "The Hitchhiker's Guide to the Galaxy":
>  > announcements are one weak measure, and then it's too late.
>  >
>  > A second scheme I can come up with is a "jump scare".
>  > Break the functionality in Fedora 37 Rawhide,
>  > make most of the affected people realize the depth of the problem,
>  > then unbreak it. Break again for Fedora 38 and never fix.
>  >
>  > This could also be extended into "let one stable release slide'.
>  > Break in 37 Rawhide, unbreak on branched off 37,
>  > but never in Rawhide.
>  >
>  > But these are all rather... crude?
>  > Sure there should be better ways,
>  > preferably something explored before.
>  > I'm all for pulling this tooth out smoothly,
>  > but I need hints on how to do it.
>  > I hope that together we can devise a better plan than these.
>  >
>  > So, how does one land a change that's bigger than a release cycle?
>  >
>  > [1] https://eprint.iacr.org/2020/014
>  > _______________________________________________
>  > devel mailing list -- devel(a)lists.fedoraproject.org
>  > To unsubscribe send an email to devel-leave(a)lists.fedoraproject.org
>  > Fedora Code of Conduct:
> https://docs.fedoraproject.org/en-US/project/code-of-conduct/
>  > List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
>  > List Archives:
> https://lists.fedoraproject.org/archives/list/[e-mail]
>  > Do not reply to spam on the list, report it:
> https://pagure.io/fedora-infrastructure
>  _______________________________________________
>  devel mailing list -- devel(a)lists.fedoraproject.org
>  To unsubscribe send an email to devel-leave(a)lists.fedoraproject.org
>  Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
>  List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
>  List Archives:
> https://lists.fedoraproject.org/archives/list/[e-mail]
>  Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure

[Stephen Gallagher]

On Wed, Apr 6, 2022 at 7:09 AM Alexander Sosedkin <asosedkin(a)redhat.com&gt; wrote:
>  On Wed, Apr 6, 2022 at 1:00 PM Vít Ondruch <vondruch(a)redhat.com&gt; wrote: 
...
>  > Alexander,
>  >
>  > Could this be enabled in ELN? This is not really question but
>  > suggestion. It is unfortunate, that ELN, while intermediate step for c9s
>  > does not have this enabled yet.
>  >
>  > For example, we are working on Ruby 3.1 for c9s [1] just to figure that
>  > while everything just works in Fedora/ELN, it does not work in c9s.
>  > Maybe working with ELN would make the migrations easier also for Fedora.
>  >
>  >
>  > Vít
>
>  It's on my plans, but it'd take some time as it's not trivial.
>  
Alexander, I can try to help you with this if you need it. Just let me know.

[Stephen Gallagher]

On Wed, Apr 6, 2022 at 10:43 AM Stephen Gallagher <sgallagh(a)redhat.com&gt; wrote:
>  On Wed, Apr 6, 2022 at 7:09 AM Alexander Sosedkin <asosedkin(a)redhat.com&gt; wrote:
>  >
>  > On Wed, Apr 6, 2022 at 1:00 PM Vít Ondruch <vondruch(a)redhat.com&gt; wrote:
>  ...
>  > > Alexander,
>  > >
>  > > Could this be enabled in ELN? This is not really question but
>  > > suggestion. It is unfortunate, that ELN, while intermediate step for c9s
>  > > does not have this enabled yet.
>  > >
>  > > For example, we are working on Ruby 3.1 for c9s [1] just to figure that
>  > > while everything just works in Fedora/ELN, it does not work in c9s.
>  > > Maybe working with ELN would make the migrations easier also for Fedora.
>  > >
>  > >
>  > > Vít
>  >
>  > It's on my plans, but it'd take some time as it's not trivial.
>  >
>
>  Alexander, I can try to help you with this if you need it. Just let me know. 
I put together a potential solution for testing this in ELN and
submitted an MR:
https://src.fedoraproject.org/rpms/crypto-policies/pull-request/10

It's a little bit heavy-handed of an approach, but it should work.

[Vít Ondruch]

Dne 06. 04. 22 v 22:29 Stephen Gallagher napsal(a):
>  On Wed, Apr 6, 2022 at 10:43 AM Stephen Gallagher
> <sgallagh(a)redhat.com&gt; wrote:
> > On Wed, Apr 6, 2022 at 7:09 AM Alexander Sosedkin <asosedkin(a)redhat.com&gt;
> wrote:
> >> On Wed, Apr 6, 2022 at 1:00 PM Vít Ondruch <vondruch(a)redhat.com&gt; wrote:
> > ...
> >>> Alexander,
> >>>
> >>> Could this be enabled in ELN? This is not really question but
> >>> suggestion. It is unfortunate, that ELN, while intermediate step for c9s
> >>> does not have this enabled yet.
> >>>
> >>> For example, we are working on Ruby 3.1 for c9s [1] just to figure that
> >>> while everything just works in Fedora/ELN, it does not work in c9s.
> >>> Maybe working with ELN would make the migrations easier also for Fedora.
> >>>
> >>>
> >>> Vít
> >> It's on my plans, but it'd take some time as it's not trivial.
> >>
> > Alexander, I can try to help you with this if you need it. Just let me know.
>  I put together a potential solution for testing this in ELN and
>  submitted an MR:
>  https://src.fedoraproject.org/rpms/crypto-policies/pull-request/10
>  

Nice!


>  It's a little bit heavy-handed of an approach, but it should work.
>  

Sorry for keeping asking naive questions, but is there a chance to do 
some mass rebuild prior this lands, to asses the impact? Better to avoid 
(complete?) breakage of ELN and give change to fix the (most 
outstanding) issues prior it lands.

Thx


Vít


>  _______________________________________________
>  devel mailing list -- devel(a)lists.fedoraproject.org
>  To unsubscribe send an email to devel-leave(a)lists.fedoraproject.org
>  Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
>  List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
>  List Archives:
> https://lists.fedoraproject.org/archives/list/[e-mail]
>  Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
>

[Stephen Gallagher]

On Thu, Apr 7, 2022 at 4:53 AM Vít Ondruch <vondruch(a)redhat.com&gt; wrote:
>  Dne 06. 04. 22 v 22:29 Stephen Gallagher napsal(a):
>  > On Wed, Apr 6, 2022 at 10:43 AM Stephen Gallagher <sgallagh(a)redhat.com&gt;
> wrote:
>  >> On Wed, Apr 6, 2022 at 7:09 AM Alexander Sosedkin <asosedkin(a)redhat.com&gt;
> wrote:
>  >>> On Wed, Apr 6, 2022 at 1:00 PM Vít Ondruch <vondruch(a)redhat.com&gt;
> wrote:
>  >> ...
>  >>>> Alexander,
>  >>>>
>  >>>> Could this be enabled in ELN? This is not really question but
>  >>>> suggestion. It is unfortunate, that ELN, while intermediate step for
> c9s
>  >>>> does not have this enabled yet.
>  >>>>
>  >>>> For example, we are working on Ruby 3.1 for c9s [1] just to figure that
>  >>>> while everything just works in Fedora/ELN, it does not work in c9s.
>  >>>> Maybe working with ELN would make the migrations easier also for
> Fedora.
>  >>>>
>  >>>>
>  >>>> Vít
>  >>> It's on my plans, but it'd take some time as it's not trivial.
>  >>>
>  >> Alexander, I can try to help you with this if you need it. Just let me know.
>  > I put together a potential solution for testing this in ELN and
>  > submitted an MR:
>  > https://src.fedoraproject.org/rpms/crypto-policies/pull-request/10
>
>
>  Nice!
>
>
>  >
>  > It's a little bit heavy-handed of an approach, but it should work.
>
>
>  Sorry for keeping asking naive questions, but is there a chance to do
>  some mass rebuild prior this lands, to asses the impact? Better to avoid
>  (complete?) breakage of ELN and give change to fix the (most
>  outstanding) issues prior it lands.
>  
I'm not sure what benefit that would have. The crypto policies are
generally applied at runtime, not build-time.

[Kamil Dudka]

On Thursday, April 7, 2022 3:45:45 PM CEST Stephen Gallagher wrote:
>  > Sorry for keeping asking naive questions, but is there a chance
> to do
>  > some mass rebuild prior this lands, to asses the impact? Better to avoid
>  > (complete?) breakage of ELN and give change to fix the (most
>  > outstanding) issues prior it lands.
>  
>  
>  I'm not sure what benefit that would have. The crypto policies are
>  generally applied at runtime, not build-time. 
The policies applied at run time can break builds though, especially when
they include regression tests that use crypto.

Kamil

[Stephen Gallagher]

On Thu, Apr 7, 2022 at 10:12 AM Kamil Dudka <kdudka(a)redhat.com&gt; wrote:
>  On Thursday, April 7, 2022 3:45:45 PM CEST Stephen Gallagher wrote:
>  > > Sorry for keeping asking naive questions, but is there a chance to do
>  > > some mass rebuild prior this lands, to asses the impact? Better to avoid
>  > > (complete?) breakage of ELN and give change to fix the (most
>  > > outstanding) issues prior it lands.
>  >
>  >
>  > I'm not sure what benefit that would have. The crypto policies are
>  > generally applied at runtime, not build-time.
>
>  The policies applied at run time can break builds though, especially when
>  they include regression tests that use crypto. 
Ah, that hadn't occurred to me. I'm still not sure it's worth running
a mass-rebuild specifically for this, though. I think it might be
better to clean things up as they come up and then piggy-back on the
next Fedora mass-rebuild.

[Clemens Lang]

Hi Stephen,

Stephen Gallagher <sgallagh(a)redhat.com&gt; wrote:

>  On Wed, Apr 6, 2022 at 10:43 AM Stephen Gallagher
> <sgallagh(a)redhat.com&gt;  
>  wrote:
>
>  I put together a potential solution for testing this in ELN and
>  submitted an MR:
>  https://src.fedoraproject.org/rpms/crypto-policies/pull-request/10
>
>  It's a little bit heavy-handed of an approach, but it should work. 
Please keep in mind that Fedora’s openssl package currently does not have
the patches that introduce the rh-allow-sha1-signatures option that would be
required to do this. Merging your pull request without this change will
cause all binaries that use OpenSSL 3.x (1.1 will ignore it) to fail to
start.

See https://bugzilla.redhat.com/show_bug.cgi?id=2070977 which tracks
implementing this. If you want to disable SHA-1 by default in ELN, we should
probably also have an option to re-enable it using an environment variable
for package build and test execution purposes. I’ll start working on this
now.


-- 
Clemens Lang
RHEL Crypto Team
Red Hat

[Zbigniew Jędrzejewski-Szmek]

On Tue, Mar 29, 2022 at 03:34:49PM -0700, Kevin Fenzi wrote:
>  On Tue, Mar 29, 2022 at 08:12:47PM +0200, Alexander Sosedkin wrote:
>  > 
>  > "You know these lights in the theaters that go out gradually?
>  >  When the guy ve-ery slo-o-owly pulls the plug out?"
>  >  - a joke from my childhood.
>  > 
>  > 
>  > Hello, it's been quiet for a while, and I've been busy
>  > but kept thinking about all the useful feedback you folks gave me.
>  > Not that it made me flesh out a perfect plan,
>  > but hopefully at least a less terrible one.
>  > 
>  > Regarding smudging the change in time,
>  > how does the following three-phaser sound?
>  
>  Might work. :) A few comments inline...
>  
>  > Phase 1 ("Wake-up Call"):
>  >   In Fedora 37, disable SHA-1 signatures verification/creation
>  >   in FUTURE policy, i.e. opt-in only.
>  >   Come up with some logging solution;
>  >   I'd prefer something non-invasive like eBPF USDT probes [2],
>  >   but maybe even stderr could work, you've been moderately convincing.
>  >   (FUTURE change is *maybe* doable in F36, but not logging.)
>  
>  Well, we just shipped beta today, so I think it's too late to land any
>  f36 changes at this point. 
>  
>  >   Announce it as a system-wide change anyway for visibility,
>  >   call for Test Days to report which apps/workflows rely on SHA-1 signatures
>  >   either from the logs
>  >   or from opting into blocking operations and seeing what starts failing hard.
>  >   That'd have to be very actively called for to make an impact,
>  >   impact that'd mostly be just maintainers thinking what will they do in
>  
>  Just a related note here, FUTURE also breaks installing anything via dnf
>  with the default metalink setup. This is because digicert (where we get
>  our *.fedoraproject.org wildcard cert) seems to always issue certs from
>  it's 2048bit CA. :( (If anyone knows how to get them to issue from a
>  newer CA that works please let me know) 
This means that probably almost nobody uses FUTURE :(

On Tue, Mar 29, 2022 at 08:05:00PM -0500, Michael Catanzaro wrote:
>  On Tue, Mar 29 2022 at 03:34:49 PM -0700, Kevin Fenzi
> <kevin(a)scrye.com&gt; wrote:
>  > Well, we just shipped beta today, so I think it's too late to land any
>  > f36 changes at this point.
>
>  This is a non-default configuration that I strongly suspect nobody or almost
>  nobody uses, except for testing. Changing this prior to F36 stable release
>  seems like no big deal. 
Considering that nobody uses FUTURE, I agree that we can do it for F36.
This effectively moves everything one release forward, which is probably
good. I don't think we want to drag this out *too* much.

>  > Phase 2 ("Jump Scare"):
>  >   As soon as f37 branch-off happens,
>  >   disable signature verification in DEFAULT in *38 rawhide*.
>  >   Cue an influx of bugreports because things get broken for all testers
>  >   and not just the ones who opt in.
>  >   I anticipate this to be the most eye-opening step
>  >   even if we test a lot in the previous phase, so to smooth it out more
>  >   we then *revert* the change in 38 before the release,
>  >   so the released Fedora behaves just like in 37
>  >   and whatever wasn't sorted out in time gets an extra cycle.
>  
>  Right before the release? 
>  Or right before Beta? 
>  or ?
>  
>  People kind of expect the beta will be something they can test and will
>  behave as the final, so changing things after beta seems like a bad idea
>  in general. That said, shipping beta with it would get a lot more
>  exposure. 
Agreed. Any user-visible changes should be done before Beta.

>  >   A second Fedora change should be filed for visibility,
>  >   but clearly stating this will not affect f38 released.
>  > 
>  > Phase 3 ("Return of the Panik"):
>  >   And then Fedora 39 comes, where the revert hasn't happened,
>  >   goes through the whole release cycle,
>  >   but this time the change goes through and reaches stable.
>  >   Again, a system-wide change, a third one for the same thing.
>  > 
>  > With the 37-38-39 numbers, that'd mean the change
>  > reaching the users in autumn 2023, with lead times of:
>  >   ~ 3.5 cycles for the most proactive developers to see this thread and panic
>  >   ~ 3 cycles for the testers to proactively report bugs (logging/opting in)
>  >   ~ 2 cycles to address everything else coming from rawhide testing
>  >     before it reaches stable by either
>  >     switching to some other algorithm,
>  >     making the users explicitly opt into trusting SHA-1 signatures somehow,
>  >     or, in the most high-profile cases,
>  >     having a widely publicised exception (and some plan for the future).
>  > 
>  > Questions:
>  >   * Do you find this smudging reasonable?
>  
>  I think it's probibly the best we can do. 
>  
>  >   * The usual tightening of the other less controversial algorithms,
>  >     should it follow the same smudging/reverting plan
>  >     since we're going through all that hassle anyway?
>  
>  I don't think they would need to have this long a runway.
>  
>  >   * Does the 37-38-39 timeframe feel right?
>  >   * Do I need to first run this contraption of a plan
>  >     by FESCo or some other smart folks?
>  
>  Well, I think run it by everyone on this list. 
>  I don't think people will hold back. ;) 
>  
>  >   * Is there a better signalling mechanism than filing 3 system-wide changes?
>  >   * What'd be the right mechanism for others to take the wheel if everything
>  >     goes sideways and the need arises to revise the plan mid-execution? 
Please just file a late change for F36. We can then manually link the
same page into F37 and F38 (or even F39 if appropriate).

>  I wonder if it would make sense to have a checkpoint before the
> revert
>  in f38, and if things look substantually less broken than we fear, we
>  just don't revert it and let it go out in f38. But that might muddy up
>  the communications.
>  
>  > Other kinds of input are, of course, also appreciated.
>  > Even the calls to magically attain the mutually exclusive goals
>  > of offering secure defaults while not breaking insecure workflows
>  > that don't offer actual solutions might serve as a useful mood check.
>  > I know it ain't the best plan. Let's figure out the right thing to do.
>  
>  Thanks for taking input on this and being willing to work out the best
>  way forward.  
Seconded. Thank you.

Zbyszek

[Alexander Sosedkin]

On Thu, Apr 7, 2022 at 9:06 PM Zbigniew Jędrzejewski-Szmek
<zbyszek(a)in.waw.pl&gt; wrote:
>  On Tue, Mar 29, 2022 at 03:34:49PM -0700, Kevin Fenzi wrote:
>  > On Tue, Mar 29, 2022 at 08:12:47PM +0200, Alexander Sosedkin wrote:
>  > >
>  > > "You know these lights in the theaters that go out gradually?
>  > >  When the guy ve-ery slo-o-owly pulls the plug out?"
>  > >  - a joke from my childhood.
>  > >
>  > >
>  > > Hello, it's been quiet for a while, and I've been busy
>  > > but kept thinking about all the useful feedback you folks gave me.
>  > > Not that it made me flesh out a perfect plan,
>  > > but hopefully at least a less terrible one.
>  > >
>  > > Regarding smudging the change in time,
>  > > how does the following three-phaser sound?
>  >
>  > Might work. :) A few comments inline...
>  >
>  > > Phase 1 ("Wake-up Call"):
>  > >   In Fedora 37, disable SHA-1 signatures verification/creation
>  > >   in FUTURE policy, i.e. opt-in only.
>  > >   Come up with some logging solution;
>  > >   I'd prefer something non-invasive like eBPF USDT probes [2],
>  > >   but maybe even stderr could work, you've been moderately convincing.
>  > >   (FUTURE change is *maybe* doable in F36, but not logging.)
>  >
>  > Well, we just shipped beta today, so I think it's too late to land any
>  > f36 changes at this point.
>  >
>  > >   Announce it as a system-wide change anyway for visibility,
>  > >   call for Test Days to report which apps/workflows rely on SHA-1 signatures
>  > >   either from the logs
>  > >   or from opting into blocking operations and seeing what starts failing hard.
>  > >   That'd have to be very actively called for to make an impact,
>  > >   impact that'd mostly be just maintainers thinking what will they do in
>  >
>  > Just a related note here, FUTURE also breaks installing anything via dnf
>  > with the default metalink setup. This is because digicert (where we get
>  > our *.fedoraproject.org wildcard cert) seems to always issue certs from
>  > it's 2048bit CA. :( (If anyone knows how to get them to issue from a
>  > newer CA that works please let me know)
>
>  This means that probably almost nobody uses FUTURE :(
>
>  On Tue, Mar 29, 2022 at 08:05:00PM -0500, Michael Catanzaro wrote:
>  > On Tue, Mar 29 2022 at 03:34:49 PM -0700, Kevin Fenzi <kevin(a)scrye.com&gt;
> wrote:
>  > > Well, we just shipped beta today, so I think it's too late to land any
>  > > f36 changes at this point.
>  >
>  > This is a non-default configuration that I strongly suspect nobody or almost
>  > nobody uses, except for testing. Changing this prior to F36 stable release
>  > seems like no big deal.
>
>  Considering that nobody uses FUTURE, I agree that we can do it for F36.
>  This effectively moves everything one release forward, which is probably
>  good. 
I'm afraid not, because there's also a logging phase proposed,
one that shouldn't go to F36 and keeps the plan from shifting forward. Meaning:
F36: FUTURE update
F37: FUTURE update + logging
F38: DEFAULT update in rawhide only
F39: DEFAULT update that reaches release
hasn't really shortened.

>  I don't think we want to drag this out *too* much. 
And neither do we want to rush it.
The more ahead of the world we would be, the more work it entails for us,
so it's a balancing act.

>  > > Phase 2 ("Jump Scare"):
>  > >   As soon as f37 branch-off happens,
>  > >   disable signature verification in DEFAULT in *38 rawhide*.
>  > >   Cue an influx of bugreports because things get broken for all testers
>  > >   and not just the ones who opt in.
>  > >   I anticipate this to be the most eye-opening step
>  > >   even if we test a lot in the previous phase, so to smooth it out more
>  > >   we then *revert* the change in 38 before the release,
>  > >   so the released Fedora behaves just like in 37
>  > >   and whatever wasn't sorted out in time gets an extra cycle.
>  >
>  > Right before the release?
>  > Or right before Beta?
>  > or ?
>  >
>  > People kind of expect the beta will be something they can test and will
>  > behave as the final, so changing things after beta seems like a bad idea
>  > in general. That said, shipping beta with it would get a lot more
>  > exposure.
>
>  Agreed. Any user-visible changes should be done before Beta.
>
>  > >   A second Fedora change should be filed for visibility,
>  > >   but clearly stating this will not affect f38 released.
>  > >
>  > > Phase 3 ("Return of the Panik"):
>  > >   And then Fedora 39 comes, where the revert hasn't happened,
>  > >   goes through the whole release cycle,
>  > >   but this time the change goes through and reaches stable.
>  > >   Again, a system-wide change, a third one for the same thing.
>  > >
>  > > With the 37-38-39 numbers, that'd mean the change
>  > > reaching the users in autumn 2023, with lead times of:
>  > >   ~ 3.5 cycles for the most proactive developers to see this thread and panic
>  > >   ~ 3 cycles for the testers to proactively report bugs (logging/opting in)
>  > >   ~ 2 cycles to address everything else coming from rawhide testing
>  > >     before it reaches stable by either
>  > >     switching to some other algorithm,
>  > >     making the users explicitly opt into trusting SHA-1 signatures somehow,
>  > >     or, in the most high-profile cases,
>  > >     having a widely publicised exception (and some plan for the future).
>  > >
>  > > Questions:
>  > >   * Do you find this smudging reasonable?
>  >
>  > I think it's probibly the best we can do.
>  >
>  > >   * The usual tightening of the other less controversial algorithms,
>  > >     should it follow the same smudging/reverting plan
>  > >     since we're going through all that hassle anyway?
>  >
>  > I don't think they would need to have this long a runway.
>  >
>  > >   * Does the 37-38-39 timeframe feel right?
>  > >   * Do I need to first run this contraption of a plan
>  > >     by FESCo or some other smart folks?
>  >
>  > Well, I think run it by everyone on this list.
>  > I don't think people will hold back. ;)
>  >
>  > >   * Is there a better signalling mechanism than filing 3 system-wide changes?
>  > >   * What'd be the right mechanism for others to take the wheel if
> everything
>  > >     goes sideways and the need arises to revise the plan mid-execution?
>
>  Please just file a late change for F36. We can then manually link the
>  same page into F37 and F38 (or even F39 if appropriate). 
Noted, 4 system-wide changes =)

>  > I wonder if it would make sense to have a checkpoint before the
> revert
>  > in f38, and if things look substantually less broken than we fear, we
>  > just don't revert it and let it go out in f38. But that might muddy up
>  > the communications.
>  >
>  > > Other kinds of input are, of course, also appreciated.
>  > > Even the calls to magically attain the mutually exclusive goals
>  > > of offering secure defaults while not breaking insecure workflows
>  > > that don't offer actual solutions might serve as a useful mood check.
>  > > I know it ain't the best plan. Let's figure out the right thing to do.
>  >
>  > Thanks for taking input on this and being willing to work out the best
>  > way forward.
>
>  Seconded. Thank you.
>
>  Zbyszek

[Vít Ondruch]

Dne 07. 04. 22 v 16:13 Stephen Gallagher napsal(a):
>  On Thu, Apr 7, 2022 at 10:12 AM Kamil Dudka <kdudka(a)redhat.com&gt;
> wrote:
> > On Thursday, April 7, 2022 3:45:45 PM CEST Stephen Gallagher wrote:
> >>> Sorry for keeping asking naive questions, but is there a chance to do
> >>> some mass rebuild prior this lands, to asses the impact? Better to avoid
> >>> (complete?) breakage of ELN and give change to fix the (most
> >>> outstanding) issues prior it lands.
> >>
> >> I'm not sure what benefit that would have. The crypto policies are
> >> generally applied at runtime, not build-time.
> > The policies applied at run time can break builds though, especially when
> > they include regression tests that use crypto.
>  

Yep, that is precisely the case. Actually it is higher chance that the 
testsuites are problem then the runtime, because the test suites trying 
to assert every code path when during runtime likely just subset of 
functionality is used (not mentioning that test suites tends to use 
weaker cryptography, because it is faster and it won't cause security 
concerns).


>  Ah, that hadn't occurred to me. I'm still not sure it's
> worth running
>  a mass-rebuild specifically for this, though. I think it might be
>  better to clean things up as they come up and then piggy-back on the
>  next Fedora mass-rebuild.
>  

At minimum, you count with broken Ruby :)


Vít

[Alexander Sosedkin]

Another status update for transparency purposes:

1. openssl-3.0.2-3 and crypto-policies-20220412-1.git97fe449
   now distrust SHA-1 signatures in FUTURE policy or NO-SHA1 subpolicy.
   Meaning that updating the packages and issuing
   `update-crypto-policies --set FUTURE` /
   `update-crypto-policies --set DEFAULT:NO-SHA1`
   can be used to preview the impact on Fedora 36 / 37.

2. A decision has been made for Fedora ELN to track CentOS Stream 9
crypto-policies.
   A side-tag rebuild has been triggered prior to the switch,
   and has found ELN to be in a pretty broken shape in general.
   Work has been temporarily stalled on that side, but I hope to get back to it.

3. I've drafted the following wiki pages so far:
   https://fedoraproject.org/wiki/Changes/StrongCryptoSettings3Forewarning1
   https://fedoraproject.org/wiki/SHA1SignaturesGuidance
   https://fedoraproject.org/wiki/WeakCryptographyException
   feedback is welcome as usual.

[Alexander Sosedkin]

Another status update:

1. USDT/eBPF tracing turned out to be a fruitful logging approach.
   Clemens Lang has kindly added USDT probes to the latest openssl builds,
   traceable with a small tool [1] available from a copr [2].
   Yes, this way it doesn't log into your face unpromptedly,
   like stderr logging would, but, in turn, it's so non-invasive
   that we're comfortable with bringing it to Fedora 36 as well.
2. crypto-policies has added FEDORA38 and TEST-FEDORA39 policies
   for not following the change and testing it early correspondingly.
3. change wiki pages have been drafted for all 4 proposed rollout phases,
   the Fedora 36 one has been marked as ChangeReadyForWrangler.

What I'd like some community help with beyond regular guidance:
1. reviews for the relevant wiki pages:
   https://fedoraproject.org/wiki/Changes/StrongCryptoSettings3Forewarning1
   https://fedoraproject.org/wiki/Changes/StrongCryptoSettings3Forewarning2
   https://fedoraproject.org/wiki/Changes/StrongCryptoSettings3Forewarning3
   https://fedoraproject.org/wiki/Changes/StrongCryptoSettings3
   https://fedoraproject.org/wiki/SHA1SignaturesGuidance
   https://fedoraproject.org/wiki/WeakCryptographyException
2. proposing the testing activities outlined there for Test Days.
   how is it done?

[1] https://gitlab.com/redhat-crypto/fedora-sha1sig-tracer
[2] https://copr.fedorainfracloud.org/coprs/asosedkin/sha1sig-tracer