Fedora Copr builders updated to Fedora 38

[Pavel Raiskup]

Hello maintainers!

Copr builders have been updated to Fedora 38 today (some old builders
might still be running F37 ATM, but when they finish the task(s) they
work on, they will be deleted). Our testsuite is passing just fine, so
you _should_ be fine too :-).  Please let us know if you have some
troubles.

There was one important change in Fedora 38 - RPM switched to the
Sequoia crypto backend.  It refuses SHA-1 in crypto;  which basically
disallows Mock to properly check EL6 GPG signatures.  To allow further
builds, we switched to gpgcheck=0 for all epel-6 chroots.  If you know a
better work-around, let me know.

Pavel

[Ondřej Budai]

RPM Sequoia's crypto policies can be configured, so you should be able to
re-enable SHA-1. However, this would be a global change, not only for
EL6... See
https://docs.rs/sequoia-policy-config/latest/sequoia_policy_config/#hash-fun...

Cheers,
Ondřej

On Thu, Jun 8, 2023 at 5:42 PM Pavel Raiskup  wrote:
> Hello maintainers!
>
> Copr builders have been updated to Fedora 38 today (some old builders
> might still be running F37 ATM, but when they finish the task(s) they
> work on, they will be deleted). Our testsuite is passing just fine, so
> you _should_ be fine too :-).  Please let us know if you have some
> troubles.
>
> There was one important change in Fedora 38 - RPM switched to the
> Sequoia crypto backend.  It refuses SHA-1 in crypto;  which basically
> disallows Mock to properly check EL6 GPG signatures.  To allow further
> builds, we switched to gpgcheck=0 for all epel-6 chroots.  If you know a
> better work-around, let me know.
>
> Pavel
>
> devel mailing list -- [e-mail]
> To unsubscribe send an email to [e-mail]
> Fedora Code of Conduct:
> https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives:
> https://lists.fedoraproject.org/archives/list/[e-mail]
> Do not reply to spam, report it:
> https://pagure.io/fedora-infrastructure/new_issue

[Pavel Raiskup]

On čtvrtek 8. června 2023 21:37:09 CEST Ondřej Budai wrote:
> RPM Sequoia's crypto policies can be configured, so you should be able to
> re-enable SHA-1. However, this would be a global change, not only for
> EL6... See
> https://docs.rs/sequoia-policy-config/latest/sequoia_policy_config/#hash-fun...
>
> Cheers,
> Ondřej

Thank you very much for the reference, Odřej!

I'm not strongly against anything; but rather than weaker policy for
everything I slightly prefer keeping the _stricter default policy_ with
_disabled gpgcheck for EL6_ (we should phase epel-6 out entirely anyway
since it's long time EOL, but we still keep it for the distro upgrade
team(s)).  This is up to the community to decide, let us know in our
issue tracker if you are concerned.

Pavel
> On Thu, Jun 8, 2023 at 5:42 PM Pavel Raiskup  wrote:
> > Hello maintainers!
> >
> > Copr builders have been updated to Fedora 38 today (some old builders
> > might still be running F37 ATM, but when they finish the task(s) they
> > work on, they will be deleted). Our testsuite is passing just fine, so
> > you _should_ be fine too :-).  Please let us know if you have some
> > troubles.
> >
> > There was one important change in Fedora 38 - RPM switched to the
> > Sequoia crypto backend.  It refuses SHA-1 in crypto;  which basically
> > disallows Mock to properly check EL6 GPG signatures.  To allow further
> > builds, we switched to gpgcheck=0 for all epel-6 chroots.  If you know a
> > better work-around, let me know.
> >
> > Pavel
> >
> > devel mailing list -- [e-mail]
> > To unsubscribe send an email to [e-mail]
> > Fedora Code of Conduct:
> > https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> > List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> > List Archives:
> > https://lists.fedoraproject.org/archives/list/[e-mail]
> > Do not reply to spam, report it:
> > https://pagure.io/fedora-infrastructure/new_issue

[Neal H. Walfield]

On Thu, 08 Jun 2023 21:37:09 +0200,
Ondřej Budai wrote:
> RPM Sequoia's crypto policies can be configured, so you should be able to re-enable SHA-1. However, this would
> be a global change, not only for EL6... See
> https://docs.rs/sequoia-policy-config/latest/sequoia_policy_config/#hash-fun...
> ...
> On Thu, Jun 8, 2023 at 5:42 PM Pavel Raiskup  wrote:
>
> Hello maintainers!
>
> Copr builders have been updated to Fedora 38 today (some old builders
>  might still be running F37 ATM, but when they finish the task(s) they
>  work on, they will be deleted). Our testsuite is passing just fine, so
>  you _should_ be fine too :-).  Please let us know if you have some
>  troubles.
>
> There was one important change in Fedora 38 - RPM switched to the
>  Sequoia crypto backend.  It refuses SHA-1 in crypto;  which basically
>  disallows Mock to properly check EL6 GPG signatures.  To allow further
>  builds, we switched to gpgcheck=0 for all epel-6 chroots.  If you know a
>  better work-around, let me know.

I find this behavior surprising.  The default policy as set by
fedora-crypto-policies is for rpm-sequoia is to accept SHA-1 (and
DSA-1024, ...):

https://gitlab.com/redhat-crypto/fedora-crypto-policies/-/blob/master/polici...

What policy are you using?

Neal

[Pavel Raiskup]

On úterý 13. června 2023 16:57:42 CEST Neal H. Walfield wrote:
> On Thu, 08 Jun 2023 21:37:09 +0200,
> Ondřej Budai wrote:
> > RPM Sequoia's crypto policies can be configured, so you should be able to re-enable SHA-1. However, this would
> > be a global change, not only for EL6... See
> > https://docs.rs/sequoia-policy-config/latest/sequoia_policy_config/#hash-fun...
> > ...
> > On Thu, Jun 8, 2023 at 5:42 PM Pavel Raiskup  wrote:
> >
> > Hello maintainers!
> >
> > Copr builders have been updated to Fedora 38 today (some old builders
> >  might still be running F37 ATM, but when they finish the task(s) they
> >  work on, they will be deleted). Our testsuite is passing just fine, so
> >  you _should_ be fine too :-).  Please let us know if you have some
> >  troubles.
> >
> > There was one important change in Fedora 38 - RPM switched to the
> >  Sequoia crypto backend.  It refuses SHA-1 in crypto;  which basically
> >  disallows Mock to properly check EL6 GPG signatures.  To allow further
> >  builds, we switched to gpgcheck=0 for all epel-6 chroots.  If you know a
> >  better work-around, let me know.
>
> I find this behavior surprising.  The default policy as set by
> fedora-crypto-policies is for rpm-sequoia is to accept SHA-1 (and
> DSA-1024, ...):
>
> https://gitlab.com/redhat-crypto/fedora-crypto-policies/-/blob/master/polici...
>
> What policy are you using?

The `DEFAULT:SHA1`, but it is weird - I can not reproduce the build
failure now.  Is something changing in the backgrounds?

So lemme try to revert back to the nogpgcheck=1 variant.  Thank you for
the hint, Neal!

Pavel
> Neal
> _______________________________________________
> devel mailing list -- [e-mail]
> To unsubscribe send an email to [e-mail]
> Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives: https://lists.fedoraproject.org/archives/list/[e-mail]
> Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue

[Pavel Raiskup]

On úterý 13. června 2023 16:57:42 CEST Neal H. Walfield wrote:
> On Thu, 08 Jun 2023 21:37:09 +0200,
> Ondřej Budai wrote:
> > RPM Sequoia's crypto policies can be configured, so you should be able to re-enable SHA-1. However, this would
> > be a global change, not only for EL6... See
> > https://docs.rs/sequoia-policy-config/latest/sequoia_policy_config/#hash-fun...
> > ...
> > On Thu, Jun 8, 2023 at 5:42 PM Pavel Raiskup  wrote:
> >
> > Hello maintainers!
> >
> > Copr builders have been updated to Fedora 38 today (some old builders
> >  might still be running F37 ATM, but when they finish the task(s) they
> >  work on, they will be deleted). Our testsuite is passing just fine, so
> >  you _should_ be fine too :-).  Please let us know if you have some
> >  troubles.
> >
> > There was one important change in Fedora 38 - RPM switched to the
> >  Sequoia crypto backend.  It refuses SHA-1 in crypto;  which basically
> >  disallows Mock to properly check EL6 GPG signatures.  To allow further
> >  builds, we switched to gpgcheck=0 for all epel-6 chroots.  If you know a
> >  better work-around, let me know.
>
> I find this behavior surprising.  The default policy as set by
> fedora-crypto-policies is for rpm-sequoia is to accept SHA-1 (and
> DSA-1024, ...):
>
> https://gitlab.com/redhat-crypto/fedora-crypto-policies/-/blob/master/polici...
>
> What policy are you using?

I was wrong.  The problem was *not* with the EPEL-6 signatures, but with
CentOS 6 signatures.  It is a bit harder to analyse, as
`sq-keyring-linter` is silent for that one:

$ sq-keyring-linter < /usr/share/distribution-gpg-keys/centos/RPM-GPG-KEY-CentOS-6
    $ echo $?
    0

Pavel
> Neal

[Neal H. Walfield]

Hi Pavel,

On Wed, 14 Jun 2023 11:27:35 +0200,
Pavel Raiskup wrote:
> On úterý 13. června 2023 16:57:42 CEST Neal H. Walfield wrote:
> > On Thu, 08 Jun 2023 21:37:09 +0200,
> > Ondřej Budai wrote:
> > > RPM Sequoia's crypto policies can be configured, so you should be able to re-enable SHA-1. However, this would
> > > be a global change, not only for EL6... See
> > > https://docs.rs/sequoia-policy-config/latest/sequoia_policy_config/#hash-fun...
> > > ...
> > > On Thu, Jun 8, 2023 at 5:42 PM Pavel Raiskup  wrote:
> > >
> > > Hello maintainers!
> > >
> > > Copr builders have been updated to Fedora 38 today (some old builders
> > >  might still be running F37 ATM, but when they finish the task(s) they
> > >  work on, they will be deleted). Our testsuite is passing just fine, so
> > >  you _should_ be fine too :-).  Please let us know if you have some
> > >  troubles.
> > >
> > > There was one important change in Fedora 38 - RPM switched to the
> > >  Sequoia crypto backend.  It refuses SHA-1 in crypto;  which basically
> > >  disallows Mock to properly check EL6 GPG signatures.  To allow further
> > >  builds, we switched to gpgcheck=0 for all epel-6 chroots.  If you know a
> > >  better work-around, let me know.
> >
> > I find this behavior surprising.  The default policy as set by
> > fedora-crypto-policies is for rpm-sequoia is to accept SHA-1 (and
> > DSA-1024, ...):
> >
> > https://gitlab.com/redhat-crypto/fedora-crypto-policies/-/blob/master/polici...
> >
> > What policy are you using?
>
> I was wrong.  The problem was *not* with the EPEL-6 signatures, but with
> CentOS 6 signatures.  It is a bit harder to analyse, as
> `sq-keyring-linter` is silent for that one:
> $ sq-keyring-linter < /usr/share/distribution-gpg-keys/centos/RPM-GPG-KEY-CentOS-6
> $ echo $?
> 0

Thanks for investigating, I opened an issue on our issue tracker about
it here:

https://gitlab.com/sequoia-pgp/keyring-linter/-/issues/20

Using https://www.centos.org/keys/RPM-GPG-KEY-CentOS-6, it appears
that the CentOS 6 key expired in July 2021.  The linter checks if a
certificate is invalid under the standard policy, but valid under the
standard policy + SHA-1.  Since the certificate is expired, it's
considered invalid in both cases, and it concludes that the
certificate doesn't have any issues.  Using faketime to examine the
certificate when it wasn't expired, we see:

$ faketime 2021-01-01 sq-keyring-linter RPM-GPG-KEY-CentOS-6
  Certificate 0946FCA2C105B9DE is not valid under the standard policy: No binding signature at time 2020-12-31T23:00:00Z
  Certificate 0946FCA2C105B9DE contains a User ID ("CentOS-6 Key (CentOS 6 Official Signing Key) ") protected by SHA-1
  Examined 1 certificate.
    0 certificates are invalid and were not linted. (GOOD)
    1 certificate was linted.
    1 of the 1 certificates (100%) has at least one issue. (BAD)
  0 of the linted certificates were revoked.
    0 of the 0 certificates has revocation certificates that are weaker than the certificate and should be recreated. (GOOD)
  0 of the linted certificates were expired.
  1 of the non-revoked linted certificate has at least one non-revoked User ID:
    1 has at least one User ID protected by SHA-1. (BAD)
    1 has all User IDs protected by SHA-1. (BAD)
  0 of the non-revoked linted certificates have at least one non-revoked, live subkey:
    0 have at least one non-revoked, live subkey with a binding signature that uses SHA-1. (GOOD)
  0 of the non-revoked linted certificates have at least one non-revoked, live, signing-capable subkey:
    0 certificates have at least one non-revoked, live, signing-capable subkey with a strong binding signature, but a backsig that uses SHA-1. (GOOD)

Neal

[Pavel Raiskup]

On čtvrtek 8. června 2023 17:42:13 CEST Pavel Raiskup wrote:
> Hello maintainers!
>
> Copr builders have been updated to Fedora 38 today (some old builders
> might still be running F37 ATM, but when they finish the task(s) they
> work on, they will be deleted). Our testsuite is passing just fine, so
> you _should_ be fine too :-).  Please let us know if you have some
> troubles.
>
> There was one important change in Fedora 38 - RPM switched to the
> Sequoia crypto backend.  It refuses SHA-1 in crypto;  which basically
> disallows Mock to properly check EL6 GPG signatures.  To allow further
> builds, we switched to gpgcheck=0 for all epel-6 chroots.  If you know a
> better work-around, let me know.

As it turns out, epel-6 config was based on CentOS 6 and EPEL 6, and the
problem was with the CentOS 6 certificate (not EPEL 6).  With the 1-line
change [1], epel-6 chroots were changed to RHEL+EPEL instead of
CentOS+EPEL, so we can again use `gpgcheck=1` (even on F38).

[1] https://pagure.io/fedora-infra/ansible/c/b9c69f5cac314c18a539c42d1ca3bf28e59...

Pavel
> Pavel

[Pavel Raiskup]

On středa 14. června 2023 12:37:24 CEST Pavel Raiskup wrote:
> On čtvrtek 8. června 2023 17:42:13 CEST Pavel Raiskup wrote:
> > Hello maintainers!
> >
> > Copr builders have been updated to Fedora 38 today (some old builders
> > might still be running F37 ATM, but when they finish the task(s) they
> > work on, they will be deleted). Our testsuite is passing just fine, so
> > you _should_ be fine too :-).  Please let us know if you have some
> > troubles.
> >
> > There was one important change in Fedora 38 - RPM switched to the
> > Sequoia crypto backend.  It refuses SHA-1 in crypto;  which basically
> > disallows Mock to properly check EL6 GPG signatures.  To allow further
> > builds, we switched to gpgcheck=0 for all epel-6 chroots.  If you know a
> > better work-around, let me know.
>
> As it turns out, epel-6 config was based on CentOS 6 and EPEL 6, and the
> problem was with the CentOS 6 certificate (not EPEL 6).  With the 1-line
> change [1], epel-6 chroots were changed to RHEL+EPEL instead of
> CentOS+EPEL, so we can again use `gpgcheck=1` (even on F38).
>
> [1] https://pagure.io/fedora-infra/ansible/c/b9c69f5cac314c18a539c42d1ca3bf28e59...

FTR, with DNF5 things just work for some reason.  Which was especially
confusing to me (dnf5 is the default package manager on my workstation):

https://github.com/rpm-software-management/dnf5/issues/617

Seems like the policy is entirely ignored with dnf5?  I'm not sure, but
if anything - worth debugging.

BTW. without the original problem in hand, this would be spotted much
later.

Pavel
> Pavel
> > Pavel