FreeIPA letsencrypt certificate problems after recent expiration of DST Root CA X3

[Stefan Fleischmann]

Hi! I've been using FreeIPA (installed without CA --no-pkinit) with letsencrypt certificate. Whenever the certificate gets renewed I install it with 
ipa-server-certinstall for both the LDAP and web server and that has been working just fine. Recently the root certificate (DST Root CA X3)
expired as mentioned here https://letsencrypt.org/docs/dst-root-ca-x3-expiration-september-2021/

Now when I try to install the new certificate I get this error:
---
CA certificate CN=DST Root CA X3,O=Digital Signature Trust Co. in /etc/letsencrypt/live/XXX/cert.pem, /etc/letsencrypt/live/XXX/privkey.pem is not valid: certutil: certificate is invalid: The certificate issuer's certificate has expired. Check your system date and time.

The ipa-server-certinstall command failed.
---
I don't understand this error message at all since the `cert.pem` file does not contain any reference to the X3 CA, so I suppose it must come from somewhere else. Does someone have an idea how to fix this?

I've already removed the root certificate with ipa-cacert-manage and added the self-signed X1 root cert, yet the same error message above keeps showing up.

[Florence Renaud]

Hi,
Let's Encrypt new chain of trust is now the following:
- *ISRG Root X1*: *C = US, O = Internet Security Research Group, CN = ISRG
Root X1*
  (was previsouly also cross signed by *DST Root CA X3*: *O = Digital
Signature Trust Co., CN = DST Root CA X3*)
- *R3*: *C = US, O = Let's Encrypt, CN = R3*
- your server cert

You mention that the cross-signed root certificate was removed with
ipa-cacert-manage, but did you also run ipa-certupdate before trying
ipa-server-certinstall? This step is mandatory in order to update the NSS
databases, as specified in the man page ipa-cacert-manage(1).

HTH,
flo

On Thu, Oct 7, 2021 at 11:44 PM Stefan Fleischmann via FreeIPA-users <
[e-mail]> wrote:
> Hi! I've been using FreeIPA (installed without CA --no-pkinit) with
> letsencrypt certificate. Whenever the certificate gets renewed I install it
> with
> ipa-server-certinstall for both the LDAP and web server and that has been
> working just fine. Recently the root certificate (DST Root CA X3)
> expired as mentioned here
> https://letsencrypt.org/docs/dst-root-ca-x3-expiration-september-2021/
> Now when I try to install the new certificate I get this error:
>
> CA certificate CN=DST Root CA X3,O=Digital Signature Trust Co. in
> /etc/letsencrypt/live/XXX/cert.pem, /etc/letsencrypt/live/XXX/privkey.pem
> is not valid: certutil: certificate is invalid: The certificate issuer's
> certificate has expired. Check your system date and time.
> The ipa-server-certinstall command failed.
>
> I don't understand this error message at all since the `cert.pem` file
> does not contain any reference to the X3 CA, so I suppose it must come from
> somewhere else. Does someone have an idea how to fix this?
>
> I've already removed the root certificate with ipa-cacert-manage and added
> the self-signed X1 root cert, yet the same error message above keeps
> showing up.
> _______________________________________________
> FreeIPA-users mailing list -- [e-mail]
> To unsubscribe send an email to [e-mail]
> Fedora Code of Conduct:
> https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives:
> https://lists.fedorahosted.org/archives/list/[e-mail]...
> Do not reply to spam on the list, report it:
> https://pagure.io/fedora-infrastructure

[Stefan Fleischmann]

I've tried that actually, but then I'm also getting an error

cannot connect to 'https://XXXX/ipa/json': [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: certificate has expired (_ssl.c:1131)
The ipa-certupdate command failed.

My certificate is still valid, so this must also be related to the expired root CA. Is there a manual way to update the NSS database? Our FreeIPA version is 4.8.10 by the way. I remember fixing the NSS db on an older installation at some time, but I haven't found the db location on this new installation yet.

I guess I could reconfigure Apache to use a different certificate or even a self-signed one to get `ipa-certupdate` working again.

[Florence Renaud]

Hi,
start by checking the certificates stored in the LDAP database (replace
$SUFFIX with your actual suffix):
# ldapsearch -D "cn=directory manager" -W -b
cn=certificates,cn=ipa,cn=etc,$SUFFIX
You will get a list of entries, one for each CA cert, each entry may
contain multiple values for the attribute "cACertificate;binary".
You need to make sure that the expected certs are present, and the
conflicting one (the cross-signed) is absent (otherwise it will get
re-added by ipa-certupdate on the other places).

Then look at the file /etc/ipa/ca.crt. You should make a backup copy of
this file and then edit it in order to remove the wrong CA cert and add the
right one. The file must contain the whole certificate chain.
The other places where the CA chain is needed are NSS databases,
/etc/ipa/nssdb, /etc/httpd/alias, /etc/dirsrv/slapd-$DOMAIN but IIRC they
can be updated later by re-running ipa-certupdate once the /etc/ipa/ca.crt
file is fixed.

Let me know if it solves your issue.
flo

On Fri, Oct 8, 2021 at 10:51 AM Stefan Fleischmann via FreeIPA-users <
[e-mail]> wrote:
> I've tried that actually, but then I'm also getting an error
>
> cannot connect to 'https://XXXX/ipa/json': [SSL:
> CERTIFICATE_VERIFY_FAILED] certificate verify failed: certificate has
> expired (_ssl.c:1131)
> The ipa-certupdate command failed.
>
> My certificate is still valid, so this must also be related to the expired
> root CA. Is there a manual way to update the NSS database? Our FreeIPA
> version is 4.8.10 by the way. I remember fixing the NSS db on an older
> installation at some time, but I haven't found the db location on this new
> installation yet.
>
> I guess I could reconfigure Apache to use a different certificate or even
> a self-signed one to get `ipa-certupdate` working again.
> _______________________________________________
> FreeIPA-users mailing list -- [e-mail]
> To unsubscribe send an email to [e-mail]
> Fedora Code of Conduct:
> https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives:
> https://lists.fedorahosted.org/archives/list/[e-mail]...
> Do not reply to spam on the list, report it:
> https://pagure.io/fedora-infrastructure

[Stefan Fleischmann]

Thank you so much for the pointers in the right direction! So actually on one of the replicas the certificate had expired just yesterday or the day before, so that posed an additional problem.

First I removed the expired X3 CA and cross-signed X1 with `ipa-cacert-manage` (using the force flag), then I added the self-signed X1 (with "-t C,," flag). Now to the task of making `ipa-certupdate` work despite the expired certificate. First I made sure as you mentioned that I have the correct (only Letsencrypt R3 and self-signed X1) in the file ` /etc/ipa/ca.crt`. Then I disabled strict ssl checks both for LDAP and HTTP by editing the following files:

/usr/lib/python3/dist-packages/ipalib/util.py
- ctx.verify_mode = ssl.CERT_REQUIRED
- ctx.check_hostname = True
- ctx.load_verify_locations(cafile)
+ ctx.verify_mode = ssl.CERT_NONE
+ ctx.check_hostname = False

(stolen from here https://github.com/freeipa/freeipa-letsencrypt/issues/25#issuecomment-761904...)

and /usr/lib/python3/dist-packages/ipapython/ipaldap.py
- conn.set_option(ldap.OPT_X_TLS_REQUIRE_CERT, ldap.OPT_X_TLS_DEMAND)           
+ conn.set_option(ldap.OPT_X_TLS_REQUIRE_CERT, ldap.OPT_X_TLS_ALLOW)

and for good measure I added "TLS_REQCERT allow" in /etc/ldap/ldap.conf.

Then the `ipa-certupdate` finally went through and I reverted the changes above. Then I could finally install the new certificate with `ipa-server-certinstall` and things seem back to normal now. But I guess I still need to be careful to not accidentally import the old chain on the next update...