Hello,
our FreeIPA was running with correct certificates for 2 years (subject "CN=ipa.hq.company,O=HQ.COMPANY"). Unfortunately, the new certificates (ocspSigningCert, auditSigningCert) were recreated with simple "CN=localhost" (automatically), i.e. the original value "CN=ipa.hq.company,O=HQ.COMPANY" was ignored by certmonger.
If you have some knowledge of the FreeIPA internals - can you point me to the right direction, so that I could debug and/or fix this rotation bug, please?
Thank you, LG
certmonger-0.79.13-3.el8.x86_64 ipa-server-4.9.6-10.module+el8.5.0+13587+92118e57.x86_64
# getcert list
Request ID '20200324213127': status: MONITORING stuck: no : CA: IPA issuer: CN=Certificate Authority,O=HQ.COMPANY subject: CN=ipa.hq.company,O=HQ.COMPANY expires: 2022-03-25 22:31:28 CET principal name: krbtgt/HQ.COMPANY@HQ.COMPANY key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment eku: id-kp-serverAuth,id-pkinit-KPKdc profile: KDCs_PKINIT_Certs pre-save command: post-save command: /usr/libexec/ipa/certmonger/renew_kdc_cert track: yes auto-renew: yes Request ID '20210120221127': status: MONITORING stuck: no : CA: dogtag-ipa-ca-renew-agent issuer: CN=Certificate Authority,O=HQ.COMPANY subject: CN=localhost expires: 2024-02-04 22:29:37 CET key usage: digitalSignature,nonRepudiation profile: caSignedLogCert pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert "auditSigningCert cert-pki-ca" track: yes auto-renew: yes Request ID '20210120221129': status: MONITORING stuck: no : CA: dogtag-ipa-ca-renew-agent issuer: CN=Certificate Authority,O=HQ.COMPANY subject: CN=localhost expires: 2024-02-04 22:28:36 CET eku: id-kp-OCSPSigning profile: caOCSPCert pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert "ocspSigningCert cert-pki-ca" track: yes auto-renew: yes <snip>
Leo Galambos via FreeIPA-users wrote:
Hello,
our FreeIPA was running with correct certificates for 2 years (subject "CN=ipa.hq.company,O=HQ.COMPANY"). Unfortunately, the new certificates (ocspSigningCert, auditSigningCert) were recreated with simple "CN=localhost" (automatically), i.e. the original value "CN=ipa.hq.company,O=HQ.COMPANY" was ignored by certmonger.
If you have some knowledge of the FreeIPA internals - can you point me to the right direction, so that I could debug and/or fix this rotation bug, please?
Thank you, LG
certmonger-0.79.13-3.el8.x86_64 ipa-server-4.9.6-10.module+el8.5.0+13587+92118e57.x86_64
# getcert list
Request ID '20200324213127': status: MONITORING stuck: no : CA: IPA issuer: CN=Certificate Authority,O=HQ.COMPANY subject: CN=ipa.hq.company,O=HQ.COMPANY expires: 2022-03-25 22:31:28 CET principal name: krbtgt/HQ.COMPANY@HQ.COMPANY key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment eku: id-kp-serverAuth,id-pkinit-KPKdc profile: KDCs_PKINIT_Certs pre-save command: post-save command: /usr/libexec/ipa/certmonger/renew_kdc_cert track: yes auto-renew: yes Request ID '20210120221127': status: MONITORING stuck: no : CA: dogtag-ipa-ca-renew-agent issuer: CN=Certificate Authority,O=HQ.COMPANY subject: CN=localhost expires: 2024-02-04 22:29:37 CET key usage: digitalSignature,nonRepudiation profile: caSignedLogCert pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert "auditSigningCert cert-pki-ca" track: yes auto-renew: yes Request ID '20210120221129': status: MONITORING stuck: no : CA: dogtag-ipa-ca-renew-agent issuer: CN=Certificate Authority,O=HQ.COMPANY subject: CN=localhost expires: 2024-02-04 22:28:36 CET eku: id-kp-OCSPSigning profile: caOCSPCert pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert "ocspSigningCert cert-pki-ca" track: yes auto-renew: yes
<snip>
Is this the renewal master? (ipa config-show | grep renewal)
You stripped out the key and certificate storage lines, can we see that as well?
The cow may be out of the barn already, but certmonger should have already been aware of the hostname when the cert was re-issued. You can determine the request file name in /var/lib/certmonger/requests by greeping for the request ID (it may or may not match the filename).
Then grep template_ from that file. At this point it may be CN=localhost but it would be interesting to see what is there.
It should be straightforward to get new certificates by using the -N <subject> option with resubmit but it would be nice to try to figure out how it got into this situation.
For example:
# getcert resubmit -i 20210120221129 -N 'CN=OCSP Subsystem,O=HQ.COMPANY' -v -w
rob
On 2/15/22 21:25, Rob Crittenden wrote:
Leo Galambos via FreeIPA-users wrote:
Hello,
our FreeIPA was running with correct certificates for 2 years (subject "CN=ipa.hq.company,O=HQ.COMPANY"). Unfortunately, the new certificates (ocspSigningCert, auditSigningCert) were recreated with simple "CN=localhost" (automatically), i.e. the original value "CN=ipa.hq.company,O=HQ.COMPANY" was ignored by certmonger.
Is this the renewal master? (ipa config-show | grep renewal)
Yes, it is: IPA CA renewal master: ipa.hq.company
You stripped out the key and certificate storage lines, can we see that as well?
Everything is default, dir /etc/pki/pki-tomcat/alias and pin are set, e.g.:
Request ID '20210120221129': status: MONITORING stuck: no key pair storage: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='ocspSigningCert cert-pki-ca',token='NSS Certificate DB',pin set certificate: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='ocspSigningCert cert-pki-ca',token='NSS Certificate DB' CA: dogtag-ipa-ca-renew-agent issuer: CN=Certificate Authority,O=HQ.COMPANY subject: CN=localhost expires: 2024-02-04 22:28:36 CET eku: id-kp-OCSPSigning profile: caOCSPCert pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert "ocspSigningCert cert-pki-ca" track: yes auto-renew: yes
^^^ this one is missing in the NSS DB now (the rotation removed it from NSS DB!), but 'auditSigningCert' seems to be fine (except of its CN=localhost). PIN is identical to /etc/pki/pki-tomcat/alias/pwdfile.txt
*** Before rotation:
# certutil -L -d alias
Certificate Nickname Trust Attributes SSL,S/MIME,JAR/XPI
caSigningCert cert-pki-ca CTu,Cu,Cu ocspSigningCert cert-pki-ca u,u,u subsystemCert cert-pki-ca u,u,u auditSigningCert cert-pki-ca u,u,Pu Server-Cert cert-pki-ca u,u,u
*** After rotation:
# certutil -L -d /etc/pki/pki-tomcat/alias
Certificate Nickname Trust Attributes SSL,S/MIME,JAR/XPI
caSigningCert cert-pki-ca CTu,Cu,Cu subsystemCert cert-pki-ca u,u,u Server-Cert cert-pki-ca u,u,u auditSigningCert cert-pki-ca u,u,Pu
# certutil -L -d /etc/pki/pki-tomcat/alias -n 'ocspSigningCert cert-pki-ca' certutil: Could not find cert: ocspSigningCert cert-pki-ca : PR_FILE_NOT_FOUND_ERROR: File not found # certutil -L -d /etc/pki/pki-tomcat/alias -n 'auditSigningCert cert-pki-ca' Certificate: Data: Version: 3 (0x2) Serial Number: 31 (0x1f) Signature Algorithm: PKCS #1 SHA-256 With RSA Encryption Issuer: "CN=Certificate Authority,O=HQ.COMPANY" Validity: Not Before: Mon Feb 14 21:29:37 2022 Not After : Sun Feb 04 21:29:37 2024 Subject: "CN=localhost"
Anyway, if I import my previous 'ocspSigningCert cert-pki-ca' back into /etc/pki/pki-tomcat/alias and rollback ca.ocsp_signing.cert and ca.ocsp_signing.certreq in CS.cfg - would it be enough to fix this?
The cow may be out of the barn already, but certmonger should have already been aware of the hostname when the cert was re-issued. You can determine the request file name in /var/lib/certmonger/requests by greeping for the request ID (it may or may not match the filename).
All parameters seems to be OK in the request files, except of the CN=localhost related to two "new" buggy certificates.
Then grep template_ from that file. At this point it may be CN=localhost but it would be interesting to see what is there.
BTW templates: auditSigningCert points to caSignedLogCert, ocspSigningCert points to caOCSPCert. Both caSignedLogCert.cfg and caOCSPCert.cfg exist in /var/lib/pki/pki-tomcat/ca/profiles/ca. None of them is owned by RPM package (O/S is RHEL 8.5). caOCSPCert.profile also exists in /var/lib/pki/pki-tomcat/conf/ca, again not owned by any RPM. caSignedLogCert.profile does not exist on my disk AFAIK.
ad ocspSigningCert) There is a correct value template_subject=CN=OCSP Subsystem,O=HQ.COMPANY
'csr' attribute in the request file includes CSR with:
Certificate Request: Data: Version: 1 (0x0) Subject: CN = localhost Subject Public Key Info: Public Key Algorithm: rsaEncryption
'cert' attribute has the respective cert.
After `getcert resubmit ...` it cannot be changed due to "status: CA_UNREACHABLE".
ad auditSigningCert) template_subject=CN=CA Audit,O=HQ.COMPANY
After `getcert resubmit ...` the 'csr' attribute seems to be OK:
Certificate Request: Data: Version: 1 (0x0) Subject: O = HQ.COMPANY, CN = CA Audit Subject Public Key Info: Public Key Algorithm: rsaEncryption
'cert' attribute is still the old one (with CN=localhost) due to "Error 7 connecting to http://ipa.hq.company:8080/ca/ee/ca/profileSubmit: Couldn't connect to server" (tomcat is down - OCSP cert is missing).
CS.cfg: ca.audit_signing.cert identical to 'auditSigningCert cert-pki-ca' (NSS DB /etc/pki/pki-tomcat/alias), ca.signing.cert is 'caSigningCert cert-pki-ca', ca.sslserver.cert is 'Server-Cert cert-pki-ca', ca.subsystem.cert is 'subsystemCert cert-pki-ca', and ca.ocsp_signing.cert is identical to 'cert' attribute of the certmonger's ocspSigningCert request file.
# ldapsearch -LLL -D 'cn=directory manager' -W -b uid=pkidbuser,ou=people,o=ipaca userCertificate description seeAlso
returns "userCertificate" identical to CS.cfg's ca.subsystem.cert. So this part of the system seems to be OK.
Summary: ocspSigningCert was rotated in CS.cfg, then it was lost, and the old one was removed from NSS DB.
During rotation, I can see this sequence in Tomcat's CA debug log:
1) 'Server-Cert cert-pki-ca' was processed with "INFO: EnrollProfile: - subject: CN=ipa.hq.company,O=HQ.COMPANY"
2) 'ocspSigningCert cert-pki-ca' came with "INFO: EnrollProfile: - subject: CN=localhost"
3) after less than 1 minute, something fails: [AuthorityMonitor] WARNING: CAEngine: Error initializing lightweight CA: Failed to update certificate; nickname conflict Failed to update certificate; nickname conflict at com.netscape.ca.CertificateAuthority.init(CertificateAuthority.java:426) at org.dogtagpki.server.ca.CAEngine.readAuthority(CAEngine.java:1441) at com.netscape.ca.AuthorityMonitor.run(AuthorityMonitor.java:162) at java.lang.Thread.run(Thread.java:750) Caused by: Failed to update certificate; nickname conflict at com.netscape.ca.CertificateAuthority.checkForNewerCert(CertificateAuthority.java:504) at com.netscape.ca.CertificateAuthority.init(CertificateAuthority.java:418) ... 3 more Caused by: org.mozilla.jss.NicknameConflictException at org.mozilla.jss.CryptoManager.importCertPackageNative(Native Method) at org.mozilla.jss.CryptoManager.importUserCACertPackage(CryptoManager.java:739) at com.netscape.ca.CertificateAuthority.checkForNewerCert(CertificateAuthority.java:480) ... 4 more
Cheers, LG
It should be straightforward to get new certificates by using the -N <subject> option with resubmit but it would be nice to try to figure out how it got into this situation.
For example:
# getcert resubmit -i 20210120221129 -N 'CN=OCSP Subsystem,O=HQ.COMPANY' -v -w
Just for record:
Unfortunately, OCSP cannot be recovered this way (NSS DB does not include OCSP cert/key pair). I tried to recover auditSigningCert, but it also fails due to dead tomcat:
# getcert resubmit -i 20210120221127 -N 'CN=CA Audit,O=HQ.COMPANY' -v -w Resubmitting "20210120221127" to "dogtag-ipa-ca-renew-agent". State GENERATING_CSR, stuck: no. State SUBMITTING, stuck: no. State CA_UNREACHABLE, stuck: no.
Summarizing.
If you have the original certs I'd definitely try restoring those. You'll need to go back in time a few days in order to restart the CA.
Once the CA comes back up I'd suggest using the -N option to be sure the subject is ok and renew the certs manually one at a time.
IPA loads profiles from LDAP which is why the on-disk doesn't seem to match up.
rob
Leo Galambos via FreeIPA-users wrote:
On 2/15/22 21:25, Rob Crittenden wrote:
Leo Galambos via FreeIPA-users wrote:
Hello,
our FreeIPA was running with correct certificates for 2 years (subject "CN=ipa.hq.company,O=HQ.COMPANY"). Unfortunately, the new certificates (ocspSigningCert, auditSigningCert) were recreated with simple "CN=localhost" (automatically), i.e. the original value "CN=ipa.hq.company,O=HQ.COMPANY" was ignored by certmonger.
Is this the renewal master? (ipa config-show | grep renewal)
Yes, it is: IPA CA renewal master: ipa.hq.company
You stripped out the key and certificate storage lines, can we see that as well?
Everything is default, dir /etc/pki/pki-tomcat/alias and pin are set, e.g.:
Request ID '20210120221129': status: MONITORING stuck: no key pair storage: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='ocspSigningCert cert-pki-ca',token='NSS Certificate DB',pin set certificate: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='ocspSigningCert cert-pki-ca',token='NSS Certificate DB' CA: dogtag-ipa-ca-renew-agent issuer: CN=Certificate Authority,O=HQ.COMPANY subject: CN=localhost expires: 2024-02-04 22:28:36 CET eku: id-kp-OCSPSigning profile: caOCSPCert pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert "ocspSigningCert cert-pki-ca" track: yes auto-renew: yes
^^^ this one is missing in the NSS DB now (the rotation removed it from NSS DB!), but 'auditSigningCert' seems to be fine (except of its CN=localhost). PIN is identical to /etc/pki/pki-tomcat/alias/pwdfile.txt
*** Before rotation:
# certutil -L -d alias
Certificate Nickname Trust Attributes SSL,S/MIME,JAR/XPI
caSigningCert cert-pki-ca CTu,Cu,Cu ocspSigningCert cert-pki-ca u,u,u subsystemCert cert-pki-ca u,u,u auditSigningCert cert-pki-ca u,u,Pu Server-Cert cert-pki-ca u,u,u
*** After rotation:
# certutil -L -d /etc/pki/pki-tomcat/alias
Certificate Nickname Trust Attributes SSL,S/MIME,JAR/XPI
caSigningCert cert-pki-ca CTu,Cu,Cu subsystemCert cert-pki-ca u,u,u Server-Cert cert-pki-ca u,u,u auditSigningCert cert-pki-ca u,u,Pu
# certutil -L -d /etc/pki/pki-tomcat/alias -n 'ocspSigningCert cert-pki-ca' certutil: Could not find cert: ocspSigningCert cert-pki-ca : PR_FILE_NOT_FOUND_ERROR: File not found # certutil -L -d /etc/pki/pki-tomcat/alias -n 'auditSigningCert cert-pki-ca' Certificate: Data: Version: 3 (0x2) Serial Number: 31 (0x1f) Signature Algorithm: PKCS #1 SHA-256 With RSA Encryption Issuer: "CN=Certificate Authority,O=HQ.COMPANY" Validity: Not Before: Mon Feb 14 21:29:37 2022 Not After : Sun Feb 04 21:29:37 2024 Subject: "CN=localhost"
Anyway, if I import my previous 'ocspSigningCert cert-pki-ca' back into /etc/pki/pki-tomcat/alias and rollback ca.ocsp_signing.cert and ca.ocsp_signing.certreq in CS.cfg - would it be enough to fix this?
The cow may be out of the barn already, but certmonger should have already been aware of the hostname when the cert was re-issued. You can determine the request file name in /var/lib/certmonger/requests by greeping for the request ID (it may or may not match the filename).
All parameters seems to be OK in the request files, except of the CN=localhost related to two "new" buggy certificates.
Then grep template_ from that file. At this point it may be CN=localhost but it would be interesting to see what is there.
BTW templates: auditSigningCert points to caSignedLogCert, ocspSigningCert points to caOCSPCert. Both caSignedLogCert.cfg and caOCSPCert.cfg exist in /var/lib/pki/pki-tomcat/ca/profiles/ca. None of them is owned by RPM package (O/S is RHEL 8.5). caOCSPCert.profile also exists in /var/lib/pki/pki-tomcat/conf/ca, again not owned by any RPM. caSignedLogCert.profile does not exist on my disk AFAIK.
ad ocspSigningCert) There is a correct value template_subject=CN=OCSP Subsystem,O=HQ.COMPANY
'csr' attribute in the request file includes CSR with:
Certificate Request: Data: Version: 1 (0x0) Subject: CN = localhost Subject Public Key Info: Public Key Algorithm: rsaEncryption
'cert' attribute has the respective cert.
After `getcert resubmit ...` it cannot be changed due to "status: CA_UNREACHABLE".
ad auditSigningCert) template_subject=CN=CA Audit,O=HQ.COMPANY
After `getcert resubmit ...` the 'csr' attribute seems to be OK:
Certificate Request: Data: Version: 1 (0x0) Subject: O = HQ.COMPANY, CN = CA Audit Subject Public Key Info: Public Key Algorithm: rsaEncryption
'cert' attribute is still the old one (with CN=localhost) due to "Error 7 connecting to http://ipa.hq.company:8080/ca/ee/ca/profileSubmit: Couldn't connect to server" (tomcat is down - OCSP cert is missing).
CS.cfg: ca.audit_signing.cert identical to 'auditSigningCert cert-pki-ca' (NSS DB /etc/pki/pki-tomcat/alias), ca.signing.cert is 'caSigningCert cert-pki-ca', ca.sslserver.cert is 'Server-Cert cert-pki-ca', ca.subsystem.cert is 'subsystemCert cert-pki-ca', and ca.ocsp_signing.cert is identical to 'cert' attribute of the certmonger's ocspSigningCert request file.
# ldapsearch -LLL -D 'cn=directory manager' -W -b uid=pkidbuser,ou=people,o=ipaca userCertificate description seeAlso
returns "userCertificate" identical to CS.cfg's ca.subsystem.cert. So this part of the system seems to be OK.
Summary: ocspSigningCert was rotated in CS.cfg, then it was lost, and the old one was removed from NSS DB.
During rotation, I can see this sequence in Tomcat's CA debug log:
- 'Server-Cert cert-pki-ca' was processed with "INFO: EnrollProfile: -
subject: CN=ipa.hq.company,O=HQ.COMPANY"
- 'ocspSigningCert cert-pki-ca' came with "INFO: EnrollProfile: -
subject: CN=localhost"
- after less than 1 minute, something fails: [AuthorityMonitor]
WARNING: CAEngine: Error initializing lightweight CA: Failed to update certificate; nickname conflict Failed to update certificate; nickname conflict at com.netscape.ca.CertificateAuthority.init(CertificateAuthority.java:426) at org.dogtagpki.server.ca.CAEngine.readAuthority(CAEngine.java:1441) at com.netscape.ca.AuthorityMonitor.run(AuthorityMonitor.java:162) at java.lang.Thread.run(Thread.java:750) Caused by: Failed to update certificate; nickname conflict at com.netscape.ca.CertificateAuthority.checkForNewerCert(CertificateAuthority.java:504)
at com.netscape.ca.CertificateAuthority.init(CertificateAuthority.java:418) ... 3 more Caused by: org.mozilla.jss.NicknameConflictException at org.mozilla.jss.CryptoManager.importCertPackageNative(Native Method) at org.mozilla.jss.CryptoManager.importUserCACertPackage(CryptoManager.java:739)
at com.netscape.ca.CertificateAuthority.checkForNewerCert(CertificateAuthority.java:480)
... 4 more
Cheers, LG
It should be straightforward to get new certificates by using the -N <subject> option with resubmit but it would be nice to try to figure out how it got into this situation.
For example:
# getcert resubmit -i 20210120221129 -N 'CN=OCSP Subsystem,O=HQ.COMPANY' -v -w
Just for record:
Unfortunately, OCSP cannot be recovered this way (NSS DB does not include OCSP cert/key pair). I tried to recover auditSigningCert, but it also fails due to dead tomcat:
# getcert resubmit -i 20210120221127 -N 'CN=CA Audit,O=HQ.COMPANY' -v -w Resubmitting "20210120221127" to "dogtag-ipa-ca-renew-agent". State GENERATING_CSR, stuck: no. State SUBMITTING, stuck: no. State CA_UNREACHABLE, stuck: no.
FreeIPA-users mailing list -- freeipa-users@lists.fedorahosted.org To unsubscribe send an email to freeipa-users-leave@lists.fedorahosted.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedorahoste...
Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
On 2/16/22 22:29, Rob Crittenden wrote:
Summarizing.
If you have the original certs I'd definitely try restoring those. You'll need to go back in time a few days in order to restart the CA.
Once the CA comes back up I'd suggest using the -N option to be sure the subject is ok and renew the certs manually one at a time.
IPA loads profiles from LDAP which is why the on-disk doesn't seem to match up.
Yup. I am just finishing the whole process and it is a real pain. I imported my previous ocspSigningCert into NSS DB, FreeIPA was alive again and started some actions, e.g.: CS.cfg was updated with the new auditSigningCert but it was not saved into NSS DB, so I had to fix this manually; subsystemCert was updated in NSS DB, but it was not propagated into pkidbuser's seeAlso in LDAP, so tomcat was dead again (it could not connect to LDAP due to bad auth), etc.
I guess the solution for a production system would be:
1. set auto-renew 'no' for auditSigningCert, ocspSigningCert, subsystemCert, (and maybe) Server-Cert
2. backup NSS DB, and renew the certificates manually
ad 1) getcert start-tracking -i $(request.id) --no-renew
ad 2) getcert resubmit -i $(request.id) -N $(find.correct.name.in.cert) -v
It will save you a lot of time.
Cheers, LG
freeipa-users@lists.fedorahosted.org
-
Leo Galambos -
Rob Crittenden