I have a cluster of 3 IPA servers, the primary server renewed the krb5kdc certificate last night, but did not include the principal name when it renewed, here is after the auto-renewal:
Number of certificates and requests being tracked: 9. Request ID '20221028185012': status: MONITORING stuck: no key pair storage: type=FILE,location='/var/kerberos/krb5kdc/kdc.key' certificate: type=FILE,location='/var/kerberos/krb5kdc/kdc.crt' CA: IPA issuer: CN=Certificate Authority,O=IPA.REDACTED subject: CN=ipa-primary.ipa.redacted,O=IPA.REDACTED issued: 2024-09-30 14:51:55 EDT expires: 2026-10-01 14:51:55 EDT dns: ipa-primary.ipa.redacted key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment eku: id-kp-serverAuth,id-pkinit-KPKdc profile: KDCs_PKINIT_Certs pre-save command: post-save command: /usr/libexec/ipa/certmonger/renew_kdc_cert track: yes auto-renew: yes
After I manually renewed with getcert resubmit, it included the principal line:
[root@ipa-primary pki]# getcert list -i 20221028185012 Number of certificates and requests being tracked: 9. Request ID '20221028185012': status: MONITORING stuck: no key pair storage: type=FILE,location='/var/kerberos/krb5kdc/kdc.key' certificate: type=FILE,location='/var/kerberos/krb5kdc/kdc.crt' CA: IPA issuer: CN=Certificate Authority,O=IPA.REDACTED subject: CN=ipa-primary.ipa.REDACTED,O=IPA.REDACTED issued: 2024-10-01 13:26:06 EDT expires: 2026-10-02 13:26:06 EDT dns: ipa-primary.ipa.REDACTED principal name: krbtgt/IPA.REDACTED@IPA.REDACTED key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment eku: id-kp-serverAuth,id-pkinit-KPKdc profile: KDCs_PKINIT_Certs pre-save command: post-save command: /usr/libexec/ipa/certmonger/renew_kdc_cert track: yes auto-renew: yes
Any idea how I can track down how or why this was missed, and how to prevent this from happening in the future?
Likely relevant version information:
[root@ipa-primary pki]# cat /etc/redhat-release CentOS Stream release 9 [root@ipa-primary pki]# rpm -qa | grep -i freeipa [root@ipa-primary pki]# rpm -qa | grep -i ipa centos-logos-ipa-90.8-1.el9.noarch ipa-client-common-4.12.2-1.el9.noarch libipa_hbac-2.9.5-4.el9.x86_64 python3-libipa_hbac-2.9.5-4.el9.x86_64 ipa-healthcheck-core-0.16-4.el9.noarch ipa-selinux-4.12.2-1.el9.noarch ipa-common-4.12.2-1.el9.noarch python3-ipalib-4.12.2-1.el9.noarch python3-ipaclient-4.12.2-1.el9.noarch sssd-ipa-2.9.5-4.el9.x86_64 ipa-client-4.12.2-1.el9.x86_64 ipa-server-common-4.12.2-1.el9.noarch python3-ipaserver-4.12.2-1.el9.noarch ipa-server-4.12.2-1.el9.x86_64 ipa-healthcheck-0.16-4.el9.noarch [root@ipa-primary pki]# rpm -qa | grep -i ssl apr-util-openssl-1.6.1-23.el9.x86_64 openssl-pkcs11-0.4.11-9.el9.x86_64 mod_ssl-2.4.62-1.el9.x86_64 perl-Net-SSLeay-1.94-1.el9.x86_64 perl-IO-Socket-SSL-2.073-2.el9.noarch openssl-libs-3.2.2-6.el9.x86_64 openssl-3.2.2-6.el9.x86_64 openssl-perl-3.2.2-6.el9.x86_64
Russ Long via FreeIPA-users wrote:
I have a cluster of 3 IPA servers, the primary server renewed the krb5kdc certificate last night, but did not include the principal name when it renewed, here is after the auto-renewal:
Number of certificates and requests being tracked: 9. Request ID '20221028185012': status: MONITORING stuck: no key pair storage: type=FILE,location='/var/kerberos/krb5kdc/kdc.key' certificate: type=FILE,location='/var/kerberos/krb5kdc/kdc.crt' CA: IPA issuer: CN=Certificate Authority,O=IPA.REDACTED subject: CN=ipa-primary.ipa.redacted,O=IPA.REDACTED issued: 2024-09-30 14:51:55 EDT expires: 2026-10-01 14:51:55 EDT dns: ipa-primary.ipa.redacted key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment eku: id-kp-serverAuth,id-pkinit-KPKdc profile: KDCs_PKINIT_Certs pre-save command: post-save command: /usr/libexec/ipa/certmonger/renew_kdc_cert track: yes auto-renew: yes
After I manually renewed with getcert resubmit, it included the principal line:
[root@ipa-primary pki]# getcert list -i 20221028185012 Number of certificates and requests being tracked: 9. Request ID '20221028185012': status: MONITORING stuck: no key pair storage: type=FILE,location='/var/kerberos/krb5kdc/kdc.key' certificate: type=FILE,location='/var/kerberos/krb5kdc/kdc.crt' CA: IPA issuer: CN=Certificate Authority,O=IPA.REDACTED subject: CN=ipa-primary.ipa.REDACTED,O=IPA.REDACTED issued: 2024-10-01 13:26:06 EDT expires: 2026-10-02 13:26:06 EDT dns: ipa-primary.ipa.REDACTED principal name: krbtgt/IPA.REDACTED@IPA.REDACTED key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment eku: id-kp-serverAuth,id-pkinit-KPKdc profile: KDCs_PKINIT_Certs pre-save command: post-save command: /usr/libexec/ipa/certmonger/renew_kdc_cert track: yes auto-renew: yes
Any idea how I can track down how or why this was missed, and how to prevent this from happening in the future?
What exactly did you do to resubmit the request?
The journal may have some information depending on what /etc/sysconfig/certmonger contains (-d2 is the newish default).
This request goes through IPA so some details may be in /var/log/httpd/error_log.
Either through the journal or the Apache log you should be able to find the CSRs that were submitted for the bad and good certs. Maybe they will be different.
Similarly the returned certificates.
rob
To resubmit the request, all I did was `getcert resubmit -i <REQUEST_ID>
Thanks for the log suggestions, I'll look at them.
Just found it in messages log.
Looks like the env var for the principal was set, but when I decode the CSR it shows no principals added.
Sep 30 14:50:17 ipa-primary certmonger[756]: 2024-09-30 14:50:17 [756] Wrote to /var/lib/certmonger/requests/20221028185012 Sep 30 14:50:17 ipa-primary certmonger[756]: 2024-09-30 14:50:17 [756] Wrote to /var/lib/certmonger/requests/20221028185012 Sep 30 14:50:17 ipa-primary certmonger[642836]: Certificate in file "/var/kerberos/krb5kdc/kdc.crt" will not be valid after 2024-10-28 14:50:12 EDT. Sep 30 14:50:17 ipa-primary certmonger[756]: 2024-09-30 14:50:17 [756] Wrote to /var/lib/certmonger/requests/20221028185012 Sep 30 14:50:17 ipa-primary certmonger[756]: 2024-09-30 14:50:17 [756] Wrote to /var/lib/certmonger/requests/20221028185012 Sep 30 14:50:17 ipa-primary certmonger[642837]: 2024-09-30 14:50:17 [642837] Error initializing NSS. Sep 30 14:50:17 ipa-primary certmonger[642837]: 2024-09-30 14:50:17 [642837] error:04000067:object identifier routines::unknown object name Sep 30 14:50:17 ipa-primary certmonger[756]: 2024-09-30 14:50:17 [756] Wrote to /var/lib/certmonger/requests/20221028185012 Sep 30 14:50:17 ipa-primary certmonger[756]: 2024-09-30 14:50:17 [756] Wrote to /var/lib/certmonger/requests/20221028185012 Sep 30 14:50:17 ipa-primary certmonger[642838]: 2024-09-30 14:50:17 [642838] Setting "CERTMONGER_REQ_SUBJECT" to "O=IPA.REDACTED,cn=ipa-primary.ipa.REDACTED" for child. Sep 30 14:50:17 ipa-primary certmonger[642838]: 2024-09-30 14:50:17 [642838] Setting "CERTMONGER_REQ_HOSTNAME" to "ipa-primary.ipa.REDACTED" for child. Sep 30 14:50:17 ipa-primary certmonger[642838]: 2024-09-30 14:50:17 [642838] Setting "CERTMONGER_REQ_PRINCIPAL" to "krbtgt/IPA.REDACTED@IPA.REDACTED" for child. Sep 30 14:50:17 ipa-primary certmonger[642838]: 2024-09-30 14:50:17 [642838] Setting "CERTMONGER_OPERATION" to "SUBMIT" for child. Sep 30 14:50:17 ipa-primary certmonger[642838]: 2024-09-30 14:50:17 [642838] Setting "CERTMONGER_CSR" to "-----BEGIN CERTIFICATE REQUEST-----
When I decode the CSR for the manual renewal I did, it includes the formerly-missing principal.
The env vars being set appear to be identical both times, but for good measure, here are the ones from the working request:
Oct 1 13:26:06 ipa-primary certmonger[6178]: 2024-10-01 13:26:06 [6178] Setting "CERTMONGER_REQ_SUBJECT" to "O=IPA.REDACTED,cn=ipa-primary.ipa.REDACTED" for child. Oct 1 13:26:06 ipa-primary certmonger[6178]: 2024-10-01 13:26:06 [6178] Setting "CERTMONGER_REQ_HOSTNAME" to "ipa-primary.ipa.REDACTED" for child. Oct 1 13:26:06 ipa-primary certmonger[6178]: 2024-10-01 13:26:06 [6178] Setting "CERTMONGER_REQ_PRINCIPAL" to "krbtgt/IPA.REDACTED@IPA.REDACTED" for child. Oct 1 13:26:06 ipa-primary certmonger[6178]: 2024-10-01 13:26:06 [6178] Setting "CERTMONGER_OPERATION" to "SUBMIT" for child. Oct 1 13:26:06 ipa-primary certmonger[6178]: 2024-10-01 13:26:06 [6178] Setting "CERTMONGER_CSR" to "-----BEGIN CERTIFICATE REQUEST-----
Russ Long via FreeIPA-users wrote:
Just found it in messages log.
Looks like the env var for the principal was set, but when I decode the CSR it shows no principals added.
Sep 30 14:50:17 ipa-primary certmonger[756]: 2024-09-30 14:50:17 [756] Wrote to /var/lib/certmonger/requests/20221028185012 Sep 30 14:50:17 ipa-primary certmonger[756]: 2024-09-30 14:50:17 [756] Wrote to /var/lib/certmonger/requests/20221028185012 Sep 30 14:50:17 ipa-primary certmonger[642836]: Certificate in file "/var/kerberos/krb5kdc/kdc.crt" will not be valid after 2024-10-28 14:50:12 EDT. Sep 30 14:50:17 ipa-primary certmonger[756]: 2024-09-30 14:50:17 [756] Wrote to /var/lib/certmonger/requests/20221028185012 Sep 30 14:50:17 ipa-primary certmonger[756]: 2024-09-30 14:50:17 [756] Wrote to /var/lib/certmonger/requests/20221028185012 Sep 30 14:50:17 ipa-primary certmonger[642837]: 2024-09-30 14:50:17 [642837] Error initializing NSS. Sep 30 14:50:17 ipa-primary certmonger[642837]: 2024-09-30 14:50:17 [642837] error:04000067:object identifier routines::unknown object name Sep 30 14:50:17 ipa-primary certmonger[756]: 2024-09-30 14:50:17 [756] Wrote to /var/lib/certmonger/requests/20221028185012 Sep 30 14:50:17 ipa-primary certmonger[756]: 2024-09-30 14:50:17 [756] Wrote to /var/lib/certmonger/requests/20221028185012 Sep 30 14:50:17 ipa-primary certmonger[642838]: 2024-09-30 14:50:17 [642838] Setting "CERTMONGER_REQ_SUBJECT" to "O=IPA.REDACTED,cn=ipa-primary.ipa.REDACTED" for child. Sep 30 14:50:17 ipa-primary certmonger[642838]: 2024-09-30 14:50:17 [642838] Setting "CERTMONGER_REQ_HOSTNAME" to "ipa-primary.ipa.REDACTED" for child. Sep 30 14:50:17 ipa-primary certmonger[642838]: 2024-09-30 14:50:17 [642838] Setting "CERTMONGER_REQ_PRINCIPAL" to "krbtgt/IPA.REDACTED@IPA.REDACTED" for child. Sep 30 14:50:17 ipa-primary certmonger[642838]: 2024-09-30 14:50:17 [642838] Setting "CERTMONGER_OPERATION" to "SUBMIT" for child. Sep 30 14:50:17 ipa-primary certmonger[642838]: 2024-09-30 14:50:17 [642838] Setting "CERTMONGER_CSR" to "-----BEGIN CERTIFICATE REQUEST-----
When I decode the CSR for the manual renewal I did, it includes the formerly-missing principal.
The env vars being set appear to be identical both times, but for good measure, here are the ones from the working request:
Oct 1 13:26:06 ipa-primary certmonger[6178]: 2024-10-01 13:26:06 [6178] Setting "CERTMONGER_REQ_SUBJECT" to "O=IPA.REDACTED,cn=ipa-primary.ipa.REDACTED" for child. Oct 1 13:26:06 ipa-primary certmonger[6178]: 2024-10-01 13:26:06 [6178] Setting "CERTMONGER_REQ_HOSTNAME" to "ipa-primary.ipa.REDACTED" for child. Oct 1 13:26:06 ipa-primary certmonger[6178]: 2024-10-01 13:26:06 [6178] Setting "CERTMONGER_REQ_PRINCIPAL" to "krbtgt/IPA.REDACTED@IPA.REDACTED" for child. Oct 1 13:26:06 ipa-primary certmonger[6178]: 2024-10-01 13:26:06 [6178] Setting "CERTMONGER_OPERATION" to "SUBMIT" for child. Oct 1 13:26:06 ipa-primary certmonger[6178]: 2024-10-01 13:26:06 [6178] Setting "CERTMONGER_CSR" to "-----BEGIN CERTIFICATE REQUEST-----
It looks like if there is an issue building a SAN extension it is simply dropped. My best guess about where this is failing, because there is no real context provided by certmonger, is after the OpenSSL extensions are generated a while loop is run "overerror = ERR_get_error()" and just prints out the errors found.
This is something I've been working to improve in certmonger but I guess I haven't gotten to this one.
It really makes no sense that the same code run over the same request would work in one case and fail in another. The request file should have remained mostly intact between, updating on the the not before/after values, the CSR and the resulting certificate. The resubmit would have needed this so we know it was there because it succeeded the second time.
I'm also not sure how IPA issued a certificate that did not contain a principal SAN. It shouldn't do this.
rob
freeipa-users@lists.fedorahosted.org
-
Rob Crittenden -
Russ Long