Hi team,
ipa-healthcheck has been a great tool for us. I run it weekly on all IPA servers via cron. This week ipa-healthcheck reported errors on all IPA servers.
Take IPA server ipa2 as an example for the investigation:
[root@ipa2 ~]# ipa-healthcheck --failures-only --output-type=human
ra.get_certificate(): EXCEPTION (Invalid Credential.)
ra.get_certificate(): EXCEPTION (Invalid Credential.)
ra.get_certificate(): EXCEPTION (Invalid Credential.)
ra.get_certificate(): EXCEPTION (Invalid Credential.)
ra.get_certificate(): EXCEPTION (Invalid Credential.)
ra.get_certificate(): EXCEPTION (Invalid Credential.)
ra.get_certificate(): EXCEPTION (Invalid Credential.)
ra.get_certificate(): EXCEPTION (Invalid Credential.)
ra.get_certificate(): EXCEPTION (Invalid Credential.)
ra.get_certificate(): EXCEPTION (Invalid Credential.)
WARNING: ipahealthcheck.ipa.certs.IPACertmongerExpirationCheck.20190425210040: Request id 20190425210040 expires in 27 days
WARNING: ipahealthcheck.ipa.certs.IPACertmongerExpirationCheck.20190425210052: Request id 20190425210052 expires in 27 days
WARNING: ipahealthcheck.ipa.certs.IPACertmongerExpirationCheck.20190425210053: Request id 20190425210053 expires in 27 days
WARNING: ipahealthcheck.ipa.certs.IPACertmongerExpirationCheck.20190425210054: Request id 20190425210054 expires in 27 days
WARNING: ipahealthcheck.ipa.certs.IPACertfileExpirationCheck.20190425210040: Request id 20190425210040 expires in 27 days
WARNING: ipahealthcheck.ipa.certs.IPACertfileExpirationCheck.20190425210052: Request id 20190425210052 expires in 27 days
WARNING: ipahealthcheck.ipa.certs.IPACertfileExpirationCheck.20190425210053: Request id 20190425210053 expires in 27 days
WARNING: ipahealthcheck.ipa.certs.IPACertfileExpirationCheck.20190425210054: Request id 20190425210054 expires in 27 days
ERROR: ipahealthcheck.ipa.certs.IPARAAgent: RA agent description does not match 2;186;CN=Certificate Authority,O=EXAMPLE.COM;CN=IPA RA,O=EXAMPLE.COM in LDAP and 2;66;CN=Certificate Authority,O=EXAMPLE.COM;CN=IPA RA,O= EXAMPLE.COM expected
ERROR: ipahealthcheck.ipa.certs.IPACertRevocation.20190425210040: Request for certificate failed, Certificate operation cannot be completed: EXCEPTION (Invalid Credential.)
ERROR: ipahealthcheck.ipa.certs.IPACertRevocation.20190425210052: Request for certificate failed, Certificate operation cannot be completed: EXCEPTION (Invalid Credential.)
ERROR: ipahealthcheck.ipa.certs.IPACertRevocation.20190425210053: Request for certificate failed, Certificate operation cannot be completed: EXCEPTION (Invalid Credential.)
ERROR: ipahealthcheck.ipa.certs.IPACertRevocation.20190425210054: Request for certificate failed, Certificate operation cannot be completed: EXCEPTION (Invalid Credential.)
ERROR: ipahealthcheck.ipa.certs.IPACertRevocation.20190425210055: Request for certificate failed, Certificate operation cannot be completed: EXCEPTION (Invalid Credential.)
ERROR: ipahealthcheck.ipa.certs.IPACertRevocation.20190425210056: Request for certificate failed, Certificate operation cannot be completed: EXCEPTION (Invalid Credential.)
ERROR: ipahealthcheck.ipa.certs.IPACertRevocation.20190425205849: Request for certificate failed, Certificate operation cannot be completed: EXCEPTION (Invalid Credential.)
ERROR: ipahealthcheck.ipa.certs.IPACertRevocation.20190425205831: Request for certificate failed, Certificate operation cannot be completed: EXCEPTION (Invalid Credential.)
ERROR: ipahealthcheck.ipa.certs.IPACertRevocation.20190425210120: Request for certificate failed, Certificate operation cannot be completed: EXCEPTION (Invalid Credential.)
ERROR: ipahealthcheck.dogtag.ca.DogtagCertsConnectivityCheck: Request for certificate failed, Certificate operation cannot be completed: EXCEPTION (Invalid Credential.)
[root@ipa2 ~]#
The list of certs:
[root@ipa2 ~]# getcert list
Number of certificates and requests being tracked: 9.
Request ID '20190425205831':
status: MONITORING
stuck: no
key pair storage: type=NSSDB,location='/etc/dirsrv/slapd-EXAMPLE-COM',nickname='Server-Cert',token='NSS Certificate DB',pinfile='/etc/dirsrv/slapd-EXAMPLE-COM/pwdfile.txt'
certificate: type=NSSDB,location='/etc/dirsrv/slapd-EXAMPLE-COM',nickname='Server-Cert',token='NSS Certificate DB'
CA: IPA
issuer: CN=Certificate Authority,O=EXAMPLE.COM
subject: CN=ipa2.example.com,O=EXAMPLE.COM
expires: 2023-03-29 21:37:22 UTC
dns: ipa2.example.com
principal name: ldap/ipa2.example.com@EXAMPLE.COM
key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment
eku: id-kp-serverAuth,id-kp-clientAuth
pre-save command:
post-save command: /usr/libexec/ipa/certmonger/restart_dirsrv EXAMPLE-COM
track: yes
auto-renew: yes
Request ID '20190425205849':
status: MONITORING
stuck: no
key pair storage: type=NSSDB,location='/etc/httpd/alias',nickname='Server-Cert',token='NSS Certificate DB',pinfile='/etc/httpd/alias/pwdfile.txt'
certificate: type=NSSDB,location='/etc/httpd/alias',nickname='Server-Cert',token='NSS Certificate DB'
CA: IPA
issuer: CN=Certificate Authority,O=EXAMPLE.COM
subject: CN=ipa2.example.com,O=EXAMPLE.COM
expires: 2023-03-29 21:37:46 UTC
dns: ipa2.example.com
principal name: HTTP/ipa2.example.com@EXAMPLE.COM
key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment
eku: id-kp-serverAuth,id-kp-clientAuth
pre-save command:
post-save command: /usr/libexec/ipa/certmonger/restart_httpd
track: yes
auto-renew: yes
Request ID '20190425210040':
status: MONITORING
stuck: no
key pair storage: type=FILE,location='/var/lib/ipa/ra-agent.key'
certificate: type=FILE,location='/var/lib/ipa/ra-agent.pem'
CA: dogtag-ipa-ca-renew-agent
issuer: CN=Certificate Authority,O=EXAMPLE.COM
subject: CN=IPA RA,O=EXAMPLE.COM
expires: 2022-05-11 03:40:55 UTC
key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment
eku: id-kp-serverAuth,id-kp-clientAuth
pre-save command: /usr/libexec/ipa/certmonger/renew_ra_cert_pre
post-save command: /usr/libexec/ipa/certmonger/renew_ra_cert
track: yes
auto-renew: yes
Request ID '20190425210052':
status: MONITORING
stuck: no
key pair storage: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='auditSigningCert cert-pki-ca',token='NSS Certificate DB',pin set
certificate: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='auditSigningCert cert-pki-ca',token='NSS Certificate DB'
CA: dogtag-ipa-ca-renew-agent
issuer: CN=Certificate Authority,O=EXAMPLE.COM
subject: CN=CA Audit,O=EXAMPLE.COM
expires: 2022-05-11 03:40:05 UTC
key usage: digitalSignature,nonRepudiation
pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad
post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert "auditSigningCert cert-pki-ca"
track: yes
auto-renew: yes
Request ID '20190425210053':
status: MONITORING
stuck: no
key pair storage: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='ocspSigningCert cert-pki-ca',token='NSS Certificate DB',pin set
certificate: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='ocspSigningCert cert-pki-ca',token='NSS Certificate DB'
CA: dogtag-ipa-ca-renew-agent
issuer: CN=Certificate Authority,O=EXAMPLE.COM
subject: CN=OCSP Subsystem,O=EXAMPLE.COM
expires: 2022-05-11 03:40:25 UTC
eku: id-kp-OCSPSigning
pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad
post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert "ocspSigningCert cert-pki-ca"
track: yes
auto-renew: yes
Request ID '20190425210054':
status: MONITORING
stuck: no
key pair storage: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='subsystemCert cert-pki-ca',token='NSS Certificate DB',pin set
certificate: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='subsystemCert cert-pki-ca',token='NSS Certificate DB'
CA: dogtag-ipa-ca-renew-agent
issuer: CN=Certificate Authority,O=EXAMPLE.COM
subject: CN=CA Subsystem,O=EXAMPLE.COM
expires: 2022-05-11 03:40:05 UTC
key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment
eku: id-kp-serverAuth,id-kp-clientAuth
pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad
post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert "subsystemCert cert-pki-ca"
track: yes
auto-renew: yes
Request ID '20190425210055':
status: MONITORING
stuck: no
key pair storage: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='caSigningCert cert-pki-ca',token='NSS Certificate DB',pin set
certificate: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='caSigningCert cert-pki-ca',token='NSS Certificate DB'
CA: dogtag-ipa-ca-renew-agent
issuer: CN=Certificate Authority,O=EXAMPLE.COM
subject: CN=Certificate Authority,O=EXAMPLE.COM
expires: 2038-06-28 21:19:45 UTC
key usage: digitalSignature,nonRepudiation,keyCertSign,cRLSign
pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad
post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert "caSigningCert cert-pki-ca"
track: yes
auto-renew: yes
Request ID '20190425210056':
status: MONITORING
stuck: no
key pair storage: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='Server-Cert cert-pki-ca',token='NSS Certificate DB',pin set
certificate: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='Server-Cert cert-pki-ca',token='NSS Certificate DB'
CA: dogtag-ipa-ca-renew-agent
issuer: CN=Certificate Authority,O=EXAMPLE.COM
subject: CN=ipa2.example.com,O=EXAMPLE.COM
expires: 2023-03-07 22:37:22 UTC
key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment
eku: id-kp-serverAuth,id-kp-clientAuth,id-kp-emailProtection
pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad
post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert "Server-Cert cert-pki-ca"
track: yes
auto-renew: yes
Request ID '20190425210120':
status: MONITORING
stuck: no
key pair storage: type=FILE,location='/var/kerberos/krb5kdc/kdc.key'
certificate: type=FILE,location='/var/kerberos/krb5kdc/kdc.crt'
CA: IPA
issuer: CN=Certificate Authority,O=EXAMPLE.COM
subject: CN=ipa2.example.com,O=EXAMPLE.COM
expires: 2023-03-29 21:37:52 UTC
principal name: krbtgt/EXAMPLE.COM@EXAMPLE.COM
key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment
eku: id-kp-serverAuth,id-pkinit-KPKdc
pre-save command:
post-save command: /usr/libexec/ipa/certmonger/renew_kdc_cert
track: yes
auto-renew: yes
[root@ipa2 ~]#
There are 4 certs which expire on 2022-05-11 which match "expires in 27 days". Take 20190425210040 as an example, we have:
WARNING: ipahealthcheck.ipa.certs.IPACertmongerExpirationCheck.20190425210040: Request id 20190425210040 expires in 27 days
WARNING: ipahealthcheck.ipa.certs.IPACertfileExpirationCheck.20190425210040: Request id 20190425210040 expires in 27 days
ERROR: ipahealthcheck.ipa.certs.IPACertRevocation.20190425210040: Request for certificate failed, Certificate operation cannot be completed: EXCEPTION (Invalid Credential.)
Request ID '20190425210040':
status: MONITORING
stuck: no
key pair storage: type=FILE,location='/var/lib/ipa/ra-agent.key'
certificate: type=FILE,location='/var/lib/ipa/ra-agent.pem'
CA: dogtag-ipa-ca-renew-agent
issuer: CN=Certificate Authority,O=EXAMPLE.COM
subject: CN=IPA RA,O=EXAMPLE.COM
expires: 2022-05-11 03:40:55 UTC
key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment
eku: id-kp-serverAuth,id-kp-clientAuth
pre-save command: /usr/libexec/ipa/certmonger/renew_ra_cert_pre
post-save command: /usr/libexec/ipa/certmonger/renew_ra_cert
track: yes
auto-renew: yes
I was able to manually renew it:
[root@ipa2 ~]# ipa-getcert resubmit -i '20190425210040'
Resubmitting "20190425210040" to "dogtag-ipa-ca-renew-agent".
[root@ipa2 ~]#
After renew, it "expires: 2024-04-02 06:09:32 UTC":
Request ID '20190425210040':
status: MONITORING
stuck: no
key pair storage: type=FILE,location='/var/lib/ipa/ra-agent.key'
certificate: type=FILE,location='/var/lib/ipa/ra-agent.pem'
CA: dogtag-ipa-ca-renew-agent
issuer: CN=Certificate Authority,O=EXAMPLE.COM
subject: CN=IPA RA,O=EXAMPLE.COM
expires: 2024-04-02 06:09:32 UTC
key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment
eku: id-kp-serverAuth,id-kp-clientAuth
pre-save command: /usr/libexec/ipa/certmonger/renew_ra_cert_pre
post-save command: /usr/libexec/ipa/certmonger/renew_ra_cert
track: yes
auto-renew: yes
How to fix the issue reported by ipa-healthcheck? And what is this issue about?
All IPA servers are at same level:
CentOS Linux release 7.9.2009 (Core)
ipa-*server*.x86_64 4.6.8-5.el7.centos.7
*slapi-nis*.x86_64 0.56.5-3.el7_9
*389-ds-base*.x86_64 1.3.10.2-12.el7_9
*389-ds-base*-libs.x86_64 1.3.10.2-12.el7_9
Many thanks!
Kathy.
I just found this post about the same or similar issue:
https://lists.fedoraproject.org/archives/list/freeipa-users@lists.fedorahost...
One detail I missed - this happens on all IPA servers BUT the renewal IPA server. I will go through ^ post to see if that applies to our situation.
Thanks.
Kathy.
On Wed, Apr 13, 2022 at 10:21 AM Kathy Zhu wrote:
Hi team,
ipa-healthcheck has been a great tool for us. I run it weekly on all IPA servers via cron. This week ipa-healthcheck reported errors on all IPA servers.
Take IPA server ipa2 as an example for the investigation:
[root@ipa2 ~]# ipa-healthcheck --failures-only --output-type=human
ra.get_certificate(): EXCEPTION (Invalid Credential.)
ra.get_certificate(): EXCEPTION (Invalid Credential.)
ra.get_certificate(): EXCEPTION (Invalid Credential.)
ra.get_certificate(): EXCEPTION (Invalid Credential.)
ra.get_certificate(): EXCEPTION (Invalid Credential.)
ra.get_certificate(): EXCEPTION (Invalid Credential.)
ra.get_certificate(): EXCEPTION (Invalid Credential.)
ra.get_certificate(): EXCEPTION (Invalid Credential.)
ra.get_certificate(): EXCEPTION (Invalid Credential.)
ra.get_certificate(): EXCEPTION (Invalid Credential.)
WARNING: ipahealthcheck.ipa.certs.IPACertmongerExpirationCheck.20190425210040: Request id 20190425210040 expires in 27 days
WARNING: ipahealthcheck.ipa.certs.IPACertmongerExpirationCheck.20190425210052: Request id 20190425210052 expires in 27 days
WARNING: ipahealthcheck.ipa.certs.IPACertmongerExpirationCheck.20190425210053: Request id 20190425210053 expires in 27 days
WARNING: ipahealthcheck.ipa.certs.IPACertmongerExpirationCheck.20190425210054: Request id 20190425210054 expires in 27 days
WARNING: ipahealthcheck.ipa.certs.IPACertfileExpirationCheck.20190425210040: Request id 20190425210040 expires in 27 days
WARNING: ipahealthcheck.ipa.certs.IPACertfileExpirationCheck.20190425210052: Request id 20190425210052 expires in 27 days
WARNING: ipahealthcheck.ipa.certs.IPACertfileExpirationCheck.20190425210053: Request id 20190425210053 expires in 27 days
WARNING: ipahealthcheck.ipa.certs.IPACertfileExpirationCheck.20190425210054: Request id 20190425210054 expires in 27 days
ERROR: ipahealthcheck.ipa.certs.IPARAAgent: RA agent description does not match 2;186;CN=Certificate Authority,O=EXAMPLE.COM;CN=IPA RA,O=EXAMPLE.COM in LDAP and 2;66;CN=Certificate Authority,O=EXAMPLE.COM;CN=IPA RA,O= EXAMPLE.COM expected
ERROR: ipahealthcheck.ipa.certs.IPACertRevocation.20190425210040: Request for certificate failed, Certificate operation cannot be completed: EXCEPTION (Invalid Credential.)
ERROR: ipahealthcheck.ipa.certs.IPACertRevocation.20190425210052: Request for certificate failed, Certificate operation cannot be completed: EXCEPTION (Invalid Credential.)
ERROR: ipahealthcheck.ipa.certs.IPACertRevocation.20190425210053: Request for certificate failed, Certificate operation cannot be completed: EXCEPTION (Invalid Credential.)
ERROR: ipahealthcheck.ipa.certs.IPACertRevocation.20190425210054: Request for certificate failed, Certificate operation cannot be completed: EXCEPTION (Invalid Credential.)
ERROR: ipahealthcheck.ipa.certs.IPACertRevocation.20190425210055: Request for certificate failed, Certificate operation cannot be completed: EXCEPTION (Invalid Credential.)
ERROR: ipahealthcheck.ipa.certs.IPACertRevocation.20190425210056: Request for certificate failed, Certificate operation cannot be completed: EXCEPTION (Invalid Credential.)
ERROR: ipahealthcheck.ipa.certs.IPACertRevocation.20190425205849: Request for certificate failed, Certificate operation cannot be completed: EXCEPTION (Invalid Credential.)
ERROR: ipahealthcheck.ipa.certs.IPACertRevocation.20190425205831: Request for certificate failed, Certificate operation cannot be completed: EXCEPTION (Invalid Credential.)
ERROR: ipahealthcheck.ipa.certs.IPACertRevocation.20190425210120: Request for certificate failed, Certificate operation cannot be completed: EXCEPTION (Invalid Credential.)
ERROR: ipahealthcheck.dogtag.ca.DogtagCertsConnectivityCheck: Request for certificate failed, Certificate operation cannot be completed: EXCEPTION (Invalid Credential.)
[root@ipa2 ~]#
The list of certs:
[root@ipa2 ~]# getcert list
Number of certificates and requests being tracked: 9.
Request ID '20190425205831':
status: MONITORING
stuck: no
key pair storage: type=NSSDB,location='/etc/dirsrv/slapd-EXAMPLE-COM',nickname='Server-Cert',token='NSS Certificate DB',pinfile='/etc/dirsrv/slapd-EXAMPLE-COM/pwdfile.txt'
certificate: type=NSSDB,location='/etc/dirsrv/slapd-EXAMPLE-COM',nickname='Server-Cert',token='NSS Certificate DB'
CA: IPA
issuer: CN=Certificate Authority,O=EXAMPLE.COM
subject: CN=ipa2.example.com,O=EXAMPLE.COM
expires: 2023-03-29 21:37:22 UTC
dns: ipa2.example.com
principal name: ldap/ipa2.example.com@EXAMPLE.COM
key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment
eku: id-kp-serverAuth,id-kp-clientAuth
pre-save command:
post-save command: /usr/libexec/ipa/certmonger/restart_dirsrv EXAMPLE-COM
track: yes
auto-renew: yes
Request ID '20190425205849':
status: MONITORING
stuck: no
key pair storage: type=NSSDB,location='/etc/httpd/alias',nickname='Server-Cert',token='NSS Certificate DB',pinfile='/etc/httpd/alias/pwdfile.txt'
certificate: type=NSSDB,location='/etc/httpd/alias',nickname='Server-Cert',token='NSS Certificate DB'
CA: IPA
issuer: CN=Certificate Authority,O=EXAMPLE.COM
subject: CN=ipa2.example.com,O=EXAMPLE.COM
expires: 2023-03-29 21:37:46 UTC
dns: ipa2.example.com
principal name: HTTP/ipa2.example.com@EXAMPLE.COM
key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment
eku: id-kp-serverAuth,id-kp-clientAuth
pre-save command:
post-save command: /usr/libexec/ipa/certmonger/restart_httpd
track: yes
auto-renew: yes
Request ID '20190425210040':
status: MONITORING
stuck: no
key pair storage: type=FILE,location='/var/lib/ipa/ra-agent.key'
certificate: type=FILE,location='/var/lib/ipa/ra-agent.pem'
CA: dogtag-ipa-ca-renew-agent
issuer: CN=Certificate Authority,O=EXAMPLE.COM
subject: CN=IPA RA,O=EXAMPLE.COM
expires: 2022-05-11 03:40:55 UTC
key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment
eku: id-kp-serverAuth,id-kp-clientAuth
pre-save command: /usr/libexec/ipa/certmonger/renew_ra_cert_pre
post-save command: /usr/libexec/ipa/certmonger/renew_ra_cert
track: yes
auto-renew: yes
Request ID '20190425210052':
status: MONITORING
stuck: no
key pair storage: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='auditSigningCert cert-pki-ca',token='NSS Certificate DB',pin set
certificate: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='auditSigningCert cert-pki-ca',token='NSS Certificate DB'
CA: dogtag-ipa-ca-renew-agent
issuer: CN=Certificate Authority,O=EXAMPLE.COM
subject: CN=CA Audit,O=EXAMPLE.COM
expires: 2022-05-11 03:40:05 UTC
key usage: digitalSignature,nonRepudiation
pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad
post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert "auditSigningCert cert-pki-ca"
track: yes
auto-renew: yes
Request ID '20190425210053':
status: MONITORING
stuck: no
key pair storage: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='ocspSigningCert cert-pki-ca',token='NSS Certificate DB',pin set
certificate: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='ocspSigningCert cert-pki-ca',token='NSS Certificate DB'
CA: dogtag-ipa-ca-renew-agent
issuer: CN=Certificate Authority,O=EXAMPLE.COM
subject: CN=OCSP Subsystem,O=EXAMPLE.COM
expires: 2022-05-11 03:40:25 UTC
eku: id-kp-OCSPSigning
pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad
post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert "ocspSigningCert cert-pki-ca"
track: yes
auto-renew: yes
Request ID '20190425210054':
status: MONITORING
stuck: no
key pair storage: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='subsystemCert cert-pki-ca',token='NSS Certificate DB',pin set
certificate: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='subsystemCert cert-pki-ca',token='NSS Certificate DB'
CA: dogtag-ipa-ca-renew-agent
issuer: CN=Certificate Authority,O=EXAMPLE.COM
subject: CN=CA Subsystem,O=EXAMPLE.COM
expires: 2022-05-11 03:40:05 UTC
key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment
eku: id-kp-serverAuth,id-kp-clientAuth
pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad
post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert "subsystemCert cert-pki-ca"
track: yes
auto-renew: yes
Request ID '20190425210055':
status: MONITORING
stuck: no
key pair storage: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='caSigningCert cert-pki-ca',token='NSS Certificate DB',pin set
certificate: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='caSigningCert cert-pki-ca',token='NSS Certificate DB'
CA: dogtag-ipa-ca-renew-agent
issuer: CN=Certificate Authority,O=EXAMPLE.COM
subject: CN=Certificate Authority,O=EXAMPLE.COM
expires: 2038-06-28 21:19:45 UTC
key usage: digitalSignature,nonRepudiation,keyCertSign,cRLSign
pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad
post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert "caSigningCert cert-pki-ca"
track: yes
auto-renew: yes
Request ID '20190425210056':
status: MONITORING
stuck: no
key pair storage: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='Server-Cert cert-pki-ca',token='NSS Certificate DB',pin set
certificate: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='Server-Cert cert-pki-ca',token='NSS Certificate DB'
CA: dogtag-ipa-ca-renew-agent
issuer: CN=Certificate Authority,O=EXAMPLE.COM
subject: CN=ipa2.example.com,O=EXAMPLE.COM
expires: 2023-03-07 22:37:22 UTC
key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment
eku: id-kp-serverAuth,id-kp-clientAuth,id-kp-emailProtection
pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad
post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert "Server-Cert cert-pki-ca"
track: yes
auto-renew: yes
Request ID '20190425210120':
status: MONITORING
stuck: no
key pair storage: type=FILE,location='/var/kerberos/krb5kdc/kdc.key'
certificate: type=FILE,location='/var/kerberos/krb5kdc/kdc.crt'
CA: IPA
issuer: CN=Certificate Authority,O=EXAMPLE.COM
subject: CN=ipa2.example.com,O=EXAMPLE.COM
expires: 2023-03-29 21:37:52 UTC
principal name: krbtgt/EXAMPLE.COM@EXAMPLE.COM
key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment
eku: id-kp-serverAuth,id-pkinit-KPKdc
pre-save command:
post-save command: /usr/libexec/ipa/certmonger/renew_kdc_cert
track: yes
auto-renew: yes
[root@ipa2 ~]#
There are 4 certs which expire on 2022-05-11 which match "expires in 27 days". Take 20190425210040 as an example, we have:
WARNING: ipahealthcheck.ipa.certs.IPACertmongerExpirationCheck.20190425210040: Request id 20190425210040 expires in 27 days
WARNING: ipahealthcheck.ipa.certs.IPACertfileExpirationCheck.20190425210040: Request id 20190425210040 expires in 27 days
ERROR: ipahealthcheck.ipa.certs.IPACertRevocation.20190425210040: Request for certificate failed, Certificate operation cannot be completed: EXCEPTION (Invalid Credential.)
Request ID '20190425210040':
status: MONITORING stuck: no key pair storage: type=FILE,location='/var/lib/ipa/ra-agent.key' certificate: type=FILE,location='/var/lib/ipa/ra-agent.pem' CA: dogtag-ipa-ca-renew-agent issuer: CN=Certificate Authority,O=EXAMPLE.COM subject: CN=IPA RA,O=EXAMPLE.COM expires: 2022-05-11 03:40:55 UTC key usage:digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment
eku: id-kp-serverAuth,id-kp-clientAuth pre-save command: /usr/libexec/ipa/certmonger/renew_ra_cert_pre post-save command: /usr/libexec/ipa/certmonger/renew_ra_cert track: yes auto-renew: yesI was able to manually renew it:
[root@ipa2 ~]# ipa-getcert resubmit -i '20190425210040'
Resubmitting "20190425210040" to "dogtag-ipa-ca-renew-agent".
[root@ipa2 ~]#
After renew, it "expires: 2024-04-02 06:09:32 UTC":
Request ID '20190425210040':
status: MONITORING stuck: no key pair storage: type=FILE,location='/var/lib/ipa/ra-agent.key' certificate: type=FILE,location='/var/lib/ipa/ra-agent.pem' CA: dogtag-ipa-ca-renew-agent issuer: CN=Certificate Authority,O=EXAMPLE.COM subject: CN=IPA RA,O=EXAMPLE.COM expires: 2024-04-02 06:09:32 UTC key usage:digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment
eku: id-kp-serverAuth,id-kp-clientAuth pre-save command: /usr/libexec/ipa/certmonger/renew_ra_cert_pre post-save command: /usr/libexec/ipa/certmonger/renew_ra_cert track: yes auto-renew: yesHow to fix the issue reported by ipa-healthcheck? And what is this issue about?
All IPA servers are at same level:
CentOS Linux release 7.9.2009 (Core)
ipa-*server*.x86_64 4.6.8-5.el7.centos.7
*slapi-nis*.x86_64 0.56.5-3.el7_9
*389-ds-base*.x86_64 1.3.10.2-12.el7_9
*389-ds-base*-libs.x86_64 1.3.10.2-12.el7_9
Many thanks!
Kathy.
This looks like the root cause:
ERROR: ipahealthcheck.ipa.certs.IPARAAgent: RA agent description does not match 2;186;CN=Certificate Authority,O=EXAMPLE.COM;CN=IPA RA,O=EXAMPLE.COM in LDAP and 2;66;CN=Certificate Authority,O=EXAMPLE.COM;CN=IPA RA,O=EXAMPLE.COM expected
It looks like an updated RA certificate was issued and was not picked up by the mother server(s).
Off the top of my head this could be:
- replication of the CA data has a problem: ipa-csreplica-manage list -v `hostname` - The updated certificate wasn't published to cn=certificates,cn=ipa,cn=etc,$SUFFIX - certmonger isn't picking up the renewal for some reason. The journal may hold clues. - something I'm forgetting
I'd start with the first two.
rob
Kathy Zhu via FreeIPA-users wrote:
I just found this post about the same or similar issue:
https://lists.fedoraproject.org/archives/list/freeipa-users@lists.fedorahost...
One detail I missed - this happens on all IPA servers BUT the renewal IPA server. I will go through ^ post to see if that applies to our situation.
Thanks.
Kathy.
On Wed, Apr 13, 2022 at 10:21 AM Kathy Zhu wrote:
Hi team, ipa-healthcheck has been a great tool for us. I run it weekly on all IPA servers via cron. This week ipa-healthcheck reported errors on all IPA servers. Take IPA server ipa2 as an example for the investigation: [root@ipa2 ~]# ipa-healthcheck --failures-only --output-type=human ra.get_certificate(): EXCEPTION (Invalid Credential.) ra.get_certificate(): EXCEPTION (Invalid Credential.) ra.get_certificate(): EXCEPTION (Invalid Credential.) ra.get_certificate(): EXCEPTION (Invalid Credential.) ra.get_certificate(): EXCEPTION (Invalid Credential.) ra.get_certificate(): EXCEPTION (Invalid Credential.) ra.get_certificate(): EXCEPTION (Invalid Credential.) ra.get_certificate(): EXCEPTION (Invalid Credential.) ra.get_certificate(): EXCEPTION (Invalid Credential.) ra.get_certificate(): EXCEPTION (Invalid Credential.) WARNING: ipahealthcheck.ipa.certs.IPACertmongerExpirationCheck.20190425210040: Request id 20190425210040 expires in 27 days WARNING: ipahealthcheck.ipa.certs.IPACertmongerExpirationCheck.20190425210052: Request id 20190425210052 expires in 27 days WARNING: ipahealthcheck.ipa.certs.IPACertmongerExpirationCheck.20190425210053: Request id 20190425210053 expires in 27 days WARNING: ipahealthcheck.ipa.certs.IPACertmongerExpirationCheck.20190425210054: Request id 20190425210054 expires in 27 days WARNING: ipahealthcheck.ipa.certs.IPACertfileExpirationCheck.20190425210040: Request id 20190425210040 expires in 27 days WARNING: ipahealthcheck.ipa.certs.IPACertfileExpirationCheck.20190425210052: Request id 20190425210052 expires in 27 days WARNING: ipahealthcheck.ipa.certs.IPACertfileExpirationCheck.20190425210053: Request id 20190425210053 expires in 27 days WARNING: ipahealthcheck.ipa.certs.IPACertfileExpirationCheck.20190425210054: Request id 20190425210054 expires in 27 days ERROR: ipahealthcheck.ipa.certs.IPARAAgent: RA agent description does not match 2;186;CN=Certificate Authority,O=EXAMPLE.COM <http://EXAMPLE.COM>;CN=IPA RA,O=EXAMPLE.COM <http://EXAMPLE.COM> in LDAP and 2;66;CN=Certificate Authority,O=EXAMPLE.COM <http://EXAMPLE.COM>;CN=IPA RA,O=EXAMPLE.COM <http://EXAMPLE.COM> expected ERROR: ipahealthcheck.ipa.certs.IPACertRevocation.20190425210040: Request for certificate failed, Certificate operation cannot be completed: EXCEPTION (Invalid Credential.) ERROR: ipahealthcheck.ipa.certs.IPACertRevocation.20190425210052: Request for certificate failed, Certificate operation cannot be completed: EXCEPTION (Invalid Credential.) ERROR: ipahealthcheck.ipa.certs.IPACertRevocation.20190425210053: Request for certificate failed, Certificate operation cannot be completed: EXCEPTION (Invalid Credential.) ERROR: ipahealthcheck.ipa.certs.IPACertRevocation.20190425210054: Request for certificate failed, Certificate operation cannot be completed: EXCEPTION (Invalid Credential.) ERROR: ipahealthcheck.ipa.certs.IPACertRevocation.20190425210055: Request for certificate failed, Certificate operation cannot be completed: EXCEPTION (Invalid Credential.) ERROR: ipahealthcheck.ipa.certs.IPACertRevocation.20190425210056: Request for certificate failed, Certificate operation cannot be completed: EXCEPTION (Invalid Credential.) ERROR: ipahealthcheck.ipa.certs.IPACertRevocation.20190425205849: Request for certificate failed, Certificate operation cannot be completed: EXCEPTION (Invalid Credential.) ERROR: ipahealthcheck.ipa.certs.IPACertRevocation.20190425205831: Request for certificate failed, Certificate operation cannot be completed: EXCEPTION (Invalid Credential.) ERROR: ipahealthcheck.ipa.certs.IPACertRevocation.20190425210120: Request for certificate failed, Certificate operation cannot be completed: EXCEPTION (Invalid Credential.) ERROR: ipahealthcheck.dogtag.ca.DogtagCertsConnectivityCheck: Request for certificate failed, Certificate operation cannot be completed: EXCEPTION (Invalid Credential.) [root@ipa2 ~]# The list of certs: [root@ipa2 ~]# getcert list Number of certificates and requests being tracked: 9. Request ID '20190425205831': status: MONITORING stuck: no key pair storage: type=NSSDB,location='/etc/dirsrv/slapd-EXAMPLE-COM',nickname='Server-Cert',token='NSS Certificate DB',pinfile='/etc/dirsrv/slapd-EXAMPLE-COM/pwdfile.txt' certificate: type=NSSDB,location='/etc/dirsrv/slapd-EXAMPLE-COM',nickname='Server-Cert',token='NSS Certificate DB' CA: IPA issuer: CN=Certificate Authority,O=EXAMPLE.COM <http://EXAMPLE.COM> subject: CN=ipa2.example.com <http://ipa2.example.com>,O=EXAMPLE.COM <http://EXAMPLE.COM> expires: 2023-03-29 21:37:22 UTC dns: ipa2.example.com <http://ipa2.example.com> principal name: ldap/ipa2.example.com@EXAMPLE.COM <mailto:ipa2.example.com@EXAMPLE.COM> key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment eku: id-kp-serverAuth,id-kp-clientAuth pre-save command: post-save command: /usr/libexec/ipa/certmonger/restart_dirsrv EXAMPLE-COM track: yes auto-renew: yes Request ID '20190425205849': status: MONITORING stuck: no key pair storage: type=NSSDB,location='/etc/httpd/alias',nickname='Server-Cert',token='NSS Certificate DB',pinfile='/etc/httpd/alias/pwdfile.txt' certificate: type=NSSDB,location='/etc/httpd/alias',nickname='Server-Cert',token='NSS Certificate DB' CA: IPA issuer: CN=Certificate Authority,O=EXAMPLE.COM <http://EXAMPLE.COM> subject: CN=ipa2.example.com <http://ipa2.example.com>,O=EXAMPLE.COM <http://EXAMPLE.COM> expires: 2023-03-29 21:37:46 UTC dns: ipa2.example.com <http://ipa2.example.com> principal name: HTTP/ipa2.example.com@EXAMPLE.COM <mailto:ipa2.example.com@EXAMPLE.COM> key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment eku: id-kp-serverAuth,id-kp-clientAuth pre-save command: post-save command: /usr/libexec/ipa/certmonger/restart_httpd track: yes auto-renew: yes Request ID '20190425210040': status: MONITORING stuck: no key pair storage: type=FILE,location='/var/lib/ipa/ra-agent.key' certificate: type=FILE,location='/var/lib/ipa/ra-agent.pem' CA: dogtag-ipa-ca-renew-agent issuer: CN=Certificate Authority,O=EXAMPLE.COM <http://EXAMPLE.COM> subject: CN=IPA RA,O=EXAMPLE.COM <http://EXAMPLE.COM> expires: 2022-05-11 03:40:55 UTC key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment eku: id-kp-serverAuth,id-kp-clientAuth pre-save command: /usr/libexec/ipa/certmonger/renew_ra_cert_pre post-save command: /usr/libexec/ipa/certmonger/renew_ra_cert track: yes auto-renew: yes Request ID '20190425210052': status: MONITORING stuck: no key pair storage: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='auditSigningCert cert-pki-ca',token='NSS Certificate DB',pin set certificate: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='auditSigningCert cert-pki-ca',token='NSS Certificate DB' CA: dogtag-ipa-ca-renew-agent issuer: CN=Certificate Authority,O=EXAMPLE.COM <http://EXAMPLE.COM> subject: CN=CA Audit,O=EXAMPLE.COM <http://EXAMPLE.COM> expires: 2022-05-11 03:40:05 UTC key usage: digitalSignature,nonRepudiation pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert "auditSigningCert cert-pki-ca" track: yes auto-renew: yes Request ID '20190425210053': status: MONITORING stuck: no key pair storage: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='ocspSigningCert cert-pki-ca',token='NSS Certificate DB',pin set certificate: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='ocspSigningCert cert-pki-ca',token='NSS Certificate DB' CA: dogtag-ipa-ca-renew-agent issuer: CN=Certificate Authority,O=EXAMPLE.COM <http://EXAMPLE.COM> subject: CN=OCSP Subsystem,O=EXAMPLE.COM <http://EXAMPLE.COM> expires: 2022-05-11 03:40:25 UTC eku: id-kp-OCSPSigning pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert "ocspSigningCert cert-pki-ca" track: yes auto-renew: yes Request ID '20190425210054': status: MONITORING stuck: no key pair storage: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='subsystemCert cert-pki-ca',token='NSS Certificate DB',pin set certificate: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='subsystemCert cert-pki-ca',token='NSS Certificate DB' CA: dogtag-ipa-ca-renew-agent issuer: CN=Certificate Authority,O=EXAMPLE.COM <http://EXAMPLE.COM> subject: CN=CA Subsystem,O=EXAMPLE.COM <http://EXAMPLE.COM> expires: 2022-05-11 03:40:05 UTC key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment eku: id-kp-serverAuth,id-kp-clientAuth pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert "subsystemCert cert-pki-ca" track: yes auto-renew: yes Request ID '20190425210055': status: MONITORING stuck: no key pair storage: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='caSigningCert cert-pki-ca',token='NSS Certificate DB',pin set certificate: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='caSigningCert cert-pki-ca',token='NSS Certificate DB' CA: dogtag-ipa-ca-renew-agent issuer: CN=Certificate Authority,O=EXAMPLE.COM <http://EXAMPLE.COM> subject: CN=Certificate Authority,O=EXAMPLE.COM <http://EXAMPLE.COM> expires: 2038-06-28 21:19:45 UTC key usage: digitalSignature,nonRepudiation,keyCertSign,cRLSign pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert "caSigningCert cert-pki-ca" track: yes auto-renew: yes Request ID '20190425210056': status: MONITORING stuck: no key pair storage: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='Server-Cert cert-pki-ca',token='NSS Certificate DB',pin set certificate: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='Server-Cert cert-pki-ca',token='NSS Certificate DB' CA: dogtag-ipa-ca-renew-agent issuer: CN=Certificate Authority,O=EXAMPLE.COM <http://EXAMPLE.COM> subject: CN=ipa2.example.com <http://ipa2.example.com>,O=EXAMPLE.COM <http://EXAMPLE.COM> expires: 2023-03-07 22:37:22 UTC key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment eku: id-kp-serverAuth,id-kp-clientAuth,id-kp-emailProtection pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert "Server-Cert cert-pki-ca" track: yes auto-renew: yes Request ID '20190425210120': status: MONITORING stuck: no key pair storage: type=FILE,location='/var/kerberos/krb5kdc/kdc.key' certificate: type=FILE,location='/var/kerberos/krb5kdc/kdc.crt' CA: IPA issuer: CN=Certificate Authority,O=EXAMPLE.COM <http://EXAMPLE.COM> subject: CN=ipa2.example.com <http://ipa2.example.com>,O=EXAMPLE.COM <http://EXAMPLE.COM> expires: 2023-03-29 21:37:52 UTC principal name: krbtgt/EXAMPLE.COM@EXAMPLE.COM <mailto:EXAMPLE.COM@EXAMPLE.COM> key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment eku: id-kp-serverAuth,id-pkinit-KPKdc pre-save command: post-save command: /usr/libexec/ipa/certmonger/renew_kdc_cert track: yes auto-renew: yes [root@ipa2 ~]# There are 4 certs which expire on 2022-05-11 which match "expires in 27 days". Take 20190425210040 as an example, we have: WARNING: ipahealthcheck.ipa.certs.IPACertmongerExpirationCheck.20190425210040: Request id 20190425210040 expires in 27 days WARNING: ipahealthcheck.ipa.certs.IPACertfileExpirationCheck.20190425210040: Request id 20190425210040 expires in 27 days ERROR: ipahealthcheck.ipa.certs.IPACertRevocation.20190425210040: Request for certificate failed, Certificate operation cannot be completed: EXCEPTION (Invalid Credential.) Request ID '20190425210040': status: MONITORING stuck: no key pair storage: type=FILE,location='/var/lib/ipa/ra-agent.key' certificate: type=FILE,location='/var/lib/ipa/ra-agent.pem' CA: dogtag-ipa-ca-renew-agent issuer: CN=Certificate Authority,O=EXAMPLE.COM <http://EXAMPLE.COM> subject: CN=IPA RA,O=EXAMPLE.COM <http://EXAMPLE.COM> expires: 2022-05-11 03:40:55 UTC key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment eku: id-kp-serverAuth,id-kp-clientAuth pre-save command: /usr/libexec/ipa/certmonger/renew_ra_cert_pre post-save command: /usr/libexec/ipa/certmonger/renew_ra_cert track: yes auto-renew: yes I was able to manually renew it: [root@ipa2 ~]# ipa-getcert resubmit -i '20190425210040' Resubmitting "20190425210040" to "dogtag-ipa-ca-renew-agent". [root@ipa2 ~]# After renew, it "expires: 2024-04-02 06:09:32 UTC": Request ID '20190425210040': status: MONITORING stuck: no key pair storage: type=FILE,location='/var/lib/ipa/ra-agent.key' certificate: type=FILE,location='/var/lib/ipa/ra-agent.pem' CA: dogtag-ipa-ca-renew-agent issuer: CN=Certificate Authority,O=EXAMPLE.COM <http://EXAMPLE.COM> subject: CN=IPA RA,O=EXAMPLE.COM <http://EXAMPLE.COM> expires: 2024-04-02 06:09:32 UTC key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment eku: id-kp-serverAuth,id-kp-clientAuth pre-save command: /usr/libexec/ipa/certmonger/renew_ra_cert_pre post-save command: /usr/libexec/ipa/certmonger/renew_ra_cert track: yes auto-renew: yes How to fix the issue reported by ipa-healthcheck? And what is this issue about? All IPA servers are at same level: CentOS Linux release 7.9.2009 (Core) ipa-*server*.x86_64 4.6.8-5.el7.centos.7 *slapi-nis*.x86_64 0.56.5-3.el7_9 *389-ds-base*.x86_64 1.3.10.2-12.el7_9 *389-ds-base*-libs.x86_64 1.3.10.2-12.el7_9 Many thanks! Kathy.
FreeIPA-users mailing list -- freeipa-users@lists.fedorahosted.org To unsubscribe send an email to freeipa-users-leave@lists.fedorahosted.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedorahoste... Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Hi Rob,
Thank you for the insight! That helped a lot!
The replication of the CA data was perfectly fine. I did the following on each IPA server except the renewal server to fix the situation:
1, restarted certmonger service, then waited a few minutes until it finished restarting pki_tomcatd (it may not restart pki_tomcatd right away), it did two restarts, see attached. 2, once see attached which confirmed the finish, ipa-healthcheck would not complain.
I noticed that this has been updated by its time stamp:
-r--r----- 1 root ipaapi 1261 Apr 13 14:13 /var/lib/ipa/ra-agent.pem
My guess of what happened is that /var/lib/ipa/ra-agent.pem had been renewed on the renewal server, the rest of the IPA servers should pick up the change, however, for a reason unknown to me, certmonger on other IPA servers failed to do so. That was why they complained of "not match". Restarting certmonger triggered this action, once done, all is fine.
Rob, please correct if my guess is wrong. Thank you for bringing this wonderful tool to Centos 7.
Many thanks.
Kathy.
[root@ipa2 ~]# systemctl status certmonger -l
● certmonger.service - Certificate monitoring and PKI enrollment
Loaded: loaded (/usr/lib/systemd/system/certmonger.service; enabled; vendor preset: disabled)
Active: active (running) since Wed 2022-04-13 14:12:18 PDT; 12min ago
Main PID: 16980 (certmonger)
CGroup: /system.slice/certmonger.service
└─16980 /usr/sbin/certmonger -S -p /var/run/certmonger.pid -n
Apr 13 14:13:05 ipa2.example.com stop_pkicad[17729]: Stopping pki_tomcatd
Apr 13 14:13:06 ipa2.example.com stop_pkicad[17729]: Stopped pki_tomcatd
Apr 13 14:13:08 ipa2.example.com renew_ca_cert[17785]: Updated trust on certificate auditSigningCert cert-pki-ca in /etc/pki/pki-tomcat/alias
Apr 13 14:13:08 ipa2.example.com renew_ca_cert[17785]: Starting pki_tomcatd
Apr 13 14:13:23 ipa2.example.com renew_ca_cert[17785]: Started pki_tomcatd
Apr 13 14:13:23 ipa2.example.com certmonger[18098]: Certificate named "auditSigningCert cert-pki-ca" in token "NSS Certificate DB" in database "/etc/pki/pki-tomcat/alias" issued by CA and saved.
Apr 13 14:13:35 ipa2.example.com stop_pkicad[18130]: Stopping pki_tomcatd
Apr 13 14:13:36 ipa2.example.com stop_pkicad[18130]: Stopped pki_tomcatd
Apr 13 14:13:38 ipa2.example.com renew_ca_cert[18185]: Starting pki_tomcatd
Apr 13 14:13:53 ipa2.example.com renew_ca_cert[18185]: Started pki_tomcatd
[root@ipa2 ~]#
On Wed, Apr 13, 2022 at 12:23 PM Rob Crittenden rcritten@redhat.com wrote:
This looks like the root cause:
ERROR: ipahealthcheck.ipa.certs.IPARAAgent: RA agent description does not match 2;186;CN=Certificate Authority,O=EXAMPLE.COM;CN=IPA RA,O=EXAMPLE.COM in LDAP and 2;66;CN=Certificate Authority,O=EXAMPLE.COM;CN=IPA RA,O=EXAMPLE.COM expected
It looks like an updated RA certificate was issued and was not picked up by the mother server(s).
Off the top of my head this could be:
- replication of the CA data has a problem: ipa-csreplica-manage list -v
`hostname`
- The updated certificate wasn't published to
cn=certificates,cn=ipa,cn=etc,$SUFFIX
- certmonger isn't picking up the renewal for some reason. The journal
may hold clues.
- something I'm forgetting
I'd start with the first two.
rob
Kathy Zhu via FreeIPA-users wrote:
I just found this post about the same or similar issue:
https://lists.fedoraproject.org/archives/list/freeipa-users@lists.fedorahost...
One detail I missed - this happens on all IPA servers BUT the renewal IPA server. I will go through ^ post to see if that applies to our situation.
Thanks.
Kathy.
On Wed, Apr 13, 2022 at 10:21 AM Kathy Zhu wrote:
Hi team, ipa-healthcheck has been a great tool for us. I run it weekly on all IPA servers via cron. This week ipa-healthcheck reported errors on all IPA servers. Take IPA server ipa2 as an example for the investigation: [root@ipa2 ~]# ipa-healthcheck --failures-only --output-type=human ra.get_certificate(): EXCEPTION (Invalid Credential.) ra.get_certificate(): EXCEPTION (Invalid Credential.) ra.get_certificate(): EXCEPTION (Invalid Credential.) ra.get_certificate(): EXCEPTION (Invalid Credential.) ra.get_certificate(): EXCEPTION (Invalid Credential.) ra.get_certificate(): EXCEPTION (Invalid Credential.) ra.get_certificate(): EXCEPTION (Invalid Credential.) ra.get_certificate(): EXCEPTION (Invalid Credential.) ra.get_certificate(): EXCEPTION (Invalid Credential.) ra.get_certificate(): EXCEPTION (Invalid Credential.) WARNING:ipahealthcheck.ipa.certs.IPACertmongerExpirationCheck.20190425210040: Request
id 20190425210040 expires in 27 days WARNING:ipahealthcheck.ipa.certs.IPACertmongerExpirationCheck.20190425210052: Request
id 20190425210052 expires in 27 days WARNING:ipahealthcheck.ipa.certs.IPACertmongerExpirationCheck.20190425210053: Request
id 20190425210053 expires in 27 days WARNING:ipahealthcheck.ipa.certs.IPACertmongerExpirationCheck.20190425210054: Request
id 20190425210054 expires in 27 days WARNING: ipahealthcheck.ipa.certs.IPACertfileExpirationCheck.20190425210040: Request id 20190425210040 expires in 27 days WARNING: ipahealthcheck.ipa.certs.IPACertfileExpirationCheck.20190425210052: Request id 20190425210052 expires in 27 days WARNING: ipahealthcheck.ipa.certs.IPACertfileExpirationCheck.20190425210053: Request id 20190425210053 expires in 27 days WARNING: ipahealthcheck.ipa.certs.IPACertfileExpirationCheck.20190425210054: Request id 20190425210054 expires in 27 days ERROR: ipahealthcheck.ipa.certs.IPARAAgent: RA agent description does not match 2;186;CN=Certificate Authority,O=EXAMPLE.COM <http://EXAMPLE.COM>;CN=IPA RA,O=EXAMPLE.COM <http://EXAMPLE.COM> in LDAP and 2;66;CN=Certificate Authority,O=EXAMPLE.COM <http://EXAMPLE.COM>;CN=IPA RA,O=EXAMPLE.COM <http://EXAMPLE.COM> expected ERROR: ipahealthcheck.ipa.certs.IPACertRevocation.20190425210040: Request for certificate failed, Certificate operation cannot be completed: EXCEPTION (Invalid Credential.) ERROR: ipahealthcheck.ipa.certs.IPACertRevocation.20190425210052: Request for certificate failed, Certificate operation cannot be completed: EXCEPTION (Invalid Credential.) ERROR: ipahealthcheck.ipa.certs.IPACertRevocation.20190425210053: Request for certificate failed, Certificate operation cannot be completed: EXCEPTION (Invalid Credential.) ERROR: ipahealthcheck.ipa.certs.IPACertRevocation.20190425210054: Request for certificate failed, Certificate operation cannot be completed: EXCEPTION (Invalid Credential.) ERROR: ipahealthcheck.ipa.certs.IPACertRevocation.20190425210055: Request for certificate failed, Certificate operation cannot be completed: EXCEPTION (Invalid Credential.) ERROR: ipahealthcheck.ipa.certs.IPACertRevocation.20190425210056: Request for certificate failed, Certificate operation cannot be completed: EXCEPTION (Invalid Credential.) ERROR: ipahealthcheck.ipa.certs.IPACertRevocation.20190425205849: Request for certificate failed, Certificate operation cannot be completed: EXCEPTION (Invalid Credential.) ERROR: ipahealthcheck.ipa.certs.IPACertRevocation.20190425205831: Request for certificate failed, Certificate operation cannot be completed: EXCEPTION (Invalid Credential.) ERROR: ipahealthcheck.ipa.certs.IPACertRevocation.20190425210120: Request for certificate failed, Certificate operation cannot be completed: EXCEPTION (Invalid Credential.) ERROR: ipahealthcheck.dogtag.ca.DogtagCertsConnectivityCheck: Request for certificate failed, Certificate operation cannot be completed: EXCEPTION (Invalid Credential.) [root@ipa2 ~]# The list of certs: [root@ipa2 ~]# getcert list Number of certificates and requests being tracked: 9. Request ID '20190425205831': status: MONITORING stuck: no key pair storage:type=NSSDB,location='/etc/dirsrv/slapd-EXAMPLE-COM',nickname='Server-Cert',token='NSS
Certificate DB',pinfile='/etc/dirsrv/slapd-EXAMPLE-COM/pwdfile.txt' certificate:type=NSSDB,location='/etc/dirsrv/slapd-EXAMPLE-COM',nickname='Server-Cert',token='NSS
Certificate DB' CA: IPA issuer: CN=Certificate Authority,O=EXAMPLE.COM <http://EXAMPLE.COM> subject: CN=ipa2.example.com <http://ipa2.example.com>,O=EXAMPLE.COM <http://EXAMPLE.COM> expires: 2023-03-29 21:37:22 UTC dns: ipa2.example.com <http://ipa2.example.com> principal name: ldap/ipa2.example.com@EXAMPLE.COM <mailto:ipa2.example.com@EXAMPLE.COM> key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment eku: id-kp-serverAuth,id-kp-clientAuth pre-save command: post-save command: /usr/libexec/ipa/certmonger/restart_dirsrv EXAMPLE-COM track: yes auto-renew: yes Request ID '20190425205849': status: MONITORING stuck: no key pair storage:type=NSSDB,location='/etc/httpd/alias',nickname='Server-Cert',token='NSS
Certificate DB',pinfile='/etc/httpd/alias/pwdfile.txt' certificate:type=NSSDB,location='/etc/httpd/alias',nickname='Server-Cert',token='NSS
Certificate DB' CA: IPA issuer: CN=Certificate Authority,O=EXAMPLE.COM <http://EXAMPLE.COM> subject: CN=ipa2.example.com <http://ipa2.example.com>,O=EXAMPLE.COM <http://EXAMPLE.COM> expires: 2023-03-29 21:37:46 UTC dns: ipa2.example.com <http://ipa2.example.com> principal name: HTTP/ipa2.example.com@EXAMPLE.COM <mailto:ipa2.example.com@EXAMPLE.COM> key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment eku: id-kp-serverAuth,id-kp-clientAuth pre-save command: post-save command: /usr/libexec/ipa/certmonger/restart_httpd track: yes auto-renew: yes Request ID '20190425210040': status: MONITORING stuck: no key pair storage: type=FILE,location='/var/lib/ipa/ra-agent.key' certificate: type=FILE,location='/var/lib/ipa/ra-agent.pem' CA: dogtag-ipa-ca-renew-agent issuer: CN=Certificate Authority,O=EXAMPLE.COM <http://EXAMPLE.COM> subject: CN=IPA RA,O=EXAMPLE.COM <http://EXAMPLE.COM> expires: 2022-05-11 03:40:55 UTC key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment eku: id-kp-serverAuth,id-kp-clientAuth pre-save command: /usr/libexec/ipa/certmonger/renew_ra_cert_pre post-save command: /usr/libexec/ipa/certmonger/renew_ra_cert track: yes auto-renew: yes Request ID '20190425210052': status: MONITORING stuck: no key pair storage:type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='auditSigningCert
cert-pki-ca',token='NSS Certificate DB',pin set certificate:type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='auditSigningCert
cert-pki-ca',token='NSS Certificate DB' CA: dogtag-ipa-ca-renew-agent issuer: CN=Certificate Authority,O=EXAMPLE.COM <http://EXAMPLE.COM> subject: CN=CA Audit,O=EXAMPLE.COM <http://EXAMPLE.COM> expires: 2022-05-11 03:40:05 UTC key usage: digitalSignature,nonRepudiation pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert "auditSigningCert cert-pki-ca" track: yes auto-renew: yes Request ID '20190425210053': status: MONITORING stuck: no key pair storage:type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='ocspSigningCert
cert-pki-ca',token='NSS Certificate DB',pin set certificate:type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='ocspSigningCert
cert-pki-ca',token='NSS Certificate DB' CA: dogtag-ipa-ca-renew-agent issuer: CN=Certificate Authority,O=EXAMPLE.COM <http://EXAMPLE.COM> subject: CN=OCSP Subsystem,O=EXAMPLE.COM <http://EXAMPLE.COM> expires: 2022-05-11 03:40:25 UTC eku: id-kp-OCSPSigning pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert "ocspSigningCert cert-pki-ca" track: yes auto-renew: yes Request ID '20190425210054': status: MONITORING stuck: no key pair storage:type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='subsystemCert
cert-pki-ca',token='NSS Certificate DB',pin set certificate:type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='subsystemCert
cert-pki-ca',token='NSS Certificate DB' CA: dogtag-ipa-ca-renew-agent issuer: CN=Certificate Authority,O=EXAMPLE.COM <http://EXAMPLE.COM> subject: CN=CA Subsystem,O=EXAMPLE.COM <http://EXAMPLE.COM> expires: 2022-05-11 03:40:05 UTC key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment eku: id-kp-serverAuth,id-kp-clientAuth pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert "subsystemCert cert-pki-ca" track: yes auto-renew: yes Request ID '20190425210055': status: MONITORING stuck: no key pair storage:type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='caSigningCert
cert-pki-ca',token='NSS Certificate DB',pin set certificate:type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='caSigningCert
cert-pki-ca',token='NSS Certificate DB' CA: dogtag-ipa-ca-renew-agent issuer: CN=Certificate Authority,O=EXAMPLE.COM <http://EXAMPLE.COM> subject: CN=Certificate Authority,O=EXAMPLE.COM <http://EXAMPLE.COM> expires: 2038-06-28 21:19:45 UTC key usage: digitalSignature,nonRepudiation,keyCertSign,cRLSign pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert "caSigningCert cert-pki-ca" track: yes auto-renew: yes Request ID '20190425210056': status: MONITORING stuck: no key pair storage:type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='Server-Cert cert-pki-ca',token='NSS
Certificate DB',pin set certificate:type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='Server-Cert cert-pki-ca',token='NSS
Certificate DB' CA: dogtag-ipa-ca-renew-agent issuer: CN=Certificate Authority,O=EXAMPLE.COM <http://EXAMPLE.COM> subject: CN=ipa2.example.com <http://ipa2.example.com>,O=EXAMPLE.COM <http://EXAMPLE.COM> expires: 2023-03-07 22:37:22 UTC key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment eku: id-kp-serverAuth,id-kp-clientAuth,id-kp-emailProtection pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert "Server-Cert cert-pki-ca" track: yes auto-renew: yes Request ID '20190425210120': status: MONITORING stuck: no key pair storage: type=FILE,location='/var/kerberos/krb5kdc/kdc.key' certificate: type=FILE,location='/var/kerberos/krb5kdc/kdc.crt' CA: IPA issuer: CN=Certificate Authority,O=EXAMPLE.COM <http://EXAMPLE.COM> subject: CN=ipa2.example.com <http://ipa2.example.com>,O=EXAMPLE.COM <http://EXAMPLE.COM> expires: 2023-03-29 21:37:52 UTC principal name: krbtgt/EXAMPLE.COM@EXAMPLE.COM <mailto:EXAMPLE.COM@EXAMPLE.COM> key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment eku: id-kp-serverAuth,id-pkinit-KPKdc pre-save command: post-save command: /usr/libexec/ipa/certmonger/renew_kdc_cert track: yes auto-renew: yes [root@ipa2 ~]# There are 4 certs which expire on 2022-05-11 which match "expires in 27 days". Take 20190425210040 as an example, we have: WARNING:ipahealthcheck.ipa.certs.IPACertmongerExpirationCheck.20190425210040: Request
id 20190425210040 expires in 27 days WARNING: ipahealthcheck.ipa.certs.IPACertfileExpirationCheck.20190425210040: Request id 20190425210040 expires in 27 days ERROR: ipahealthcheck.ipa.certs.IPACertRevocation.20190425210040: Request for certificate failed, Certificate operation cannot be completed: EXCEPTION (Invalid Credential.) Request ID '20190425210040': status: MONITORING stuck: no key pair storage:type=FILE,location='/var/lib/ipa/ra-agent.key'
certificate: type=FILE,location='/var/lib/ipa/ra-agent.pem' CA: dogtag-ipa-ca-renew-agent issuer: CN=Certificate Authority,O=EXAMPLE.COM <http://EXAMPLE.COM> subject: CN=IPA RA,O=EXAMPLE.COM <http://EXAMPLE.COM> expires: 2022-05-11 03:40:55 UTC key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment eku: id-kp-serverAuth,id-kp-clientAuth pre-save command:/usr/libexec/ipa/certmonger/renew_ra_cert_pre
post-save command: /usr/libexec/ipa/certmonger/renew_ra_cert track: yes auto-renew: yes I was able to manually renew it: [root@ipa2 ~]# ipa-getcert resubmit -i '20190425210040' Resubmitting "20190425210040" to "dogtag-ipa-ca-renew-agent". [root@ipa2 ~]# After renew, it "expires: 2024-04-02 06:09:32 UTC": Request ID '20190425210040': status: MONITORING stuck: no key pair storage:type=FILE,location='/var/lib/ipa/ra-agent.key'
certificate: type=FILE,location='/var/lib/ipa/ra-agent.pem' CA: dogtag-ipa-ca-renew-agent issuer: CN=Certificate Authority,O=EXAMPLE.COM <http://EXAMPLE.COM> subject: CN=IPA RA,O=EXAMPLE.COM <http://EXAMPLE.COM> expires: 2024-04-02 06:09:32 UTC key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment eku: id-kp-serverAuth,id-kp-clientAuth pre-save command:/usr/libexec/ipa/certmonger/renew_ra_cert_pre
post-save command: /usr/libexec/ipa/certmonger/renew_ra_cert track: yes auto-renew: yes How to fix the issue reported by ipa-healthcheck? And what is this issue about? All IPA servers are at same level: CentOS Linux release 7.9.2009 (Core) ipa-*server*.x86_64 4.6.8-5.el7.centos.7 *slapi-nis*.x86_64 0.56.5-3.el7_9 *389-ds-base*.x86_64 1.3.10.2-12.el7_9 *389-ds-base*-libs.x86_64 1.3.10.2-12.el7_9 Many thanks! Kathy.
FreeIPA-users mailing list -- freeipa-users@lists.fedorahosted.org To unsubscribe send an email to
freeipa-users-leave@lists.fedorahosted.org
Fedora Code of Conduct:
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives:
https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedorahoste...
Do not reply to spam on the list, report it:
There was a bug in certmonger where it slowly leaked file descriptors over time. So if certmonger ran for an extended period, usually months, it would eventually run out. certmonger uses helpers to do renewals so with no descriptors, forks would fail and hence renewals. This is BZ https://bugzilla.redhat.com/show_bug.cgi?id=1992439 and fixed in 0.78.4-16 in RHEL 7.9.
That is my guess as to what happened. Just restarting an older certmonger would fix it as it would get a whole new batch of available descriptors and allow issuance to complete.
I'm glad to hear that ipa-healthcheck helped you avoid a painful certificate expiration issue.
rob
Kathy Zhu via FreeIPA-users wrote:
Hi Rob,
Thank you for the insight! That helped a lot!
The replication of the CA data was perfectly fine. I did the following on each IPA server except the renewal server to fix the situation:
1, restarted certmonger service, then waited a few minutes until it finished restarting pki_tomcatd (it may not restart pki_tomcatd right away), it did two restarts, see attached. 2, once see attached which confirmed the finish, ipa-healthcheck would not complain.
I noticed that this has been updated by its time stamp:
-r--r----- 1 root ipaapi 1261 Apr 13 14:13 /var/lib/ipa/ra-agent.pem
My guess of what happened is that /var/lib/ipa/ra-agent.pem had been renewed on the renewal server, the rest of the IPA servers should pick up the change, however, for a reason unknown to me, certmonger on other IPA servers failed to do so. That was why they complained of "not match". Restarting certmonger triggered this action, once done, all is fine.
Rob, please correct if my guess is wrong. Thank you for bringing this wonderful tool to Centos 7.
Many thanks.
Kathy.
[root@ipa2 ~]# systemctl status certmonger -l
● certmonger.service - Certificate monitoring and PKI enrollment
Loaded: loaded (/usr/lib/systemd/system/certmonger.service; enabled; vendor preset: disabled)
Active: active (running) since Wed 2022-04-13 14:12:18 PDT; 12min ago
Main PID: 16980 (certmonger)
CGroup: /system.slice/certmonger.service
└─16980 /usr/sbin/certmonger -S -p /var/run/certmonger.pid -n
Apr 13 14:13:05 ipa2.example.com http://ipa2.example.com stop_pkicad[17729]: Stopping pki_tomcatd
Apr 13 14:13:06 ipa2.example.com http://ipa2.example.com stop_pkicad[17729]: Stopped pki_tomcatd
Apr 13 14:13:08 ipa2.example.com http://ipa2.example.com renew_ca_cert[17785]: Updated trust on certificate auditSigningCert cert-pki-ca in /etc/pki/pki-tomcat/alias
Apr 13 14:13:08 ipa2.example.com http://ipa2.example.com renew_ca_cert[17785]: Starting pki_tomcatd
Apr 13 14:13:23 ipa2.example.com http://ipa2.example.com renew_ca_cert[17785]: Started pki_tomcatd
Apr 13 14:13:23 ipa2.example.com http://ipa2.example.com certmonger[18098]: Certificate named "auditSigningCert cert-pki-ca" in token "NSS Certificate DB" in database "/etc/pki/pki-tomcat/alias" issued by CA and saved.
Apr 13 14:13:35 ipa2.example.com http://ipa2.example.com stop_pkicad[18130]: Stopping pki_tomcatd
Apr 13 14:13:36 ipa2.example.com http://ipa2.example.com stop_pkicad[18130]: Stopped pki_tomcatd
Apr 13 14:13:38 ipa2.example.com http://ipa2.example.com renew_ca_cert[18185]: Starting pki_tomcatd
Apr 13 14:13:53 ipa2.example.com http://ipa2.example.com renew_ca_cert[18185]: Started pki_tomcatd
[root@ipa2 ~]#
On Wed, Apr 13, 2022 at 12:23 PM Rob Crittenden <rcritten@redhat.com mailto:rcritten@redhat.com> wrote:
This looks like the root cause: ERROR: ipahealthcheck.ipa.certs.IPARAAgent: RA agent description does not match 2;186;CN=Certificate Authority,O=EXAMPLE.COM <http://EXAMPLE.COM>;CN=IPA RA,O=EXAMPLE.COM <http://EXAMPLE.COM> in LDAP and 2;66;CN=Certificate Authority,O=EXAMPLE.COM <http://EXAMPLE.COM>;CN=IPA RA,O=EXAMPLE.COM <http://EXAMPLE.COM> expected It looks like an updated RA certificate was issued and was not picked up by the mother server(s). Off the top of my head this could be: - replication of the CA data has a problem: ipa-csreplica-manage list -v `hostname` - The updated certificate wasn't published to cn=certificates,cn=ipa,cn=etc,$SUFFIX - certmonger isn't picking up the renewal for some reason. The journal may hold clues. - something I'm forgetting I'd start with the first two. rob Kathy Zhu via FreeIPA-users wrote: > I just found this post about the same or similar issue: > > https://lists.fedoraproject.org/archives/list/freeipa-users@lists.fedorahosted.org/thread/DFEMDNWSCE4FDDFRDUCZYYIIOIUC3RFD/ > > One detail I missed - this happens on all IPA servers BUT the renewal > IPA server. I will go through ^ post to see if that applies to our > situation. > > Thanks. > > Kathy. > > > On Wed, Apr 13, 2022 at 10:21 AM Kathy Zhu wrote: > > Hi team, > > > ipa-healthcheck has been a great tool for us. I run it weekly on all > IPA servers via cron. This week ipa-healthcheck reported errors on > all IPA servers. > > > Take IPA server ipa2 as an example for the investigation: > > > > [root@ipa2 ~]# ipa-healthcheck --failures-only --output-type=human > > ra.get_certificate(): EXCEPTION (Invalid Credential.) > > ra.get_certificate(): EXCEPTION (Invalid Credential.) > > ra.get_certificate(): EXCEPTION (Invalid Credential.) > > ra.get_certificate(): EXCEPTION (Invalid Credential.) > > ra.get_certificate(): EXCEPTION (Invalid Credential.) > > ra.get_certificate(): EXCEPTION (Invalid Credential.) > > ra.get_certificate(): EXCEPTION (Invalid Credential.) > > ra.get_certificate(): EXCEPTION (Invalid Credential.) > > ra.get_certificate(): EXCEPTION (Invalid Credential.) > > ra.get_certificate(): EXCEPTION (Invalid Credential.) > > WARNING: > ipahealthcheck.ipa.certs.IPACertmongerExpirationCheck.20190425210040: Request > id 20190425210040 expires in 27 days > > WARNING: > ipahealthcheck.ipa.certs.IPACertmongerExpirationCheck.20190425210052: Request > id 20190425210052 expires in 27 days > > WARNING: > ipahealthcheck.ipa.certs.IPACertmongerExpirationCheck.20190425210053: Request > id 20190425210053 expires in 27 days > > WARNING: > ipahealthcheck.ipa.certs.IPACertmongerExpirationCheck.20190425210054: Request > id 20190425210054 expires in 27 days > > WARNING: > ipahealthcheck.ipa.certs.IPACertfileExpirationCheck.20190425210040: > Request id 20190425210040 expires in 27 days > > WARNING: > ipahealthcheck.ipa.certs.IPACertfileExpirationCheck.20190425210052: > Request id 20190425210052 expires in 27 days > > WARNING: > ipahealthcheck.ipa.certs.IPACertfileExpirationCheck.20190425210053: > Request id 20190425210053 expires in 27 days > > WARNING: > ipahealthcheck.ipa.certs.IPACertfileExpirationCheck.20190425210054: > Request id 20190425210054 expires in 27 days > > ERROR: ipahealthcheck.ipa.certs.IPARAAgent: RA agent description > does not match 2;186;CN=Certificate Authority,O=EXAMPLE.COM <http://EXAMPLE.COM> > <http://EXAMPLE.COM>;CN=IPA RA,O=EXAMPLE.COM <http://EXAMPLE.COM> <http://EXAMPLE.COM> in > LDAP and 2;66;CN=Certificate Authority,O=EXAMPLE.COM <http://EXAMPLE.COM> > <http://EXAMPLE.COM>;CN=IPA RA,O=EXAMPLE.COM <http://EXAMPLE.COM> <http://EXAMPLE.COM> > expected > > ERROR: ipahealthcheck.ipa.certs.IPACertRevocation.20190425210040: > Request for certificate failed, Certificate operation cannot be > completed: EXCEPTION (Invalid Credential.) > > ERROR: ipahealthcheck.ipa.certs.IPACertRevocation.20190425210052: > Request for certificate failed, Certificate operation cannot be > completed: EXCEPTION (Invalid Credential.) > > ERROR: ipahealthcheck.ipa.certs.IPACertRevocation.20190425210053: > Request for certificate failed, Certificate operation cannot be > completed: EXCEPTION (Invalid Credential.) > > ERROR: ipahealthcheck.ipa.certs.IPACertRevocation.20190425210054: > Request for certificate failed, Certificate operation cannot be > completed: EXCEPTION (Invalid Credential.) > > ERROR: ipahealthcheck.ipa.certs.IPACertRevocation.20190425210055: > Request for certificate failed, Certificate operation cannot be > completed: EXCEPTION (Invalid Credential.) > > ERROR: ipahealthcheck.ipa.certs.IPACertRevocation.20190425210056: > Request for certificate failed, Certificate operation cannot be > completed: EXCEPTION (Invalid Credential.) > > ERROR: ipahealthcheck.ipa.certs.IPACertRevocation.20190425205849: > Request for certificate failed, Certificate operation cannot be > completed: EXCEPTION (Invalid Credential.) > > ERROR: ipahealthcheck.ipa.certs.IPACertRevocation.20190425205831: > Request for certificate failed, Certificate operation cannot be > completed: EXCEPTION (Invalid Credential.) > > ERROR: ipahealthcheck.ipa.certs.IPACertRevocation.20190425210120: > Request for certificate failed, Certificate operation cannot be > completed: EXCEPTION (Invalid Credential.) > > ERROR: ipahealthcheck.dogtag.ca <http://ipahealthcheck.dogtag.ca>.DogtagCertsConnectivityCheck: > Request for certificate failed, Certificate operation cannot be > completed: EXCEPTION (Invalid Credential.) > > [root@ipa2 ~]# > > > > The list of certs: > > > [root@ipa2 ~]# getcert list > > Number of certificates and requests being tracked: 9. > > Request ID '20190425205831': > > status: MONITORING > > stuck: no > > key pair storage: > type=NSSDB,location='/etc/dirsrv/slapd-EXAMPLE-COM',nickname='Server-Cert',token='NSS > Certificate DB',pinfile='/etc/dirsrv/slapd-EXAMPLE-COM/pwdfile.txt' > > certificate: > type=NSSDB,location='/etc/dirsrv/slapd-EXAMPLE-COM',nickname='Server-Cert',token='NSS > Certificate DB' > > CA: IPA > > issuer: CN=Certificate Authority,O=EXAMPLE.COM <http://EXAMPLE.COM> <http://EXAMPLE.COM> > > subject: CN=ipa2.example.com <http://ipa2.example.com> <http://ipa2.example.com>,O=EXAMPLE.COM <http://EXAMPLE.COM> > <http://EXAMPLE.COM> > > expires: 2023-03-29 21:37:22 UTC > > dns: ipa2.example.com <http://ipa2.example.com> <http://ipa2.example.com> > > principal name: ldap/ipa2.example.com@EXAMPLE.COM <mailto:ipa2.example.com@EXAMPLE.COM> > <mailto:ipa2.example.com@EXAMPLE.COM <mailto:ipa2.example.com@EXAMPLE.COM>> > > key usage: > digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment > > eku: id-kp-serverAuth,id-kp-clientAuth > > pre-save command: > > post-save command: /usr/libexec/ipa/certmonger/restart_dirsrv > EXAMPLE-COM > > track: yes > > auto-renew: yes > > Request ID '20190425205849': > > status: MONITORING > > stuck: no > > key pair storage: > type=NSSDB,location='/etc/httpd/alias',nickname='Server-Cert',token='NSS > Certificate DB',pinfile='/etc/httpd/alias/pwdfile.txt' > > certificate: > type=NSSDB,location='/etc/httpd/alias',nickname='Server-Cert',token='NSS > Certificate DB' > > CA: IPA > > issuer: CN=Certificate Authority,O=EXAMPLE.COM <http://EXAMPLE.COM> <http://EXAMPLE.COM> > > subject: CN=ipa2.example.com <http://ipa2.example.com> <http://ipa2.example.com>,O=EXAMPLE.COM <http://EXAMPLE.COM> > <http://EXAMPLE.COM> > > expires: 2023-03-29 21:37:46 UTC > > dns: ipa2.example.com <http://ipa2.example.com> <http://ipa2.example.com> > > principal name: HTTP/ipa2.example.com@EXAMPLE.COM <mailto:ipa2.example.com@EXAMPLE.COM> > <mailto:ipa2.example.com@EXAMPLE.COM <mailto:ipa2.example.com@EXAMPLE.COM>> > > key usage: > digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment > > eku: id-kp-serverAuth,id-kp-clientAuth > > pre-save command: > > post-save command: /usr/libexec/ipa/certmonger/restart_httpd > > track: yes > > auto-renew: yes > > Request ID '20190425210040': > > status: MONITORING > > stuck: no > > key pair storage: type=FILE,location='/var/lib/ipa/ra-agent.key' > > certificate: type=FILE,location='/var/lib/ipa/ra-agent.pem' > > CA: dogtag-ipa-ca-renew-agent > > issuer: CN=Certificate Authority,O=EXAMPLE.COM <http://EXAMPLE.COM> <http://EXAMPLE.COM> > > subject: CN=IPA RA,O=EXAMPLE.COM <http://EXAMPLE.COM> <http://EXAMPLE.COM> > > expires: 2022-05-11 03:40:55 UTC > > key usage: > digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment > > eku: id-kp-serverAuth,id-kp-clientAuth > > pre-save command: /usr/libexec/ipa/certmonger/renew_ra_cert_pre > > post-save command: /usr/libexec/ipa/certmonger/renew_ra_cert > > track: yes > > auto-renew: yes > > Request ID '20190425210052': > > status: MONITORING > > stuck: no > > key pair storage: > type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='auditSigningCert > cert-pki-ca',token='NSS Certificate DB',pin set > > certificate: > type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='auditSigningCert > cert-pki-ca',token='NSS Certificate DB' > > CA: dogtag-ipa-ca-renew-agent > > issuer: CN=Certificate Authority,O=EXAMPLE.COM <http://EXAMPLE.COM> <http://EXAMPLE.COM> > > subject: CN=CA Audit,O=EXAMPLE.COM <http://EXAMPLE.COM> <http://EXAMPLE.COM> > > expires: 2022-05-11 03:40:05 UTC > > key usage: digitalSignature,nonRepudiation > > pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad > > post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert > "auditSigningCert cert-pki-ca" > > track: yes > > auto-renew: yes > > Request ID '20190425210053': > > status: MONITORING > > stuck: no > > key pair storage: > type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='ocspSigningCert > cert-pki-ca',token='NSS Certificate DB',pin set > > certificate: > type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='ocspSigningCert > cert-pki-ca',token='NSS Certificate DB' > > CA: dogtag-ipa-ca-renew-agent > > issuer: CN=Certificate Authority,O=EXAMPLE.COM <http://EXAMPLE.COM> <http://EXAMPLE.COM> > > subject: CN=OCSP Subsystem,O=EXAMPLE.COM <http://EXAMPLE.COM> <http://EXAMPLE.COM> > > expires: 2022-05-11 03:40:25 UTC > > eku: id-kp-OCSPSigning > > pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad > > post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert > "ocspSigningCert cert-pki-ca" > > track: yes > > auto-renew: yes > > Request ID '20190425210054': > > status: MONITORING > > stuck: no > > key pair storage: > type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='subsystemCert > cert-pki-ca',token='NSS Certificate DB',pin set > > certificate: > type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='subsystemCert > cert-pki-ca',token='NSS Certificate DB' > > CA: dogtag-ipa-ca-renew-agent > > issuer: CN=Certificate Authority,O=EXAMPLE.COM <http://EXAMPLE.COM> <http://EXAMPLE.COM> > > subject: CN=CA Subsystem,O=EXAMPLE.COM <http://EXAMPLE.COM> <http://EXAMPLE.COM> > > expires: 2022-05-11 03:40:05 UTC > > key usage: > digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment > > eku: id-kp-serverAuth,id-kp-clientAuth > > pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad > > post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert > "subsystemCert cert-pki-ca" > > track: yes > > auto-renew: yes > > Request ID '20190425210055': > > status: MONITORING > > stuck: no > > key pair storage: > type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='caSigningCert > cert-pki-ca',token='NSS Certificate DB',pin set > > certificate: > type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='caSigningCert > cert-pki-ca',token='NSS Certificate DB' > > CA: dogtag-ipa-ca-renew-agent > > issuer: CN=Certificate Authority,O=EXAMPLE.COM <http://EXAMPLE.COM> <http://EXAMPLE.COM> > > subject: CN=Certificate Authority,O=EXAMPLE.COM <http://EXAMPLE.COM> <http://EXAMPLE.COM> > > expires: 2038-06-28 21:19:45 UTC > > key usage: digitalSignature,nonRepudiation,keyCertSign,cRLSign > > pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad > > post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert > "caSigningCert cert-pki-ca" > > track: yes > > auto-renew: yes > > Request ID '20190425210056': > > status: MONITORING > > stuck: no > > key pair storage: > type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='Server-Cert cert-pki-ca',token='NSS > Certificate DB',pin set > > certificate: > type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='Server-Cert cert-pki-ca',token='NSS > Certificate DB' > > CA: dogtag-ipa-ca-renew-agent > > issuer: CN=Certificate Authority,O=EXAMPLE.COM <http://EXAMPLE.COM> <http://EXAMPLE.COM> > > subject: CN=ipa2.example.com <http://ipa2.example.com> <http://ipa2.example.com>,O=EXAMPLE.COM <http://EXAMPLE.COM> > <http://EXAMPLE.COM> > > expires: 2023-03-07 22:37:22 UTC > > key usage: > digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment > > eku: id-kp-serverAuth,id-kp-clientAuth,id-kp-emailProtection > > pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad > > post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert > "Server-Cert cert-pki-ca" > > track: yes > > auto-renew: yes > > Request ID '20190425210120': > > status: MONITORING > > stuck: no > > key pair storage: type=FILE,location='/var/kerberos/krb5kdc/kdc.key' > > certificate: type=FILE,location='/var/kerberos/krb5kdc/kdc.crt' > > CA: IPA > > issuer: CN=Certificate Authority,O=EXAMPLE.COM <http://EXAMPLE.COM> <http://EXAMPLE.COM> > > subject: CN=ipa2.example.com <http://ipa2.example.com> <http://ipa2.example.com>,O=EXAMPLE.COM <http://EXAMPLE.COM> > <http://EXAMPLE.COM> > > expires: 2023-03-29 21:37:52 UTC > > principal name: krbtgt/EXAMPLE.COM@EXAMPLE.COM <mailto:EXAMPLE.COM@EXAMPLE.COM> > <mailto:EXAMPLE.COM@EXAMPLE.COM <mailto:EXAMPLE.COM@EXAMPLE.COM>> > > key usage: > digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment > > eku: id-kp-serverAuth,id-pkinit-KPKdc > > pre-save command: > > post-save command: /usr/libexec/ipa/certmonger/renew_kdc_cert > > track: yes > > auto-renew: yes > > [root@ipa2 ~]# > > > > > There are 4 certs which expire on 2022-05-11 which match "expires in > 27 days". Take 20190425210040 as an example, we have: > > > > > WARNING: > ipahealthcheck.ipa.certs.IPACertmongerExpirationCheck.20190425210040: Request > id 20190425210040 expires in 27 days > > WARNING: > ipahealthcheck.ipa.certs.IPACertfileExpirationCheck.20190425210040: > Request id 20190425210040 expires in 27 days > > ERROR: ipahealthcheck.ipa.certs.IPACertRevocation.20190425210040: > Request for certificate failed, Certificate operation cannot be > completed: EXCEPTION (Invalid Credential.) > > > Request ID '20190425210040': > > status: MONITORING > > stuck: no > > key pair storage: type=FILE,location='/var/lib/ipa/ra-agent.key' > > certificate: type=FILE,location='/var/lib/ipa/ra-agent.pem' > > CA: dogtag-ipa-ca-renew-agent > > issuer: CN=Certificate Authority,O=EXAMPLE.COM <http://EXAMPLE.COM> > <http://EXAMPLE.COM> > > subject: CN=IPA RA,O=EXAMPLE.COM <http://EXAMPLE.COM> <http://EXAMPLE.COM> > > expires: 2022-05-11 03:40:55 UTC > > key usage: > digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment > > eku: id-kp-serverAuth,id-kp-clientAuth > > pre-save command: /usr/libexec/ipa/certmonger/renew_ra_cert_pre > > post-save command: /usr/libexec/ipa/certmonger/renew_ra_cert > > track: yes > > auto-renew: yes > > > > I was able to manually renew it: > > > > [root@ipa2 ~]# ipa-getcert resubmit -i '20190425210040' > > Resubmitting "20190425210040" to "dogtag-ipa-ca-renew-agent". > > [root@ipa2 ~]# > > > > After renew, it "expires: 2024-04-02 06:09:32 UTC": > > > > Request ID '20190425210040': > > status: MONITORING > > stuck: no > > key pair storage: type=FILE,location='/var/lib/ipa/ra-agent.key' > > certificate: type=FILE,location='/var/lib/ipa/ra-agent.pem' > > CA: dogtag-ipa-ca-renew-agent > > issuer: CN=Certificate Authority,O=EXAMPLE.COM <http://EXAMPLE.COM> > <http://EXAMPLE.COM> > > subject: CN=IPA RA,O=EXAMPLE.COM <http://EXAMPLE.COM> <http://EXAMPLE.COM> > > expires: 2024-04-02 06:09:32 UTC > > key usage: > digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment > > eku: id-kp-serverAuth,id-kp-clientAuth > > pre-save command: /usr/libexec/ipa/certmonger/renew_ra_cert_pre > > post-save command: /usr/libexec/ipa/certmonger/renew_ra_cert > > track: yes > > auto-renew: yes > > > > How to fix the issue reported by ipa-healthcheck? And what is this > issue about? > > > All IPA servers are at same level: > > > CentOS Linux release 7.9.2009 (Core) > > ipa-*server*.x86_64 4.6.8-5.el7.centos.7 > > *slapi-nis*.x86_64 0.56.5-3.el7_9 > > *389-ds-base*.x86_64 1.3.10.2-12.el7_9 > > *389-ds-base*-libs.x86_64 1.3.10.2-12.el7_9 > > > Many thanks! > > > Kathy. > > > _______________________________________________ > FreeIPA-users mailing list -- freeipa-users@lists.fedorahosted.org <mailto:freeipa-users@lists.fedorahosted.org> > To unsubscribe send an email to freeipa-users-leave@lists.fedorahosted.org <mailto:freeipa-users-leave@lists.fedorahosted.org> > Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ > List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines > List Archives: https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedorahosted.org > Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure >
FreeIPA-users mailing list -- freeipa-users@lists.fedorahosted.org To unsubscribe send an email to freeipa-users-leave@lists.fedorahosted.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedorahoste... Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
freeipa-users@lists.fedorahosted.org
-
Kathy Zhu -
Rob Crittenden