Hi Rob,
Thank you for the insight! That helped a lot!
The replication of the CA data was perfectly fine. I did the following on
each IPA server except the renewal server to fix the situation:
1, restarted certmonger service, then waited a few minutes until it
finished restarting pki_tomcatd (it may not restart pki_tomcatd right
away), it did two restarts, see attached.
2, once see attached which confirmed the finish, ipa-healthcheck would not
complain.
I noticed that this has been updated by its time stamp:
-r--r----- 1 root ipaapi 1261 Apr 13 14:13 /var/lib/ipa/ra-agent.pem
My guess of what happened is that /var/lib/ipa/ra-agent.pem had been
renewed on the renewal server, the rest of the IPA servers should pick up
the change, however, for a reason unknown to me, certmonger on other IPA
servers failed to do so. That was why they complained of "not match".
Restarting certmonger triggered this action, once done, all is fine.
Rob, please correct if my guess is wrong. Thank you for bringing this
wonderful tool to Centos 7.
Many thanks.
Kathy.
[root@ipa2 ~]# systemctl status certmonger -l
● certmonger.service - Certificate monitoring and PKI enrollment
Loaded: loaded (/usr/lib/systemd/system/certmonger.service; enabled;
vendor preset: disabled)
Active: active (running) since Wed 2022-04-13 14:12:18 PDT; 12min ago
Main PID: 16980 (certmonger)
CGroup: /system.slice/certmonger.service
└─16980 /usr/sbin/certmonger -S -p /var/run/certmonger.pid -n
Apr 13 14:13:05
renew_ca_cert[17785]: Updated trust on
certificate auditSigningCert cert-pki-ca in /etc/pki/pki-tomcat/alias
Apr 13 14:13:08
certmonger[18098]: Certificate named
"auditSigningCert cert-pki-ca" in token "NSS Certificate DB" in
database
"/etc/pki/pki-tomcat/alias" issued by CA and saved.
Apr 13 14:13:35
renew_ca_cert[18185]: Started pki_tomcatd
[root@ipa2 ~]#
On Wed, Apr 13, 2022 at 12:23 PM Rob Crittenden <rcritten(a)redhat.com> wrote:
This looks like the root cause:
ERROR: ipahealthcheck.ipa.certs.IPARAAgent: RA agent description does
not match 2;186;CN=Certificate Authority,O=EXAMPLE.COM;CN=IPA
RA,O=EXAMPLE.COM in LDAP and 2;66;CN=Certificate
Authority,O=EXAMPLE.COM;CN=IPA
RA,O=EXAMPLE.COM expected
It looks like an updated RA certificate was issued and was not picked up
by the mother server(s).
Off the top of my head this could be:
- replication of the CA data has a problem: ipa-csreplica-manage list -v
`hostname`
- The updated certificate wasn't published to
cn=certificates,cn=ipa,cn=etc,$SUFFIX
- certmonger isn't picking up the renewal for some reason. The journal
may hold clues.
- something I'm forgetting
I'd start with the first two.
rob
Kathy Zhu via FreeIPA-users wrote:
> I just found this post about the same or similar issue:
>
>
https://lists.fedoraproject.org/archives/list/freeipa-users@lists.fedorah...
>
> One detail I missed - this happens on all IPA servers BUT the renewal
> IPA server. I will go through ^ post to see if that applies to our
> situation.
>
> Thanks.
>
> Kathy.
>
>
> On Wed, Apr 13, 2022 at 10:21 AM Kathy Zhu wrote:
>
> Hi team,
>
>
> ipa-healthcheck has been a great tool for us. I run it weekly on all
> IPA servers via cron. This week ipa-healthcheck reported errors on
> all IPA servers.
>
>
> Take IPA server ipa2 as an example for the investigation:
>
>
>
> [root@ipa2 ~]# ipa-healthcheck --failures-only --output-type=human
>
> ra.get_certificate(): EXCEPTION (Invalid Credential.)
>
> ra.get_certificate(): EXCEPTION (Invalid Credential.)
>
> ra.get_certificate(): EXCEPTION (Invalid Credential.)
>
> ra.get_certificate(): EXCEPTION (Invalid Credential.)
>
> ra.get_certificate(): EXCEPTION (Invalid Credential.)
>
> ra.get_certificate(): EXCEPTION (Invalid Credential.)
>
> ra.get_certificate(): EXCEPTION (Invalid Credential.)
>
> ra.get_certificate(): EXCEPTION (Invalid Credential.)
>
> ra.get_certificate(): EXCEPTION (Invalid Credential.)
>
> ra.get_certificate(): EXCEPTION (Invalid Credential.)
>
> WARNING:
>
ipahealthcheck.ipa.certs.IPACertmongerExpirationCheck.20190425210040:
Request
> id 20190425210040 expires in 27 days
>
> WARNING:
>
ipahealthcheck.ipa.certs.IPACertmongerExpirationCheck.20190425210052:
Request
> id 20190425210052 expires in 27 days
>
> WARNING:
>
ipahealthcheck.ipa.certs.IPACertmongerExpirationCheck.20190425210053:
Request
> id 20190425210053 expires in 27 days
>
> WARNING:
>
ipahealthcheck.ipa.certs.IPACertmongerExpirationCheck.20190425210054:
Request
> id 20190425210054 expires in 27 days
>
> WARNING:
> ipahealthcheck.ipa.certs.IPACertfileExpirationCheck.20190425210040:
> Request id 20190425210040 expires in 27 days
>
> WARNING:
> ipahealthcheck.ipa.certs.IPACertfileExpirationCheck.20190425210052:
> Request id 20190425210052 expires in 27 days
>
> WARNING:
> ipahealthcheck.ipa.certs.IPACertfileExpirationCheck.20190425210053:
> Request id 20190425210053 expires in 27 days
>
> WARNING:
> ipahealthcheck.ipa.certs.IPACertfileExpirationCheck.20190425210054:
> Request id 20190425210054 expires in 27 days
>
> ERROR: ipahealthcheck.ipa.certs.IPARAAgent: RA agent description
> does not match 2;186;CN=Certificate
Authority,O=EXAMPLE.COM
> <
http://EXAMPLE.COM>;CN=IPA RA,O=EXAMPLE.COM <
http://EXAMPLE.COM>
in
> LDAP and 2;66;CN=Certificate
Authority,O=EXAMPLE.COM
> <
http://EXAMPLE.COM>;CN=IPA RA,O=EXAMPLE.COM <
http://EXAMPLE.COM>
> expected
>
> ERROR: ipahealthcheck.ipa.certs.IPACertRevocation.20190425210040:
> Request for certificate failed, Certificate operation cannot be
> completed: EXCEPTION (Invalid Credential.)
>
> ERROR: ipahealthcheck.ipa.certs.IPACertRevocation.20190425210052:
> Request for certificate failed, Certificate operation cannot be
> completed: EXCEPTION (Invalid Credential.)
>
> ERROR: ipahealthcheck.ipa.certs.IPACertRevocation.20190425210053:
> Request for certificate failed, Certificate operation cannot be
> completed: EXCEPTION (Invalid Credential.)
>
> ERROR: ipahealthcheck.ipa.certs.IPACertRevocation.20190425210054:
> Request for certificate failed, Certificate operation cannot be
> completed: EXCEPTION (Invalid Credential.)
>
> ERROR: ipahealthcheck.ipa.certs.IPACertRevocation.20190425210055:
> Request for certificate failed, Certificate operation cannot be
> completed: EXCEPTION (Invalid Credential.)
>
> ERROR: ipahealthcheck.ipa.certs.IPACertRevocation.20190425210056:
> Request for certificate failed, Certificate operation cannot be
> completed: EXCEPTION (Invalid Credential.)
>
> ERROR: ipahealthcheck.ipa.certs.IPACertRevocation.20190425205849:
> Request for certificate failed, Certificate operation cannot be
> completed: EXCEPTION (Invalid Credential.)
>
> ERROR: ipahealthcheck.ipa.certs.IPACertRevocation.20190425205831:
> Request for certificate failed, Certificate operation cannot be
> completed: EXCEPTION (Invalid Credential.)
>
> ERROR: ipahealthcheck.ipa.certs.IPACertRevocation.20190425210120:
> Request for certificate failed, Certificate operation cannot be
> completed: EXCEPTION (Invalid Credential.)
>
> ERROR: ipahealthcheck.dogtag.ca.DogtagCertsConnectivityCheck:
> Request for certificate failed, Certificate operation cannot be
> completed: EXCEPTION (Invalid Credential.)
>
> [root@ipa2 ~]#
>
>
>
> The list of certs:
>
>
> [root@ipa2 ~]# getcert list
>
> Number of certificates and requests being tracked: 9.
>
> Request ID '20190425205831':
>
> status: MONITORING
>
> stuck: no
>
> key pair storage:
>
type=NSSDB,location='/etc/dirsrv/slapd-EXAMPLE-COM',nickname='Server-Cert',token='NSS
> Certificate DB',pinfile='/etc/dirsrv/slapd-EXAMPLE-COM/pwdfile.txt'
>
> certificate:
>
type=NSSDB,location='/etc/dirsrv/slapd-EXAMPLE-COM',nickname='Server-Cert',token='NSS
> Certificate DB'
>
> CA: IPA
>
> issuer: CN=Certificate
Authority,O=EXAMPLE.COM <
http://EXAMPLE.COM>
>
> subject:
CN=ipa2.example.com <
http://ipa2.example.com>,O=EXAMPLE.COM
> <
http://EXAMPLE.COM>
>
> expires: 2023-03-29 21:37:22 UTC
>
> dns:
ipa2.example.com <
http://ipa2.example.com>
>
> principal name: ldap/ipa2.example.com(a)EXAMPLE.COM
> <mailto:ipa2.example.com@EXAMPLE.COM>
>
> key usage:
> digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment
>
> eku: id-kp-serverAuth,id-kp-clientAuth
>
> pre-save command:
>
> post-save command: /usr/libexec/ipa/certmonger/restart_dirsrv
> EXAMPLE-COM
>
> track: yes
>
> auto-renew: yes
>
> Request ID '20190425205849':
>
> status: MONITORING
>
> stuck: no
>
> key pair storage:
>
type=NSSDB,location='/etc/httpd/alias',nickname='Server-Cert',token='NSS
> Certificate DB',pinfile='/etc/httpd/alias/pwdfile.txt'
>
> certificate:
>
type=NSSDB,location='/etc/httpd/alias',nickname='Server-Cert',token='NSS
> Certificate DB'
>
> CA: IPA
>
> issuer: CN=Certificate
Authority,O=EXAMPLE.COM <
http://EXAMPLE.COM>
>
> subject:
CN=ipa2.example.com <
http://ipa2.example.com>,O=EXAMPLE.COM
> <
http://EXAMPLE.COM>
>
> expires: 2023-03-29 21:37:46 UTC
>
> dns:
ipa2.example.com <
http://ipa2.example.com>
>
> principal name: HTTP/ipa2.example.com(a)EXAMPLE.COM
> <mailto:ipa2.example.com@EXAMPLE.COM>
>
> key usage:
> digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment
>
> eku: id-kp-serverAuth,id-kp-clientAuth
>
> pre-save command:
>
> post-save command: /usr/libexec/ipa/certmonger/restart_httpd
>
> track: yes
>
> auto-renew: yes
>
> Request ID '20190425210040':
>
> status: MONITORING
>
> stuck: no
>
> key pair storage: type=FILE,location='/var/lib/ipa/ra-agent.key'
>
> certificate: type=FILE,location='/var/lib/ipa/ra-agent.pem'
>
> CA: dogtag-ipa-ca-renew-agent
>
> issuer: CN=Certificate
Authority,O=EXAMPLE.COM <
http://EXAMPLE.COM>
>
> subject: CN=IPA
RA,O=EXAMPLE.COM <
http://EXAMPLE.COM>
>
> expires: 2022-05-11 03:40:55 UTC
>
> key usage:
> digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment
>
> eku: id-kp-serverAuth,id-kp-clientAuth
>
> pre-save command: /usr/libexec/ipa/certmonger/renew_ra_cert_pre
>
> post-save command: /usr/libexec/ipa/certmonger/renew_ra_cert
>
> track: yes
>
> auto-renew: yes
>
> Request ID '20190425210052':
>
> status: MONITORING
>
> stuck: no
>
> key pair storage:
>
type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='auditSigningCert
> cert-pki-ca',token='NSS Certificate DB',pin set
>
> certificate:
>
type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='auditSigningCert
> cert-pki-ca',token='NSS Certificate DB'
>
> CA: dogtag-ipa-ca-renew-agent
>
> issuer: CN=Certificate
Authority,O=EXAMPLE.COM <
http://EXAMPLE.COM>
>
> subject: CN=CA
Audit,O=EXAMPLE.COM <
http://EXAMPLE.COM>
>
> expires: 2022-05-11 03:40:05 UTC
>
> key usage: digitalSignature,nonRepudiation
>
> pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad
>
> post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert
> "auditSigningCert cert-pki-ca"
>
> track: yes
>
> auto-renew: yes
>
> Request ID '20190425210053':
>
> status: MONITORING
>
> stuck: no
>
> key pair storage:
>
type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='ocspSigningCert
> cert-pki-ca',token='NSS Certificate DB',pin set
>
> certificate:
>
type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='ocspSigningCert
> cert-pki-ca',token='NSS Certificate DB'
>
> CA: dogtag-ipa-ca-renew-agent
>
> issuer: CN=Certificate
Authority,O=EXAMPLE.COM <
http://EXAMPLE.COM>
>
> subject: CN=OCSP
Subsystem,O=EXAMPLE.COM <
http://EXAMPLE.COM>
>
> expires: 2022-05-11 03:40:25 UTC
>
> eku: id-kp-OCSPSigning
>
> pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad
>
> post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert
> "ocspSigningCert cert-pki-ca"
>
> track: yes
>
> auto-renew: yes
>
> Request ID '20190425210054':
>
> status: MONITORING
>
> stuck: no
>
> key pair storage:
>
type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='subsystemCert
> cert-pki-ca',token='NSS Certificate DB',pin set
>
> certificate:
>
type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='subsystemCert
> cert-pki-ca',token='NSS Certificate DB'
>
> CA: dogtag-ipa-ca-renew-agent
>
> issuer: CN=Certificate
Authority,O=EXAMPLE.COM <
http://EXAMPLE.COM>
>
> subject: CN=CA
Subsystem,O=EXAMPLE.COM <
http://EXAMPLE.COM>
>
> expires: 2022-05-11 03:40:05 UTC
>
> key usage:
> digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment
>
> eku: id-kp-serverAuth,id-kp-clientAuth
>
> pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad
>
> post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert
> "subsystemCert cert-pki-ca"
>
> track: yes
>
> auto-renew: yes
>
> Request ID '20190425210055':
>
> status: MONITORING
>
> stuck: no
>
> key pair storage:
>
type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='caSigningCert
> cert-pki-ca',token='NSS Certificate DB',pin set
>
> certificate:
>
type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='caSigningCert
> cert-pki-ca',token='NSS Certificate DB'
>
> CA: dogtag-ipa-ca-renew-agent
>
> issuer: CN=Certificate
Authority,O=EXAMPLE.COM <
http://EXAMPLE.COM>
>
> subject: CN=Certificate
Authority,O=EXAMPLE.COM <
http://EXAMPLE.COM>
>
> expires: 2038-06-28 21:19:45 UTC
>
> key usage: digitalSignature,nonRepudiation,keyCertSign,cRLSign
>
> pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad
>
> post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert
> "caSigningCert cert-pki-ca"
>
> track: yes
>
> auto-renew: yes
>
> Request ID '20190425210056':
>
> status: MONITORING
>
> stuck: no
>
> key pair storage:
>
type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='Server-Cert
cert-pki-ca',token='NSS
> Certificate DB',pin set
>
> certificate:
>
type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='Server-Cert
cert-pki-ca',token='NSS
> Certificate DB'
>
> CA: dogtag-ipa-ca-renew-agent
>
> issuer: CN=Certificate
Authority,O=EXAMPLE.COM <
http://EXAMPLE.COM>
>
> subject:
CN=ipa2.example.com <
http://ipa2.example.com>,O=EXAMPLE.COM
> <
http://EXAMPLE.COM>
>
> expires: 2023-03-07 22:37:22 UTC
>
> key usage:
> digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment
>
> eku: id-kp-serverAuth,id-kp-clientAuth,id-kp-emailProtection
>
> pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad
>
> post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert
> "Server-Cert cert-pki-ca"
>
> track: yes
>
> auto-renew: yes
>
> Request ID '20190425210120':
>
> status: MONITORING
>
> stuck: no
>
> key pair storage: type=FILE,location='/var/kerberos/krb5kdc/kdc.key'
>
> certificate: type=FILE,location='/var/kerberos/krb5kdc/kdc.crt'
>
> CA: IPA
>
> issuer: CN=Certificate
Authority,O=EXAMPLE.COM <
http://EXAMPLE.COM>
>
> subject:
CN=ipa2.example.com <
http://ipa2.example.com>,O=EXAMPLE.COM
> <
http://EXAMPLE.COM>
>
> expires: 2023-03-29 21:37:52 UTC
>
> principal name: krbtgt/EXAMPLE.COM(a)EXAMPLE.COM
> <mailto:EXAMPLE.COM@EXAMPLE.COM>
>
> key usage:
> digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment
>
> eku: id-kp-serverAuth,id-pkinit-KPKdc
>
> pre-save command:
>
> post-save command: /usr/libexec/ipa/certmonger/renew_kdc_cert
>
> track: yes
>
> auto-renew: yes
>
> [root@ipa2 ~]#
>
>
>
>
> There are 4 certs which expire on 2022-05-11 which match "expires in
> 27 days". Take 20190425210040 as an example, we have:
>
>
>
>
> WARNING:
>
ipahealthcheck.ipa.certs.IPACertmongerExpirationCheck.20190425210040:
Request
> id 20190425210040 expires in 27 days
>
> WARNING:
> ipahealthcheck.ipa.certs.IPACertfileExpirationCheck.20190425210040:
> Request id 20190425210040 expires in 27 days
>
> ERROR: ipahealthcheck.ipa.certs.IPACertRevocation.20190425210040:
> Request for certificate failed, Certificate operation cannot be
> completed: EXCEPTION (Invalid Credential.)
>
>
> Request ID '20190425210040':
>
> status: MONITORING
>
> stuck: no
>
> key pair storage:
type=FILE,location='/var/lib/ipa/ra-agent.key'
>
> certificate: type=FILE,location='/var/lib/ipa/ra-agent.pem'
>
> CA: dogtag-ipa-ca-renew-agent
>
> issuer: CN=Certificate
Authority,O=EXAMPLE.COM
> <
http://EXAMPLE.COM>
>
> subject: CN=IPA
RA,O=EXAMPLE.COM <
http://EXAMPLE.COM>
>
> expires: 2022-05-11 03:40:55 UTC
>
> key usage:
> digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment
>
> eku: id-kp-serverAuth,id-kp-clientAuth
>
> pre-save command:
/usr/libexec/ipa/certmonger/renew_ra_cert_pre
>
> post-save command: /usr/libexec/ipa/certmonger/renew_ra_cert
>
> track: yes
>
> auto-renew: yes
>
>
>
> I was able to manually renew it:
>
>
>
> [root@ipa2 ~]# ipa-getcert resubmit -i '20190425210040'
>
> Resubmitting "20190425210040" to
"dogtag-ipa-ca-renew-agent".
>
> [root@ipa2 ~]#
>
>
>
> After renew, it "expires: 2024-04-02 06:09:32 UTC":
>
>
>
> Request ID '20190425210040':
>
> status: MONITORING
>
> stuck: no
>
> key pair storage:
type=FILE,location='/var/lib/ipa/ra-agent.key'
>
> certificate: type=FILE,location='/var/lib/ipa/ra-agent.pem'
>
> CA: dogtag-ipa-ca-renew-agent
>
> issuer: CN=Certificate
Authority,O=EXAMPLE.COM
> <
http://EXAMPLE.COM>
>
> subject: CN=IPA
RA,O=EXAMPLE.COM <
http://EXAMPLE.COM>
>
> expires: 2024-04-02 06:09:32 UTC
>
> key usage:
> digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment
>
> eku: id-kp-serverAuth,id-kp-clientAuth
>
> pre-save command:
/usr/libexec/ipa/certmonger/renew_ra_cert_pre
>
> post-save command: /usr/libexec/ipa/certmonger/renew_ra_cert
>
> track: yes
>
> auto-renew: yes
>
>
>
> How to fix the issue reported by ipa-healthcheck? And what is this
> issue about?
>
>
> All IPA servers are at same level:
>
>
> CentOS Linux release 7.9.2009 (Core)
>
> ipa-*server*.x86_64 4.6.8-5.el7.centos.7
>
> *slapi-nis*.x86_64 0.56.5-3.el7_9
>
> *389-ds-base*.x86_64 1.3.10.2-12.el7_9
>
> *389-ds-base*-libs.x86_64 1.3.10.2-12.el7_9
>
>
> Many thanks!
>
>
> Kathy.
>
>
> _______________________________________________
> FreeIPA-users mailing list -- freeipa-users(a)lists.fedorahosted.org
> To unsubscribe send an email to
freeipa-users-leave(a)lists.fedorahosted.org
> Fedora Code of Conduct:
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> List Guidelines:
https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives:
https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedoraho...
> Do not reply to spam on the list, report it:
https://pagure.io/fedora-infrastructure
>