ipa-healthcheck - RA agent description does not match 2 and Invalid Credential

[Kathy Zhu]

Hi team,


ipa-healthcheck has been a great tool for us. I run it weekly on all IPA
servers via cron. This week ipa-healthcheck reported errors on all IPA
servers.


Take IPA server ipa2 as an example for the investigation:



[root@ipa2 ~]# ipa-healthcheck --failures-only --output-type=human

ra.get_certificate(): EXCEPTION (Invalid Credential.)

ra.get_certificate(): EXCEPTION (Invalid Credential.)

ra.get_certificate(): EXCEPTION (Invalid Credential.)

ra.get_certificate(): EXCEPTION (Invalid Credential.)

ra.get_certificate(): EXCEPTION (Invalid Credential.)

ra.get_certificate(): EXCEPTION (Invalid Credential.)

ra.get_certificate(): EXCEPTION (Invalid Credential.)

ra.get_certificate(): EXCEPTION (Invalid Credential.)

ra.get_certificate(): EXCEPTION (Invalid Credential.)

ra.get_certificate(): EXCEPTION (Invalid Credential.)

WARNING:
ipahealthcheck.ipa.certs.IPACertmongerExpirationCheck.20190425210040:
Request id 20190425210040 expires in 27 days

WARNING:
ipahealthcheck.ipa.certs.IPACertmongerExpirationCheck.20190425210052:
Request id 20190425210052 expires in 27 days

WARNING:
ipahealthcheck.ipa.certs.IPACertmongerExpirationCheck.20190425210053:
Request id 20190425210053 expires in 27 days

WARNING:
ipahealthcheck.ipa.certs.IPACertmongerExpirationCheck.20190425210054:
Request id 20190425210054 expires in 27 days

WARNING:
ipahealthcheck.ipa.certs.IPACertfileExpirationCheck.20190425210040: Request
id 20190425210040 expires in 27 days

WARNING:
ipahealthcheck.ipa.certs.IPACertfileExpirationCheck.20190425210052: Request
id 20190425210052 expires in 27 days

WARNING:
ipahealthcheck.ipa.certs.IPACertfileExpirationCheck.20190425210053: Request
id 20190425210053 expires in 27 days

WARNING:
ipahealthcheck.ipa.certs.IPACertfileExpirationCheck.20190425210054: Request
id 20190425210054 expires in 27 days

ERROR: ipahealthcheck.ipa.certs.IPARAAgent: RA agent description does not
match 2;186;CN=Certificate Authority,O=EXAMPLE.COM;CN=IPA RA,O=EXAMPLE.COM
in LDAP and 2;66;CN=Certificate Authority,O=EXAMPLE.COM;CN=IPA RA,O=
EXAMPLE.COM expected

ERROR: ipahealthcheck.ipa.certs.IPACertRevocation.20190425210040: Request
for certificate failed, Certificate operation cannot be completed:
EXCEPTION (Invalid Credential.)

ERROR: ipahealthcheck.ipa.certs.IPACertRevocation.20190425210052: Request
for certificate failed, Certificate operation cannot be completed:
EXCEPTION (Invalid Credential.)

ERROR: ipahealthcheck.ipa.certs.IPACertRevocation.20190425210053: Request
for certificate failed, Certificate operation cannot be completed:
EXCEPTION (Invalid Credential.)

ERROR: ipahealthcheck.ipa.certs.IPACertRevocation.20190425210054: Request
for certificate failed, Certificate operation cannot be completed:
EXCEPTION (Invalid Credential.)

ERROR: ipahealthcheck.ipa.certs.IPACertRevocation.20190425210055: Request
for certificate failed, Certificate operation cannot be completed:
EXCEPTION (Invalid Credential.)

ERROR: ipahealthcheck.ipa.certs.IPACertRevocation.20190425210056: Request
for certificate failed, Certificate operation cannot be completed:
EXCEPTION (Invalid Credential.)

ERROR: ipahealthcheck.ipa.certs.IPACertRevocation.20190425205849: Request
for certificate failed, Certificate operation cannot be completed:
EXCEPTION (Invalid Credential.)

ERROR: ipahealthcheck.ipa.certs.IPACertRevocation.20190425205831: Request
for certificate failed, Certificate operation cannot be completed:
EXCEPTION (Invalid Credential.)

ERROR: ipahealthcheck.ipa.certs.IPACertRevocation.20190425210120: Request
for certificate failed, Certificate operation cannot be completed:
EXCEPTION (Invalid Credential.)

ERROR: ipahealthcheck.dogtag.ca.DogtagCertsConnectivityCheck: Request for
certificate failed, Certificate operation cannot be completed: EXCEPTION
(Invalid Credential.)

[root@ipa2 ~]#



The list of certs:


[root@ipa2 ~]# getcert list

Number of certificates and requests being tracked: 9.

Request ID '20190425205831':

status: MONITORING

stuck: no

key pair storage:
type=NSSDB,location='/etc/dirsrv/slapd-EXAMPLE-COM',nickname='Server-Cert',token='NSS
Certificate DB',pinfile='/etc/dirsrv/slapd-EXAMPLE-COM/pwdfile.txt'

certificate:
type=NSSDB,location='/etc/dirsrv/slapd-EXAMPLE-COM',nickname='Server-Cert',token='NSS
Certificate DB'

CA: IPA

issuer: CN=Certificate Authority,O=EXAMPLE.COM

subject: CN=ipa2.example.com,O=EXAMPLE.COM

expires: 2023-03-29 21:37:22 UTC

dns: ipa2.example.com

principal name: ldap/ipa2.example.com(a)EXAMPLE.COM

key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment

eku: id-kp-serverAuth,id-kp-clientAuth

pre-save command:

post-save command: /usr/libexec/ipa/certmonger/restart_dirsrv EXAMPLE-COM

track: yes

auto-renew: yes

Request ID '20190425205849':

status: MONITORING

stuck: no

key pair storage:
type=NSSDB,location='/etc/httpd/alias',nickname='Server-Cert',token='NSS
Certificate DB',pinfile='/etc/httpd/alias/pwdfile.txt'

certificate:
type=NSSDB,location='/etc/httpd/alias',nickname='Server-Cert',token='NSS
Certificate DB'

CA: IPA

issuer: CN=Certificate Authority,O=EXAMPLE.COM

subject: CN=ipa2.example.com,O=EXAMPLE.COM

expires: 2023-03-29 21:37:46 UTC

dns: ipa2.example.com

principal name: HTTP/ipa2.example.com(a)EXAMPLE.COM

key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment

eku: id-kp-serverAuth,id-kp-clientAuth

pre-save command:

post-save command: /usr/libexec/ipa/certmonger/restart_httpd

track: yes

auto-renew: yes

Request ID '20190425210040':

status: MONITORING

stuck: no

key pair storage: type=FILE,location='/var/lib/ipa/ra-agent.key'

certificate: type=FILE,location='/var/lib/ipa/ra-agent.pem'

CA: dogtag-ipa-ca-renew-agent

issuer: CN=Certificate Authority,O=EXAMPLE.COM

subject: CN=IPA RA,O=EXAMPLE.COM

expires: 2022-05-11 03:40:55 UTC

key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment

eku: id-kp-serverAuth,id-kp-clientAuth

pre-save command: /usr/libexec/ipa/certmonger/renew_ra_cert_pre

post-save command: /usr/libexec/ipa/certmonger/renew_ra_cert

track: yes

auto-renew: yes

Request ID '20190425210052':

status: MONITORING

stuck: no

key pair storage:
type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='auditSigningCert
cert-pki-ca',token='NSS Certificate DB',pin set

certificate:
type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='auditSigningCert
cert-pki-ca',token='NSS Certificate DB'

CA: dogtag-ipa-ca-renew-agent

issuer: CN=Certificate Authority,O=EXAMPLE.COM

subject: CN=CA Audit,O=EXAMPLE.COM

expires: 2022-05-11 03:40:05 UTC

key usage: digitalSignature,nonRepudiation

pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad

post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert
"auditSigningCert cert-pki-ca"

track: yes

auto-renew: yes

Request ID '20190425210053':

status: MONITORING

stuck: no

key pair storage:
type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='ocspSigningCert
cert-pki-ca',token='NSS Certificate DB',pin set

certificate:
type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='ocspSigningCert
cert-pki-ca',token='NSS Certificate DB'

CA: dogtag-ipa-ca-renew-agent

issuer: CN=Certificate Authority,O=EXAMPLE.COM

subject: CN=OCSP Subsystem,O=EXAMPLE.COM

expires: 2022-05-11 03:40:25 UTC

eku: id-kp-OCSPSigning

pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad

post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert
"ocspSigningCert cert-pki-ca"

track: yes

auto-renew: yes

Request ID '20190425210054':

status: MONITORING

stuck: no

key pair storage:
type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='subsystemCert
cert-pki-ca',token='NSS Certificate DB',pin set

certificate:
type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='subsystemCert
cert-pki-ca',token='NSS Certificate DB'

CA: dogtag-ipa-ca-renew-agent

issuer: CN=Certificate Authority,O=EXAMPLE.COM

subject: CN=CA Subsystem,O=EXAMPLE.COM

expires: 2022-05-11 03:40:05 UTC

key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment

eku: id-kp-serverAuth,id-kp-clientAuth

pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad

post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert "subsystemCert
cert-pki-ca"

track: yes

auto-renew: yes

Request ID '20190425210055':

status: MONITORING

stuck: no

key pair storage:
type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='caSigningCert
cert-pki-ca',token='NSS Certificate DB',pin set

certificate:
type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='caSigningCert
cert-pki-ca',token='NSS Certificate DB'

CA: dogtag-ipa-ca-renew-agent

issuer: CN=Certificate Authority,O=EXAMPLE.COM

subject: CN=Certificate Authority,O=EXAMPLE.COM

expires: 2038-06-28 21:19:45 UTC

key usage: digitalSignature,nonRepudiation,keyCertSign,cRLSign

pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad

post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert "caSigningCert
cert-pki-ca"

track: yes

auto-renew: yes

Request ID '20190425210056':

status: MONITORING

stuck: no

key pair storage:
type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='Server-Cert
cert-pki-ca',token='NSS Certificate DB',pin set

certificate:
type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='Server-Cert
cert-pki-ca',token='NSS Certificate DB'

CA: dogtag-ipa-ca-renew-agent

issuer: CN=Certificate Authority,O=EXAMPLE.COM

subject: CN=ipa2.example.com,O=EXAMPLE.COM

expires: 2023-03-07 22:37:22 UTC

key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment

eku: id-kp-serverAuth,id-kp-clientAuth,id-kp-emailProtection

pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad

post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert "Server-Cert
cert-pki-ca"

track: yes

auto-renew: yes

Request ID '20190425210120':

status: MONITORING

stuck: no

key pair storage: type=FILE,location='/var/kerberos/krb5kdc/kdc.key'

certificate: type=FILE,location='/var/kerberos/krb5kdc/kdc.crt'

CA: IPA

issuer: CN=Certificate Authority,O=EXAMPLE.COM

subject: CN=ipa2.example.com,O=EXAMPLE.COM

expires: 2023-03-29 21:37:52 UTC

principal name: krbtgt/EXAMPLE.COM(a)EXAMPLE.COM

key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment

eku: id-kp-serverAuth,id-pkinit-KPKdc

pre-save command:

post-save command: /usr/libexec/ipa/certmonger/renew_kdc_cert

track: yes

auto-renew: yes

[root@ipa2 ~]#




There are 4 certs which expire on 2022-05-11 which match "expires in 27
days". Take 20190425210040 as an example, we have:




WARNING:
ipahealthcheck.ipa.certs.IPACertmongerExpirationCheck.20190425210040:
Request id 20190425210040 expires in 27 days

WARNING:
ipahealthcheck.ipa.certs.IPACertfileExpirationCheck.20190425210040: Request
id 20190425210040 expires in 27 days

ERROR: ipahealthcheck.ipa.certs.IPACertRevocation.20190425210040: Request
for certificate failed, Certificate operation cannot be completed:
EXCEPTION (Invalid Credential.)


Request ID '20190425210040':

        status: MONITORING

        stuck: no

        key pair storage: type=FILE,location='/var/lib/ipa/ra-agent.key'

        certificate: type=FILE,location='/var/lib/ipa/ra-agent.pem'

        CA: dogtag-ipa-ca-renew-agent

        issuer: CN=Certificate Authority,O=EXAMPLE.COM

        subject: CN=IPA RA,O=EXAMPLE.COM

        expires: 2022-05-11 03:40:55 UTC

        key usage:
digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment

        eku: id-kp-serverAuth,id-kp-clientAuth

        pre-save command: /usr/libexec/ipa/certmonger/renew_ra_cert_pre

        post-save command: /usr/libexec/ipa/certmonger/renew_ra_cert

        track: yes

        auto-renew: yes



I was able to manually renew it:



[root@ipa2 ~]# ipa-getcert resubmit -i '20190425210040'

Resubmitting "20190425210040" to "dogtag-ipa-ca-renew-agent".

[root@ipa2 ~]#



After renew, it "expires: 2024-04-02 06:09:32 UTC":



Request ID '20190425210040':

        status: MONITORING

        stuck: no

        key pair storage: type=FILE,location='/var/lib/ipa/ra-agent.key'

        certificate: type=FILE,location='/var/lib/ipa/ra-agent.pem'

        CA: dogtag-ipa-ca-renew-agent

        issuer: CN=Certificate Authority,O=EXAMPLE.COM

        subject: CN=IPA RA,O=EXAMPLE.COM

        expires: 2024-04-02 06:09:32 UTC

        key usage:
digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment

        eku: id-kp-serverAuth,id-kp-clientAuth

        pre-save command: /usr/libexec/ipa/certmonger/renew_ra_cert_pre

        post-save command: /usr/libexec/ipa/certmonger/renew_ra_cert

        track: yes

        auto-renew: yes



How to fix the issue reported by ipa-healthcheck? And what is this issue
about?


All IPA servers are at same level:


CentOS Linux release 7.9.2009 (Core)

ipa-*server*.x86_64                     4.6.8-5.el7.centos.7

*slapi-nis*.x86_64                      0.56.5-3.el7_9

*389-ds-base*.x86_64                    1.3.10.2-12.el7_9

*389-ds-base*-libs.x86_64               1.3.10.2-12.el7_9


Many thanks!


Kathy.

[Kathy Zhu]

I just found this post about the same or similar issue:

https://lists.fedoraproject.org/archives/list/[e-mail]...

One detail I missed - this happens on all IPA servers BUT the renewal IPA
server. I will go through ^ post to see if that applies to our situation.

Thanks.

Kathy.


On Wed, Apr 13, 2022 at 10:21 AM Kathy Zhu  wrote:

>  Hi team,
>
>
>  ipa-healthcheck has been a great tool for us. I run it weekly on all IPA
>  servers via cron. This week ipa-healthcheck reported errors on all IPA
>  servers.
>
>
>  Take IPA server ipa2 as an example for the investigation:
>
>
>
>  [root@ipa2 ~]# ipa-healthcheck --failures-only --output-type=human
>
>  ra.get_certificate(): EXCEPTION (Invalid Credential.)
>
>  ra.get_certificate(): EXCEPTION (Invalid Credential.)
>
>  ra.get_certificate(): EXCEPTION (Invalid Credential.)
>
>  ra.get_certificate(): EXCEPTION (Invalid Credential.)
>
>  ra.get_certificate(): EXCEPTION (Invalid Credential.)
>
>  ra.get_certificate(): EXCEPTION (Invalid Credential.)
>
>  ra.get_certificate(): EXCEPTION (Invalid Credential.)
>
>  ra.get_certificate(): EXCEPTION (Invalid Credential.)
>
>  ra.get_certificate(): EXCEPTION (Invalid Credential.)
>
>  ra.get_certificate(): EXCEPTION (Invalid Credential.)
>
>  WARNING:
>  ipahealthcheck.ipa.certs.IPACertmongerExpirationCheck.20190425210040:
>  Request id 20190425210040 expires in 27 days
>
>  WARNING:
>  ipahealthcheck.ipa.certs.IPACertmongerExpirationCheck.20190425210052:
>  Request id 20190425210052 expires in 27 days
>
>  WARNING:
>  ipahealthcheck.ipa.certs.IPACertmongerExpirationCheck.20190425210053:
>  Request id 20190425210053 expires in 27 days
>
>  WARNING:
>  ipahealthcheck.ipa.certs.IPACertmongerExpirationCheck.20190425210054:
>  Request id 20190425210054 expires in 27 days
>
>  WARNING:
>  ipahealthcheck.ipa.certs.IPACertfileExpirationCheck.20190425210040: Request
>  id 20190425210040 expires in 27 days
>
>  WARNING:
>  ipahealthcheck.ipa.certs.IPACertfileExpirationCheck.20190425210052: Request
>  id 20190425210052 expires in 27 days
>
>  WARNING:
>  ipahealthcheck.ipa.certs.IPACertfileExpirationCheck.20190425210053: Request
>  id 20190425210053 expires in 27 days
>
>  WARNING:
>  ipahealthcheck.ipa.certs.IPACertfileExpirationCheck.20190425210054: Request
>  id 20190425210054 expires in 27 days
>
>  ERROR: ipahealthcheck.ipa.certs.IPARAAgent: RA agent description does not
>  match 2;186;CN=Certificate Authority,O=EXAMPLE.COM;CN=IPA RA,O=EXAMPLE.COM
>  in LDAP and 2;66;CN=Certificate Authority,O=EXAMPLE.COM;CN=IPA RA,O=
>  EXAMPLE.COM expected
>
>  ERROR: ipahealthcheck.ipa.certs.IPACertRevocation.20190425210040: Request
>  for certificate failed, Certificate operation cannot be completed:
>  EXCEPTION (Invalid Credential.)
>
>  ERROR: ipahealthcheck.ipa.certs.IPACertRevocation.20190425210052: Request
>  for certificate failed, Certificate operation cannot be completed:
>  EXCEPTION (Invalid Credential.)
>
>  ERROR: ipahealthcheck.ipa.certs.IPACertRevocation.20190425210053: Request
>  for certificate failed, Certificate operation cannot be completed:
>  EXCEPTION (Invalid Credential.)
>
>  ERROR: ipahealthcheck.ipa.certs.IPACertRevocation.20190425210054: Request
>  for certificate failed, Certificate operation cannot be completed:
>  EXCEPTION (Invalid Credential.)
>
>  ERROR: ipahealthcheck.ipa.certs.IPACertRevocation.20190425210055: Request
>  for certificate failed, Certificate operation cannot be completed:
>  EXCEPTION (Invalid Credential.)
>
>  ERROR: ipahealthcheck.ipa.certs.IPACertRevocation.20190425210056: Request
>  for certificate failed, Certificate operation cannot be completed:
>  EXCEPTION (Invalid Credential.)
>
>  ERROR: ipahealthcheck.ipa.certs.IPACertRevocation.20190425205849: Request
>  for certificate failed, Certificate operation cannot be completed:
>  EXCEPTION (Invalid Credential.)
>
>  ERROR: ipahealthcheck.ipa.certs.IPACertRevocation.20190425205831: Request
>  for certificate failed, Certificate operation cannot be completed:
>  EXCEPTION (Invalid Credential.)
>
>  ERROR: ipahealthcheck.ipa.certs.IPACertRevocation.20190425210120: Request
>  for certificate failed, Certificate operation cannot be completed:
>  EXCEPTION (Invalid Credential.)
>
>  ERROR: ipahealthcheck.dogtag.ca.DogtagCertsConnectivityCheck: Request for
>  certificate failed, Certificate operation cannot be completed: EXCEPTION
>  (Invalid Credential.)
>
>  [root@ipa2 ~]#
>
>
>
>  The list of certs:
>
>
>  [root@ipa2 ~]# getcert list
>
>  Number of certificates and requests being tracked: 9.
>
>  Request ID '20190425205831':
>
>  status: MONITORING
>
>  stuck: no
>
>  key pair storage:
>
> type=NSSDB,location='/etc/dirsrv/slapd-EXAMPLE-COM',nickname='Server-Cert',token='NSS
>  Certificate DB',pinfile='/etc/dirsrv/slapd-EXAMPLE-COM/pwdfile.txt'
>
>  certificate:
>
> type=NSSDB,location='/etc/dirsrv/slapd-EXAMPLE-COM',nickname='Server-Cert',token='NSS
>  Certificate DB'
>
>  CA: IPA
>
>  issuer: CN=Certificate Authority,O=EXAMPLE.COM
>
>  subject: CN=ipa2.example.com,O=EXAMPLE.COM
>
>  expires: 2023-03-29 21:37:22 UTC
>
>  dns: ipa2.example.com
>
>  principal name: ldap/ipa2.example.com(a)EXAMPLE.COM
>
>  key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment
>
>  eku: id-kp-serverAuth,id-kp-clientAuth
>
>  pre-save command:
>
>  post-save command: /usr/libexec/ipa/certmonger/restart_dirsrv EXAMPLE-COM
>
>  track: yes
>
>  auto-renew: yes
>
>  Request ID '20190425205849':
>
>  status: MONITORING
>
>  stuck: no
>
>  key pair storage:
>
> type=NSSDB,location='/etc/httpd/alias',nickname='Server-Cert',token='NSS
>  Certificate DB',pinfile='/etc/httpd/alias/pwdfile.txt'
>
>  certificate:
>
> type=NSSDB,location='/etc/httpd/alias',nickname='Server-Cert',token='NSS
>  Certificate DB'
>
>  CA: IPA
>
>  issuer: CN=Certificate Authority,O=EXAMPLE.COM
>
>  subject: CN=ipa2.example.com,O=EXAMPLE.COM
>
>  expires: 2023-03-29 21:37:46 UTC
>
>  dns: ipa2.example.com
>
>  principal name: HTTP/ipa2.example.com(a)EXAMPLE.COM
>
>  key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment
>
>  eku: id-kp-serverAuth,id-kp-clientAuth
>
>  pre-save command:
>
>  post-save command: /usr/libexec/ipa/certmonger/restart_httpd
>
>  track: yes
>
>  auto-renew: yes
>
>  Request ID '20190425210040':
>
>  status: MONITORING
>
>  stuck: no
>
>  key pair storage: type=FILE,location='/var/lib/ipa/ra-agent.key'
>
>  certificate: type=FILE,location='/var/lib/ipa/ra-agent.pem'
>
>  CA: dogtag-ipa-ca-renew-agent
>
>  issuer: CN=Certificate Authority,O=EXAMPLE.COM
>
>  subject: CN=IPA RA,O=EXAMPLE.COM
>
>  expires: 2022-05-11 03:40:55 UTC
>
>  key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment
>
>  eku: id-kp-serverAuth,id-kp-clientAuth
>
>  pre-save command: /usr/libexec/ipa/certmonger/renew_ra_cert_pre
>
>  post-save command: /usr/libexec/ipa/certmonger/renew_ra_cert
>
>  track: yes
>
>  auto-renew: yes
>
>  Request ID '20190425210052':
>
>  status: MONITORING
>
>  stuck: no
>
>  key pair storage:
>  type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='auditSigningCert
>  cert-pki-ca',token='NSS Certificate DB',pin set
>
>  certificate:
>  type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='auditSigningCert
>  cert-pki-ca',token='NSS Certificate DB'
>
>  CA: dogtag-ipa-ca-renew-agent
>
>  issuer: CN=Certificate Authority,O=EXAMPLE.COM
>
>  subject: CN=CA Audit,O=EXAMPLE.COM
>
>  expires: 2022-05-11 03:40:05 UTC
>
>  key usage: digitalSignature,nonRepudiation
>
>  pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad
>
>  post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert
>  "auditSigningCert cert-pki-ca"
>
>  track: yes
>
>  auto-renew: yes
>
>  Request ID '20190425210053':
>
>  status: MONITORING
>
>  stuck: no
>
>  key pair storage:
>  type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='ocspSigningCert
>  cert-pki-ca',token='NSS Certificate DB',pin set
>
>  certificate:
>  type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='ocspSigningCert
>  cert-pki-ca',token='NSS Certificate DB'
>
>  CA: dogtag-ipa-ca-renew-agent
>
>  issuer: CN=Certificate Authority,O=EXAMPLE.COM
>
>  subject: CN=OCSP Subsystem,O=EXAMPLE.COM
>
>  expires: 2022-05-11 03:40:25 UTC
>
>  eku: id-kp-OCSPSigning
>
>  pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad
>
>  post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert
>  "ocspSigningCert cert-pki-ca"
>
>  track: yes
>
>  auto-renew: yes
>
>  Request ID '20190425210054':
>
>  status: MONITORING
>
>  stuck: no
>
>  key pair storage:
>  type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='subsystemCert
>  cert-pki-ca',token='NSS Certificate DB',pin set
>
>  certificate:
>  type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='subsystemCert
>  cert-pki-ca',token='NSS Certificate DB'
>
>  CA: dogtag-ipa-ca-renew-agent
>
>  issuer: CN=Certificate Authority,O=EXAMPLE.COM
>
>  subject: CN=CA Subsystem,O=EXAMPLE.COM
>
>  expires: 2022-05-11 03:40:05 UTC
>
>  key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment
>
>  eku: id-kp-serverAuth,id-kp-clientAuth
>
>  pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad
>
>  post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert
>  "subsystemCert cert-pki-ca"
>
>  track: yes
>
>  auto-renew: yes
>
>  Request ID '20190425210055':
>
>  status: MONITORING
>
>  stuck: no
>
>  key pair storage:
>  type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='caSigningCert
>  cert-pki-ca',token='NSS Certificate DB',pin set
>
>  certificate:
>  type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='caSigningCert
>  cert-pki-ca',token='NSS Certificate DB'
>
>  CA: dogtag-ipa-ca-renew-agent
>
>  issuer: CN=Certificate Authority,O=EXAMPLE.COM
>
>  subject: CN=Certificate Authority,O=EXAMPLE.COM
>
>  expires: 2038-06-28 21:19:45 UTC
>
>  key usage: digitalSignature,nonRepudiation,keyCertSign,cRLSign
>
>  pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad
>
>  post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert
>  "caSigningCert cert-pki-ca"
>
>  track: yes
>
>  auto-renew: yes
>
>  Request ID '20190425210056':
>
>  status: MONITORING
>
>  stuck: no
>
>  key pair storage:
>  type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='Server-Cert
>  cert-pki-ca',token='NSS Certificate DB',pin set
>
>  certificate:
>  type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='Server-Cert
>  cert-pki-ca',token='NSS Certificate DB'
>
>  CA: dogtag-ipa-ca-renew-agent
>
>  issuer: CN=Certificate Authority,O=EXAMPLE.COM
>
>  subject: CN=ipa2.example.com,O=EXAMPLE.COM
>
>  expires: 2023-03-07 22:37:22 UTC
>
>  key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment
>
>  eku: id-kp-serverAuth,id-kp-clientAuth,id-kp-emailProtection
>
>  pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad
>
>  post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert "Server-Cert
>  cert-pki-ca"
>
>  track: yes
>
>  auto-renew: yes
>
>  Request ID '20190425210120':
>
>  status: MONITORING
>
>  stuck: no
>
>  key pair storage: type=FILE,location='/var/kerberos/krb5kdc/kdc.key'
>
>  certificate: type=FILE,location='/var/kerberos/krb5kdc/kdc.crt'
>
>  CA: IPA
>
>  issuer: CN=Certificate Authority,O=EXAMPLE.COM
>
>  subject: CN=ipa2.example.com,O=EXAMPLE.COM
>
>  expires: 2023-03-29 21:37:52 UTC
>
>  principal name: krbtgt/EXAMPLE.COM(a)EXAMPLE.COM
>
>  key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment
>
>  eku: id-kp-serverAuth,id-pkinit-KPKdc
>
>  pre-save command:
>
>  post-save command: /usr/libexec/ipa/certmonger/renew_kdc_cert
>
>  track: yes
>
>  auto-renew: yes
>
>  [root@ipa2 ~]#
>
>
>
>
>  There are 4 certs which expire on 2022-05-11 which match "expires in 27
>  days". Take 20190425210040 as an example, we have:
>
>
>
>
>  WARNING:
>  ipahealthcheck.ipa.certs.IPACertmongerExpirationCheck.20190425210040:
>  Request id 20190425210040 expires in 27 days
>
>  WARNING:
>  ipahealthcheck.ipa.certs.IPACertfileExpirationCheck.20190425210040: Request
>  id 20190425210040 expires in 27 days
>
>  ERROR: ipahealthcheck.ipa.certs.IPACertRevocation.20190425210040: Request
>  for certificate failed, Certificate operation cannot be completed:
>  EXCEPTION (Invalid Credential.)
>
>
>  Request ID '20190425210040':
>
>          status: MONITORING
>
>          stuck: no
>
>          key pair storage: type=FILE,location='/var/lib/ipa/ra-agent.key'
>
>          certificate: type=FILE,location='/var/lib/ipa/ra-agent.pem'
>
>          CA: dogtag-ipa-ca-renew-agent
>
>          issuer: CN=Certificate Authority,O=EXAMPLE.COM
>
>          subject: CN=IPA RA,O=EXAMPLE.COM
>
>          expires: 2022-05-11 03:40:55 UTC
>
>          key usage:
>  digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment
>
>          eku: id-kp-serverAuth,id-kp-clientAuth
>
>          pre-save command: /usr/libexec/ipa/certmonger/renew_ra_cert_pre
>
>          post-save command: /usr/libexec/ipa/certmonger/renew_ra_cert
>
>          track: yes
>
>          auto-renew: yes
>
>
>
>  I was able to manually renew it:
>
>
>
>  [root@ipa2 ~]# ipa-getcert resubmit -i '20190425210040'
>
>  Resubmitting "20190425210040" to "dogtag-ipa-ca-renew-agent".
>
>  [root@ipa2 ~]#
>
>
>
>  After renew, it "expires: 2024-04-02 06:09:32 UTC":
>
>
>
>  Request ID '20190425210040':
>
>          status: MONITORING
>
>          stuck: no
>
>          key pair storage: type=FILE,location='/var/lib/ipa/ra-agent.key'
>
>          certificate: type=FILE,location='/var/lib/ipa/ra-agent.pem'
>
>          CA: dogtag-ipa-ca-renew-agent
>
>          issuer: CN=Certificate Authority,O=EXAMPLE.COM
>
>          subject: CN=IPA RA,O=EXAMPLE.COM
>
>          expires: 2024-04-02 06:09:32 UTC
>
>          key usage:
>  digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment
>
>          eku: id-kp-serverAuth,id-kp-clientAuth
>
>          pre-save command: /usr/libexec/ipa/certmonger/renew_ra_cert_pre
>
>          post-save command: /usr/libexec/ipa/certmonger/renew_ra_cert
>
>          track: yes
>
>          auto-renew: yes
>
>
>
>  How to fix the issue reported by ipa-healthcheck? And what is this issue
>  about?
>
>
>  All IPA servers are at same level:
>
>
>  CentOS Linux release 7.9.2009 (Core)
>
>  ipa-*server*.x86_64                     4.6.8-5.el7.centos.7
>
>  *slapi-nis*.x86_64                      0.56.5-3.el7_9
>
>  *389-ds-base*.x86_64                    1.3.10.2-12.el7_9
>
>  *389-ds-base*-libs.x86_64               1.3.10.2-12.el7_9
>
>
>  Many thanks!
>
>
>  Kathy.
>

[Rob Crittenden]

This looks like the root cause:

ERROR: ipahealthcheck.ipa.certs.IPARAAgent: RA agent description does
not match 2;186;CN=Certificate Authority,O=EXAMPLE.COM;CN=IPA
RA,O=EXAMPLE.COM in LDAP and 2;66;CN=Certificate
Authority,O=EXAMPLE.COM;CN=IPA RA,O=EXAMPLE.COM expected

It looks like an updated RA certificate was issued and was not picked up
by the mother server(s).

Off the top of my head this could be:

- replication of the CA data has a problem: ipa-csreplica-manage list -v
`hostname`
- The updated certificate wasn't published to
cn=certificates,cn=ipa,cn=etc,$SUFFIX
- certmonger isn't picking up the renewal for some reason. The journal
may hold clues.
- something I'm forgetting

I'd start with the first two.

rob


Kathy Zhu via FreeIPA-users wrote:
>  I just found this post about the same or similar issue: 
>  
>
> https://lists.fedoraproject.org/archives/list/[e-mail]...
>  
>  One detail I missed - this happens on all IPA servers BUT the renewal
>  IPA server. I will go through ^ post to see if that applies to our
>  situation. 
>  
>  Thanks. 
>  
>  Kathy. 
>  
>  
>  On Wed, Apr 13, 2022 at 10:21 AM Kathy Zhu  wrote:
>  
>      Hi team, 
>  
>  
>      ipa-healthcheck has been a great tool for us. I run it weekly on all
>      IPA servers via cron. This week ipa-healthcheck reported errors on
>      all IPA servers. 
>  
>  
>      Take IPA server ipa2 as an example for the investigation: 
>  
>  
>  
>      [root@ipa2 ~]# ipa-healthcheck --failures-only --output-type=human
>  
>      ra.get_certificate(): EXCEPTION (Invalid Credential.)
>  
>      ra.get_certificate(): EXCEPTION (Invalid Credential.)
>  
>      ra.get_certificate(): EXCEPTION (Invalid Credential.)
>  
>      ra.get_certificate(): EXCEPTION (Invalid Credential.)
>  
>      ra.get_certificate(): EXCEPTION (Invalid Credential.)
>  
>      ra.get_certificate(): EXCEPTION (Invalid Credential.)
>  
>      ra.get_certificate(): EXCEPTION (Invalid Credential.)
>  
>      ra.get_certificate(): EXCEPTION (Invalid Credential.)
>  
>      ra.get_certificate(): EXCEPTION (Invalid Credential.)
>  
>      ra.get_certificate(): EXCEPTION (Invalid Credential.)
>  
>      WARNING:
>      ipahealthcheck.ipa.certs.IPACertmongerExpirationCheck.20190425210040: Request
>      id 20190425210040 expires in 27 days
>  
>      WARNING:
>      ipahealthcheck.ipa.certs.IPACertmongerExpirationCheck.20190425210052: Request
>      id 20190425210052 expires in 27 days
>  
>      WARNING:
>      ipahealthcheck.ipa.certs.IPACertmongerExpirationCheck.20190425210053: Request
>      id 20190425210053 expires in 27 days
>  
>      WARNING:
>      ipahealthcheck.ipa.certs.IPACertmongerExpirationCheck.20190425210054: Request
>      id 20190425210054 expires in 27 days
>  
>      WARNING:
>      ipahealthcheck.ipa.certs.IPACertfileExpirationCheck.20190425210040:
>      Request id 20190425210040 expires in 27 days
>  
>      WARNING:
>      ipahealthcheck.ipa.certs.IPACertfileExpirationCheck.20190425210052:
>      Request id 20190425210052 expires in 27 days
>  
>      WARNING:
>      ipahealthcheck.ipa.certs.IPACertfileExpirationCheck.20190425210053:
>      Request id 20190425210053 expires in 27 days
>  
>      WARNING:
>      ipahealthcheck.ipa.certs.IPACertfileExpirationCheck.20190425210054:
>      Request id 20190425210054 expires in 27 days
>  
>      ERROR: ipahealthcheck.ipa.certs.IPARAAgent: RA agent description
>      does not match 2;186;CN=Certificate Authority,O=EXAMPLE.COM
>      <http://EXAMPLE.COM>;CN=IPA RA,O=EXAMPLE.COM <http://EXAMPLE.COM> in
>      LDAP and 2;66;CN=Certificate Authority,O=EXAMPLE.COM
>      <http://EXAMPLE.COM>;CN=IPA RA,O=EXAMPLE.COM <http://EXAMPLE.COM>
>      expected
>  
>      ERROR: ipahealthcheck.ipa.certs.IPACertRevocation.20190425210040:
>      Request for certificate failed, Certificate operation cannot be
>      completed: EXCEPTION (Invalid Credential.)
>  
>      ERROR: ipahealthcheck.ipa.certs.IPACertRevocation.20190425210052:
>      Request for certificate failed, Certificate operation cannot be
>      completed: EXCEPTION (Invalid Credential.)
>  
>      ERROR: ipahealthcheck.ipa.certs.IPACertRevocation.20190425210053:
>      Request for certificate failed, Certificate operation cannot be
>      completed: EXCEPTION (Invalid Credential.)
>  
>      ERROR: ipahealthcheck.ipa.certs.IPACertRevocation.20190425210054:
>      Request for certificate failed, Certificate operation cannot be
>      completed: EXCEPTION (Invalid Credential.)
>  
>      ERROR: ipahealthcheck.ipa.certs.IPACertRevocation.20190425210055:
>      Request for certificate failed, Certificate operation cannot be
>      completed: EXCEPTION (Invalid Credential.)
>  
>      ERROR: ipahealthcheck.ipa.certs.IPACertRevocation.20190425210056:
>      Request for certificate failed, Certificate operation cannot be
>      completed: EXCEPTION (Invalid Credential.)
>  
>      ERROR: ipahealthcheck.ipa.certs.IPACertRevocation.20190425205849:
>      Request for certificate failed, Certificate operation cannot be
>      completed: EXCEPTION (Invalid Credential.)
>  
>      ERROR: ipahealthcheck.ipa.certs.IPACertRevocation.20190425205831:
>      Request for certificate failed, Certificate operation cannot be
>      completed: EXCEPTION (Invalid Credential.)
>  
>      ERROR: ipahealthcheck.ipa.certs.IPACertRevocation.20190425210120:
>      Request for certificate failed, Certificate operation cannot be
>      completed: EXCEPTION (Invalid Credential.)
>  
>      ERROR: ipahealthcheck.dogtag.ca.DogtagCertsConnectivityCheck:
>      Request for certificate failed, Certificate operation cannot be
>      completed: EXCEPTION (Invalid Credential.)
>  
>      [root@ipa2 ~]# 
>  
>  
>  
>      The list of certs: 
>  
>  
>      [root@ipa2 ~]# getcert list 
>  
>      Number of certificates and requests being tracked: 9.
>  
>      Request ID '20190425205831':
>  
>      status: MONITORING
>  
>      stuck: no
>  
>      key pair storage:
>     
> type=NSSDB,location='/etc/dirsrv/slapd-EXAMPLE-COM',nickname='Server-Cert',token='NSS
>      Certificate DB',pinfile='/etc/dirsrv/slapd-EXAMPLE-COM/pwdfile.txt'
>  
>      certificate:
>     
> type=NSSDB,location='/etc/dirsrv/slapd-EXAMPLE-COM',nickname='Server-Cert',token='NSS
>      Certificate DB'
>  
>      CA: IPA
>  
>      issuer: CN=Certificate Authority,O=EXAMPLE.COM <http://EXAMPLE.COM>
>  
>      subject: CN=ipa2.example.com <http://ipa2.example.com>,O=EXAMPLE.COM
>      <http://EXAMPLE.COM>
>  
>      expires: 2023-03-29 21:37:22 UTC
>  
>      dns: ipa2.example.com <http://ipa2.example.com>
>  
>      principal name: ldap/ipa2.example.com(a)EXAMPLE.COM
>      <mailto:[e-mail]>
>  
>      key usage:
>      digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment
>  
>      eku: id-kp-serverAuth,id-kp-clientAuth
>  
>      pre-save command: 
>  
>      post-save command: /usr/libexec/ipa/certmonger/restart_dirsrv
>      EXAMPLE-COM
>  
>      track: yes
>  
>      auto-renew: yes
>  
>      Request ID '20190425205849':
>  
>      status: MONITORING
>  
>      stuck: no
>  
>      key pair storage:
>     
> type=NSSDB,location='/etc/httpd/alias',nickname='Server-Cert',token='NSS
>      Certificate DB',pinfile='/etc/httpd/alias/pwdfile.txt'
>  
>      certificate:
>     
> type=NSSDB,location='/etc/httpd/alias',nickname='Server-Cert',token='NSS
>      Certificate DB'
>  
>      CA: IPA
>  
>      issuer: CN=Certificate Authority,O=EXAMPLE.COM <http://EXAMPLE.COM>
>  
>      subject: CN=ipa2.example.com <http://ipa2.example.com>,O=EXAMPLE.COM
>      <http://EXAMPLE.COM>
>  
>      expires: 2023-03-29 21:37:46 UTC
>  
>      dns: ipa2.example.com <http://ipa2.example.com>
>  
>      principal name: HTTP/ipa2.example.com(a)EXAMPLE.COM
>      <mailto:[e-mail]>
>  
>      key usage:
>      digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment
>  
>      eku: id-kp-serverAuth,id-kp-clientAuth
>  
>      pre-save command: 
>  
>      post-save command: /usr/libexec/ipa/certmonger/restart_httpd
>  
>      track: yes
>  
>      auto-renew: yes
>  
>      Request ID '20190425210040':
>  
>      status: MONITORING
>  
>      stuck: no
>  
>      key pair storage: type=FILE,location='/var/lib/ipa/ra-agent.key'
>  
>      certificate: type=FILE,location='/var/lib/ipa/ra-agent.pem'
>  
>      CA: dogtag-ipa-ca-renew-agent
>  
>      issuer: CN=Certificate Authority,O=EXAMPLE.COM <http://EXAMPLE.COM>
>  
>      subject: CN=IPA RA,O=EXAMPLE.COM <http://EXAMPLE.COM>
>  
>      expires: 2022-05-11 03:40:55 UTC
>  
>      key usage:
>      digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment
>  
>      eku: id-kp-serverAuth,id-kp-clientAuth
>  
>      pre-save command: /usr/libexec/ipa/certmonger/renew_ra_cert_pre
>  
>      post-save command: /usr/libexec/ipa/certmonger/renew_ra_cert
>  
>      track: yes
>  
>      auto-renew: yes
>  
>      Request ID '20190425210052':
>  
>      status: MONITORING
>  
>      stuck: no
>  
>      key pair storage:
>     
> type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='auditSigningCert
>      cert-pki-ca',token='NSS Certificate DB',pin set
>  
>      certificate:
>     
> type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='auditSigningCert
>      cert-pki-ca',token='NSS Certificate DB'
>  
>      CA: dogtag-ipa-ca-renew-agent
>  
>      issuer: CN=Certificate Authority,O=EXAMPLE.COM <http://EXAMPLE.COM>
>  
>      subject: CN=CA Audit,O=EXAMPLE.COM <http://EXAMPLE.COM>
>  
>      expires: 2022-05-11 03:40:05 UTC
>  
>      key usage: digitalSignature,nonRepudiation
>  
>      pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad
>  
>      post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert
>      "auditSigningCert cert-pki-ca"
>  
>      track: yes
>  
>      auto-renew: yes
>  
>      Request ID '20190425210053':
>  
>      status: MONITORING
>  
>      stuck: no
>  
>      key pair storage:
>     
> type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='ocspSigningCert
>      cert-pki-ca',token='NSS Certificate DB',pin set
>  
>      certificate:
>     
> type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='ocspSigningCert
>      cert-pki-ca',token='NSS Certificate DB'
>  
>      CA: dogtag-ipa-ca-renew-agent
>  
>      issuer: CN=Certificate Authority,O=EXAMPLE.COM <http://EXAMPLE.COM>
>  
>      subject: CN=OCSP Subsystem,O=EXAMPLE.COM <http://EXAMPLE.COM>
>  
>      expires: 2022-05-11 03:40:25 UTC
>  
>      eku: id-kp-OCSPSigning
>  
>      pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad
>  
>      post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert
>      "ocspSigningCert cert-pki-ca"
>  
>      track: yes
>  
>      auto-renew: yes
>  
>      Request ID '20190425210054':
>  
>      status: MONITORING
>  
>      stuck: no
>  
>      key pair storage:
>      type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='subsystemCert
>      cert-pki-ca',token='NSS Certificate DB',pin set
>  
>      certificate:
>      type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='subsystemCert
>      cert-pki-ca',token='NSS Certificate DB'
>  
>      CA: dogtag-ipa-ca-renew-agent
>  
>      issuer: CN=Certificate Authority,O=EXAMPLE.COM <http://EXAMPLE.COM>
>  
>      subject: CN=CA Subsystem,O=EXAMPLE.COM <http://EXAMPLE.COM>
>  
>      expires: 2022-05-11 03:40:05 UTC
>  
>      key usage:
>      digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment
>  
>      eku: id-kp-serverAuth,id-kp-clientAuth
>  
>      pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad
>  
>      post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert
>      "subsystemCert cert-pki-ca"
>  
>      track: yes
>  
>      auto-renew: yes
>  
>      Request ID '20190425210055':
>  
>      status: MONITORING
>  
>      stuck: no
>  
>      key pair storage:
>      type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='caSigningCert
>      cert-pki-ca',token='NSS Certificate DB',pin set
>  
>      certificate:
>      type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='caSigningCert
>      cert-pki-ca',token='NSS Certificate DB'
>  
>      CA: dogtag-ipa-ca-renew-agent
>  
>      issuer: CN=Certificate Authority,O=EXAMPLE.COM <http://EXAMPLE.COM>
>  
>      subject: CN=Certificate Authority,O=EXAMPLE.COM <http://EXAMPLE.COM>
>  
>      expires: 2038-06-28 21:19:45 UTC
>  
>      key usage: digitalSignature,nonRepudiation,keyCertSign,cRLSign
>  
>      pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad
>  
>      post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert
>      "caSigningCert cert-pki-ca"
>  
>      track: yes
>  
>      auto-renew: yes
>  
>      Request ID '20190425210056':
>  
>      status: MONITORING
>  
>      stuck: no
>  
>      key pair storage:
>      type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='Server-Cert
> cert-pki-ca',token='NSS
>      Certificate DB',pin set
>  
>      certificate:
>      type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='Server-Cert
> cert-pki-ca',token='NSS
>      Certificate DB'
>  
>      CA: dogtag-ipa-ca-renew-agent
>  
>      issuer: CN=Certificate Authority,O=EXAMPLE.COM <http://EXAMPLE.COM>
>  
>      subject: CN=ipa2.example.com <http://ipa2.example.com>,O=EXAMPLE.COM
>      <http://EXAMPLE.COM>
>  
>      expires: 2023-03-07 22:37:22 UTC
>  
>      key usage:
>      digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment
>  
>      eku: id-kp-serverAuth,id-kp-clientAuth,id-kp-emailProtection
>  
>      pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad
>  
>      post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert
>      "Server-Cert cert-pki-ca"
>  
>      track: yes
>  
>      auto-renew: yes
>  
>      Request ID '20190425210120':
>  
>      status: MONITORING
>  
>      stuck: no
>  
>      key pair storage: type=FILE,location='/var/kerberos/krb5kdc/kdc.key'
>  
>      certificate: type=FILE,location='/var/kerberos/krb5kdc/kdc.crt'
>  
>      CA: IPA
>  
>      issuer: CN=Certificate Authority,O=EXAMPLE.COM <http://EXAMPLE.COM>
>  
>      subject: CN=ipa2.example.com <http://ipa2.example.com>,O=EXAMPLE.COM
>      <http://EXAMPLE.COM>
>  
>      expires: 2023-03-29 21:37:52 UTC
>  
>      principal name: krbtgt/EXAMPLE.COM(a)EXAMPLE.COM
>      <mailto:[e-mail]>
>  
>      key usage:
>      digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment
>  
>      eku: id-kp-serverAuth,id-pkinit-KPKdc
>  
>      pre-save command: 
>  
>      post-save command: /usr/libexec/ipa/certmonger/renew_kdc_cert
>  
>      track: yes
>  
>      auto-renew: yes
>  
>      [root@ipa2 ~]# 
>  
>  
>  
>  
>      There are 4 certs which expire on 2022-05-11 which match "expires in
>      27 days". Take 20190425210040 as an example, we have: 
>  
>       
>  
>  
>      WARNING:
>      ipahealthcheck.ipa.certs.IPACertmongerExpirationCheck.20190425210040: Request
>      id 20190425210040 expires in 27 days
>  
>      WARNING:
>      ipahealthcheck.ipa.certs.IPACertfileExpirationCheck.20190425210040:
>      Request id 20190425210040 expires in 27 days
>  
>      ERROR: ipahealthcheck.ipa.certs.IPACertRevocation.20190425210040:
>      Request for certificate failed, Certificate operation cannot be
>      completed: EXCEPTION (Invalid Credential.)
>  
>  
>      Request ID '20190425210040':
>  
>              status: MONITORING
>  
>              stuck: no
>  
>              key pair storage: type=FILE,location='/var/lib/ipa/ra-agent.key'
>  
>              certificate: type=FILE,location='/var/lib/ipa/ra-agent.pem'
>  
>              CA: dogtag-ipa-ca-renew-agent
>  
>              issuer: CN=Certificate Authority,O=EXAMPLE.COM
>      <http://EXAMPLE.COM>
>  
>              subject: CN=IPA RA,O=EXAMPLE.COM <http://EXAMPLE.COM>
>  
>              expires: 2022-05-11 03:40:55 UTC
>  
>              key usage:
>      digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment
>  
>              eku: id-kp-serverAuth,id-kp-clientAuth
>  
>              pre-save command: /usr/libexec/ipa/certmonger/renew_ra_cert_pre
>  
>              post-save command: /usr/libexec/ipa/certmonger/renew_ra_cert
>  
>              track: yes
>  
>              auto-renew: yes 
>  
>  
>  
>      I was able to manually renew it: 
>  
>  
>  
>      [root@ipa2 ~]# ipa-getcert resubmit -i '20190425210040'
>  
>      Resubmitting "20190425210040" to "dogtag-ipa-ca-renew-agent".
>  
>      [root@ipa2 ~]# 
>  
>  
>  
>      After renew, it "expires: 2024-04-02 06:09:32 UTC": 
>  
>  
>  
>      Request ID '20190425210040':
>  
>              status: MONITORING
>  
>              stuck: no
>  
>              key pair storage: type=FILE,location='/var/lib/ipa/ra-agent.key'
>  
>              certificate: type=FILE,location='/var/lib/ipa/ra-agent.pem'
>  
>              CA: dogtag-ipa-ca-renew-agent
>  
>              issuer: CN=Certificate Authority,O=EXAMPLE.COM
>      <http://EXAMPLE.COM>
>  
>              subject: CN=IPA RA,O=EXAMPLE.COM <http://EXAMPLE.COM>
>  
>              expires: 2024-04-02 06:09:32 UTC
>  
>              key usage:
>      digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment
>  
>              eku: id-kp-serverAuth,id-kp-clientAuth
>  
>              pre-save command: /usr/libexec/ipa/certmonger/renew_ra_cert_pre
>  
>              post-save command: /usr/libexec/ipa/certmonger/renew_ra_cert
>  
>              track: yes
>  
>              auto-renew: yes
>  
>  
>  
>      How to fix the issue reported by ipa-healthcheck? And what is this
>      issue about? 
>  
>  
>      All IPA servers are at same level: 
>  
>  
>      CentOS Linux release 7.9.2009 (Core)
>  
>      ipa-*server*.x86_64                     4.6.8-5.el7.centos.7 
>  
>      *slapi-nis*.x86_64                      0.56.5-3.el7_9 
>  
>      *389-ds-base*.x86_64                    1.3.10.2-12.el7_9  
>  
>      *389-ds-base*-libs.x86_64               1.3.10.2-12.el7_9 
>  
>  
>      Many thanks! 
>  
>  
>      Kathy. 
>  
>  
>  _______________________________________________
>  FreeIPA-users mailing list -- freeipa-users(a)lists.fedorahosted.org
>  To unsubscribe send an email to freeipa-users-leave(a)lists.fedorahosted.org
>  Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
>  List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
>  List Archives:
> https://lists.fedorahosted.org/archives/list/[e-mail]...
>  Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
>

[Kathy Zhu]

Hi Rob,

Thank you for the insight! That helped a lot!

The replication of the CA data was perfectly fine. I did the following on
each IPA server except the renewal server to fix the situation:

1, restarted certmonger service, then waited a few minutes until it
finished restarting pki_tomcatd (it may not restart pki_tomcatd right
away), it did two restarts, see attached.
2, once see attached which confirmed the finish, ipa-healthcheck would not
complain.

I noticed that this has been updated by its time stamp:

-r--r----- 1 root ipaapi 1261 Apr 13 14:13 /var/lib/ipa/ra-agent.pem


My guess of what happened is that /var/lib/ipa/ra-agent.pem had been
renewed on the renewal server, the rest of the IPA servers should pick up
the change, however, for a reason unknown to me, certmonger on other IPA
servers failed to do so. That was why they complained of "not match".
Restarting certmonger triggered this action, once done, all is fine.


Rob, please correct if my guess is wrong. Thank you for bringing this
wonderful tool to Centos 7.


Many thanks.


Kathy.


[root@ipa2 ~]# systemctl status certmonger -l

● certmonger.service - Certificate monitoring and PKI enrollment

   Loaded: loaded (/usr/lib/systemd/system/certmonger.service; enabled;
vendor preset: disabled)

   Active: active (running) since Wed 2022-04-13 14:12:18 PDT; 12min ago

 Main PID: 16980 (certmonger)

   CGroup: /system.slice/certmonger.service

           └─16980 /usr/sbin/certmonger -S -p /var/run/certmonger.pid -n


Apr 13 14:13:05 ipa2.example.com stop_pkicad[17729]: Stopping pki_tomcatd

Apr 13 14:13:06 ipa2.example.com stop_pkicad[17729]: Stopped pki_tomcatd

Apr 13 14:13:08 ipa2.example.com renew_ca_cert[17785]: Updated trust on
certificate auditSigningCert cert-pki-ca in /etc/pki/pki-tomcat/alias

Apr 13 14:13:08 ipa2.example.com renew_ca_cert[17785]: Starting pki_tomcatd

Apr 13 14:13:23 ipa2.example.com renew_ca_cert[17785]: Started pki_tomcatd

Apr 13 14:13:23 ipa2.example.com certmonger[18098]: Certificate named
"auditSigningCert cert-pki-ca" in token "NSS Certificate DB" in
database
"/etc/pki/pki-tomcat/alias" issued by CA and saved.

Apr 13 14:13:35 ipa2.example.com stop_pkicad[18130]: Stopping pki_tomcatd

Apr 13 14:13:36 ipa2.example.com stop_pkicad[18130]: Stopped pki_tomcatd

Apr 13 14:13:38 ipa2.example.com renew_ca_cert[18185]: Starting pki_tomcatd

Apr 13 14:13:53 ipa2.example.com renew_ca_cert[18185]: Started pki_tomcatd

[root@ipa2 ~]#


On Wed, Apr 13, 2022 at 12:23 PM Rob Crittenden <rcritten(a)redhat.com&gt; wrote:

>  This looks like the root cause:
>
>  ERROR: ipahealthcheck.ipa.certs.IPARAAgent: RA agent description does
>  not match 2;186;CN=Certificate Authority,O=EXAMPLE.COM;CN=IPA
>  RA,O=EXAMPLE.COM in LDAP and 2;66;CN=Certificate
>  Authority,O=EXAMPLE.COM;CN=IPA RA,O=EXAMPLE.COM expected
>
>  It looks like an updated RA certificate was issued and was not picked up
>  by the mother server(s).
>
>  Off the top of my head this could be:
>
>  - replication of the CA data has a problem: ipa-csreplica-manage list -v
>  `hostname`
>  - The updated certificate wasn't published to
>  cn=certificates,cn=ipa,cn=etc,$SUFFIX
>  - certmonger isn't picking up the renewal for some reason. The journal
>  may hold clues.
>  - something I'm forgetting
>
>  I'd start with the first two.
>
>  rob
>
>
>  Kathy Zhu via FreeIPA-users wrote:
>  > I just found this post about the same or similar issue:
>  >
>  >
>
> https://lists.fedoraproject.org/archives/list/[e-mail]...
>  >
>  > One detail I missed - this happens on all IPA servers BUT the renewal
>  > IPA server. I will go through ^ post to see if that applies to our
>  > situation.
>  >
>  > Thanks.
>  >
>  > Kathy.
>  >
>  >
>  > On Wed, Apr 13, 2022 at 10:21 AM Kathy Zhu  wrote:
>  >
>  >     Hi team,
>  >
>  >
>  >     ipa-healthcheck has been a great tool for us. I run it weekly on all
>  >     IPA servers via cron. This week ipa-healthcheck reported errors on
>  >     all IPA servers.
>  >
>  >
>  >     Take IPA server ipa2 as an example for the investigation:
>  >
>  >
>  >
>  >     [root@ipa2 ~]# ipa-healthcheck --failures-only --output-type=human
>  >
>  >     ra.get_certificate(): EXCEPTION (Invalid Credential.)
>  >
>  >     ra.get_certificate(): EXCEPTION (Invalid Credential.)
>  >
>  >     ra.get_certificate(): EXCEPTION (Invalid Credential.)
>  >
>  >     ra.get_certificate(): EXCEPTION (Invalid Credential.)
>  >
>  >     ra.get_certificate(): EXCEPTION (Invalid Credential.)
>  >
>  >     ra.get_certificate(): EXCEPTION (Invalid Credential.)
>  >
>  >     ra.get_certificate(): EXCEPTION (Invalid Credential.)
>  >
>  >     ra.get_certificate(): EXCEPTION (Invalid Credential.)
>  >
>  >     ra.get_certificate(): EXCEPTION (Invalid Credential.)
>  >
>  >     ra.get_certificate(): EXCEPTION (Invalid Credential.)
>  >
>  >     WARNING:
>  >
>   ipahealthcheck.ipa.certs.IPACertmongerExpirationCheck.20190425210040:
>  Request
>  >     id 20190425210040 expires in 27 days
>  >
>  >     WARNING:
>  >
>   ipahealthcheck.ipa.certs.IPACertmongerExpirationCheck.20190425210052:
>  Request
>  >     id 20190425210052 expires in 27 days
>  >
>  >     WARNING:
>  >
>   ipahealthcheck.ipa.certs.IPACertmongerExpirationCheck.20190425210053:
>  Request
>  >     id 20190425210053 expires in 27 days
>  >
>  >     WARNING:
>  >
>   ipahealthcheck.ipa.certs.IPACertmongerExpirationCheck.20190425210054:
>  Request
>  >     id 20190425210054 expires in 27 days
>  >
>  >     WARNING:
>  >     ipahealthcheck.ipa.certs.IPACertfileExpirationCheck.20190425210040:
>  >     Request id 20190425210040 expires in 27 days
>  >
>  >     WARNING:
>  >     ipahealthcheck.ipa.certs.IPACertfileExpirationCheck.20190425210052:
>  >     Request id 20190425210052 expires in 27 days
>  >
>  >     WARNING:
>  >     ipahealthcheck.ipa.certs.IPACertfileExpirationCheck.20190425210053:
>  >     Request id 20190425210053 expires in 27 days
>  >
>  >     WARNING:
>  >     ipahealthcheck.ipa.certs.IPACertfileExpirationCheck.20190425210054:
>  >     Request id 20190425210054 expires in 27 days
>  >
>  >     ERROR: ipahealthcheck.ipa.certs.IPARAAgent: RA agent description
>  >     does not match 2;186;CN=Certificate Authority,O=EXAMPLE.COM
>  >     <http://EXAMPLE.COM>;CN=IPA RA,O=EXAMPLE.COM <http://EXAMPLE.COM>
> in
>  >     LDAP and 2;66;CN=Certificate Authority,O=EXAMPLE.COM
>  >     <http://EXAMPLE.COM>;CN=IPA RA,O=EXAMPLE.COM <http://EXAMPLE.COM>
>  >     expected
>  >
>  >     ERROR: ipahealthcheck.ipa.certs.IPACertRevocation.20190425210040:
>  >     Request for certificate failed, Certificate operation cannot be
>  >     completed: EXCEPTION (Invalid Credential.)
>  >
>  >     ERROR: ipahealthcheck.ipa.certs.IPACertRevocation.20190425210052:
>  >     Request for certificate failed, Certificate operation cannot be
>  >     completed: EXCEPTION (Invalid Credential.)
>  >
>  >     ERROR: ipahealthcheck.ipa.certs.IPACertRevocation.20190425210053:
>  >     Request for certificate failed, Certificate operation cannot be
>  >     completed: EXCEPTION (Invalid Credential.)
>  >
>  >     ERROR: ipahealthcheck.ipa.certs.IPACertRevocation.20190425210054:
>  >     Request for certificate failed, Certificate operation cannot be
>  >     completed: EXCEPTION (Invalid Credential.)
>  >
>  >     ERROR: ipahealthcheck.ipa.certs.IPACertRevocation.20190425210055:
>  >     Request for certificate failed, Certificate operation cannot be
>  >     completed: EXCEPTION (Invalid Credential.)
>  >
>  >     ERROR: ipahealthcheck.ipa.certs.IPACertRevocation.20190425210056:
>  >     Request for certificate failed, Certificate operation cannot be
>  >     completed: EXCEPTION (Invalid Credential.)
>  >
>  >     ERROR: ipahealthcheck.ipa.certs.IPACertRevocation.20190425205849:
>  >     Request for certificate failed, Certificate operation cannot be
>  >     completed: EXCEPTION (Invalid Credential.)
>  >
>  >     ERROR: ipahealthcheck.ipa.certs.IPACertRevocation.20190425205831:
>  >     Request for certificate failed, Certificate operation cannot be
>  >     completed: EXCEPTION (Invalid Credential.)
>  >
>  >     ERROR: ipahealthcheck.ipa.certs.IPACertRevocation.20190425210120:
>  >     Request for certificate failed, Certificate operation cannot be
>  >     completed: EXCEPTION (Invalid Credential.)
>  >
>  >     ERROR: ipahealthcheck.dogtag.ca.DogtagCertsConnectivityCheck:
>  >     Request for certificate failed, Certificate operation cannot be
>  >     completed: EXCEPTION (Invalid Credential.)
>  >
>  >     [root@ipa2 ~]#
>  >
>  >
>  >
>  >     The list of certs:
>  >
>  >
>  >     [root@ipa2 ~]# getcert list
>  >
>  >     Number of certificates and requests being tracked: 9.
>  >
>  >     Request ID '20190425205831':
>  >
>  >     status: MONITORING
>  >
>  >     stuck: no
>  >
>  >     key pair storage:
>  >
>  
> type=NSSDB,location='/etc/dirsrv/slapd-EXAMPLE-COM',nickname='Server-Cert',token='NSS
>  >     Certificate DB',pinfile='/etc/dirsrv/slapd-EXAMPLE-COM/pwdfile.txt'
>  >
>  >     certificate:
>  >
>  
> type=NSSDB,location='/etc/dirsrv/slapd-EXAMPLE-COM',nickname='Server-Cert',token='NSS
>  >     Certificate DB'
>  >
>  >     CA: IPA
>  >
>  >     issuer: CN=Certificate Authority,O=EXAMPLE.COM <http://EXAMPLE.COM>
>  >
>  >     subject: CN=ipa2.example.com <http://ipa2.example.com>,O=EXAMPLE.COM
>  >     <http://EXAMPLE.COM>
>  >
>  >     expires: 2023-03-29 21:37:22 UTC
>  >
>  >     dns: ipa2.example.com <http://ipa2.example.com>
>  >
>  >     principal name: ldap/ipa2.example.com(a)EXAMPLE.COM
>  >     <mailto:[e-mail]>
>  >
>  >     key usage:
>  >     digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment
>  >
>  >     eku: id-kp-serverAuth,id-kp-clientAuth
>  >
>  >     pre-save command:
>  >
>  >     post-save command: /usr/libexec/ipa/certmonger/restart_dirsrv
>  >     EXAMPLE-COM
>  >
>  >     track: yes
>  >
>  >     auto-renew: yes
>  >
>  >     Request ID '20190425205849':
>  >
>  >     status: MONITORING
>  >
>  >     stuck: no
>  >
>  >     key pair storage:
>  >
>  
> type=NSSDB,location='/etc/httpd/alias',nickname='Server-Cert',token='NSS
>  >     Certificate DB',pinfile='/etc/httpd/alias/pwdfile.txt'
>  >
>  >     certificate:
>  >
>  
> type=NSSDB,location='/etc/httpd/alias',nickname='Server-Cert',token='NSS
>  >     Certificate DB'
>  >
>  >     CA: IPA
>  >
>  >     issuer: CN=Certificate Authority,O=EXAMPLE.COM <http://EXAMPLE.COM>
>  >
>  >     subject: CN=ipa2.example.com <http://ipa2.example.com>,O=EXAMPLE.COM
>  >     <http://EXAMPLE.COM>
>  >
>  >     expires: 2023-03-29 21:37:46 UTC
>  >
>  >     dns: ipa2.example.com <http://ipa2.example.com>
>  >
>  >     principal name: HTTP/ipa2.example.com(a)EXAMPLE.COM
>  >     <mailto:[e-mail]>
>  >
>  >     key usage:
>  >     digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment
>  >
>  >     eku: id-kp-serverAuth,id-kp-clientAuth
>  >
>  >     pre-save command:
>  >
>  >     post-save command: /usr/libexec/ipa/certmonger/restart_httpd
>  >
>  >     track: yes
>  >
>  >     auto-renew: yes
>  >
>  >     Request ID '20190425210040':
>  >
>  >     status: MONITORING
>  >
>  >     stuck: no
>  >
>  >     key pair storage: type=FILE,location='/var/lib/ipa/ra-agent.key'
>  >
>  >     certificate: type=FILE,location='/var/lib/ipa/ra-agent.pem'
>  >
>  >     CA: dogtag-ipa-ca-renew-agent
>  >
>  >     issuer: CN=Certificate Authority,O=EXAMPLE.COM <http://EXAMPLE.COM>
>  >
>  >     subject: CN=IPA RA,O=EXAMPLE.COM <http://EXAMPLE.COM>
>  >
>  >     expires: 2022-05-11 03:40:55 UTC
>  >
>  >     key usage:
>  >     digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment
>  >
>  >     eku: id-kp-serverAuth,id-kp-clientAuth
>  >
>  >     pre-save command: /usr/libexec/ipa/certmonger/renew_ra_cert_pre
>  >
>  >     post-save command: /usr/libexec/ipa/certmonger/renew_ra_cert
>  >
>  >     track: yes
>  >
>  >     auto-renew: yes
>  >
>  >     Request ID '20190425210052':
>  >
>  >     status: MONITORING
>  >
>  >     stuck: no
>  >
>  >     key pair storage:
>  >
>   type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='auditSigningCert
>  >     cert-pki-ca',token='NSS Certificate DB',pin set
>  >
>  >     certificate:
>  >
>   type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='auditSigningCert
>  >     cert-pki-ca',token='NSS Certificate DB'
>  >
>  >     CA: dogtag-ipa-ca-renew-agent
>  >
>  >     issuer: CN=Certificate Authority,O=EXAMPLE.COM <http://EXAMPLE.COM>
>  >
>  >     subject: CN=CA Audit,O=EXAMPLE.COM <http://EXAMPLE.COM>
>  >
>  >     expires: 2022-05-11 03:40:05 UTC
>  >
>  >     key usage: digitalSignature,nonRepudiation
>  >
>  >     pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad
>  >
>  >     post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert
>  >     "auditSigningCert cert-pki-ca"
>  >
>  >     track: yes
>  >
>  >     auto-renew: yes
>  >
>  >     Request ID '20190425210053':
>  >
>  >     status: MONITORING
>  >
>  >     stuck: no
>  >
>  >     key pair storage:
>  >
>   type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='ocspSigningCert
>  >     cert-pki-ca',token='NSS Certificate DB',pin set
>  >
>  >     certificate:
>  >
>   type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='ocspSigningCert
>  >     cert-pki-ca',token='NSS Certificate DB'
>  >
>  >     CA: dogtag-ipa-ca-renew-agent
>  >
>  >     issuer: CN=Certificate Authority,O=EXAMPLE.COM <http://EXAMPLE.COM>
>  >
>  >     subject: CN=OCSP Subsystem,O=EXAMPLE.COM <http://EXAMPLE.COM>
>  >
>  >     expires: 2022-05-11 03:40:25 UTC
>  >
>  >     eku: id-kp-OCSPSigning
>  >
>  >     pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad
>  >
>  >     post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert
>  >     "ocspSigningCert cert-pki-ca"
>  >
>  >     track: yes
>  >
>  >     auto-renew: yes
>  >
>  >     Request ID '20190425210054':
>  >
>  >     status: MONITORING
>  >
>  >     stuck: no
>  >
>  >     key pair storage:
>  >
>   type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='subsystemCert
>  >     cert-pki-ca',token='NSS Certificate DB',pin set
>  >
>  >     certificate:
>  >
>   type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='subsystemCert
>  >     cert-pki-ca',token='NSS Certificate DB'
>  >
>  >     CA: dogtag-ipa-ca-renew-agent
>  >
>  >     issuer: CN=Certificate Authority,O=EXAMPLE.COM <http://EXAMPLE.COM>
>  >
>  >     subject: CN=CA Subsystem,O=EXAMPLE.COM <http://EXAMPLE.COM>
>  >
>  >     expires: 2022-05-11 03:40:05 UTC
>  >
>  >     key usage:
>  >     digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment
>  >
>  >     eku: id-kp-serverAuth,id-kp-clientAuth
>  >
>  >     pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad
>  >
>  >     post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert
>  >     "subsystemCert cert-pki-ca"
>  >
>  >     track: yes
>  >
>  >     auto-renew: yes
>  >
>  >     Request ID '20190425210055':
>  >
>  >     status: MONITORING
>  >
>  >     stuck: no
>  >
>  >     key pair storage:
>  >
>   type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='caSigningCert
>  >     cert-pki-ca',token='NSS Certificate DB',pin set
>  >
>  >     certificate:
>  >
>   type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='caSigningCert
>  >     cert-pki-ca',token='NSS Certificate DB'
>  >
>  >     CA: dogtag-ipa-ca-renew-agent
>  >
>  >     issuer: CN=Certificate Authority,O=EXAMPLE.COM <http://EXAMPLE.COM>
>  >
>  >     subject: CN=Certificate Authority,O=EXAMPLE.COM <http://EXAMPLE.COM>
>  >
>  >     expires: 2038-06-28 21:19:45 UTC
>  >
>  >     key usage: digitalSignature,nonRepudiation,keyCertSign,cRLSign
>  >
>  >     pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad
>  >
>  >     post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert
>  >     "caSigningCert cert-pki-ca"
>  >
>  >     track: yes
>  >
>  >     auto-renew: yes
>  >
>  >     Request ID '20190425210056':
>  >
>  >     status: MONITORING
>  >
>  >     stuck: no
>  >
>  >     key pair storage:
>  >
>   type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='Server-Cert
>  cert-pki-ca',token='NSS
>  >     Certificate DB',pin set
>  >
>  >     certificate:
>  >
>   type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='Server-Cert
>  cert-pki-ca',token='NSS
>  >     Certificate DB'
>  >
>  >     CA: dogtag-ipa-ca-renew-agent
>  >
>  >     issuer: CN=Certificate Authority,O=EXAMPLE.COM <http://EXAMPLE.COM>
>  >
>  >     subject: CN=ipa2.example.com <http://ipa2.example.com>,O=EXAMPLE.COM
>  >     <http://EXAMPLE.COM>
>  >
>  >     expires: 2023-03-07 22:37:22 UTC
>  >
>  >     key usage:
>  >     digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment
>  >
>  >     eku: id-kp-serverAuth,id-kp-clientAuth,id-kp-emailProtection
>  >
>  >     pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad
>  >
>  >     post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert
>  >     "Server-Cert cert-pki-ca"
>  >
>  >     track: yes
>  >
>  >     auto-renew: yes
>  >
>  >     Request ID '20190425210120':
>  >
>  >     status: MONITORING
>  >
>  >     stuck: no
>  >
>  >     key pair storage: type=FILE,location='/var/kerberos/krb5kdc/kdc.key'
>  >
>  >     certificate: type=FILE,location='/var/kerberos/krb5kdc/kdc.crt'
>  >
>  >     CA: IPA
>  >
>  >     issuer: CN=Certificate Authority,O=EXAMPLE.COM <http://EXAMPLE.COM>
>  >
>  >     subject: CN=ipa2.example.com <http://ipa2.example.com>,O=EXAMPLE.COM
>  >     <http://EXAMPLE.COM>
>  >
>  >     expires: 2023-03-29 21:37:52 UTC
>  >
>  >     principal name: krbtgt/EXAMPLE.COM(a)EXAMPLE.COM
>  >     <mailto:[e-mail]>
>  >
>  >     key usage:
>  >     digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment
>  >
>  >     eku: id-kp-serverAuth,id-pkinit-KPKdc
>  >
>  >     pre-save command:
>  >
>  >     post-save command: /usr/libexec/ipa/certmonger/renew_kdc_cert
>  >
>  >     track: yes
>  >
>  >     auto-renew: yes
>  >
>  >     [root@ipa2 ~]#
>  >
>  >
>  >
>  >
>  >     There are 4 certs which expire on 2022-05-11 which match "expires in
>  >     27 days". Take 20190425210040 as an example, we have:
>  >
>  >
>  >
>  >
>  >     WARNING:
>  >
>   ipahealthcheck.ipa.certs.IPACertmongerExpirationCheck.20190425210040:
>  Request
>  >     id 20190425210040 expires in 27 days
>  >
>  >     WARNING:
>  >     ipahealthcheck.ipa.certs.IPACertfileExpirationCheck.20190425210040:
>  >     Request id 20190425210040 expires in 27 days
>  >
>  >     ERROR: ipahealthcheck.ipa.certs.IPACertRevocation.20190425210040:
>  >     Request for certificate failed, Certificate operation cannot be
>  >     completed: EXCEPTION (Invalid Credential.)
>  >
>  >
>  >     Request ID '20190425210040':
>  >
>  >             status: MONITORING
>  >
>  >             stuck: no
>  >
>  >             key pair storage:
>  type=FILE,location='/var/lib/ipa/ra-agent.key'
>  >
>  >             certificate: type=FILE,location='/var/lib/ipa/ra-agent.pem'
>  >
>  >             CA: dogtag-ipa-ca-renew-agent
>  >
>  >             issuer: CN=Certificate Authority,O=EXAMPLE.COM
>  >     <http://EXAMPLE.COM>
>  >
>  >             subject: CN=IPA RA,O=EXAMPLE.COM <http://EXAMPLE.COM>
>  >
>  >             expires: 2022-05-11 03:40:55 UTC
>  >
>  >             key usage:
>  >     digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment
>  >
>  >             eku: id-kp-serverAuth,id-kp-clientAuth
>  >
>  >             pre-save command:
>  /usr/libexec/ipa/certmonger/renew_ra_cert_pre
>  >
>  >             post-save command: /usr/libexec/ipa/certmonger/renew_ra_cert
>  >
>  >             track: yes
>  >
>  >             auto-renew: yes
>  >
>  >
>  >
>  >     I was able to manually renew it:
>  >
>  >
>  >
>  >     [root@ipa2 ~]# ipa-getcert resubmit -i '20190425210040'
>  >
>  >     Resubmitting "20190425210040" to
> "dogtag-ipa-ca-renew-agent".
>  >
>  >     [root@ipa2 ~]#
>  >
>  >
>  >
>  >     After renew, it "expires: 2024-04-02 06:09:32 UTC":
>  >
>  >
>  >
>  >     Request ID '20190425210040':
>  >
>  >             status: MONITORING
>  >
>  >             stuck: no
>  >
>  >             key pair storage:
>  type=FILE,location='/var/lib/ipa/ra-agent.key'
>  >
>  >             certificate: type=FILE,location='/var/lib/ipa/ra-agent.pem'
>  >
>  >             CA: dogtag-ipa-ca-renew-agent
>  >
>  >             issuer: CN=Certificate Authority,O=EXAMPLE.COM
>  >     <http://EXAMPLE.COM>
>  >
>  >             subject: CN=IPA RA,O=EXAMPLE.COM <http://EXAMPLE.COM>
>  >
>  >             expires: 2024-04-02 06:09:32 UTC
>  >
>  >             key usage:
>  >     digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment
>  >
>  >             eku: id-kp-serverAuth,id-kp-clientAuth
>  >
>  >             pre-save command:
>  /usr/libexec/ipa/certmonger/renew_ra_cert_pre
>  >
>  >             post-save command: /usr/libexec/ipa/certmonger/renew_ra_cert
>  >
>  >             track: yes
>  >
>  >             auto-renew: yes
>  >
>  >
>  >
>  >     How to fix the issue reported by ipa-healthcheck? And what is this
>  >     issue about?
>  >
>  >
>  >     All IPA servers are at same level:
>  >
>  >
>  >     CentOS Linux release 7.9.2009 (Core)
>  >
>  >     ipa-*server*.x86_64                     4.6.8-5.el7.centos.7
>  >
>  >     *slapi-nis*.x86_64                      0.56.5-3.el7_9
>  >
>  >     *389-ds-base*.x86_64                    1.3.10.2-12.el7_9
>  >
>  >     *389-ds-base*-libs.x86_64               1.3.10.2-12.el7_9
>  >
>  >
>  >     Many thanks!
>  >
>  >
>  >     Kathy.
>  >
>  >
>  > _______________________________________________
>  > FreeIPA-users mailing list -- freeipa-users(a)lists.fedorahosted.org
>  > To unsubscribe send an email to
>  freeipa-users-leave(a)lists.fedorahosted.org
>  > Fedora Code of Conduct:
>  https://docs.fedoraproject.org/en-US/project/code-of-conduct/
>  > List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
>  > List Archives:
>  https://lists.fedorahosted.org/archives/list/[e-mail]...
>  > Do not reply to spam on the list, report it:
>  https://pagure.io/fedora-infrastructure
>  >
>
>

[Rob Crittenden]

There was a bug in certmonger where it slowly leaked file descriptors
over time. So if certmonger ran for an extended period, usually months,
it would eventually run out. certmonger uses helpers to do renewals so
with no descriptors, forks would fail and hence renewals. This is BZ
https://bugzilla.redhat.com/show_bug.cgi?id=1992439 and fixed in
0.78.4-16 in RHEL 7.9.

That is my guess as to what happened. Just restarting an older
certmonger would fix it as it would get a whole new batch of available
descriptors and allow issuance to complete.

I'm glad to hear that ipa-healthcheck helped you avoid a painful
certificate expiration issue.

rob

Kathy Zhu via FreeIPA-users wrote:
>  Hi Rob, 
>  
>  Thank you for the insight! That helped a lot! 
>  
>  The replication of the CA data was perfectly fine. I did the following
>  on each IPA server except the renewal server to fix the situation: 
>  
>  1, restarted certmonger service, then waited a few minutes until it
>  finished restarting pki_tomcatd (it may not restart pki_tomcatd right
>  away), it did two restarts, see attached. 
>  2, once see attached which confirmed the finish, ipa-healthcheck would
>  not complain. 
>  
>  I noticed that this has been updated by its time stamp: 
>  
>  -r--r----- 1 root ipaapi 1261 Apr 13 14:13 /var/lib/ipa/ra-agent.pem
>  
>  
>  My guess of what happened is that /var/lib/ipa/ra-agent.pem had been
>  renewed on the renewal server, the rest of the IPA servers should pick
>  up the change, however, for a reason unknown to me, certmonger on
>  other IPA servers failed to do so. That was why they complained of "not
>  match".  Restarting certmonger triggered this action, once done, all is
>  fine.  
>  
>  
>  Rob, please correct if my guess is wrong. Thank you for bringing this
>  wonderful tool to Centos 7.  
>  
>  
>  Many thanks. 
>  
>  
>  Kathy.  
>  
>  
>  
>  [root@ipa2 ~]# systemctl status certmonger -l
>  
>  ● certmonger.service - Certificate monitoring and PKI enrollment
>  
>     Loaded: loaded (/usr/lib/systemd/system/certmonger.service; enabled;
>  vendor preset: disabled)
>  
>     Active: active (running) since Wed 2022-04-13 14:12:18 PDT; 12min ago
>  
>   Main PID: 16980 (certmonger)
>  
>     CGroup: /system.slice/certmonger.service
>  
>             └─16980 /usr/sbin/certmonger -S -p /var/run/certmonger.pid -n
>  
>  
>  Apr 13 14:13:05 ipa2.example.com <http://ipa2.example.com>
>  stop_pkicad[17729]: Stopping pki_tomcatd
>  
>  Apr 13 14:13:06 ipa2.example.com <http://ipa2.example.com>
>  stop_pkicad[17729]: Stopped pki_tomcatd
>  
>  Apr 13 14:13:08 ipa2.example.com <http://ipa2.example.com>
>  renew_ca_cert[17785]: Updated trust on certificate auditSigningCert
>  cert-pki-ca in /etc/pki/pki-tomcat/alias
>  
>  Apr 13 14:13:08 ipa2.example.com <http://ipa2.example.com>
>  renew_ca_cert[17785]: Starting pki_tomcatd
>  
>  Apr 13 14:13:23 ipa2.example.com <http://ipa2.example.com>
>  renew_ca_cert[17785]: Started pki_tomcatd
>  
>  Apr 13 14:13:23 ipa2.example.com <http://ipa2.example.com>
>  certmonger[18098]: Certificate named "auditSigningCert cert-pki-ca" in
>  token "NSS Certificate DB" in database "/etc/pki/pki-tomcat/alias"
>  issued by CA and saved.
>  
>  Apr 13 14:13:35 ipa2.example.com <http://ipa2.example.com>
>  stop_pkicad[18130]: Stopping pki_tomcatd
>  
>  Apr 13 14:13:36 ipa2.example.com <http://ipa2.example.com>
>  stop_pkicad[18130]: Stopped pki_tomcatd
>  
>  Apr 13 14:13:38 ipa2.example.com <http://ipa2.example.com>
>  renew_ca_cert[18185]: Starting pki_tomcatd
>  
>  Apr 13 14:13:53 ipa2.example.com <http://ipa2.example.com>
>  renew_ca_cert[18185]: Started pki_tomcatd
>  
>  [root@ipa2 ~]#
>  
>  
>  
>  On Wed, Apr 13, 2022 at 12:23 PM Rob Crittenden <rcritten(a)redhat.com
>  <mailto:[e-mail]>> wrote:
>  
>      This looks like the root cause:
>  
>      ERROR: ipahealthcheck.ipa.certs.IPARAAgent: RA agent description does
>      not match 2;186;CN=Certificate Authority,O=EXAMPLE.COM
>      <http://EXAMPLE.COM>;CN=IPA
>      RA,O=EXAMPLE.COM <http://EXAMPLE.COM> in LDAP and 2;66;CN=Certificate
>      Authority,O=EXAMPLE.COM <http://EXAMPLE.COM>;CN=IPA RA,O=EXAMPLE.COM
>      <http://EXAMPLE.COM> expected
>  
>      It looks like an updated RA certificate was issued and was not picked up
>      by the mother server(s).
>  
>      Off the top of my head this could be:
>  
>      - replication of the CA data has a problem: ipa-csreplica-manage list -v
>      `hostname`
>      - The updated certificate wasn't published to
>      cn=certificates,cn=ipa,cn=etc,$SUFFIX
>      - certmonger isn't picking up the renewal for some reason. The journal
>      may hold clues.
>      - something I'm forgetting
>  
>      I'd start with the first two.
>  
>      rob
>  
>  
>      Kathy Zhu via FreeIPA-users wrote:
>      > I just found this post about the same or similar issue: 
>      >
>      >
>     
> https://lists.fedoraproject.org/archives/list/[e-mail]...
>      >
>      > One detail I missed - this happens on all IPA servers BUT the renewal
>      > IPA server. I will go through ^ post to see if that applies to our
>      > situation. 
>      >
>      > Thanks. 
>      >
>      > Kathy. 
>      >
>      >
>      > On Wed, Apr 13, 2022 at 10:21 AM Kathy Zhu  wrote:
>      >
>      >     Hi team, 
>      >
>      >
>      >     ipa-healthcheck has been a great tool for us. I run it weekly
>      on all
>      >     IPA servers via cron. This week ipa-healthcheck reported errors on
>      >     all IPA servers. 
>      >
>      >
>      >     Take IPA server ipa2 as an example for the investigation: 
>      >
>      >
>      >
>      >     [root@ipa2 ~]# ipa-healthcheck --failures-only --output-type=human
>      >
>      >     ra.get_certificate(): EXCEPTION (Invalid Credential.)
>      >
>      >     ra.get_certificate(): EXCEPTION (Invalid Credential.)
>      >
>      >     ra.get_certificate(): EXCEPTION (Invalid Credential.)
>      >
>      >     ra.get_certificate(): EXCEPTION (Invalid Credential.)
>      >
>      >     ra.get_certificate(): EXCEPTION (Invalid Credential.)
>      >
>      >     ra.get_certificate(): EXCEPTION (Invalid Credential.)
>      >
>      >     ra.get_certificate(): EXCEPTION (Invalid Credential.)
>      >
>      >     ra.get_certificate(): EXCEPTION (Invalid Credential.)
>      >
>      >     ra.get_certificate(): EXCEPTION (Invalid Credential.)
>      >
>      >     ra.get_certificate(): EXCEPTION (Invalid Credential.)
>      >
>      >     WARNING:
>      >   
>       ipahealthcheck.ipa.certs.IPACertmongerExpirationCheck.20190425210040:
>      Request
>      >     id 20190425210040 expires in 27 days
>      >
>      >     WARNING:
>      >   
>       ipahealthcheck.ipa.certs.IPACertmongerExpirationCheck.20190425210052:
>      Request
>      >     id 20190425210052 expires in 27 days
>      >
>      >     WARNING:
>      >   
>       ipahealthcheck.ipa.certs.IPACertmongerExpirationCheck.20190425210053:
>      Request
>      >     id 20190425210053 expires in 27 days
>      >
>      >     WARNING:
>      >   
>       ipahealthcheck.ipa.certs.IPACertmongerExpirationCheck.20190425210054:
>      Request
>      >     id 20190425210054 expires in 27 days
>      >
>      >     WARNING:
>      >   
>       ipahealthcheck.ipa.certs.IPACertfileExpirationCheck.20190425210040:
>      >     Request id 20190425210040 expires in 27 days
>      >
>      >     WARNING:
>      >   
>       ipahealthcheck.ipa.certs.IPACertfileExpirationCheck.20190425210052:
>      >     Request id 20190425210052 expires in 27 days
>      >
>      >     WARNING:
>      >   
>       ipahealthcheck.ipa.certs.IPACertfileExpirationCheck.20190425210053:
>      >     Request id 20190425210053 expires in 27 days
>      >
>      >     WARNING:
>      >   
>       ipahealthcheck.ipa.certs.IPACertfileExpirationCheck.20190425210054:
>      >     Request id 20190425210054 expires in 27 days
>      >
>      >     ERROR: ipahealthcheck.ipa.certs.IPARAAgent: RA agent description
>      >     does not match 2;186;CN=Certificate Authority,O=EXAMPLE.COM
>      <http://EXAMPLE.COM>
>      >     <http://EXAMPLE.COM>;CN=IPA RA,O=EXAMPLE.COM
>      <http://EXAMPLE.COM> <http://EXAMPLE.COM> in
>      >     LDAP and 2;66;CN=Certificate Authority,O=EXAMPLE.COM
>      <http://EXAMPLE.COM>
>      >     <http://EXAMPLE.COM>;CN=IPA RA,O=EXAMPLE.COM
>      <http://EXAMPLE.COM> <http://EXAMPLE.COM>
>      >     expected
>      >
>      >     ERROR: ipahealthcheck.ipa.certs.IPACertRevocation.20190425210040:
>      >     Request for certificate failed, Certificate operation cannot be
>      >     completed: EXCEPTION (Invalid Credential.)
>      >
>      >     ERROR: ipahealthcheck.ipa.certs.IPACertRevocation.20190425210052:
>      >     Request for certificate failed, Certificate operation cannot be
>      >     completed: EXCEPTION (Invalid Credential.)
>      >
>      >     ERROR: ipahealthcheck.ipa.certs.IPACertRevocation.20190425210053:
>      >     Request for certificate failed, Certificate operation cannot be
>      >     completed: EXCEPTION (Invalid Credential.)
>      >
>      >     ERROR: ipahealthcheck.ipa.certs.IPACertRevocation.20190425210054:
>      >     Request for certificate failed, Certificate operation cannot be
>      >     completed: EXCEPTION (Invalid Credential.)
>      >
>      >     ERROR: ipahealthcheck.ipa.certs.IPACertRevocation.20190425210055:
>      >     Request for certificate failed, Certificate operation cannot be
>      >     completed: EXCEPTION (Invalid Credential.)
>      >
>      >     ERROR: ipahealthcheck.ipa.certs.IPACertRevocation.20190425210056:
>      >     Request for certificate failed, Certificate operation cannot be
>      >     completed: EXCEPTION (Invalid Credential.)
>      >
>      >     ERROR: ipahealthcheck.ipa.certs.IPACertRevocation.20190425205849:
>      >     Request for certificate failed, Certificate operation cannot be
>      >     completed: EXCEPTION (Invalid Credential.)
>      >
>      >     ERROR: ipahealthcheck.ipa.certs.IPACertRevocation.20190425205831:
>      >     Request for certificate failed, Certificate operation cannot be
>      >     completed: EXCEPTION (Invalid Credential.)
>      >
>      >     ERROR: ipahealthcheck.ipa.certs.IPACertRevocation.20190425210120:
>      >     Request for certificate failed, Certificate operation cannot be
>      >     completed: EXCEPTION (Invalid Credential.)
>      >
>      >     ERROR: ipahealthcheck.dogtag.ca
>      <http://ipahealthcheck.dogtag.ca>.DogtagCertsConnectivityCheck:
>      >     Request for certificate failed, Certificate operation cannot be
>      >     completed: EXCEPTION (Invalid Credential.)
>      >
>      >     [root@ipa2 ~]# 
>      >
>      >
>      >
>      >     The list of certs: 
>      >
>      >
>      >     [root@ipa2 ~]# getcert list 
>      >
>      >     Number of certificates and requests being tracked: 9.
>      >
>      >     Request ID '20190425205831':
>      >
>      >     status: MONITORING
>      >
>      >     stuck: no
>      >
>      >     key pair storage:
>      >   
>     
>  type=NSSDB,location='/etc/dirsrv/slapd-EXAMPLE-COM',nickname='Server-Cert',token='NSS
>      >     Certificate
>      DB',pinfile='/etc/dirsrv/slapd-EXAMPLE-COM/pwdfile.txt'
>      >
>      >     certificate:
>      >   
>     
>  type=NSSDB,location='/etc/dirsrv/slapd-EXAMPLE-COM',nickname='Server-Cert',token='NSS
>      >     Certificate DB'
>      >
>      >     CA: IPA
>      >
>      >     issuer: CN=Certificate Authority,O=EXAMPLE.COM
>      <http://EXAMPLE.COM> <http://EXAMPLE.COM>
>      >
>      >     subject: CN=ipa2.example.com <http://ipa2.example.com>
>      <http://ipa2.example.com>,O=EXAMPLE.COM <http://EXAMPLE.COM>
>      >     <http://EXAMPLE.COM>
>      >
>      >     expires: 2023-03-29 21:37:22 UTC
>      >
>      >     dns: ipa2.example.com <http://ipa2.example.com>
>      <http://ipa2.example.com>
>      >
>      >     principal name: ldap/ipa2.example.com(a)EXAMPLE.COM
>      <mailto:[e-mail]>
>      >     <mailto:[e-mail]
>      <mailto:[e-mail]>>
>      >
>      >     key usage:
>      >     digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment
>      >
>      >     eku: id-kp-serverAuth,id-kp-clientAuth
>      >
>      >     pre-save command: 
>      >
>      >     post-save command: /usr/libexec/ipa/certmonger/restart_dirsrv
>      >     EXAMPLE-COM
>      >
>      >     track: yes
>      >
>      >     auto-renew: yes
>      >
>      >     Request ID '20190425205849':
>      >
>      >     status: MONITORING
>      >
>      >     stuck: no
>      >
>      >     key pair storage:
>      >   
>     
>  type=NSSDB,location='/etc/httpd/alias',nickname='Server-Cert',token='NSS
>      >     Certificate DB',pinfile='/etc/httpd/alias/pwdfile.txt'
>      >
>      >     certificate:
>      >   
>     
>  type=NSSDB,location='/etc/httpd/alias',nickname='Server-Cert',token='NSS
>      >     Certificate DB'
>      >
>      >     CA: IPA
>      >
>      >     issuer: CN=Certificate Authority,O=EXAMPLE.COM
>      <http://EXAMPLE.COM> <http://EXAMPLE.COM>
>      >
>      >     subject: CN=ipa2.example.com <http://ipa2.example.com>
>      <http://ipa2.example.com>,O=EXAMPLE.COM <http://EXAMPLE.COM>
>      >     <http://EXAMPLE.COM>
>      >
>      >     expires: 2023-03-29 21:37:46 UTC
>      >
>      >     dns: ipa2.example.com <http://ipa2.example.com>
>      <http://ipa2.example.com>
>      >
>      >     principal name: HTTP/ipa2.example.com(a)EXAMPLE.COM
>      <mailto:[e-mail]>
>      >     <mailto:[e-mail]
>      <mailto:[e-mail]>>
>      >
>      >     key usage:
>      >     digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment
>      >
>      >     eku: id-kp-serverAuth,id-kp-clientAuth
>      >
>      >     pre-save command: 
>      >
>      >     post-save command: /usr/libexec/ipa/certmonger/restart_httpd
>      >
>      >     track: yes
>      >
>      >     auto-renew: yes
>      >
>      >     Request ID '20190425210040':
>      >
>      >     status: MONITORING
>      >
>      >     stuck: no
>      >
>      >     key pair storage: type=FILE,location='/var/lib/ipa/ra-agent.key'
>      >
>      >     certificate: type=FILE,location='/var/lib/ipa/ra-agent.pem'
>      >
>      >     CA: dogtag-ipa-ca-renew-agent
>      >
>      >     issuer: CN=Certificate Authority,O=EXAMPLE.COM
>      <http://EXAMPLE.COM> <http://EXAMPLE.COM>
>      >
>      >     subject: CN=IPA RA,O=EXAMPLE.COM <http://EXAMPLE.COM>
>      <http://EXAMPLE.COM>
>      >
>      >     expires: 2022-05-11 03:40:55 UTC
>      >
>      >     key usage:
>      >     digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment
>      >
>      >     eku: id-kp-serverAuth,id-kp-clientAuth
>      >
>      >     pre-save command: /usr/libexec/ipa/certmonger/renew_ra_cert_pre
>      >
>      >     post-save command: /usr/libexec/ipa/certmonger/renew_ra_cert
>      >
>      >     track: yes
>      >
>      >     auto-renew: yes
>      >
>      >     Request ID '20190425210052':
>      >
>      >     status: MONITORING
>      >
>      >     stuck: no
>      >
>      >     key pair storage:
>      >   
>     
>  type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='auditSigningCert
>      >     cert-pki-ca',token='NSS Certificate DB',pin set
>      >
>      >     certificate:
>      >   
>     
>  type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='auditSigningCert
>      >     cert-pki-ca',token='NSS Certificate DB'
>      >
>      >     CA: dogtag-ipa-ca-renew-agent
>      >
>      >     issuer: CN=Certificate Authority,O=EXAMPLE.COM
>      <http://EXAMPLE.COM> <http://EXAMPLE.COM>
>      >
>      >     subject: CN=CA Audit,O=EXAMPLE.COM <http://EXAMPLE.COM>
>      <http://EXAMPLE.COM>
>      >
>      >     expires: 2022-05-11 03:40:05 UTC
>      >
>      >     key usage: digitalSignature,nonRepudiation
>      >
>      >     pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad
>      >
>      >     post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert
>      >     "auditSigningCert cert-pki-ca"
>      >
>      >     track: yes
>      >
>      >     auto-renew: yes
>      >
>      >     Request ID '20190425210053':
>      >
>      >     status: MONITORING
>      >
>      >     stuck: no
>      >
>      >     key pair storage:
>      >   
>     
>  type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='ocspSigningCert
>      >     cert-pki-ca',token='NSS Certificate DB',pin set
>      >
>      >     certificate:
>      >   
>     
>  type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='ocspSigningCert
>      >     cert-pki-ca',token='NSS Certificate DB'
>      >
>      >     CA: dogtag-ipa-ca-renew-agent
>      >
>      >     issuer: CN=Certificate Authority,O=EXAMPLE.COM
>      <http://EXAMPLE.COM> <http://EXAMPLE.COM>
>      >
>      >     subject: CN=OCSP Subsystem,O=EXAMPLE.COM <http://EXAMPLE.COM>
>      <http://EXAMPLE.COM>
>      >
>      >     expires: 2022-05-11 03:40:25 UTC
>      >
>      >     eku: id-kp-OCSPSigning
>      >
>      >     pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad
>      >
>      >     post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert
>      >     "ocspSigningCert cert-pki-ca"
>      >
>      >     track: yes
>      >
>      >     auto-renew: yes
>      >
>      >     Request ID '20190425210054':
>      >
>      >     status: MONITORING
>      >
>      >     stuck: no
>      >
>      >     key pair storage:
>      >   
>       type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='subsystemCert
>      >     cert-pki-ca',token='NSS Certificate DB',pin set
>      >
>      >     certificate:
>      >   
>       type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='subsystemCert
>      >     cert-pki-ca',token='NSS Certificate DB'
>      >
>      >     CA: dogtag-ipa-ca-renew-agent
>      >
>      >     issuer: CN=Certificate Authority,O=EXAMPLE.COM
>      <http://EXAMPLE.COM> <http://EXAMPLE.COM>
>      >
>      >     subject: CN=CA Subsystem,O=EXAMPLE.COM <http://EXAMPLE.COM>
>      <http://EXAMPLE.COM>
>      >
>      >     expires: 2022-05-11 03:40:05 UTC
>      >
>      >     key usage:
>      >     digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment
>      >
>      >     eku: id-kp-serverAuth,id-kp-clientAuth
>      >
>      >     pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad
>      >
>      >     post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert
>      >     "subsystemCert cert-pki-ca"
>      >
>      >     track: yes
>      >
>      >     auto-renew: yes
>      >
>      >     Request ID '20190425210055':
>      >
>      >     status: MONITORING
>      >
>      >     stuck: no
>      >
>      >     key pair storage:
>      >   
>       type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='caSigningCert
>      >     cert-pki-ca',token='NSS Certificate DB',pin set
>      >
>      >     certificate:
>      >   
>       type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='caSigningCert
>      >     cert-pki-ca',token='NSS Certificate DB'
>      >
>      >     CA: dogtag-ipa-ca-renew-agent
>      >
>      >     issuer: CN=Certificate Authority,O=EXAMPLE.COM
>      <http://EXAMPLE.COM> <http://EXAMPLE.COM>
>      >
>      >     subject: CN=Certificate Authority,O=EXAMPLE.COM
>      <http://EXAMPLE.COM> <http://EXAMPLE.COM>
>      >
>      >     expires: 2038-06-28 21:19:45 UTC
>      >
>      >     key usage: digitalSignature,nonRepudiation,keyCertSign,cRLSign
>      >
>      >     pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad
>      >
>      >     post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert
>      >     "caSigningCert cert-pki-ca"
>      >
>      >     track: yes
>      >
>      >     auto-renew: yes
>      >
>      >     Request ID '20190425210056':
>      >
>      >     status: MONITORING
>      >
>      >     stuck: no
>      >
>      >     key pair storage:
>      >   
>       type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='Server-Cert
>      cert-pki-ca',token='NSS
>      >     Certificate DB',pin set
>      >
>      >     certificate:
>      >   
>       type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='Server-Cert
>      cert-pki-ca',token='NSS
>      >     Certificate DB'
>      >
>      >     CA: dogtag-ipa-ca-renew-agent
>      >
>      >     issuer: CN=Certificate Authority,O=EXAMPLE.COM
>      <http://EXAMPLE.COM> <http://EXAMPLE.COM>
>      >
>      >     subject: CN=ipa2.example.com <http://ipa2.example.com>
>      <http://ipa2.example.com>,O=EXAMPLE.COM <http://EXAMPLE.COM>
>      >     <http://EXAMPLE.COM>
>      >
>      >     expires: 2023-03-07 22:37:22 UTC
>      >
>      >     key usage:
>      >     digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment
>      >
>      >     eku: id-kp-serverAuth,id-kp-clientAuth,id-kp-emailProtection
>      >
>      >     pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad
>      >
>      >     post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert
>      >     "Server-Cert cert-pki-ca"
>      >
>      >     track: yes
>      >
>      >     auto-renew: yes
>      >
>      >     Request ID '20190425210120':
>      >
>      >     status: MONITORING
>      >
>      >     stuck: no
>      >
>      >     key pair storage:
>      type=FILE,location='/var/kerberos/krb5kdc/kdc.key'
>      >
>      >     certificate: type=FILE,location='/var/kerberos/krb5kdc/kdc.crt'
>      >
>      >     CA: IPA
>      >
>      >     issuer: CN=Certificate Authority,O=EXAMPLE.COM
>      <http://EXAMPLE.COM> <http://EXAMPLE.COM>
>      >
>      >     subject: CN=ipa2.example.com <http://ipa2.example.com>
>      <http://ipa2.example.com>,O=EXAMPLE.COM <http://EXAMPLE.COM>
>      >     <http://EXAMPLE.COM>
>      >
>      >     expires: 2023-03-29 21:37:52 UTC
>      >
>      >     principal name: krbtgt/EXAMPLE.COM(a)EXAMPLE.COM
>      <mailto:[e-mail]>
>      >     <mailto:[e-mail]
> <mailto:[e-mail]>>
>      >
>      >     key usage:
>      >     digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment
>      >
>      >     eku: id-kp-serverAuth,id-pkinit-KPKdc
>      >
>      >     pre-save command: 
>      >
>      >     post-save command: /usr/libexec/ipa/certmonger/renew_kdc_cert
>      >
>      >     track: yes
>      >
>      >     auto-renew: yes
>      >
>      >     [root@ipa2 ~]# 
>      >
>      >
>      >
>      >
>      >     There are 4 certs which expire on 2022-05-11 which match
>      "expires in
>      >     27 days". Take 20190425210040 as an example, we have: 
>      >
>      >      
>      >
>      >
>      >     WARNING:
>      >   
>       ipahealthcheck.ipa.certs.IPACertmongerExpirationCheck.20190425210040:
>      Request
>      >     id 20190425210040 expires in 27 days
>      >
>      >     WARNING:
>      >   
>       ipahealthcheck.ipa.certs.IPACertfileExpirationCheck.20190425210040:
>      >     Request id 20190425210040 expires in 27 days
>      >
>      >     ERROR: ipahealthcheck.ipa.certs.IPACertRevocation.20190425210040:
>      >     Request for certificate failed, Certificate operation cannot be
>      >     completed: EXCEPTION (Invalid Credential.)
>      >
>      >
>      >     Request ID '20190425210040':
>      >
>      >             status: MONITORING
>      >
>      >             stuck: no
>      >
>      >             key pair storage:
>      type=FILE,location='/var/lib/ipa/ra-agent.key'
>      >
>      >             certificate:
>      type=FILE,location='/var/lib/ipa/ra-agent.pem'
>      >
>      >             CA: dogtag-ipa-ca-renew-agent
>      >
>      >             issuer: CN=Certificate Authority,O=EXAMPLE.COM
>      <http://EXAMPLE.COM>
>      >     <http://EXAMPLE.COM>
>      >
>      >             subject: CN=IPA RA,O=EXAMPLE.COM <http://EXAMPLE.COM>
>      <http://EXAMPLE.COM>
>      >
>      >             expires: 2022-05-11 03:40:55 UTC
>      >
>      >             key usage:
>      >     digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment
>      >
>      >             eku: id-kp-serverAuth,id-kp-clientAuth
>      >
>      >             pre-save command:
>      /usr/libexec/ipa/certmonger/renew_ra_cert_pre
>      >
>      >             post-save command:
>      /usr/libexec/ipa/certmonger/renew_ra_cert
>      >
>      >             track: yes
>      >
>      >             auto-renew: yes 
>      >
>      >
>      >
>      >     I was able to manually renew it: 
>      >
>      >
>      >
>      >     [root@ipa2 ~]# ipa-getcert resubmit -i '20190425210040'
>      >
>      >     Resubmitting "20190425210040" to
> "dogtag-ipa-ca-renew-agent".
>      >
>      >     [root@ipa2 ~]# 
>      >
>      >
>      >
>      >     After renew, it "expires: 2024-04-02 06:09:32 UTC": 
>      >
>      >
>      >
>      >     Request ID '20190425210040':
>      >
>      >             status: MONITORING
>      >
>      >             stuck: no
>      >
>      >             key pair storage:
>      type=FILE,location='/var/lib/ipa/ra-agent.key'
>      >
>      >             certificate:
>      type=FILE,location='/var/lib/ipa/ra-agent.pem'
>      >
>      >             CA: dogtag-ipa-ca-renew-agent
>      >
>      >             issuer: CN=Certificate Authority,O=EXAMPLE.COM
>      <http://EXAMPLE.COM>
>      >     <http://EXAMPLE.COM>
>      >
>      >             subject: CN=IPA RA,O=EXAMPLE.COM <http://EXAMPLE.COM>
>      <http://EXAMPLE.COM>
>      >
>      >             expires: 2024-04-02 06:09:32 UTC
>      >
>      >             key usage:
>      >     digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment
>      >
>      >             eku: id-kp-serverAuth,id-kp-clientAuth
>      >
>      >             pre-save command:
>      /usr/libexec/ipa/certmonger/renew_ra_cert_pre
>      >
>      >             post-save command:
>      /usr/libexec/ipa/certmonger/renew_ra_cert
>      >
>      >             track: yes
>      >
>      >             auto-renew: yes
>      >
>      >
>      >
>      >     How to fix the issue reported by ipa-healthcheck? And what is this
>      >     issue about? 
>      >
>      >
>      >     All IPA servers are at same level: 
>      >
>      >
>      >     CentOS Linux release 7.9.2009 (Core)
>      >
>      >     ipa-*server*.x86_64                     4.6.8-5.el7.centos.7 
>      >
>      >     *slapi-nis*.x86_64                      0.56.5-3.el7_9 
>      >
>      >     *389-ds-base*.x86_64                    1.3.10.2-12.el7_9  
>      >
>      >     *389-ds-base*-libs.x86_64               1.3.10.2-12.el7_9 
>      >
>      >
>      >     Many thanks! 
>      >
>      >
>      >     Kathy. 
>      >
>      >
>      > _______________________________________________
>      > FreeIPA-users mailing list -- freeipa-users(a)lists.fedorahosted.org
>      <mailto:[e-mail]>
>      > To unsubscribe send an email to
>      freeipa-users-leave(a)lists.fedorahosted.org
>      <mailto:[e-mail]>
>      > Fedora Code of Conduct:
>      https://docs.fedoraproject.org/en-US/project/code-of-conduct/
>      > List Guidelines:
>      https://fedoraproject.org/wiki/Mailing_list_guidelines
>      > List Archives:
>      https://lists.fedorahosted.org/archives/list/[e-mail]...
>      > Do not reply to spam on the list, report it:
>      https://pagure.io/fedora-infrastructure
>      >
>  
>  
>  _______________________________________________
>  FreeIPA-users mailing list -- freeipa-users(a)lists.fedorahosted.org
>  To unsubscribe send an email to freeipa-users-leave(a)lists.fedorahosted.org
>  Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
>  List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
>  List Archives:
> https://lists.fedorahosted.org/archives/list/[e-mail]...
>  Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
>