CA Subsystem certificate

[Travis West]

The person who set this up is no longer available.  We have 6 IPA servers in a cluster, all set as MASTER.  All servers are running IPA v. 4.6.4.
On 8 March the CA Subsystem certificate expired.  When looking at the certificate I noticed it had an incorrect Common Name, which may be why it didn't renew.
I checked the pki-tomcat CS.cfg and the two lines
ca.subsystem.cert - Has cert with incorrect hostname listed
ca.subsystem.certreq - Has cert request for correct ca subsystem cert (Common  Name CA Subsystem)

I tried removing the errant ca subsystem cert from the NSS DB in pki-tomcat/alias and was successful.  I then tried to request a new SubSystem Cert using this command

getcert request -I CASubsystem -c dogtag-ipa-renew-agent -d /etc/pki/pki-tomcat/alias -n 'subsystemCert cert-pki-ca' -N 'cn=CA Subsystem,o=IPA.*****.NET' -P 'PIN_FROM_FILE' -t 'NSS Certificate DB'

And that seems to at least have issued the request because 'getcert list' shows the request, but with a CA_REJECTED message.

If I do an ldapsearch for the certificate, it shows the the correct cert with CN=CA Subystem, but the one that expired on 8 March.

How can I get a valid CA Subsystem cert again so I can start the CA on all IPA servers?

[Rob Crittenden]

Travis West via FreeIPA-users wrote:
> The person who set this up is no longer available.  We have 6 IPA servers in a cluster, all set as MASTER.  All servers are running IPA v. 4.6.4.
> On 8 March the CA Subsystem certificate expired.  When looking at the certificate I noticed it had an incorrect Common Name, which may be why it didn't renew.
> I checked the pki-tomcat CS.cfg and the two lines
> ca.subsystem.cert - Has cert with incorrect hostname listed
> ca.subsystem.certreq - Has cert request for correct ca subsystem cert (Common  Name CA Subsystem)
>
> I tried removing the errant ca subsystem cert from the NSS DB in pki-tomcat/alias and was successful.  I then tried to request a new SubSystem Cert using this command
>
> getcert request -I CASubsystem -c dogtag-ipa-renew-agent -d /etc/pki/pki-tomcat/alias -n 'subsystemCert cert-pki-ca' -N 'cn=CA Subsystem,o=IPA.*****.NET' -P 'PIN_FROM_FILE' -t 'NSS Certificate DB'
>
> And that seems to at least have issued the request because 'getcert list' shows the request, but with a CA_REJECTED message.
>
> If I do an ldapsearch for the certificate, it shows the the correct cert with CN=CA Subystem, but the one that expired on 8 March.
>
> How can I get a valid CA Subsystem cert again so I can start the CA on all IPA servers?

You are running a quite old version, on RHEL/CentOS 7.6 I presume? Later
versions provide a tool to address this, ipa-cert-fix, which isn't
available in 4.6.4.

You need to do a couple of things:
1. identify which server  is your renewal master. ipa config-show will
tell you. Otherwise you can restore to: ldapsearch -LLL -Q -Y GSSAPI -b
cn=masters,cn=ipa,cn=etc,dc=example,dc=test
"(ipaConfigString=caRenewalMaster)"

2. On the renewal master make sure it is tracking at least 8
certificates: getcert list will tell you. If it is then run getcert list
| grep expires

This will tell you how many certs have or will expire soon. All the CA
subsystem-related certificates expire at the same time so its likely
that multiple have expired.

Removing the cert wasn't a good idea. I don't think it will be
catastrophic to renewal but manually tweaking the cert database is not
generally recommended.

So assuming you have multiple certificates that all expired on the 8th
then what typically works is to disable chronyd/ntpd and use the date
command to go back in time. Restart all IPA services and then certmonger
and then watch it to see if the certificates are renewed.

rob

[Travis West]

Yes, running on CentOS 7.9.  Just for testing sake I did update one of the servers to a later version of IPA that has ipa-cert-fix, and tried running it, it returned this (CN partially censored)

# ipa-cert-fix

The following certificates will be renewed:

Dogtag subsystem certificate:
  Subject: CN=vault-hv1.***,O=IPA.****.NET
  Serial:  1073676328
  Expires: 2021-03-08 04:56:05

IPA IPA RA certificate:
  Subject: CN=iso1.***,O=IPA.***.NET
  Serial:  1073676325
  Expires: 2021-03-08 03:28:16

Enter "yes" to proceed: yes
Proceeding.
Command 'pki-server cert-fix --ldapi-socket /var/run/slapd-IPA-***-NET.socket --agent-uid ipara --cert subsystem --extra-cert 1073676325' returned non-zero exit status 1

So even still, when running ipactl start, pki-tomcat won't start.

Here we see that the CN are wrong on both of these certificates.  I still don't know how that happened.
I can certainly roll back all 6 servers to before I removed the errant CA Subsystem cert, but it still won't renew.

Then there is the discrepancy between what this shows and what I see if I do
ldapsearch -LLL -D 'cn=directory manager' -W -b uid=pkidbuser,ou=people,o=ipaca userCertificate description seeAlso

It returns the cert with the correct CN of CA Subsystem.

Then in /etc/pki/pki-tomcat/ca/CS.cfg, the two lines

ca.subsystem.cert - Has incorrect cert with errant CN from above
ca.subsystem.certreq - Has request with correct CN

So to me the problem stems from this incorrect CN that seems to be in the NSS DB in /etc/pki/pki-tomcat/alias while and ldapsearch returns the correct CN name certificate that expired back on 8  March of this  year.

For the IPA RA certificate, I was able to fix that and now certutil list on all 6 servers has that being monitored and not expiring until 2026.  It's just this CA Subsystem cert that I am stumped on.

[Travis West]

I've restored the Renewal Master from before I started changing this.  If I run getcert list I do see 9 certificates being tracked.
None of the system certs seem to expire at the same time, but they also all have incorrect Common Name in the Subject.  The RA cert is also expired and has an incorrect Common Name in the Subject

# getcert list
Number of certificates and requests being tracked: 9.
Request ID '20190322031541':
        status: MONITORING
        stuck: no
        key pair storage: type=NSSDB,location='/etc/dirsrv/slapd-IPA-****-NET',nickname='Server-Cert',token='NSS Certificate DB',pinfile='/etc/dirsrv/slapd-IPA-****-NET/pwdfile.txt'
        certificate: type=NSSDB,location='/etc/dirsrv/slapd-IPA-****-NET',nickname='Server-Cert',token='NSS Certificate DB'
        CA: IPA
        issuer: CN=Certificate Authority,O=IPA.****.NET
        subject: CN=ipa1-sea2.ipa.****.net,O=IPA.****.NET
        expires: 2025-01-26 11:37:18 UTC
        dns: ipa1-sea2.ipa.****.net
        principal name: ldap/ipa1-sea2.ipa.****.net@IPA.****.NET
        key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment
        eku: id-kp-serverAuth,id-kp-clientAuth
        pre-save command:
        post-save command: /usr/libexec/ipa/certmonger/restart_dirsrv IPA-****-NET
        track: yes
        auto-renew: yes
Request ID '20190322031615':
        status: MONITORING
        stuck: no
        key pair storage: type=NSSDB,location='/etc/httpd/alias',nickname='Server-Cert',token='NSS Certificate DB',pinfile='/etc/httpd/alias/pwdfile.txt'
        certificate: type=NSSDB,location='/etc/httpd/alias',nickname='Server-Cert',token='NSS Certificate DB'
        CA: IPA
        issuer: CN=Certificate Authority,O=IPA.****.NET
        subject: CN=ipa1-sea2.ipa.****.net,O=IPA.****.NET
        expires: 2025-01-26 11:37:04 UTC
        dns: ipa1-sea2.ipa.****.net
        principal name: HTTP/ipa1-sea2.ipa.****.net@IPA.****.NET
        key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment
        eku: id-kp-serverAuth,id-kp-clientAuth
        pre-save command:
        post-save command: /usr/libexec/ipa/certmonger/restart_httpd
        track: yes
        auto-renew: yes
Request ID '20190322032004':
        status: CA_UNREACHABLE
        ca-error: Internal error
        stuck: no
        key pair storage: type=FILE,location='/var/lib/ipa/ra-agent.key'
        certificate: type=FILE,location='/var/lib/ipa/ra-agent.pem'
        CA: dogtag-ipa-ca-renew-agent
        issuer: CN=Certificate Authority,O=IPA.****.NET
        subject: CN=iso1.sea2.****.net,O=IPA.****.NET
        expires: 2021-03-08 03:28:16 UTC
        dns: iso1.sea2.****.net
        principal name: HOST/iso1.sea2.****.net@IPA.****.NET
        key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment
        eku: id-kp-serverAuth,id-kp-clientAuth
        pre-save command: /usr/libexec/ipa/certmonger/renew_ra_cert_pre
        post-save command: /usr/libexec/ipa/certmonger/renew_ra_cert
        track: yes
        auto-renew: yes
Request ID '20190322032029':
        status: MONITORING
        stuck: no
        key pair storage: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='auditSigningCert cert-pki-ca',token='NSS Certificate DB',pin set
        certificate: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='auditSigningCert cert-pki-ca',token='NSS Certificate DB'
        CA: dogtag-ipa-ca-renew-agent
        issuer: CN=Certificate Authority,O=IPA.****.NET
        subject: CN=mbc-hv1.sea2.****.net,O=IPA.****.NET
        expires: 2026-02-10 23:07:57 UTC
        dns: mbc-hv1.sea2.****.net
        principal name: HOST/mbc-hv1.sea2.****.net@IPA.****.NET
        key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment
        eku: id-kp-serverAuth,id-kp-clientAuth
        pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad
        post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert "auditSigningCert cert-pki-ca"
        track: yes
        auto-renew: yes
Request ID '20190322032030':
        status: MONITORING
        stuck: no
        key pair storage: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='ocspSigningCert cert-pki-ca',token='NSS Certificate DB',pin set
        certificate: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='ocspSigningCert cert-pki-ca',token='NSS Certificate DB'
        CA: dogtag-ipa-ca-renew-agent
        issuer: CN=Certificate Authority,O=IPA.****.NET
        subject: CN=vault-backup2.sea2.****.net,O=IPA.****.NET
        expires: 2026-02-10 23:08:07 UTC
        dns: vault-backup2.sea2.****.net
        principal name: HOST/vault-backup2.sea2.****.net@IPA.****.NET
        key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment
        eku: id-kp-serverAuth,id-kp-clientAuth
        pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad
        post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert "ocspSigningCert cert-pki-ca"
        track: yes
        auto-renew: yes
Request ID '20190322032031':
        status: NEED_CSR_GEN_TOKEN
        stuck: yes
        key pair storage: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='subsystemCert cert-pki-ca',token='NSS Certificate DB',pin set
        certificate: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='subsystemCert cert-pki-ca',token='NSS Certificate DB'
        CA: dogtag-ipa-ca-renew-agent
        issuer: CN=Certificate Authority,O=IPA.****.NET
        subject: CN=vault-hv1.sea2.****.net,O=IPA.****.NET
        expires: 2021-03-08 04:56:05 UTC
        dns: vault-hv1.sea2.****.net
        principal name: HOST/vault-hv1.sea2.****.net@IPA.****.NET
        key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment
        eku: id-kp-serverAuth,id-kp-clientAuth
        pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad
        post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert "subsystemCert cert-pki-ca"
        track: yes
        auto-renew: yes
Request ID '20190322032032':
        status: MONITORING
        stuck: no
        key pair storage: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='caSigningCert cert-pki-ca',token='NSS Certificate DB',pin set
        certificate: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='caSigningCert cert-pki-ca',token='NSS Certificate DB'
        CA: dogtag-ipa-ca-renew-agent
        issuer: CN=Certificate Authority,O=IPA.****.NET
        subject: CN=Certificate Authority,O=IPA.****.NET
        expires: 2037-03-21 04:43:44 UTC
        key usage: digitalSignature,nonRepudiation,keyCertSign,cRLSign
        pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad
        post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert "caSigningCert cert-pki-ca"
        track: yes
        auto-renew: yes
Request ID '20190322032033':
        status: MONITORING
        stuck: no
        key pair storage: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='Server-Cert cert-pki-ca',token='NSS Certificate DB',pin set
        certificate: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='Server-Cert cert-pki-ca',token='NSS Certificate DB'
        CA: dogtag-ipa-ca-renew-agent
        issuer: CN=Certificate Authority,O=IPA.****.NET
        subject: CN=ipa1-sea2.ipa.****.net,O=IPA.****.NET
        expires: 2024-12-24 11:37:06 UTC
        key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment
        eku: id-kp-serverAuth,id-kp-clientAuth,id-kp-emailProtection
        pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad
        post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert "Server-Cert cert-pki-ca"
        track: yes
        auto-renew: yes
Request ID '20190322032117':
        status: MONITORING
        stuck: no
        key pair storage: type=FILE,location='/var/kerberos/krb5kdc/kdc.key'
        certificate: type=FILE,location='/var/kerberos/krb5kdc/kdc.crt'
        CA: IPA
        issuer: CN=Certificate Authority,O=IPA.****.NET
        subject: CN=ipa1-sea2.ipa.****.net,O=IPA.****.NET
        expires: 2025-01-26 11:41:35 UTC
        principal name: krbtgt/IPA.****.NET@IPA.****.NET
        key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment
        eku: id-kp-serverAuth,id-pkinit-KPKdc
        pre-save command:
        post-save command: /usr/libexec/ipa/certmonger/renew_kdc_cert
        track: yes
        auto-renew: yes

I'm just not sure how to get out of this pickle.  Since the subject Common Name of the certificates is incorrect, I don't think setting the time back will solve this.
I could potentially do an IPA data only backup (my understanding is that this doesn't include system certs).  Then reinstall each of the 6 servers, install IPA again and restore the data backup.  I believe there may be problems with this method as the /etc/ipa/ca.crt will likely change which I believe would affect the 389 hosts that use IPA.

[Rob Crittenden]

Travis West via FreeIPA-users wrote:
> I've restored the Renewal Master from before I started changing this.  If I run getcert list I do see 9 certificates being tracked.
> None of the system certs seem to expire at the same time, but they also all have incorrect Common Name in the Subject.  The RA cert is also expired and has an incorrect Common Name in the Subject
>
> # getcert list
> Number of certificates and requests being tracked: 9.
> Request ID '20190322031541':
>         status: MONITORING
>         stuck: no
>         key pair storage: type=NSSDB,location='/etc/dirsrv/slapd-IPA-****-NET',nickname='Server-Cert',token='NSS Certificate DB',pinfile='/etc/dirsrv/slapd-IPA-****-NET/pwdfile.txt'
>         certificate: type=NSSDB,location='/etc/dirsrv/slapd-IPA-****-NET',nickname='Server-Cert',token='NSS Certificate DB'
>         CA: IPA
>         issuer: CN=Certificate Authority,O=IPA.****.NET
>         subject: CN=ipa1-sea2.ipa.****.net,O=IPA.****.NET
>         expires: 2025-01-26 11:37:18 UTC
>         dns: ipa1-sea2.ipa.****.net
>         principal name: ldap/ipa1-sea2.ipa.****.net@IPA.****.NET
>         key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment
>         eku: id-kp-serverAuth,id-kp-clientAuth
>         pre-save command:
>         post-save command: /usr/libexec/ipa/certmonger/restart_dirsrv IPA-****-NET
>         track: yes
>         auto-renew: yes
> Request ID '20190322031615':
>         status: MONITORING
>         stuck: no
>         key pair storage: type=NSSDB,location='/etc/httpd/alias',nickname='Server-Cert',token='NSS Certificate DB',pinfile='/etc/httpd/alias/pwdfile.txt'
>         certificate: type=NSSDB,location='/etc/httpd/alias',nickname='Server-Cert',token='NSS Certificate DB'
>         CA: IPA
>         issuer: CN=Certificate Authority,O=IPA.****.NET
>         subject: CN=ipa1-sea2.ipa.****.net,O=IPA.****.NET
>         expires: 2025-01-26 11:37:04 UTC
>         dns: ipa1-sea2.ipa.****.net
>         principal name: HTTP/ipa1-sea2.ipa.****.net@IPA.****.NET
>         key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment
>         eku: id-kp-serverAuth,id-kp-clientAuth
>         pre-save command:
>         post-save command: /usr/libexec/ipa/certmonger/restart_httpd
>         track: yes
>         auto-renew: yes
> Request ID '20190322032004':
>         status: CA_UNREACHABLE
>         ca-error: Internal error
>         stuck: no
>         key pair storage: type=FILE,location='/var/lib/ipa/ra-agent.key'
>         certificate: type=FILE,location='/var/lib/ipa/ra-agent.pem'
>         CA: dogtag-ipa-ca-renew-agent
>         issuer: CN=Certificate Authority,O=IPA.****.NET
>         subject: CN=iso1.sea2.****.net,O=IPA.****.NET
>         expires: 2021-03-08 03:28:16 UTC
>         dns: iso1.sea2.****.net
>         principal name: HOST/iso1.sea2.****.net@IPA.****.NET
>         key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment
>         eku: id-kp-serverAuth,id-kp-clientAuth
>         pre-save command: /usr/libexec/ipa/certmonger/renew_ra_cert_pre
>         post-save command: /usr/libexec/ipa/certmonger/renew_ra_cert
>         track: yes
>         auto-renew: yes
> Request ID '20190322032029':
>         status: MONITORING
>         stuck: no
>         key pair storage: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='auditSigningCert cert-pki-ca',token='NSS Certificate DB',pin set
>         certificate: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='auditSigningCert cert-pki-ca',token='NSS Certificate DB'
>         CA: dogtag-ipa-ca-renew-agent
>         issuer: CN=Certificate Authority,O=IPA.****.NET
>         subject: CN=mbc-hv1.sea2.****.net,O=IPA.****.NET
>         expires: 2026-02-10 23:07:57 UTC
>         dns: mbc-hv1.sea2.****.net
>         principal name: HOST/mbc-hv1.sea2.****.net@IPA.****.NET
>         key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment
>         eku: id-kp-serverAuth,id-kp-clientAuth
>         pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad
>         post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert "auditSigningCert cert-pki-ca"
>         track: yes
>         auto-renew: yes
> Request ID '20190322032030':
>         status: MONITORING
>         stuck: no
>         key pair storage: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='ocspSigningCert cert-pki-ca',token='NSS Certificate DB',pin set
>         certificate: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='ocspSigningCert cert-pki-ca',token='NSS Certificate DB'
>         CA: dogtag-ipa-ca-renew-agent
>         issuer: CN=Certificate Authority,O=IPA.****.NET
>         subject: CN=vault-backup2.sea2.****.net,O=IPA.****.NET
>         expires: 2026-02-10 23:08:07 UTC
>         dns: vault-backup2.sea2.****.net
>         principal name: HOST/vault-backup2.sea2.****.net@IPA.****.NET
>         key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment
>         eku: id-kp-serverAuth,id-kp-clientAuth
>         pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad
>         post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert "ocspSigningCert cert-pki-ca"
>         track: yes
>         auto-renew: yes
> Request ID '20190322032031':
>         status: NEED_CSR_GEN_TOKEN
>         stuck: yes
>         key pair storage: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='subsystemCert cert-pki-ca',token='NSS Certificate DB',pin set
>         certificate: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='subsystemCert cert-pki-ca',token='NSS Certificate DB'
>         CA: dogtag-ipa-ca-renew-agent
>         issuer: CN=Certificate Authority,O=IPA.****.NET
>         subject: CN=vault-hv1.sea2.****.net,O=IPA.****.NET
>         expires: 2021-03-08 04:56:05 UTC
>         dns: vault-hv1.sea2.****.net
>         principal name: HOST/vault-hv1.sea2.****.net@IPA.****.NET
>         key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment
>         eku: id-kp-serverAuth,id-kp-clientAuth
>         pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad
>         post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert "subsystemCert cert-pki-ca"
>         track: yes
>         auto-renew: yes
> Request ID '20190322032032':
>         status: MONITORING
>         stuck: no
>         key pair storage: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='caSigningCert cert-pki-ca',token='NSS Certificate DB',pin set
>         certificate: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='caSigningCert cert-pki-ca',token='NSS Certificate DB'
>         CA: dogtag-ipa-ca-renew-agent
>         issuer: CN=Certificate Authority,O=IPA.****.NET
>         subject: CN=Certificate Authority,O=IPA.****.NET
>         expires: 2037-03-21 04:43:44 UTC
>         key usage: digitalSignature,nonRepudiation,keyCertSign,cRLSign
>         pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad
>         post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert "caSigningCert cert-pki-ca"
>         track: yes
>         auto-renew: yes
> Request ID '20190322032033':
>         status: MONITORING
>         stuck: no
>         key pair storage: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='Server-Cert cert-pki-ca',token='NSS Certificate DB',pin set
>         certificate: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='Server-Cert cert-pki-ca',token='NSS Certificate DB'
>         CA: dogtag-ipa-ca-renew-agent
>         issuer: CN=Certificate Authority,O=IPA.****.NET
>         subject: CN=ipa1-sea2.ipa.****.net,O=IPA.****.NET
>         expires: 2024-12-24 11:37:06 UTC
>         key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment
>         eku: id-kp-serverAuth,id-kp-clientAuth,id-kp-emailProtection
>         pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad
>         post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert "Server-Cert cert-pki-ca"
>         track: yes
>         auto-renew: yes
> Request ID '20190322032117':
>         status: MONITORING
>         stuck: no
>         key pair storage: type=FILE,location='/var/kerberos/krb5kdc/kdc.key'
>         certificate: type=FILE,location='/var/kerberos/krb5kdc/kdc.crt'
>         CA: IPA
>         issuer: CN=Certificate Authority,O=IPA.****.NET
>         subject: CN=ipa1-sea2.ipa.****.net,O=IPA.****.NET
>         expires: 2025-01-26 11:41:35 UTC
>         principal name: krbtgt/IPA.****.NET@IPA.****.NET
>         key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment
>         eku: id-kp-serverAuth,id-pkinit-KPKdc
>         pre-save command:
>         post-save command: /usr/libexec/ipa/certmonger/renew_kdc_cert
>         track: yes
>         auto-renew: yes

I think the root cause is the certmonger tracking has been modified.
There should be no principal associated with any request other than
HTTP, ldap and pkint. The host principal is associated with most of the
CA-related certs. I've never seen this before. It is also possible
whoever messed with the tracking added -N to set the subject to
CN=$FQDN. That would explain the bad subjects.

There is likely no going back in time for this one as the cert validity
ranges are from 2021-2026 somehow. certmonger doesn't report on issued
time so you'd have to look at each cert using certutil to see if there
is a time where all the certs are still valid and see if the CA will start.
> I'm just not sure how to get out of this pickle.  Since the subject Common Name of the certificates is incorrect, I don't think setting the time back will solve this.
> I could potentially do an IPA data only backup (my understanding is that this doesn't include system certs).  Then reinstall each of the 6 servers, install IPA again and restore the data backup.  I believe there may be problems with this method as the /etc/ipa/ca.crt will likely change which I believe would affect the 389 hosts that use IPA.

The Kerberos keys would change which would mean a non-functional system
after the restore. Plus all the clients.

rob

[Travis West]

Rob,

I did manage to find valid .p12 files for ocsp and ca subsystem with an expiry date of 2019.  Is there any way to recover this installation by inserting these?
Alternatively is there a way to delete these errant certs from the NSS DB  in /etc/pki/pki-tomcat/alias and get new ones issued with the correct information?

Or am I way off base here and this is a case where I'm going to have to do a fresh install of IPA, add all users, hosts, etc.. back?

Thank you

[Travis West]

Hm, it looks like on one of the IPA servers I have a copy of /etc/pki/pki-tomcat/alias from 2019.  I'm not sure if I can use this somehow to get back to normal by restoring then setting the time back and trying to renew the certs.

I guess I don't know for sure if this has the correctly named certificates in them, so if there is a way to determine that.

[Travis West]

I've just found an old p12 file from 2019.  I was able to extract the key from that and it does match the CA Subystem cert that expired 8 March that is listed in LDAP.
So if I could somehow generate a new certificate with this and import into the NSS DB for /etc/pki/pki-tomcat/alias would that at least get the CA started?

[Rob Crittenden]

Travis West via FreeIPA-users wrote:
> I've just found an old p12 file from 2019.  I was able to extract the key from that and it does match the CA Subystem cert that expired 8 March that is listed in LDAP.
> So if I could somehow generate a new certificate with this and import into the NSS DB for /etc/pki/pki-tomcat/alias would that at least get the CA started?

Perhaps. It will be complicated because you'll need to move time
multiple times (e.g. start in 2019, renew, move to 2021-ish, renew, move
to 2023-ish, renew).

First you need to fix your certmonger tracking or its likely to fail
again. Back in 2019 when things are running then executing
ipa-server-upgrade should repair the bad tracking.

rob

[Travis West]

Over the weekend I was able to find the CA cert and matching key.  So I was able to generate a new certificate using these have them signed correctly.  Here is how I did that (subsystem cert as an example)

CSR gen
openssl req -new -sha256 -key subsystem.key -subj "/CN=CA Subsystem /O=IPA.***.NET" -out subsystem.csr

Cert gen
openssl x509 -req  -in subsystem.csr -CA ca.crt -CAkey ca.key -set_serial 4 -out subsystem.crt -days 3650 -sha256 -extfile openssl.cnf

create p12
openssl pkcs12  -export -out subsystem.p12 -inkey subsystem.key -in subsystem.crt -certfile ca.crt -name subsystemCert cert-pki-ca
import p12 to NSS DB
pk12util -d . -i subsystem.p12 -n "subsystemCert cert-pki-ca"

The 'extfile' contains some of the v3 attributes

$ cat openssl.cnf
subjectKeyIdentifier=hash
authorityKeyIdentifier=keyid,issuer

keyUsage = critical, nonRepudiation, digitalSignature

This morning I imported the auditSigningCert, subsystemCert, and ocpsSigning certs to /etc/pki/pki-tomcat/alias and the trust attributes are correct.

Then I tried adding them back to certmonger for tracking, and they are now being tracked.
Request ID '20240401141044':
        status: MONITORING
        stuck: no
        key pair storage: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='subsystemCert cert-pki-ca',token='NSS Certificate DB',pin set
        certificate: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='subsystemCert cert-pki-ca',token='NSS Certificate DB'
        CA: dogtag-ipa-ca-renew-agent
        issuer: CN=Certificate Authority,O=IPA.****.NET
        subject: O="IPA.****.NET ",CN="CA Subsystem "
        expires: 2034-03-30 11:10:54 UTC
        key usage: digitalSignature,nonRepudiation
        pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad
        post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert "subsystemCert cert-pki-ca"
        track: yes
        auto-renew: yes
Request ID '20240401141327':
        status: MONITORING
        stuck: no
        key pair storage: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='ocspSigningCert cert-pki-ca',token='NSS Certificate DB',pin set
        certificate: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='ocspSigningCert cert-pki-ca',token='NSS Certificate DB'
        CA: dogtag-ipa-ca-renew-agent
        issuer: CN=Certificate Authority,O=IPA.****.NET
        subject: O="IPA.****.NET ",CN="OCSP Subsystem "
        expires: 2034-03-30 10:59:25 UTC
        key usage: digitalSignature,nonRepudiation
        pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad
        post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert "ocspSigningCert cert-pki-ca"
        track: yes
        auto-renew: yes
Request ID '20240401145826':
        status: MONITORING
        stuck: no
        key pair storage: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='auditSigningCert cert-pki-ca',token='NSS Certificate DB',pin set
        certificate: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='auditSigningCert cert-pki-ca',token='NSS Certificate DB'
        CA: dogtag-ipa-ca-renew-agent
        issuer: CN=Certificate Authority,O=IPA.****.NET
        subject: O="IPA.****.NET ",CN="CA Audit "
        expires: 2034-03-30 11:05:14 UTC
        key usage: digitalSignature,nonRepudiation
        pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad
        post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert "auditSigningCert cert-pki-ca"
        track: yes
        auto-renew: yes

However, after getting them tracked again, the NSS DB appears to have two copies (?)

# certutil -K -d .
certutil: Checking token "NSS Certificate DB" in slot "NSS User Private Key and Certificate Services"
Enter Password or Pin for "NSS Certificate DB":
< 0> rsa      d326b4d65770485d4e0652590101cb7327be0835   caSigningCert cert-pki-ca
< 1> rsa      f5544801e45007862e7593febbeba32c6931b100   subsystemCert cert-pki-ca
< 2> rsa      c13cdf1ff7588fbf7b8a25f7ce3e56d5ae0450cd   ocspSigningCert cert-pki-ca
< 3> rsa      99fffc1c7d251e95374aa15db210aa994c9452ef   NSS Certificate DB:Server-Cert cert-pki-ca
< 4> rsa      75ff858e34df66b838167a31c4d4e12ef76b0044   auditSigningCert cert-pki-ca
< 5> rsa      623e08407bf1fbace5146c7413e343935a987243   subsystemCert cert-pki-ca
< 6> rsa      2c62bcd9a61f0db2288c0e85c9c4f316793df98a   ocspSigningCert cert-pki-ca

But here only shows one, with correct trust attributes

# certutil -L -d .

Certificate Nickname                                         Trust Attributes
                                                             SSL,S/MIME,JAR/XPI

subsystemCert cert-pki-ca                                    u,u,u
ocspSigningCert cert-pki-ca                                  u,u,u
caSigningCert cert-pki-ca                                    CTu,Cu,Cu
Server-Cert cert-pki-ca                                      u,u,u
auditSigningCert cert-pki-ca                                 u,u,Pu

I also updated the subsystemCert in LDAP so that matches (both cert and serial)

I am still unable to get pki-tomcat to start when I run 'ipactl start'  but if I check the service using systemctl it appears to be running

Clearly there is still something I'm missing.

[Travis West]

I noticed some issues with the newly generated certs in my previous update.  I have again generated new certs that this time have the Subject correct.
I can delete the bad certs that contain errant Principal from /etc/pki/pki-tomcat/alias and import the new ones.  Then update the CA Subsystem Certificate in LDAP using ldapmodify and an updatecert.ldif, as well as update the cert and certreq values in /etc/pki/pki-tomcat/ca/CS.cfg.

What I am unclear on is how to fix the certmonger tracking so it isn't tracking the certs with the errant Principal.

[Travis West]

Okay, I've sort of fixed the tracking, but there is still an issue I can't seem to solve.  Here is the tracking now for the Audit, OCSP, and Subsystem certificates

Number of certificates and requests being tracked: 9.
Request ID '20190322032029':
        status: MONITORING
        stuck: no
        key pair storage: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='auditSigningCert cert-pki-ca',token='NSS Certificate DB',pin set
        certificate: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='auditSigningCert cert-pki-ca',token='NSS Certificate DB'
        CA: dogtag-ipa-ca-renew-agent
        issuer: CN=Certificate Authority,O=IPA.****.NET
        subject: O=IPA.****.NET,CN="CA Audit "
        expires: 2034-03-31 14:24:53 UTC
        key usage: digitalSignature,nonRepudiation
        pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad
        post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert "auditSigningCert cert-pki-ca"
        track: yes
        auto-renew: yes
[root@ipa1-sea2 ~]# getcert list -i 20190322032030
Number of certificates and requests being tracked: 9.
Request ID '20190322032030':
        status: MONITORING
        stuck: no
        key pair storage: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='ocspSigningCert cert-pki-ca',token='NSS Certificate DB',pin set
        certificate: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='ocspSigningCert cert-pki-ca',token='NSS Certificate DB'
        CA: dogtag-ipa-ca-renew-agent
        issuer: CN=Certificate Authority,O=IPA.****.NET
        subject: O=IPA.****.NET,CN="OCSP Subsystem "
        expires: 2034-03-31 14:15:41 UTC
        key usage: digitalSignature,nonRepudiation
        pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad
        post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert "ocspSigningCert cert-pki-ca"
        track: yes
        auto-renew: yes
[root@ipa1-sea2 ~]# getcert list -i 20190322032031
Number of certificates and requests being tracked: 9.
Request ID '20190322032031':
        status: MONITORING
        stuck: no
        key pair storage: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='subsystemCert cert-pki-ca',token='NSS Certificate DB',pin set
        certificate: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='subsystemCert cert-pki-ca',token='NSS Certificate DB'
        CA: dogtag-ipa-ca-renew-agent
        issuer: CN=Certificate Authority,O=IPA.****.NET
        subject: O=IPA.****.NET,CN="CA Subsystem "
        expires: 2034-03-31 14:40:33 UTC
        key usage: digitalSignature,nonRepudiation
        pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad
        post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert "subsystemCert cert-pki-ca"
        track: yes

In each of these the Subject line has the CN and O backwards.  If I look at the certificates themselves, they have it listed correctly

# openssl pkcs12 -info -in audit.p12
MAC Iteration 2048
MAC verified OK
PKCS7 Encrypted data: pbeWithSHA1And40BitRC2-CBC, Iteration 2048
Certificate bag
Bag Attributes
    localKeyID: A8 74 8A 94 58 C0 9E 28 3F 55 B9 F7 AC 9D 78 33 8E D3 C6 E3
    friendlyName: auditSigningCert cert-pki-ca
subject=/CN=CA Audit /O=IPA.****.NET
issuer=/O=IPA.****.NET/CN=Certificate Authority

So I'm confused as to how the 'getcert' output has the items in Subject reversed.

[Rob Crittenden]

Travis West via FreeIPA-users wrote:
> Okay, I've sort of fixed the tracking, but there is still an issue I can't seem to solve.  Here is the tracking now for the Audit, OCSP, and Subsystem certificates
>
> Number of certificates and requests being tracked: 9.
> Request ID '20190322032029':
>         status: MONITORING
>         stuck: no
>         key pair storage: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='auditSigningCert cert-pki-ca',token='NSS Certificate DB',pin set
>         certificate: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='auditSigningCert cert-pki-ca',token='NSS Certificate DB'
>         CA: dogtag-ipa-ca-renew-agent
>         issuer: CN=Certificate Authority,O=IPA.****.NET
>         subject: O=IPA.****.NET,CN="CA Audit "
>         expires: 2034-03-31 14:24:53 UTC
>         key usage: digitalSignature,nonRepudiation
>         pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad
>         post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert "auditSigningCert cert-pki-ca"
>         track: yes
>         auto-renew: yes
> [root@ipa1-sea2 ~]# getcert list -i 20190322032030
> Number of certificates and requests being tracked: 9.
> Request ID '20190322032030':
>         status: MONITORING
>         stuck: no
>         key pair storage: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='ocspSigningCert cert-pki-ca',token='NSS Certificate DB',pin set
>         certificate: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='ocspSigningCert cert-pki-ca',token='NSS Certificate DB'
>         CA: dogtag-ipa-ca-renew-agent
>         issuer: CN=Certificate Authority,O=IPA.****.NET
>         subject: O=IPA.****.NET,CN="OCSP Subsystem "
>         expires: 2034-03-31 14:15:41 UTC
>         key usage: digitalSignature,nonRepudiation
>         pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad
>         post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert "ocspSigningCert cert-pki-ca"
>         track: yes
>         auto-renew: yes
> [root@ipa1-sea2 ~]# getcert list -i 20190322032031
> Number of certificates and requests being tracked: 9.
> Request ID '20190322032031':
>         status: MONITORING
>         stuck: no
>         key pair storage: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='subsystemCert cert-pki-ca',token='NSS Certificate DB',pin set
>         certificate: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='subsystemCert cert-pki-ca',token='NSS Certificate DB'
>         CA: dogtag-ipa-ca-renew-agent
>         issuer: CN=Certificate Authority,O=IPA.****.NET
>         subject: O=IPA.****.NET,CN="CA Subsystem "
>         expires: 2034-03-31 14:40:33 UTC
>         key usage: digitalSignature,nonRepudiation
>         pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad
>         post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert "subsystemCert cert-pki-ca"
>         track: yes
>
> In each of these the Subject line has the CN and O backwards.  If I look at the certificates themselves, they have it listed correctly
>
> # openssl pkcs12 -info -in audit.p12
> MAC Iteration 2048
> MAC verified OK
> PKCS7 Encrypted data: pbeWithSHA1And40BitRC2-CBC, Iteration 2048
> Certificate bag
> Bag Attributes
>     localKeyID: A8 74 8A 94 58 C0 9E 28 3F 55 B9 F7 AC 9D 78 33 8E D3 C6 E3
>     friendlyName: auditSigningCert cert-pki-ca
> subject=/CN=CA Audit /O=IPA.****.NET
> issuer=/O=IPA.****.NET/CN=Certificate Authority
>
> So I'm confused as to how the 'getcert' output has the items in Subject reversed.

The OpenSSL and NSS libraries merely display the data differently. It's
fine.

But you still have an issue with the certificates. You have a trailing
space after at least the audit, subsystem and OCSP certs. I think you
tried to quote only that when generating the subject rather than the
entire thing.

So

O=IPA.****.NET,"CN=CA Audit "

rather than

"O=IPA.****.NET,CN=CA Audit"

Once the certificates are valid you can try running ipa-server-upgrade.
It should repair bad tracking.

But with the strange subjects I'm not sure what will happen. What I do
know is that "CA Audit " != "CA Audit" in a subject.

rob

[Travis West]

This was the command I used to generate the CSRs

openssl req -new -sha256 -key subsystem.key -subj "/CN=CA Subsystem /O=IPA.****.NET" -out subsystem.csr

But I guess that results in the extra space.  So perhaps it should be

openssl req -new -sha256 -key subsystem.key -subj "/CN=CA Subsystem,/O=IPA.***.NET" -out subsystem.csr  (?)

Apologies for all the questions that might seem pretty basic.  Just trying to get this figured out.

[Travis West]

Okay, I've generated new certs that don't have the extra space.  Once those were imported to the NSS DB I also updated the CS.cfg with the new cert and certreq vaules for OCSP, Audit, and Subsystem.
I also did an ldapsearch for the Subsystem certificate to make sure it matches.  I then tried to run ipa-server-upgrade, but it failed.

Tracking Requests:

Request ID '20190322032031':
        status: MONITORING
        stuck: no
        key pair storage: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='subsystemCert cert-pki-ca',token='NSS Certificate DB',pin set
        certificate: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='subsystemCert cert-pki-ca',token='NSS Certificate DB'
        CA: dogtag-ipa-ca-renew-agent
        issuer: CN=Certificate Authority,O=IPA.****.NET
        subject: O=IPA.****.NET,CN=CA Subsystem
        expires: 2034-03-31 17:57:15 UTC
        key usage: digitalSignature,nonRepudiation
        pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad
        post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert "subsystemCert cert-pki-ca"
        track: yes
        auto-renew: yes

Request ID '20190322032030':
        status: MONITORING
        stuck: no
        key pair storage: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='ocspSigningCert cert-pki-ca',token='NSS Certificate DB',pin set
        certificate: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='ocspSigningCert cert-pki-ca',token='NSS Certificate DB'
        CA: dogtag-ipa-ca-renew-agent
        issuer: CN=Certificate Authority,O=IPA.****.NET
        subject: O=IPA.****.NET,CN=OCSP Subsystem
        expires: 2034-03-31 18:02:29 UTC
        key usage: digitalSignature,nonRepudiation
        pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad
        post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert "ocspSigningCert cert-pki-ca"
        track: yes
        auto-renew: yes

Request ID '20190322032029':
        status: MONITORING
        stuck: no
        key pair storage: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='auditSigningCert cert-pki-ca',token='NSS Certificate DB',pin set
        certificate: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='auditSigningCert cert-pki-ca',token='NSS Certificate DB'
        CA: dogtag-ipa-ca-renew-agent
        issuer: CN=Certificate Authority,O=IPA.****.NET
        subject: O=IPA.****.NET,CN=CA Audit
        expires: 2034-03-31 18:00:11 UTC
        key usage: digitalSignature,nonRepudiation
        pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad
        post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert "auditSigningCert cert-pki-ca"
        track: yes
        auto-renew: yes

Subsystem in LDAP matches the NSS DB

# ldapsearch -LLL -D 'cn=directory manager' -W -b uid=pkidbuser,ou=people,o=ipaca userCertificate description seeAlso
Enter LDAP Password:
dn: uid=pkidbuser,ou=people,o=ipaca
userCertificate:: MIIDNjCCA...EyISxo3w==
description: 2;4;CN=Certificate Authority,O=IPA.****.NET;CN=CA Subsystem,O=IPA.***.NET
seeAlso: CN=CA Subsystem,O=IPA****.NET

[root@ipa1-sea2 log]# certutil -L -d /etc/pki/pki-tomcat/alias -n 'subsystemCert cert-pki-ca' -a
-----BEGIN CERTIFICATE-----
MIIDNjCCA...EyISxo3w==
-----END CERTIFICATE-----
[root@ipa1-sea2 log]# certutil -L -d /etc/pki/pki-tomcat/alias -n 'subsystemCert cert-pki-ca' | grep Serial
        Serial Number: 4 (0x4)

*note the Serial in LDAP is '4' while in NSS DB it shows as 4 (0x4)  not sure if this is the issue.

Output of ipa-server-upgrade

# ipa-server-upgrade
Upgrading IPA:. Estimated time: 1 minute 30 seconds
  [1/11]: stopping directory server
  [2/11]: saving configuration
  [3/11]: disabling listeners
  [4/11]: enabling DS global lock
  [5/11]: disabling Schema Compat
  [6/11]: starting directory server
  [7/11]: updating schema
  [8/11]: upgrading server
  [9/11]: stopping directory server
  [10/11]: restoring configuration
  [11/11]: starting directory server
Done.
Update complete
Upgrading IPA services
Upgrading the configuration of the IPA services
[Verifying that root certificate is published]
[Migrate CRL publish directory]
Publish directory already set to new location
[Verifying that CA proxy configuration is correct]
IPA server upgrade failed: Inspect /var/log/ipaupgrade.log and run command ipa-server-upgrade manually.
CA did not start in 300.0s

Output in the /var/log/pki/pki-tomcat/ca/system log while the ugprade was running

2024-04-02T18:30:11Z DEBUG response body '<html><head><title>Apache Tomcat/7.0.76 - Error report</title><style><!--H1 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:22px;} H2 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:16px;} H3 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:14px;} BODY {font-family:Tahoma,Arial,sans-serif;color:black;background-color:white;} B {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;} P {font-family:Tahoma,Arial,sans-serif;background:white;color:black;font-size:12px;}A {color : black;}A.name {color : black;}HR {color : #525D76;}--></style> </head><body><h1>HTTP Status 500 - Subsystem unavailable</h1><HR size="1" noshade="noshade"><p><b>type</b> Exception report</p><p><b>message</b> <u>Subsystem unavailable</u></p><p><b>description</b> <u>The server encountered an internal error that prevented it from fulfilling this requ
 est.</u></p><p><b>exception</b> <pre>javax.ws.rs.ServiceUnavailableException: Subsystem unavailable\n\tcom.netscape.cms.tomcat.ProxyRealm.findSecurityConstraints(ProxyRealm.java:145)\n\torg.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:500)\n\torg.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:103)\n\torg.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:962)\n\torg.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:445)\n\torg.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1087)\n\torg.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:637)\n\torg.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:316)\n\tjava.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)\n\tjava.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)\n\torg.apache.tomcat.util.threads.TaskThrea
 d$WrappingRunnable.run(TaskThread.java:61)\n\tjava.lang.Thread.run(Thread.java:748)\n</pre></p><p><b>note</b> <u>The full stack trace of the root cause is available in the Apache Tomcat/7.0.76 logs.</u></p><HR size="1" noshade="noshade"><h3>Apache Tomcat/7.0.76</h3></body></html>'
2024-04-02T18:30:11Z DEBUG The CA status is: check interrupted due to error: Retrieving CA status failed with status 500
2024-04-02T18:30:11Z DEBUG Waiting for CA to start...
2024-04-02T18:30:12Z DEBUG request POST http://ipa1-sea2.ipa.****.net:8080/ca/admin/ca/getStatus
2024-04-02T18:30:12Z DEBUG request body ''
2024-04-02T18:30:12Z DEBUG response status 500
2024-04-02T18:30:12Z DEBUG response headers Server: Apache-Coyote/1.1
Content-Type: text/html;charset=utf-8
Content-Language: en
Content-Length: 2208
Date: Tue, 02 Apr 2024 18:30:12 GMT
Connection: close

2024-04-02T18:30:12Z DEBUG response body '<html><head><title>Apache Tomcat/7.0.76 - Error report</title><style><!--H1 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:22px;} H2 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:16px;} H3 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:14px;} BODY {font-family:Tahoma,Arial,sans-serif;color:black;background-color:white;} B {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;} P {font-family:Tahoma,Arial,sans-serif;background:white;color:black;font-size:12px;}A {color : black;}A.name {color : black;}HR {color : #525D76;}--></style> </head><body><h1>HTTP Status 500 - Subsystem unavailable</h1><HR size="1" noshade="noshade"><p><b>type</b> Exception report</p><p><b>message</b> <u>Subsystem unavailable</u></p><p><b>description</b> <u>The server encountered an internal error that prevented it from fulfilling this requ
 est.</u></p><p><b>exception</b> <pre>javax.ws.rs.ServiceUnavailableException: Subsystem unavailable\n\tcom.netscape.cms.tomcat.ProxyRealm.findSecurityConstraints(ProxyRealm.java:145)\n\torg.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:500)\n\torg.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:103)\n\torg.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:962)\n\torg.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:445)\n\torg.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1087)\n\torg.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:637)\n\torg.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:316)\n\tjava.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)\n\tjava.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)\n\torg.apache.tomcat.util.threads.TaskThrea
 d$WrappingRunnable.run(TaskThread.java:61)\n\tjava.lang.Thread.run(Thread.java:748)\n</pre></p><p><b>note</b> <u>The full stack trace of the root cause is available in the Apache Tomcat/7.0.76 logs.</u></p><HR size="1" noshade="noshade"><h3>Apache Tomcat/7.0.76</h3></body></html>'
2024-04-02T18:30:12Z DEBUG The CA status is: check interrupted due to error: Retrieving CA status failed with status 500
2024-04-02T18:30:12Z DEBUG Waiting for CA to start...

[Florence Blanc-Renaud]

Hi,

On Tue, Apr 2, 2024 at 8:50 PM Travis West via FreeIPA-users <
[e-mail]> wrote:
> Okay, I've generated new certs that don't have the extra space.  Once
> those were imported to the NSS DB I also updated the CS.cfg with the new
> cert and certreq vaules for OCSP, Audit, and Subsystem.
> I also did an ldapsearch for the Subsystem certificate to make sure it
> matches.  I then tried to run ipa-server-upgrade, but it failed.
>
> Tracking Requests:
>
> Request ID '20190322032031':
>         status: MONITORING
>         stuck: no
>         key pair storage:
> type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='subsystemCert
> cert-pki-ca',token='NSS Certificate DB',pin set
>         certificate:
> type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='subsystemCert
> cert-pki-ca',token='NSS Certificate DB'
>         CA: dogtag-ipa-ca-renew-agent
>         issuer: CN=Certificate Authority,O=IPA.****.NET
>         subject: O=IPA.****.NET,CN=CA Subsystem

As Rob wrote, it's not a problem that getcert list, OpenssL and NSS
libraries show the subject in a DN order (RFC2253) or DN reverse order, but
I find it suspect that issuer and subject have picked inconsistent order.
In my f35 instance, getcert list shows the following:
issuer: CN=Certificate Authority,O=IPA.TEST
subject: CN=CA Subsystem,O=IPA.TEST

flo

expires: 2034-03-31 17:57:15 UTC
>     key usage: digitalSignature,nonRepudiation
>     pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad
>     post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert
>
> "subsystemCert cert-pki-ca"
>         track: yes
>         auto-renew: yes
>
> Request ID '20190322032030':
>         status: MONITORING
>         stuck: no
>         key pair storage:
> type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='ocspSigningCert
> cert-pki-ca',token='NSS Certificate DB',pin set
>         certificate:
> type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='ocspSigningCert
> cert-pki-ca',token='NSS Certificate DB'
>         CA: dogtag-ipa-ca-renew-agent
>         issuer: CN=Certificate Authority,O=IPA.****.NET
>         subject: O=IPA.****.NET,CN=OCSP Subsystem
>         expires: 2034-03-31 18:02:29 UTC
>         key usage: digitalSignature,nonRepudiation
>         pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad
>         post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert
> "ocspSigningCert cert-pki-ca"
>         track: yes
>         auto-renew: yes
>
> Request ID '20190322032029':
>         status: MONITORING
>         stuck: no
>         key pair storage:
> type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='auditSigningCert
> cert-pki-ca',token='NSS Certificate DB',pin set
>         certificate:
> type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='auditSigningCert
> cert-pki-ca',token='NSS Certificate DB'
>         CA: dogtag-ipa-ca-renew-agent
>         issuer: CN=Certificate Authority,O=IPA.****.NET
>         subject: O=IPA.****.NET,CN=CA Audit
>         expires: 2034-03-31 18:00:11 UTC
>         key usage: digitalSignature,nonRepudiation
>         pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad
>         post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert
> "auditSigningCert cert-pki-ca"
>         track: yes
>         auto-renew: yes
>
> Subsystem in LDAP matches the NSS DB
>
> # ldapsearch -LLL -D 'cn=directory manager' -W -b
> uid=pkidbuser,ou=people,o=ipaca userCertificate description seeAlso
> Enter LDAP Password:
> dn: uid=pkidbuser,ou=people,o=ipaca
> userCertificate:: MIIDNjCCA...EyISxo3w==
> description: 2;4;CN=Certificate Authority,O=IPA.****.NET;CN=CA
> Subsystem,O=IPA.***.NET
> seeAlso: CN=CA Subsystem,O=IPA****.NET
>
> [root@ipa1-sea2 log]# certutil -L -d /etc/pki/pki-tomcat/alias -n
> 'subsystemCert cert-pki-ca' -a
> -----BEGIN CERTIFICATE-----
> MIIDNjCCA...EyISxo3w==
> -----END CERTIFICATE-----
> [root@ipa1-sea2 log]# certutil -L -d /etc/pki/pki-tomcat/alias -n
> 'subsystemCert cert-pki-ca' | grep Serial
>         Serial Number: 4 (0x4)
>
> *note the Serial in LDAP is '4' while in NSS DB it shows as 4 (0x4)  not
> sure if this is the issue.
>
> Output of ipa-server-upgrade
>
> # ipa-server-upgrade
> Upgrading IPA:. Estimated time: 1 minute 30 seconds
>   [1/11]: stopping directory server
>   [2/11]: saving configuration
>   [3/11]: disabling listeners
>   [4/11]: enabling DS global lock
>   [5/11]: disabling Schema Compat
>   [6/11]: starting directory server
>   [7/11]: updating schema
>   [8/11]: upgrading server
>   [9/11]: stopping directory server
>   [10/11]: restoring configuration
>   [11/11]: starting directory server
> Done.
> Update complete
> Upgrading IPA services
> Upgrading the configuration of the IPA services
> [Verifying that root certificate is published]
> [Migrate CRL publish directory]
> Publish directory already set to new location
> [Verifying that CA proxy configuration is correct]
> IPA server upgrade failed: Inspect /var/log/ipaupgrade.log and run command
> ipa-server-upgrade manually.
> CA did not start in 300.0s
>
> Output in the /var/log/pki/pki-tomcat/ca/system log while the ugprade was
> running
>
> 2024-04-02T18:30:11Z DEBUG response body '<html><head><title>Apache
> Tomcat/7.0.76 - Error report</title><style><!--H1
> {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:22px;}
> H2
> {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:16px;}
> H3
> {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:14px;}
> BODY
> {font-family:Tahoma,Arial,sans-serif;color:black;background-color:white;} B
> {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;}
> P
> {font-family:Tahoma,Arial,sans-serif;background:white;color:black;font-size:12px;}A
> {color : black;}A.name {color : black;}HR {color : #525D76;}--></style>
>
> </head><body><h1>HTTP Status 500 - Subsystem unavailable</h1><HR size="1"
> noshade="noshade"><p><b>type</b> Exception report</p><p><b>message</b>
> <u>Subsystem unavailable</u></p><p><b>description</b> <u>The server
> encountered an internal error that prevented it from fulfilling this requ
>  est.</u></p><p><b>exception</b> <pre>javax.ws.rs.ServiceUnavailableException:
> Subsystem
> unavailable\n\tcom.netscape.cms.tomcat.ProxyRealm.findSecurityConstraints(ProxyRealm.java:145)\n\torg.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:500)\n\torg.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:103)\n\torg.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:962)\n\torg.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:445)\n\torg.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1087)\n\torg.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:637)\n\torg.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:316)\n\tjava.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)\n\tjava.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)\n\torg.apache.tomcat.util.threads.TaskThrea
>  d$WrappingRunnable.run(TaskThread.java:61)\n\tjava.lang.Thread.run(Thread.java:748)\n</pre></p><p><b>note</b>
> <u>The full stack trace of the root cause is available in the Apache
> Tomcat/7.0.76 logs.</u></p><HR size="1" noshade="noshade"><h3>Apache
> Tomcat/7.0.76</h3></body></html>'
> 2024-04-02T18:30:11Z DEBUG The CA status is: check interrupted due to
> error: Retrieving CA status failed with status 500
> 2024-04-02T18:30:11Z DEBUG Waiting for CA to start...
> 2024-04-02T18:30:12Z DEBUG request POST http://ipa1-sea2.ipa.
> ****.net:8080/ca/admin/ca/getStatus
> 2024-04-02T18:30:12Z DEBUG request body ''
> 2024-04-02T18:30:12Z DEBUG response status 500
> 2024-04-02T18:30:12Z DEBUG response headers Server: Apache-Coyote/1.1
> Content-Type: text/html;charset=utf-8
> Content-Language: en
> Content-Length: 2208
> Date: Tue, 02 Apr 2024 18:30:12 GMT
> Connection: close
>
> 2024-04-02T18:30:12Z DEBUG response body '<html><head><title>Apache
> Tomcat/7.0.76 - Error report</title><style><!--H1
> {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:22px;}
> H2
> {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:16px;}
> H3
> {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:14px;}
> BODY
> {font-family:Tahoma,Arial,sans-serif;color:black;background-color:white;} B
> {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;}
> P
> {font-family:Tahoma,Arial,sans-serif;background:white;color:black;font-size:12px;}A
> {color : black;}A.name {color : black;}HR {color : #525D76;}--></style>
>
> </head><body><h1>HTTP Status 500 - Subsystem unavailable</h1><HR size="1"
> noshade="noshade"><p><b>type</b> Exception report</p><p><b>message</b>
> <u>Subsystem unavailable</u></p><p><b>description</b> <u>The server
> encountered an internal error that prevented it from fulfilling this requ
>  est.</u></p><p><b>exception</b> <pre>javax.ws.rs.ServiceUnavailableException:
> Subsystem
> unavailable\n\tcom.netscape.cms.tomcat.ProxyRealm.findSecurityConstraints(ProxyRealm.java:145)\n\torg.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:500)\n\torg.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:103)\n\torg.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:962)\n\torg.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:445)\n\torg.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1087)\n\torg.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:637)\n\torg.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:316)\n\tjava.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)\n\tjava.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)\n\torg.apache.tomcat.util.threads.TaskThrea
>  d$WrappingRunnable.run(TaskThread.java:61)\n\tjava.lang.Thread.run(Thread.java:748)\n</pre></p><p><b>note</b>
> <u>The full stack trace of the root cause is available in the Apache
> Tomcat/7.0.76 logs.</u></p><HR size="1" noshade="noshade"><h3>Apache
> Tomcat/7.0.76</h3></body></html>'
> 2024-04-02T18:30:12Z DEBUG The CA status is: check interrupted due to
> error: Retrieving CA status failed with status 500
> 2024-04-02T18:30:12Z DEBUG Waiting for CA to start...
>
> --
> _______________________________________________
> FreeIPA-users mailing list -- [e-mail]
> To unsubscribe send an email to [e-mail]
> Fedora Code of Conduct:
> https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives:
> https://lists.fedorahosted.org/archives/list/[e-mail]...
> Do not reply to spam, report it:
> https://pagure.io/fedora-infrastructure/new_issue

[Travis West]

> Hi,
>
> On Tue, Apr 2, 2024 at 8:50 PM Travis West via FreeIPA-users <
> freeipa-users(a)lists.fedorahosted.org&gt; wrote:
>
> As Rob wrote, it's not a problem that getcert list, OpenssL and NSS
> libraries show the subject in a DN order (RFC2253) or DN reverse order, but
> I find it suspect that issuer and subject have picked inconsistent order.
> In my f35 instance, getcert list shows the following:
> issuer: CN=Certificate Authority,O=IPA.TEST
> subject: CN=CA Subsystem,O=IPA.TEST

I'm not sure I follow.  My getcert list output looks like that, except the CN and O are reversed in the Subject line

issuer: CN=Certificate Authority,O=IPA.****.NET
subject: O=IPA.****.NET,CN=OCSP Subsystem

issuer: CN=Certificate Authority,O=IPA.****.NET
subject: O=IPA.****.NET,CN=CA Subsystem

issuer: CN=Certificate Authority,O=IPA.****.NET
subject: O=IPA.****.NET,CN=CA Audit

[Florence Blanc-Renaud]

On Wed, Apr 3, 2024 at 5:24 AM Travis West via FreeIPA-users <
[e-mail]> wrote:
> > Hi,
> >
> > On Tue, Apr 2, 2024 at 8:50 PM Travis West via FreeIPA-users <
> > freeipa-users(a)lists.fedorahosted.org&gt; wrote:
> >
> > As Rob wrote, it's not a problem that getcert list, OpenssL and NSS
> > libraries show the subject in a DN order (RFC2253) or DN reverse order,
>
> but
> > I find it suspect that issuer and subject have picked inconsistent order.
> > In my f35 instance, getcert list shows the following:
> > issuer: CN=Certificate Authority,O=IPA.TEST
> > subject: CN=CA Subsystem,O=IPA.TEST
>
> I'm not sure I follow.  My getcert list output looks like that, except the
> CN and O are reversed in the Subject line

That's exactly my point. I would expect subject and issuer to display the
components in the same order (ending with O=IPA.****.NET). The subject was
provided to openssl req command, you can try to provide it in the reverse
order.

flo
> issuer: CN=Certificate Authority,O=IPA.****.NET
> subject: O=IPA.****.NET,CN=OCSP Subsystem
>
> issuer: CN=Certificate Authority,O=IPA.****.NET
> subject: O=IPA.****.NET,CN=CA Subsystem
>
> issuer: CN=Certificate Authority,O=IPA.****.NET
> subject: O=IPA.****.NET,CN=CA Audit
> --
> _______________________________________________
> FreeIPA-users mailing list -- [e-mail]
> To unsubscribe send an email to [e-mail]
> Fedora Code of Conduct:
> https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives:
> https://lists.fedorahosted.org/archives/list/[e-mail]...
> Do not reply to spam, report it:
> https://pagure.io/fedora-infrastructure/new_issue

[Travis West]

> On Wed, Apr 3, 2024 at 5:24 AM Travis West via FreeIPA-users <
> freeipa-users(a)lists.fedorahosted.org&gt; wrote:
>
> That's exactly my point. I would expect subject and issuer to display the
> components in the same order (ending with O=IPA.****.NET). The subject was
> provided to openssl req command, you can try to provide it in the reverse
> order.

If I look at the p12 file I created from the it has them listed in the correct order for Subject, but the Issuer line is reversed from what getcert shows

subject=/CN=OCSP Subsystem/O=IPA.****.NET
issuer=/O=IPA.****.NET/CN=Certificate Authority

subject=/CN=CA Subsystem/O=IPA.****.NET
issuer=/O=IPA.****.NET/CN=Certificate Authority

subject=/CN=CA Audit/O=IPA.****.NET
issuer=/O=IPA.****.NET/CN=Certificate Authority

The CSR was created using this command

openssl req -new -sha256 -key ocsp.key -subj "/CN=OCSP Subsystem /O=IPA.SUPERB.NET" -out ocsp.csr

The certificate was requested using this command

x509 -req -in ocsp.csr -CA ca.crt -CAkey ca.key -set_serial 2 -out ocsp.crt -days 3650 -sha256

So you're saying in that CSR req to swap CN and O for that -subj flag?

[Travis West]

Swapping the O and CN in the req did the trick for the getcert list output

Request ID '20190322032031':
        status: MONITORING
        stuck: no
        key pair storage: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='subsystemCert cert-pki-ca',token='NSS Certificate DB',pin set
        certificate: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='subsystemCert cert-pki-ca',token='NSS Certificate DB'
        CA: dogtag-ipa-ca-renew-agent
        issuer: CN=Certificate Authority,O=IPA.****.NET
        subject: CN=CA Subsystem,O=IPA.****.NET
        expires: 2034-04-01 11:35:47 UTC
        key usage: digitalSignature,nonRepudiation
        pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad
        post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert "subsystemCert cert-pki-ca"
        track: yes
        auto-renew: yes

Request ID '20190322032030':
        status: MONITORING
        stuck: no
        key pair storage: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='ocspSigningCert cert-pki-ca',token='NSS Certificate DB',pin set
        certificate: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='ocspSigningCert cert-pki-ca',token='NSS Certificate DB'
        CA: dogtag-ipa-ca-renew-agent
        issuer: CN=Certificate Authority,O=IPA.****.NET
        subject: CN=OCSP Subsystem,O=IPA.****.NET
        expires: 2034-04-01 11:32:48 UTC
        key usage: digitalSignature,nonRepudiation
        pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad
        post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert "ocspSigningCert cert-pki-ca"
        track: yes
        auto-renew: yes

Request ID '20190322032029':
        status: MONITORING
        stuck: no
        key pair storage: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='auditSigningCert cert-pki-ca',token='NSS Certificate DB',pin set
        certificate: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='auditSigningCert cert-pki-ca',token='NSS Certificate DB'
        CA: dogtag-ipa-ca-renew-agent
        issuer: CN=Certificate Authority,O=IPA.****.NET
        subject: CN=CA Audit,O=IPA.****.NET
        expires: 2034-04-01 11:38:26 UTC
        key usage: digitalSignature,nonRepudiation
        pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad
        post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert "auditSigningCert cert-pki-ca"
        track: yes
        auto-renew: yes

I then updated LDAP with the new CA Subsystem cert, so that and the serial for it  match

# ldapsearch -LLL -D 'cn=directory manager' -W -b uid=pkidbuser,ou=people,o=ipaca userCertificate description seeAlso
Enter LDAP Password:
dn: uid=pkidbuser,ou=people,o=ipaca
userCertificate:: MIIDNj....RXOm8Q==
description: 2;4;CN=Certificate Authority,O=IPA.****.NET;CN=CA Subsystem,O=IPA.****.NET
seeAlso: CN=CA Subsystem,O=IPA.****.NET

# certutil -L -d /etc/pki/pki-tomcat/alias -n 'subsystemCert cert-pki-ca' -a
-----BEGIN CERTIFICATE-----
MIIDNj....RXOm8Q==
-----END CERTIFICATE-----

# certutil -L -d /etc/pki/pki-tomcat/alias -n 'subsystemCert cert-pki-ca' | grep Serial
        Serial Number: 4 (0x4)

After this I tried an 'ipactl restart  --ignore-service-failures' but pki-tomcat  still failed to start.  So I tried manually stopping that service using systemctl stop [e-mail] then issuing an 'ipactl start  --ignore-service-failures.
This time all services seem to have started

# ipactl start  --ignore-service-failures
Starting Directory Service
Starting krb5kdc Service
Starting kadmin Service
Starting named Service
Starting httpd Service
Starting ipa-custodia Service
Starting pki-tomcatd Service
Starting ipa-otpd Service
Starting ipa-dnskeysyncd Service
ipa: INFO: The ipactl command was successful

If I login to the UI I can now browse to Authentication > Certificates, where as before I got an error when going here.

So far so good.  Now, I've got 5 other servers in this cluster, all denoted as Master, with this server set as the CA Renewal Master.  Do I need to repeat the certificate import steps on the other 5 servers or is there a way to replicate over the new certificates to the other hosts?

[Travis West]

Spoke too soon.  If I try to get a new certificate on an enrolled host I get this

status: CA_UNREACHABLE
ca-error: Server at https://ipa1-sea2.ipa.****.net/ipa/xml failed request, will retry: 907 (RPC failed at server.  cannot connect to 'https://ipa1-sea2.ipa.****.net:443/ca/rest/account/login': [SSL: SSL_HANDSHAKE_FAILURE] ssl handshake failure (_ssl.c:1822)).

This reflected in the UI if I go to Authentication > Certificates > Certificate Authorities where I see the same error.

The IPA server listed there is the one where all services started via ipactl start in my previous update.

[Rob Crittenden]

Travis West via FreeIPA-users wrote:
> Spoke too soon.  If I try to get a new certificate on an enrolled host I get this
>
> status: CA_UNREACHABLE
> ca-error: Server at https://ipa1-sea2.ipa.****.net/ipa/xml failed request, will retry: 907 (RPC failed at server.  cannot connect to 'https://ipa1-sea2.ipa.****.net:443/ca/rest/account/login': [SSL: SSL_HANDSHAKE_FAILURE] ssl handshake failure (_ssl.c:1822)).
>
> This reflected in the UI if I go to Authentication > Certificates > Certificate Authorities where I see the same error.
>
> The IPA server listed there is the one where all services started via ipactl start in my previous update.

I think you need to take a look at fresh logs to see what failed. It may
point to why.

I assume you went back in time to 2019 and then leaped forward 2 years
at a pop, renewing as you went, and now it's present day?

rob

[Travis West]

No I didn't go back in time, I generated new certificates and imported them to NSS DB after deleting the ones that contained Principles that had other hosts listed.
I then updated the CS.cfg with the cert and certreq values, and made sure the CA Subsystem cert in NSS DB matched what is in  LDAP.

I'm not sure what logs to look at.  /etc/pki/pki-tomcat/ca/selftest has no errors /etc/pki/pki-tomcat/ca/system has the last error from before I got ipa to fully start.  The debug log has a lot of information, but nothing that looks like an error.

I've got no expired certs

# getcert list | grep expires
        expires: 2025-01-26 11:37:18 UTC
        expires: 2025-01-26 11:37:04 UTC
        expires: 2026-03-12 13:24:44 UTC
        expires: 2034-04-01 11:38:26 UTC
        expires: 2034-04-01 11:32:48 UTC
        expires: 2034-04-01 11:35:47 UTC
        expires: 2037-03-21 04:43:44 UTC
        expires: 2024-12-24 11:37:06 UTC
        expires: 2025-01-26 11:41:35 UTC

Trust attributes all look correct in /etc/pki/pki-tomcat/alias
# certutil -L -d .

Certificate Nickname                                         Trust Attributes
                                                             SSL,S/MIME,JAR/XPI

subsystemCert cert-pki-ca                                    u,u,u
ocspSigningCert cert-pki-ca                                  u,u,u
caSigningCert cert-pki-ca                                    CTu,Cu,Cu
Server-Cert cert-pki-ca                                      u,u,u
auditSigningCert cert-pki-ca                                 u,u,Pu

Certmonger tracking shows correct now with the Subject having the CN and O in the correct order.

[Travis West]

In the apache error log I found this that is generated when, in the UI, I try to access Authentication > Certificates  > Certificate Authorities.

[Wed Apr 03 16:33:28.439180 2024] [:error] [pid 19048] ipa: INFO: [jsonserver_session] twest@IPA.****.NET: cert_find(None, version=u'2.230'): SUCCESS
[Wed Apr 03 16:33:30.661528 2024] [:warn] [pid 19601] [client IP.ADD.RE.SS:61691] failed to set perms (3140) on file (/var/run/ipa/ccaches/twest@IPA.****.NET)!, referer: https://ipa1-sea2.ipa.****.net/ipa/ui/
[Wed Apr 03 16:33:30.720054 2024] [:error] [pid 19047] ipa: INFO: [jsonserver_session] twest@IPA.****.NET: ca_find(u'', sizelimit=0, version=u'2.230', pkey_only=True): SUCCESS
[Wed Apr 03 16:33:30.731584 2024] [:warn] [pid 19601] [client IP.ADD.RE.SS:61691] failed to set perms (3140) on file (/var/run/ipa/ccaches/twest@IPA.****.NET)!, referer: https://ipa1-sea2.ipa.****.net/ipa/ui/
[Wed Apr 03 16:33:30.831428 2024] [:error] [pid 19055] Bad remote server certificate: -8179
[Wed Apr 03 16:33:30.831479 2024] [:error] [pid 19055] SSL Library Error: -8179 Certificate is signed by an unknown issuer
[Wed Apr 03 16:33:30.831557 2024] [:error] [pid 19055] Re-negotiation handshake failed: Not accepted by client!?
[Wed Apr 03 16:33:30.831672 2024] [:error] [pid 19055] SSL Library Error: -12116 Unknown
[Wed Apr 03 16:33:30.832809 2024] [:error] [pid 19048] ipa: INFO: twest@IPA.****.NET: batch: ca_show(u'ipa'): NetworkError
[Wed Apr 03 16:33:30.833300 2024] [:error] [pid 19048] ipa: INFO: [jsonserver_session] twest@IPA.****.NET: batch(({u'params': ([u'ipa'], {}), u'method': u'ca_show'},), version=u'2.230'): SUCCESS

but no indication of which certificate it is complaining about.  I thought maybe the IPA RA cert, but that is definitely signed by this CA and doesn't expires on 2026.
The certs I generated and imported to /etc/pki/pki-tomcat/alias are also signed by the CA.

[Rob Crittenden]

Travis West via FreeIPA-users wrote:
> In the apache error log I found this that is generated when, in the UI, I try to access Authentication > Certificates  > Certificate Authorities.
>
> [Wed Apr 03 16:33:28.439180 2024] [:error] [pid 19048] ipa: INFO: [jsonserver_session] twest@IPA.****.NET: cert_find(None, version=u'2.230'): SUCCESS
> [Wed Apr 03 16:33:30.661528 2024] [:warn] [pid 19601] [client IP.ADD.RE.SS:61691] failed to set perms (3140) on file (/var/run/ipa/ccaches/twest@IPA.****.NET)!, referer: https://ipa1-sea2.ipa.****.net/ipa/ui/
> [Wed Apr 03 16:33:30.720054 2024] [:error] [pid 19047] ipa: INFO: [jsonserver_session] twest@IPA.****.NET: ca_find(u'', sizelimit=0, version=u'2.230', pkey_only=True): SUCCESS
> [Wed Apr 03 16:33:30.731584 2024] [:warn] [pid 19601] [client IP.ADD.RE.SS:61691] failed to set perms (3140) on file (/var/run/ipa/ccaches/twest@IPA.****.NET)!, referer: https://ipa1-sea2.ipa.****.net/ipa/ui/
> [Wed Apr 03 16:33:30.831428 2024] [:error] [pid 19055] Bad remote server certificate: -8179
> [Wed Apr 03 16:33:30.831479 2024] [:error] [pid 19055] SSL Library Error: -8179 Certificate is signed by an unknown issuer
> [Wed Apr 03 16:33:30.831557 2024] [:error] [pid 19055] Re-negotiation handshake failed: Not accepted by client!?
> [Wed Apr 03 16:33:30.831672 2024] [:error] [pid 19055] SSL Library Error: -12116 Unknown
> [Wed Apr 03 16:33:30.832809 2024] [:error] [pid 19048] ipa: INFO: twest@IPA.****.NET: batch: ca_show(u'ipa'): NetworkError
> [Wed Apr 03 16:33:30.833300 2024] [:error] [pid 19048] ipa: INFO: [jsonserver_session] twest@IPA.****.NET: batch(({u'params': ([u'ipa'], {}), u'method': u'ca_show'},), version=u'2.230'): SUCCESS
>
> but no indication of which certificate it is complaining about.  I thought maybe the IPA RA cert, but that is definitely signed by this CA and doesn't expires on 2026.
> The certs I generated and imported to /etc/pki/pki-tomcat/alias are also signed by the CA.

Apache, via the IPA API, is acting as the client in this case. So Apache
doesn't trust the CA certificate (unlikely), or the Server-Cert cert-pki-ca.

You can validate it directly with:

# certutil -V -u V -d /etc/pki/pki-tomcat/alias -n 'Server-Cert
cert-pki-ca' -e -f /etc/pki/pki-tomcat/alias/pwdfile.txt

Also, given the subject issues you ran into I guess I'd also verify that
the ASN.1 is correct in the issued certificates. This will be easier
since you have them as PEM files already:

# openssl asn1parse -inform pem -in /path/to/cert.pem

In the output you should see each component of the issuer and subject
broken out like:

...
   37:d=5  hl=2 l=   3 prim: OBJECT            :organizationName
   42:d=5  hl=2 l=  12 prim: UTF8STRING        :EXAMPLE.TEST
   56:d=3  hl=2 l=  30 cons: SET
   58:d=4  hl=2 l=  28 cons: SEQUENCE
   60:d=5  hl=2 l=   3 prim: OBJECT            :commonName
   65:d=5  hl=2 l=  21 prim: UTF8STRING        :Certificate Authority
   88:d=2  hl=2 l=  30 cons: SEQUENCE
   90:d=3  hl=2 l=  13 prim: UTCTIME           :240221205457Z
  105:d=3  hl=2 l=  13 prim: UTCTIME           :260221205457Z
  120:d=2  hl=2 l=  50 cons: SEQUENCE
  122:d=3  hl=2 l=  21 cons: SET
  124:d=4  hl=2 l=  19 cons: SEQUENCE
  126:d=5  hl=2 l=   3 prim: OBJECT            :organizationName
  131:d=5  hl=2 l=  12 prim: UTF8STRING        :EXAMPLE.TEST
  145:d=3  hl=2 l=  25 cons: SET
  147:d=4  hl=2 l=  23 cons: SEQUENCE
  149:d=5  hl=2 l=   3 prim: OBJECT            :commonName
  154:d=5  hl=2 l=  16 prim: UTF8STRING        :ipa.example.test
...

And finally, and this might be kinda nutty, but you can use certmonger
to force issue a new certificate using the resubmit command. I'd
snapshot things but that could be a way to get freshly issued certs that
might play more nicely with others.

rob

[Travis West]

Here is the output of validation

# certutil -V -u V -d /etc/pki/pki-tomcat/alias -n 'Server-Cert cert-pki-ca' -e -f /etc/pki/pki-tomcat/alias/pwdfile.txt
certutil: certificate is valid

And for the asn.1 of the Audit, OCSP, Subsystem, and RA certs

$ openssl asn1parse -inform pem -in audit.crt
   37:d=5  hl=2 l=   3 prim: OBJECT            :organizationName
   42:d=5  hl=2 l=  14 prim: UTF8STRING        :IPA.****.NET
   58:d=3  hl=2 l=  30 cons: SET
   60:d=4  hl=2 l=  28 cons: SEQUENCE
   62:d=5  hl=2 l=   3 prim: OBJECT            :commonName
   67:d=5  hl=2 l=  21 prim: UTF8STRING        :Certificate Authority
   90:d=2  hl=2 l=  30 cons: SEQUENCE
   92:d=3  hl=2 l=  13 prim: UTCTIME           :240403113826Z
  107:d=3  hl=2 l=  13 prim: UTCTIME           :340401113826Z
  122:d=2  hl=2 l=  44 cons: SEQUENCE
  124:d=3  hl=2 l=  23 cons: SET
  126:d=4  hl=2 l=  21 cons: SEQUENCE
  128:d=5  hl=2 l=   3 prim: OBJECT            :organizationName
  133:d=5  hl=2 l=  14 prim: UTF8STRING        :IPA.****.NET
  149:d=3  hl=2 l=  17 cons: SET
  151:d=4  hl=2 l=  15 cons: SEQUENCE
  153:d=5  hl=2 l=   3 prim: OBJECT            :commonName
  158:d=5  hl=2 l=   8 prim: UTF8STRING        :CA Audit

$ openssl asn1parse -inform pem -in subsystem.crt
    37:d=5  hl=2 l=   3 prim: OBJECT            :organizationName
   42:d=5  hl=2 l=  14 prim: UTF8STRING        :IPA.****.NET
   58:d=3  hl=2 l=  30 cons: SET
   60:d=4  hl=2 l=  28 cons: SEQUENCE
   62:d=5  hl=2 l=   3 prim: OBJECT            :commonName
   67:d=5  hl=2 l=  21 prim: UTF8STRING        :Certificate Authority
   90:d=2  hl=2 l=  30 cons: SEQUENCE
   92:d=3  hl=2 l=  13 prim: UTCTIME           :240403113547Z
  107:d=3  hl=2 l=  13 prim: UTCTIME           :340401113547Z
  122:d=2  hl=2 l=  48 cons: SEQUENCE
  124:d=3  hl=2 l=  23 cons: SET
  126:d=4  hl=2 l=  21 cons: SEQUENCE
  128:d=5  hl=2 l=   3 prim: OBJECT            :organizationName
  133:d=5  hl=2 l=  14 prim: UTF8STRING        :IPA.****.NET
  149:d=3  hl=2 l=  21 cons: SET
  151:d=4  hl=2 l=  19 cons: SEQUENCE
  153:d=5  hl=2 l=   3 prim: OBJECT            :commonName
  158:d=5  hl=2 l=  12 prim: UTF8STRING        :CA Subsystem

$ openssl asn1parse -inform pem -in ocsp.crt
   37:d=5  hl=2 l=   3 prim: OBJECT            :organizationName
   42:d=5  hl=2 l=  14 prim: UTF8STRING        :IPA.****.NET
   58:d=3  hl=2 l=  30 cons: SET
   60:d=4  hl=2 l=  28 cons: SEQUENCE
   62:d=5  hl=2 l=   3 prim: OBJECT            :commonName
   67:d=5  hl=2 l=  21 prim: UTF8STRING        :Certificate Authority
   90:d=2  hl=2 l=  30 cons: SEQUENCE
   92:d=3  hl=2 l=  13 prim: UTCTIME           :240403113248Z
  107:d=3  hl=2 l=  13 prim: UTCTIME           :340401113248Z
  122:d=2  hl=2 l=  50 cons: SEQUENCE
  124:d=3  hl=2 l=  23 cons: SET
  126:d=4  hl=2 l=  21 cons: SEQUENCE
  128:d=5  hl=2 l=   3 prim: OBJECT            :organizationName
  133:d=5  hl=2 l=  14 prim: UTF8STRING        :IPA.****.NET
  149:d=3  hl=2 l=  23 cons: SET
  151:d=4  hl=2 l=  21 cons: SEQUENCE
  153:d=5  hl=2 l=   3 prim: OBJECT            :commonName
  158:d=5  hl=2 l=  14 prim: UTF8STRING        :OCSP Subsystem

$ openssl asn1parse -inform pem -in ra-agent.pem
   37:d=5  hl=2 l=   3 prim: OBJECT            :organizationName
   42:d=5  hl=2 l=  14 prim: UTF8STRING        :IPA.****.NET
   58:d=3  hl=2 l=  30 cons: SET
   60:d=4  hl=2 l=  28 cons: SEQUENCE
   62:d=5  hl=2 l=   3 prim: OBJECT            :commonName
   67:d=5  hl=2 l=  21 prim: UTF8STRING        :Certificate Authority
   90:d=2  hl=2 l=  30 cons: SEQUENCE
   92:d=3  hl=2 l=  13 prim: UTCTIME           :240322132444Z
  107:d=3  hl=2 l=  13 prim: UTCTIME           :260312132444Z
  122:d=2  hl=2 l=  42 cons: SEQUENCE
  124:d=3  hl=2 l=  23 cons: SET
  126:d=4  hl=2 l=  21 cons: SEQUENCE
  128:d=5  hl=2 l=   3 prim: OBJECT            :organizationName
  133:d=5  hl=2 l=  14 prim: UTF8STRING        :IPA.****.NET
  149:d=3  hl=2 l=  15 cons: SET
  151:d=4  hl=2 l=  13 cons: SEQUENCE
  153:d=5  hl=2 l=   3 prim: OBJECT            :commonName
  158:d=5  hl=2 l=   6 prim: PRINTABLESTRING   :IPA RA

I tried a resubmit on the ra-agent cert with getcert and this was the result

Request ID '20190322032004':
        status: CA_UNREACHABLE
        ca-error: Error 35 connecting to https://ipa1-sea2.ipa.****.net:8443/ca/agent/ca/profileReview: SSL connect error.
        stuck: no

[Travis West]

This morning I tried running ipa-server-upgrade to see if that would help.  It ultimately failed, but in a different spot and with a different error:

2024-04-04T11:36:42Z DEBUG The CA status is: running
2024-04-04T11:36:42Z INFO [Ensuring CA is using LDAPProfileSubsystem]
2024-04-04T11:36:42Z INFO [Migrating certificate profiles to LDAP]
2024-04-04T11:36:42Z DEBUG Created connection context.ldap2_140461768893264
2024-04-04T11:36:42Z DEBUG flushing ldapi://%2fvar%2frun%2fslapd-IPA-****-NET.socket from SchemaCache
2024-04-04T11:36:42Z DEBUG retrieving schema for SchemaCache url=ldapi://%2fvar%2frun%2fslapd-IPA-****-NET.socket conn=<ldap.ldapobject.SimpleLDAPObject instance at 0x7fbfcdd14098>
2024-04-04T11:36:42Z DEBUG Destroyed connection context.ldap2_140461768893264
2024-04-04T11:36:42Z DEBUG request GET https://ipa1-sea2.ipa.****.net:8443/ca/rest/account/login
2024-04-04T11:36:42Z DEBUG request body ''
2024-04-04T11:36:42Z DEBUG httplib request failed:
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/ipapython/dogtag.py", line 220, in _httplib_request
    conn.request(method, uri, body=request_body, headers=headers)
  File "/usr/lib64/python2.7/httplib.py", line 1041, in request
    self._send_request(method, url, body, headers)
  File "/usr/lib64/python2.7/httplib.py", line 1075, in _send_request
    self.endheaders(body)
  File "/usr/lib64/python2.7/httplib.py", line 1037, in endheaders
    self._send_output(message_body)
  File "/usr/lib64/python2.7/httplib.py", line 881, in _send_output
    self.send(msg)
  File "/usr/lib64/python2.7/httplib.py", line 843, in send
    self.connect()
  File "/usr/lib64/python2.7/httplib.py", line 1260, in connect
    server_hostname=sni_hostname)
  File "/usr/lib64/python2.7/ssl.py", line 348, in wrap_socket
    _context=self)
  File "/usr/lib64/python2.7/ssl.py", line 609, in __init__
    self.do_handshake()
  File "/usr/lib64/python2.7/ssl.py", line 831, in do_handshake
    self._sslobj.do_handshake()
SSLError: [SSL: TLSV1_ALERT_UNKNOWN_CA] tlsv1 alert unknown ca (_ssl.c:618)
2024-04-04T11:36:42Z ERROR IPA server upgrade failed: Inspect /var/log/ipaupgrade.log and run command ipa-server-upgrade manually.
2024-04-04T11:36:42Z DEBUG   File "/usr/lib/python2.7/site-packages/ipapython/admintool.py", line 178, in execute
    return_value = self.run()
  File "/usr/lib/python2.7/site-packages/ipaserver/install/ipa_server_upgrade.py", line 54, in run
    server.upgrade()
  File "/usr/lib/python2.7/site-packages/ipaserver/install/server/upgrade.py", line 2085, in upgrade
    upgrade_configuration()
  File "/usr/lib/python2.7/site-packages/ipaserver/install/server/upgrade.py", line 1952, in upgrade_configuration
    ca_enable_ldap_profile_subsystem(ca)
  File "/usr/lib/python2.7/site-packages/ipaserver/install/server/upgrade.py", line 396, in ca_enable_ldap_profile_subsystem
    cainstance.migrate_profiles_to_ldap()
  File "/usr/lib/python2.7/site-packages/ipaserver/install/cainstance.py", line 1814, in migrate_profiles_to_ldap
    _create_dogtag_profile(profile_id, profile_data, overwrite=False)
  File "/usr/lib/python2.7/site-packages/ipaserver/install/cainstance.py", line 1820, in _create_dogtag_profile
    with api.Backend.ra_certprofile as profile_api:
  File "/usr/lib/python2.7/site-packages/ipaserver/plugins/dogtag.py", line 1298, in __enter__
    method='GET'
  File "/usr/lib/python2.7/site-packages/ipapython/dogtag.py", line 167, in https_request
    method=method, headers=headers)
  File "/usr/lib/python2.7/site-packages/ipapython/dogtag.py", line 229, in _httplib_request
    raise NetworkError(uri=uri, error=str(e))

2024-04-04T11:36:42Z DEBUG The ipa-server-upgrade command failed, exception: NetworkError: cannot connect to 'https://ipa1-sea2.ipa.****.net:8443/ca/rest/account/login': [SSL: TLSV1_ALERT_UNKNOWN_CA] tlsv1 alert unknown ca (_ssl.c:618)
2024-04-04T11:36:42Z ERROR Unexpected error - see /var/log/ipaupgrade.log for details:

Again with the 'unknown ca' message.  I've confirmed that the ca.crt is the same that is listed as the caSigngingCert in /etc/pki/pki-tomcat/alias and is the one found at /etc/ipa/ca.crt.
I believe my output of asn.1 for each certificate also shows all the certificates signed by the CA, so I'm not sure what certificate it's complaining about coming from an unknown CA.

[Travis West]

Rob,

I installed the ipa-healthcheck that you got to work on CentOS 7, and run it.  Got a couple of errors regarding the RA Agent cert:

[
  {
    "source": "ipahealthcheck.ipa.certs",
    "kw": {
      "msg": "Certificate validation for /var/lib/ipa/ra-agent.pem failed: ",
      "reason": "",
      "key": "/var/lib/ipa/ra-agent.pem"
    },
    "uuid": "a855346c-4998-4415-a819-ce83048e174e",
    "duration": "0.100214",
    "when": "20240404141916Z",
    "check": "IPAOpenSSLChainValidation",
    "result": "ERROR"
  },
  {
    "source": "ipahealthcheck.ipa.certs",
    "kw": {
      "msg": "RA agent not found in LDAP"
    },
    "uuid": "b6efdb6c-ca33-4421-bdc5-c449e7d64591",
    "duration": "0.027569",
    "when": "20240404141916Z",
    "check": "IPARAAgent",
    "result": "ERROR"
  }

That first error, I'm not sure about what kind of validation it's performing.  In my asn.1 output earlier I did include the ra-agent.pem and it looks like it's correctly signed.
As far as the "RA agent not found in LDAP", it looks to me like it is, and it matches the cert in /var/lib/ipa/ra-agent.pem

# ldapsearch -D "cn=directory manager" -W -b uid=ipara,ou=people,o=ipaca
Enter LDAP Password:
# extended LDIF
#
# LDAPv3
# base <uid=ipara,ou=people,o=ipaca> with scope subtree
# filter: (objectclass=*)
# requesting: ALL
#

# ipara, people, ipaca
dn: uid=ipara,ou=people,o=ipaca
description: 2;7;CN=Certificate Authority,O=IPA.****.NET;CN=IPA RA,O=IPA.****.NET
userCertificate:: MIID6j...ssifAg==
uid: ipara
sn: ipara
usertype: agentType
userstate: 1
objectClass: cmsuser
objectClass: top
objectClass: organizationalPerson
objectClass: inetOrgPerson
objectClass: person
cn: ipara

# search result
search: 2
result: 0 Success

# numResponses: 2
# numEntries: 1

# cat ra-agent.pem
-----BEGIN CERTIFICATE-----
MIID6j...ssifAg==
-----END CERTIFICATE-----

[Rob Crittenden]

Travis West via FreeIPA-users wrote:
> Rob,
>
> I installed the ipa-healthcheck that you got to work on CentOS 7, and run it.  Got a couple of errors regarding the RA Agent cert:
>
> [
>   {
>     "source": "ipahealthcheck.ipa.certs",
>     "kw": {
>       "msg": "Certificate validation for /var/lib/ipa/ra-agent.pem failed: ",
>       "reason": "",
>       "key": "/var/lib/ipa/ra-agent.pem"
>     },
>     "uuid": "a855346c-4998-4415-a819-ce83048e174e",
>     "duration": "0.100214",
>     "when": "20240404141916Z",
>     "check": "IPAOpenSSLChainValidation",
>     "result": "ERROR"
>   },
>   {
>     "source": "ipahealthcheck.ipa.certs",
>     "kw": {
>       "msg": "RA agent not found in LDAP"
>     },
>     "uuid": "b6efdb6c-ca33-4421-bdc5-c449e7d64591",
>     "duration": "0.027569",
>     "when": "20240404141916Z",
>     "check": "IPARAAgent",
>     "result": "ERROR"
>   }

It runs: openssl verify -verbose -show_chain -CAfile /etc/ipa/ca.crt
/var/lib/ipa/ra-agent.pem
> That first error, I'm not sure about what kind of validation it's performing.  In my asn.1 output earlier I did include the ra-agent.pem and it looks like it's correctly signed.
> As far as the "RA agent not found in LDAP", it looks to me like it is, and it matches the cert in /var/lib/ipa/ra-agent.pem
>
> # ldapsearch -D "cn=directory manager" -W -b uid=ipara,ou=people,o=ipaca
> Enter LDAP Password:
> # extended LDIF
> #
> # LDAPv3
> # base <uid=ipara,ou=people,o=ipaca> with scope subtree
> # filter: (objectclass=*)
> # requesting: ALL
> #
>
> # ipara, people, ipaca
> dn: uid=ipara,ou=people,o=ipaca
> description: 2;7;CN=Certificate Authority,O=IPA.****.NET;CN=IPA RA,O=IPA.****.NET
> userCertificate:: MIID6j...ssifAg==
> uid: ipara
> sn: ipara
> usertype: agentType
> userstate: 1
> objectClass: cmsuser
> objectClass: top
> objectClass: organizationalPerson
> objectClass: inetOrgPerson
> objectClass: person
> cn: ipara
>
> # search result
> search: 2
> result: 0 Success
>
> # numResponses: 2
> # numEntries: 1
>
> # cat ra-agent.pem
> -----BEGIN CERTIFICATE-----
> MIID6j...ssifAg==
> -----END CERTIFICATE-----

Watch the 389-ds access log (buffer) while healthcheck runs. You should
see the failed search and the reason may be enlightening (or not).

You can also add --debug to the command and may be that will help.

rob

[Travis West]

If I run that command manually it doesn't appear to do anything except output 'recognized usages"
If I try it without the -show_chain flag I get

# openssl verify -verbose -CAfile /etc/ipa/ca.crt /var/lib/ipa/ra-agent.pem
/var/lib/ipa/ra-agent.pem: O = IPA.****.NET, CN = IPA RA
error 20 at 0 depth lookup:unable to get local issuer certificate

The only information in the access log while healthcheck is running is a number of these

[04/Apr/2024:15:09:46 +0000] "POST https://ipa1-sea2.ipa.****.net:443/ca/agent/ca/displayBySerial HTTP/1.1" 403 229

But those coincide with the healthcheck checking other certificates managed by certmonger where the error shown by healthcheck is
 [SSL: SSL_HANDSHAKE_FAILURE] ssl handshake failure (_ssl.c:1822)",

[Travis West]

I spun up a new server and did a fresh install of IPA.  On that server if I run the command I get a better result

# openssl verify -verbose -show_chain -CAfile /etc/ipa/ca.crt /var/lib/ipa/ra-agent.pem	
/var/lib/ipa/ra-agent.pem: OK
Chain:
depth=0: O = AUTH.****.NET, CN = IPA RA (untrusted)
depth=1: O = AUTH.****.NET, CN = Certificate Authority

So I must be missing something with the RA cert.  It's definitely in LDAP.  I've read that it should also be present in /etc/httpd/alias/ NSS DB, but that directory is empty on the fresh install so I cannot confirm.
The ASN.1 appears to be correct on the ra-agent.pem when I check

$ openssl asn1parse -inform pem -in ra-agent.pem
37:d=5 hl=2 l= 3 prim: OBJECT :organizationName
42:d=5 hl=2 l= 14 prim: UTF8STRING :IPA.****.NET
58:d=3 hl=2 l= 30 cons: SET
60:d=4 hl=2 l= 28 cons: SEQUENCE
62:d=5 hl=2 l= 3 prim: OBJECT :commonName
67:d=5 hl=2 l= 21 prim: UTF8STRING :Certificate Authority
90:d=2 hl=2 l= 30 cons: SEQUENCE
92:d=3 hl=2 l= 13 prim: UTCTIME :240322132444Z
107:d=3 hl=2 l= 13 prim: UTCTIME :260312132444Z
122:d=2 hl=2 l= 42 cons: SEQUENCE
124:d=3 hl=2 l= 23 cons: SET
126:d=4 hl=2 l= 21 cons: SEQUENCE
128:d=5 hl=2 l= 3 prim: OBJECT :organizationName
133:d=5 hl=2 l= 14 prim: UTF8STRING :IPA.****.NET
149:d=3 hl=2 l= 15 cons: SET
151:d=4 hl=2 l= 13 cons: SEQUENCE
153:d=5 hl=2 l= 3 prim: OBJECT :commonName
158:d=5 hl=2 l= 6 prim: PRINTABLESTRING :IPA RA

This was another cert that had an incorrect Principle attached and was regenerated.  I may have messed up something there, but I'm not sure what.
I do have a copy of the ra-agent.pem (and matching key) with the correct Principle from 2019.  I can put this in place on the broken server, but even with rolling the time back I'm not sure it will get renewed.

[Travis West]

This morning I thought I had found what I was missing, import the new RA cert to ~/.dogtag/nssdb, which I've done and now all the places I know about the RA cert matches.

# certutil -L -d /root/.dogtag/nssdb

Certificate Nickname                                         Trust Attributes
                                                             SSL,S/MIME,JAR/XPI

Certificate Authority - IPA.****.NET                       CT,C,C
IPA RA - IPA.****.NET                                      u,u,u

# certutil -L -d /root/.dogtag/nssdb -n "IPA RA - IPA.****.NET" -a
-----BEGIN CERTIFICATE-----
MIID6jCC...ssifAg==
-----END CERTIFICATE-----

# certutil -L -d /root/.dogtag/nssdb -n "IPA RA - IPA.****.NET" | grep Serial
        Serial Number: 7 (0x7)

# ldapsearch -D "cn=directory manager" -W -b uid=ipara,ou=people,o=ipaca
Enter LDAP Password:
# extended LDIF
#
# LDAPv3
# base <uid=ipara,ou=people,o=ipaca> with scope subtree
# filter: (objectclass=*)
# requesting: ALL
#

# ipara, people, ipaca
dn: uid=ipara,ou=people,o=ipaca
description: 2;7;CN=Certificate Authority,O=IPA.****.NET;CN=IPA RA,O=IPA.****.NET
userCertificate:: MIID6jCC...ssifAg==
uid: ipara
sn: ipara
usertype: agentType
userstate: 1
objectClass: cmsuser
objectClass: top
objectClass: organizationalPerson
objectClass: inetOrgPerson
objectClass: person
cn: ipara

# search result
search: 2
result: 0 Success

# numResponses: 2
# numEntries: 1

# cat /var/lib/ipa/ra-agent.pem
-----BEGIN CERTIFICATE-----
MIID6jCC...ssifAg==
-----END CERTIFICATE-----

but the openssl verify command with the -show_chain flag still seems to fail

]# openssl verify -verbose -show_chain -CAfile /etc/ipa/ca.crt /var/lib/ipa/ra-agent.pem
usage: verify [-verbose] [-CApath path] [-CAfile file] [-trusted_first] [-purpose purpose] [-crl_check] [-no_alt_chains] [-attime timestamp] [-engine e] cert1 cert2 ...
recognized usages:
        sslclient       SSL client
        sslserver       SSL server
        nssslserver     Netscape SSL server
        smimesign       S/MIME signing
        smimeencrypt    S/MIME encryption
        crlsign         CRL signing
        any             Any Purpose
        ocsphelper      OCSP helper
        timestampsign   Time Stamp signing

[Travis West]

The problem was definitely the ra-agent.pem.  I generated a new one and imported it to ~/.dogtag/nssdb, LDAP and placed the pem and key in /var/lib/ipa/

Now I can verify the certificate with the openssl verify command.  Additionally the error in the UI is gone and running an 'ipa cert-show 1' works and doesn't return the error I was seeing.

The last piece here is replicating the new certificates to other 5 hosts in the cluster.  Is there a method to do that or should I import the new certs manually on the other hosts?

[Rob Crittenden]

Travis West via FreeIPA-users wrote:
> The problem was definitely the ra-agent.pem.  I generated a new one and imported it to ~/.dogtag/nssdb, LDAP and placed the pem and key in /var/lib/ipa/
>
> Now I can verify the certificate with the openssl verify command.  Additionally the error in the UI is gone and running an 'ipa cert-show 1' works and doesn't return the error I was seeing.
>
> The last piece here is replicating the new certificates to other 5 hosts in the cluster.  Is there a method to do that or should I import the new certs manually on the other hosts?

If you put the certificates into
cn=<nickname>,cn=ca_renewal,cn=ipa,cn=etc,$SUFFIX then the other servers
will pick them up assuming that replication is working.

rob

[Travis West]

Thanks Rob!  New certs are all replicated and all IPA services are started on all 6 servers.
I can perform 'ipa cert-show 1' on all 6 and get the expected result.

As a sanity check I did run the ipa-healthcheck on all 6 servers.  One of them came back fine, the other 5 returned

[
  {
    "source": "ipahealthcheck.ipa.dna",
    "kw": {
      "msg": "No DNA range defined. If no masters define a range then users and groups cannot be created.",
      "range_start": 0,
      "next_start": 0,
      "next_max": 0,
      "range_max": 0
    },
    "uuid": "70636197-0b3e-4424-b509-1aa7f8be084d",
    "duration": "0.706384",
    "when": "20240405170045Z",
    "check": "IPADNARangeCheck",
    "result": "WARNING"
  }
]

Now it's just a WARNING, and since the one didn't return it (they're all denoted as MASTER) maybe it's okay?

[Rob Crittenden]

Travis West via FreeIPA-users wrote:
> Thanks Rob!  New certs are all replicated and all IPA services are started on all 6 servers.
> I can perform 'ipa cert-show 1' on all 6 and get the expected result.
>
> As a sanity check I did run the ipa-healthcheck on all 6 servers.  One of them came back fine, the other 5 returned
>
> [
>   {
>     "source": "ipahealthcheck.ipa.dna",
>     "kw": {
>       "msg": "No DNA range defined. If no masters define a range then users and groups cannot be created.",
>       "range_start": 0,
>       "next_start": 0,
>       "next_max": 0,
>       "range_max": 0
>     },
>     "uuid": "70636197-0b3e-4424-b509-1aa7f8be084d",
>     "duration": "0.706384",
>     "when": "20240405170045Z",
>     "check": "IPADNARangeCheck",
>     "result": "WARNING"
>   }
> ]
>
> Now it's just a WARNING, and since the one didn't return it (they're all denoted as MASTER) maybe it's okay?

It just means that when you add users or groups you do it against the
same IPA server. If you do it on others then it will split the range
between them as needed. Not a bad thing but it gets complex if you add
and remove a lot of servers, particularly older ones. I made changes a
few years ago to try to capture ranges that would otherwise be lost but
it's sort of a best effort kind of thing.

The purpose if this is to ensure that at least one server has a range.
Currently healthcheck only validates the server it is running on and
doesn't do much cluster-wide checking.

rob

[Travis West]

Alright, so 'ipa idrange-find' returns the same values on all 6 servers.

However, ldapsearch -x -D 'cn=Directory Manager' -W -b 'cn=Posix IDs,cn=Distributed Numeric Assignment Plugin,cn=plugins,cn=config'

returns different results on 1 (the one where I don't get that warning with the healthcheck)  The other 5 return

dnaMagicRegen: -1
dnaMaxValue: 1100
dnaNextValue: 1101
dnaScope: dc=ipa,dc=superb,dc=net
dnaSharedCfgDN: cn=posix-ids,cn=dna,cn=ipa,cn=etc,dc=ipa,dc=superb,dc=net
dnaThreshold: 500
dnaType: uidNumber
dnaType: gidNumber
objectClass: top
objectClass: extensibleObject

Which seems to match your blog post from 2015 about this.

Since I cannot be sure which IPA server will be used when enrolling new hosts, would it be best to try to fix this?  I suppose the same can be said for when new users are added.   If done manually I can be sure it will be done on the same host, but we have an internal system that also creates the user in IPA and I think that would just use whichever one is closest.

[Travis West]

I tried adding a test user on one of the servers that returned that warning and the new user didn't appear on the others.
So maybe replication is broken.

[Travis West]

I take that back.  Replication is working on 4/6 servers.

If I add a user on any of those 4 it shows up on the other 3.  The 2 outliers don't seem to pick up the new user.  If I check the 2 outliers I get this

Error (18) Replication error acquiring replica: Incremental update transient error.  Backing off, will retry update later. (transient error)

Which seems to be saying that it's just delayed, which can sometimes happen in an MMR setup.  I will recheck these 2 later to see if they eventually pick up the new test user I've created and that is present on 4 of them.