Howto renew certificates with external CA?

[Harald Husemann]

Hello IPA-experts,

we are running FreeIPA version 4.4.0 with an external CA (our own one), 
everything was working fine until the CA certificate expired which 
happened at January 13th.
Since i was on vacation and the basic functions were still available 
no-one created a new certificate, so, it's now my task.
As explained in 
https://www.freeipa.org/page/Howto/CA_Certificate_Renewal, I've reset 
the time to January 10th, created a new certificate which is valid from 
2017 to 2023, and installed it with ipa-cacert-manage.
Afterwards, I did an ipa-certupdate, the server certificates were 
updated and the cert8.db in /etc/httpd/alias contains the new valid CA. 
But, the expiration date of the certificate itself is still January 
13th, so, the certificate is still expired:

root@mat-ipa-master-1:~$ /usr/bin/certutil -d /etc/httpd/alias -L -n 
"MATERNA-COM.DE IPA CA"
Certificate:
     Data:
         Version: 3 (0x2)
         Serial Number: 36 (0x24)
         Signature Algorithm: PKCS #1 SHA-256 With RSA Encryption
         Issuer: "E=[e-mail],CN=Materna OC CA,OU=OC RZ,O=Materna 
GmbH,
             L=Dortmund,ST=NRW,C=DE"
         Validity:
             Not Before: Mon Jan 23 14:45:00 2017
             Not After : Mon Jan 23 14:45:00 2023
         Subject: "CN=Certificate Authority,O=MATERNA-COM.DE"
        (...)
     Certificate Trust Flags:
         SSL Flags:
             Valid CA
             Trusted CA
             Trusted Client CA
         Email Flags:
             Valid CA
             Trusted CA
         Object Signing Flags:
             Valid CA
             Trusted CA

Certificate:
     Data:
         Version: 3 (0x2)
         Serial Number: 23 (0x17)
         Signature Algorithm: PKCS #1 SHA-256 With RSA Encryption
         Issuer: "E=[e-mail],CN=Materna OC CA,OU=OC RZ,O=Materna 
GmbH,
             L=Dortmund,ST=NRW,C=DE"
         Validity:
             Not Before: Fri Jan 13 14:45:00 2017
             Not After : Sat Jan 13 14:45:00 2018
         Subject: "CN=Certificate Authority,O=MATERNA-COM.DE"
     (...)

root@mat-ipa-master-1:~$

I have only checked this one, but I'd suppose the others are also not 
updated. AFAIK certmonger is responsible the renewal, so, I've restarted 
it and hoped it would grab my certificate and renew it - but it seems 
there is a problem, journalctl -u certmonger gives

Jan 24 11:22:43 mat-ipa-master-1.materna-com.de systemd[1]: Starting 
Certificate monitoring and PKI enrollment...
Jan 24 11:22:44 mat-ipa-master-1.materna-com.de systemd[1]: Started 
Certificate monitoring and PKI enrollment.
Jan 24 11:22:48 mat-ipa-master-1.materna-com.de certmonger[1026]: 
2018-01-24 11:22:48 [1026] Error setting up ccache for "host" service on 
client using default keytab: Cannot contact any KDC for realm 
'MATERNA-COM.DE'.
Jan 24 11:22:48 mat-ipa-master-1.materna-com.de certmonger[1026]: 
2018-01-24 11:22:48 [1026] Error setting up ccache for "host" service on 
client using default keytab: Cannot contact any KDC for realm 
'MATERNA-COM.DE'.
Jan 24 11:22:58 mat-ipa-master-1.materna-com.de certmonger[1026]: 
2018-01-24 11:22:58 [1026] Error 7 connecting to 
https://mat-ipa-master-1.materna-com.de:8443/ca/agent/ca/profileReview: 
Couldn't connect to server.
Jan 24 11:23:00 mat-ipa-master-1.materna-com.de 
dogtag-ipa-ca-renew-agent-submit[2282]: Traceback (most recent call last):
File "/usr/libexec/certmonger/dogtag-ipa-ca-renew-agent-submit", line 
511, in <module>
sys.exit(main())
File "/usr/libexec/certmonger/dogtag-ipa-ca-renew-agent-submit", line 
490, in main
ipautil.kinit_keytab(principal, paths.KRB5_KEYTAB, ccache_filename)
File "/usr/lib/python2.7/site-packages/ipapython/ipautil.py", line 1314, 
in kinit_keytab
cred = gssapi.Credentials(name=name, store=store, usage='initiate')
File "/usr/lib64/python2.7/site-packages/gssapi/creds.py", line 64, in 
__new__
store=store)
File "/usr/lib64/python2.7/site-packages/gssapi/creds.py", line 148, in 
acquire
usage)
File "ext_cred_store.pyx", line 182, in 
gssapi.raw.ext_cred_store.acquire_cred_from 
(gssapi/raw/ext_cred_store.c:1732)
GSSError: Major (851968): Unspecified GSS failure.  Minor code may 
provide more information, Minor (2529639068): Cannot contact any KDC for 
realm 'MA
Jan 24 11:23:00 mat-ipa-master-1.materna-com.de certmonger[1026]: 
2018-01-24 11:23:00 [1026] Internal error

Any help is greatly appreciated since I'm stuck here... If it helps, I 
have a clean backup of the IPA master which was written yesterday 
evening, so, I can use this one to "start over" if I've already mixed up 
things.

Thanks and kind regards from Germany,

Harald

[Florence Blanc-Renaud]

On 01/24/2018 12:35 PM, Harald Husemann via FreeIPA-users wrote:
> Hello IPA-experts,
>
> we are running FreeIPA version 4.4.0 with an external CA (our own one), 
> everything was working fine until the CA certificate expired which 
> happened at January 13th.
> Since i was on vacation and the basic functions were still available 
> no-one created a new certificate, so, it's now my task.
> As explained in 
> https://www.freeipa.org/page/Howto/CA_Certificate_Renewal, I've reset 
> the time to January 10th, created a new certificate which is valid from 
> 2017 to 2023, and installed it with ipa-cacert-manage.
> Afterwards, I did an ipa-certupdate, the server certificates were 
> updated and the cert8.db in /etc/httpd/alias contains the new valid CA. 
> But, the expiration date of the certificate itself is still January 
> 13th, so, the certificate is still expired:
>
> root@mat-ipa-master-1:~$ /usr/bin/certutil -d /etc/httpd/alias -L -n 
> "MATERNA-COM.DE IPA CA"
> Certificate:
>      Data:
>          Version: 3 (0x2)
>          Serial Number: 36 (0x24)
>          Signature Algorithm: PKCS #1 SHA-256 With RSA Encryption
>          Issuer: "E=[e-mail],CN=Materna OC CA,OU=OC RZ,O=Materna 
> GmbH,
>              L=Dortmund,ST=NRW,C=DE"
>          Validity:
>              Not Before: Mon Jan 23 14:45:00 2017
>              Not After : Mon Jan 23 14:45:00 2023
>          Subject: "CN=Certificate Authority,O=MATERNA-COM.DE"
>         (...)
>      Certificate Trust Flags:
>          SSL Flags:
>              Valid CA
>              Trusted CA
>              Trusted Client CA
>          Email Flags:
>              Valid CA
>              Trusted CA
>          Object Signing Flags:
>              Valid CA
>              Trusted CA
>
> Certificate:
>      Data:
>          Version: 3 (0x2)
>          Serial Number: 23 (0x17)
>          Signature Algorithm: PKCS #1 SHA-256 With RSA Encryption
>          Issuer: "E=[e-mail],CN=Materna OC CA,OU=OC RZ,O=Materna 
> GmbH,
>              L=Dortmund,ST=NRW,C=DE"
>          Validity:
>              Not Before: Fri Jan 13 14:45:00 2017
>              Not After : Sat Jan 13 14:45:00 2018
>          Subject: "CN=Certificate Authority,O=MATERNA-COM.DE"
>      (...)
>
> root@mat-ipa-master-1:~$

Hi,

in the above output we can see 2 different certificates for 
"CN=Certificate Authority,O=MATERNA-COM.DE", which is an expected 
behavior: the database still contains the old one (Not After: Sat Jan 13 
14:45:00 2018) but also contains the new one (Not After : Mon Jan 23 
14:45:00 2023). So from this point of view, IPA CA cert was properly 
renewed and distributed to the httpd NSS database.
> I have only checked this one, but I'd suppose the others are also not 
> updated. AFAIK certmonger is responsible the renewal, so, I've restarted 
> it and hoped it would grab my certificate and renew it - but it seems 
> there is a problem, journalctl -u certmonger gives
>
> Jan 24 11:22:43 mat-ipa-master-1.materna-com.de systemd[1]: Starting 
> Certificate monitoring and PKI enrollment...
> Jan 24 11:22:44 mat-ipa-master-1.materna-com.de systemd[1]: Started 
> Certificate monitoring and PKI enrollment.
> Jan 24 11:22:48 mat-ipa-master-1.materna-com.de certmonger[1026]: 
> 2018-01-24 11:22:48 [1026] Error setting up ccache for "host" service on 
> client using default keytab: Cannot contact any KDC for realm 
> 'MATERNA-COM.DE'.
> Jan 24 11:22:48 mat-ipa-master-1.materna-com.de certmonger[1026]: 
> 2018-01-24 11:22:48 [1026] Error setting up ccache for "host" service on 
> client using default keytab: Cannot contact any KDC for realm 
> 'MATERNA-COM.DE'.
> Jan 24 11:22:58 mat-ipa-master-1.materna-com.de certmonger[1026]: 
> 2018-01-24 11:22:58 [1026] Error 7 connecting to 
> https://mat-ipa-master-1.materna-com.de:8443/ca/agent/ca/profileReview: 
> Couldn't connect to server.
> Jan 24 11:23:00 mat-ipa-master-1.materna-com.de 
> dogtag-ipa-ca-renew-agent-submit[2282]: Traceback (most recent call last):
> File "/usr/libexec/certmonger/dogtag-ipa-ca-renew-agent-submit", line 
> 511, in <module>
> sys.exit(main())
> File "/usr/libexec/certmonger/dogtag-ipa-ca-renew-agent-submit", line 
> 490, in main
> ipautil.kinit_keytab(principal, paths.KRB5_KEYTAB, ccache_filename)
> File "/usr/lib/python2.7/site-packages/ipapython/ipautil.py", line 1314, 
> in kinit_keytab
> cred = gssapi.Credentials(name=name, store=store, usage='initiate')
> File "/usr/lib64/python2.7/site-packages/gssapi/creds.py", line 64, in 
> __new__
> store=store)
> File "/usr/lib64/python2.7/site-packages/gssapi/creds.py", line 148, in 
> acquire
> usage)
> File "ext_cred_store.pyx", line 182, in 
> gssapi.raw.ext_cred_store.acquire_cred_from 
> (gssapi/raw/ext_cred_store.c:1732)
> GSSError: Major (851968): Unspecified GSS failure.  Minor code may 
> provide more information, Minor (2529639068): Cannot contact any KDC for 
> realm 'MA
> Jan 24 11:23:00 mat-ipa-master-1.materna-com.de certmonger[1026]: 
> 2018-01-24 11:23:00 [1026] Internal error

The traceback is generated by the helper launched to renew IPA CA. This 
helper authenticates using /etc/krb5.keytab but according to the traces, 
was unable to reach the Kerberos server.
Can you manually try to perform
$ sudo kinit -kt /etc/krb5.keytab
and check its output?

Flo
> Any help is greatly appreciated since I'm stuck here... If it helps, I 
> have a clean backup of the IPA master which was written yesterday 
> evening, so, I can use this one to "start over" if I've already mixed up 
> things.
>
> Thanks and kind regards from Germany,
>
> Harald
> _______________________________________________
> FreeIPA-users mailing list -- [e-mail]
> To unsubscribe send an email to [e-mail]

Hello Flo,

thanks for your answer, and for the explanation of the certutil output. I have tried your suggestion, first with sudo:

hhuseman@mat-ipa-master-1:~$ sudo kinit -kt /etc/krb5.keytab
[sudo] password for hhuseman:
Sorry, try again.
[sudo] password for hhuseman:
Sorry, try again.
[sudo] password for hhuseman:
sudo: 2 incorrect password attempts

I'm quite sure my password is correct, so it seems there's something broken here also, since sudo worked before the certificate update. My next try was running the command as root:

hhuseman@mat-ipa-master-1:~$ su -
Password:
root@mat-ipa-master-1:~$ kinit -kt /etc/krb5.keytab
root@mat-ipa-master-1:~$ exit
logout

As you see, there is no output at all, so I tried it again with -V:

root@mat-ipa-master-1:~$ kinit -V -kt /etc/krb5.keytab
Using existing cache: persistent:0:krb_ccache_VPUg94b
Using principal: host/[e-mail]
Using keytab: /etc/krb5.keytab
Authenticated to Kerberos v5
root@mat-ipa-master-1:~$

I have also re-checked the certificate which is issued by the HTTPS-Server in my browser, it is still the old one. 
And, I've tried to get the list of certificates with ipa-getcert:

root@mat-ipa-master-1:~$ ipa-getcert list
Number of certificates and requests being tracked: 5.
Request ID '20170303080146':
        status: CA_UNREACHABLE
        ca-error: Server at https://mat-ipa-master-1.materna-com.de/ipa/xml failed request, will retry: -504 (libcurl failed to execute the HTTP POST transaction, explaining:  Peer's Certificate has expired.).
        stuck: no
        key pair storage: type=NSSDB,location='/etc/dirsrv/slapd-MATERNA-COM-DE',nickname='Server-Cert',token='NSS Certificate DB',pinfile='/etc/dirsrv/slapd-MATERNA-COM-DE/pwdfile.txt'
        certificate: type=NSSDB,location='/etc/dirsrv/slapd-MATERNA-COM-DE',nickname='Server-Cert',token='NSS Certificate DB'
        CA: IPA
        issuer: CN=Certificate Authority,O=MATERNA-COM.DE
        subject: CN=mat-ipa-master-1.materna-com.de,O=MATERNA-COM.DE
        expires: 2018-01-13 14:45:00 UTC
        key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment
        eku: id-kp-serverAuth,id-kp-clientAuth
        pre-save command:
        post-save command: /usr/libexec/ipa/certmonger/restart_dirsrv MATERNA-COM-DE
        track: yes
        auto-renew: yes

Interesting, since the date was still reset to January 11th, so, even the old certificate should be valid:
root@mat-ipa-master-1:~$ date
Thu Jan 11 05:22:21 CET 2018

Nevertheless, I've set the date to actual time by sync'ing it to our NTP-Server:

root@mat-ipa-master-1:~$ ntpdate omcix
24 Jan 19:09:00 ntpdate[32699]: step time server 172.30.96.6 offset 1172766.789568 sec
root@mat-ipa-master-1:~$ date
Wed Jan 24 19:09:06 CET 2018

But, ipa-getcert list is still falling:

root@mat-ipa-master-1:~$ ipa-getcert list
Number of certificates and requests being tracked: 5.
Request ID '20170303080146':
        status: NEED_TO_SUBMIT
        ca-error: Server at https://mat-ipa-master-1.materna-com.de/ipa/xml failed request, will retry: -504 (libcurl failed to execute the HTTP POST transaction, explaining:  Peer's Certificate has expired.).
        stuck: no
        key pair storage: type=NSSDB,location='/etc/dirsrv/slapd-MATERNA-COM-DE',nickname='Server-Cert',token='NSS Certificate DB',pinfile='/etc/dirsrv/slapd-MATERNA-COM-DE/pwdfile.txt'
        certificate: type=NSSDB,location='/etc/dirsrv/slapd-MATERNA-COM-DE',nickname='Server-Cert',token='NSS Certificate DB'
        CA: IPA
        issuer: CN=Certificate Authority,O=MATERNA-COM.DE
        subject: CN=mat-ipa-master-1.materna-com.de,O=MATERNA-COM.DE
        expires: 2018-01-13 14:45:00 UTC
        key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment
        eku: id-kp-serverAuth,id-kp-clientAuth
        pre-save command:
        post-save command: /usr/libexec/ipa/certmonger/restart_dirsrv MATERNA-COM-DE
        track: yes
        auto-renew: yes
root@mat-ipa-master-1:~$

To ensure everything's running I've issued an ipactl:

root@mat-ipa-master-1:~$ ipactl status
Directory Service: RUNNING
krb5kdc Service: RUNNING
kadmin Service: RUNNING
ipa_memcached Service: RUNNING
httpd Service: RUNNING
ipa-custodia Service: RUNNING
pki-tomcatd Service: STOPPED
ipa-otpd Service: RUNNING
ipa: INFO: The ipactl command was successful
root@mat-ipa-master-1:~$

So it seems everything's ok except of the PKI, I've tried to restart it, but it fails:

root@mat-ipa-master-1:~$ ipactl start pki-tomcatd
You must specify one action
root@mat-ipa-master-1:~$ ipactl start
Existing service file detected!
Assuming stale, cleaning and proceeding
Starting Directory Service
Starting krb5kdc Service
Starting kadmin Service
Starting ipa_memcached Service
Starting httpd Service
Starting ipa-custodia Service
Starting pki-tomcatd Service
Failed to start pki-tomcatd Service
Shutting down
Hint: You can use --ignore-service-failure option for forced start in case that a non-critical service failed
Aborting ipactl
root@mat-ipa-master-1:~$

I hope this helps to track down the problem a bit...

Many thanks and regards from Germany,

Harald

[Florence Blanc-Renaud]

On 01/24/2018 07:35 PM, Harald.Husemann--- via FreeIPA-users wrote:
> Hello Flo,
>
> thanks for your answer, and for the explanation of the certutil output. I have tried your suggestion, first with sudo:
>
> hhuseman@mat-ipa-master-1:~$ sudo kinit -kt /etc/krb5.keytab
> [sudo] password for hhuseman:
> Sorry, try again.
> [sudo] password for hhuseman:
> Sorry, try again.
> [sudo] password for hhuseman:
> sudo: 2 incorrect password attempts
>
> I'm quite sure my password is correct, so it seems there's something broken here also, since sudo worked before the certificate update. My next try was running the command as root:
>
> hhuseman@mat-ipa-master-1:~$ su -
> Password:
> root@mat-ipa-master-1:~$ kinit -kt /etc/krb5.keytab
> root@mat-ipa-master-1:~$ exit
> logout
>
> As you see, there is no output at all, so I tried it again with -V:
>
> root@mat-ipa-master-1:~$ kinit -V -kt /etc/krb5.keytab
> Using existing cache: persistent:0:krb_ccache_VPUg94b
> Using principal: host/[e-mail]
> Using keytab: /etc/krb5.keytab
> Authenticated to Kerberos v5
> root@mat-ipa-master-1:~$
>
> I have also re-checked the certificate which is issued by the HTTPS-Server in my browser, it is still the old one.
> And, I've tried to get the list of certificates with ipa-getcert:
>
> root@mat-ipa-master-1:~$ ipa-getcert list
> Number of certificates and requests being tracked: 5.
> Request ID '20170303080146':
>          status: CA_UNREACHABLE
>          ca-error: Server at https://mat-ipa-master-1.materna-com.de/ipa/xml failed request, will retry: -504 (libcurl failed to execute the HTTP POST transaction, explaining:  Peer's Certificate has expired.).
>          stuck: no
>          key pair storage: type=NSSDB,location='/etc/dirsrv/slapd-MATERNA-COM-DE',nickname='Server-Cert',token='NSS Certificate DB',pinfile='/etc/dirsrv/slapd-MATERNA-COM-DE/pwdfile.txt'
>          certificate: type=NSSDB,location='/etc/dirsrv/slapd-MATERNA-COM-DE',nickname='Server-Cert',token='NSS Certificate DB'
>          CA: IPA
>          issuer: CN=Certificate Authority,O=MATERNA-COM.DE
>          subject: CN=mat-ipa-master-1.materna-com.de,O=MATERNA-COM.DE
>          expires: 2018-01-13 14:45:00 UTC
>          key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment
>          eku: id-kp-serverAuth,id-kp-clientAuth
>          pre-save command:
>          post-save command: /usr/libexec/ipa/certmonger/restart_dirsrv MATERNA-COM-DE
>          track: yes
>          auto-renew: yes
>
> Interesting, since the date was still reset to January 11th, so, even the old certificate should be valid:
> root@mat-ipa-master-1:~$ date
> Thu Jan 11 05:22:21 CET 2018
>
> Nevertheless, I've set the date to actual time by sync'ing it to our NTP-Server:
>
> root@mat-ipa-master-1:~$ ntpdate omcix
> 24 Jan 19:09:00 ntpdate[32699]: step time server 172.30.96.6 offset 1172766.789568 sec
> root@mat-ipa-master-1:~$ date
> Wed Jan 24 19:09:06 CET 2018
>
> But, ipa-getcert list is still falling:
>
> root@mat-ipa-master-1:~$ ipa-getcert list
> Number of certificates and requests being tracked: 5.
> Request ID '20170303080146':
>          status: NEED_TO_SUBMIT
>          ca-error: Server at https://mat-ipa-master-1.materna-com.de/ipa/xml failed request, will retry: -504 (libcurl failed to execute the HTTP POST transaction, explaining:  Peer's Certificate has expired.).
>          stuck: no
>          key pair storage: type=NSSDB,location='/etc/dirsrv/slapd-MATERNA-COM-DE',nickname='Server-Cert',token='NSS Certificate DB',pinfile='/etc/dirsrv/slapd-MATERNA-COM-DE/pwdfile.txt'
>          certificate: type=NSSDB,location='/etc/dirsrv/slapd-MATERNA-COM-DE',nickname='Server-Cert',token='NSS Certificate DB'
>          CA: IPA
>          issuer: CN=Certificate Authority,O=MATERNA-COM.DE
>          subject: CN=mat-ipa-master-1.materna-com.de,O=MATERNA-COM.DE
>          expires: 2018-01-13 14:45:00 UTC
>          key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment
>          eku: id-kp-serverAuth,id-kp-clientAuth
>          pre-save command:
>          post-save command: /usr/libexec/ipa/certmonger/restart_dirsrv MATERNA-COM-DE
>          track: yes
>          auto-renew: yes
> root@mat-ipa-master-1:~$
>
> To ensure everything's running I've issued an ipactl:
>
> root@mat-ipa-master-1:~$ ipactl status
> Directory Service: RUNNING
> krb5kdc Service: RUNNING
> kadmin Service: RUNNING
> ipa_memcached Service: RUNNING
> httpd Service: RUNNING
> ipa-custodia Service: RUNNING
> pki-tomcatd Service: STOPPED
> ipa-otpd Service: RUNNING
> ipa: INFO: The ipactl command was successful
> root@mat-ipa-master-1:~$
>
> So it seems everything's ok except of the PKI, I've tried to restart it, but it fails:
>
> root@mat-ipa-master-1:~$ ipactl start pki-tomcatd
> You must specify one action
> root@mat-ipa-master-1:~$ ipactl start
> Existing service file detected!
> Assuming stale, cleaning and proceeding
> Starting Directory Service
> Starting krb5kdc Service
> Starting kadmin Service
> Starting ipa_memcached Service
> Starting httpd Service
> Starting ipa-custodia Service
> Starting pki-tomcatd Service
> Failed to start pki-tomcatd Service
> Shutting down
> Hint: You can use --ignore-service-failure option for forced start in case that a non-critical service failed
> Aborting ipactl
> root@mat-ipa-master-1:~$
>
> I hope this helps to track down the problem a bit...

pki-tomcat may fail to start if it's unable to authenticate to the LDAP 
server (LDAP is used as data store by pki-tomcat, and authentication is 
done with the cert 'subsystemCert cert-pki-ca' that is stored in 
/etc/pki/pki-tomcat/alias).

You will need to check if this cert is still valid. Sometimes during 
renewal, the cert properly gets downloaded to the NSS db 
/etc/pki/pki-tomcat/alias but is not propagated to the LDAP server. You 
need to compare the cert in the NSS DB and the value for the ldap entry 
uid=pkidbuser,ou=people,o=ipaca. More information can be found in this 
blog:https://floblanc.wordpress.com/2017/09/11/troubleshooting-freeipa-pki-tomcat...

But before jumping to the conclusion, please read pki-tomcat logs in 
/var/log/pki/pki-tomcat/ca/debug and check if the issue is indeed coming 
from an expired subsystemCert cert-pki-ca certificate.

Flo
> Many thanks and regards from Germany,
>
> Harald

[Harald Husemann]

Hello Flo,

and thanks again for your response. First of all, I've figured out that 
the package "pki-symkey" was missing, so I've installed it with yum. 
Now, according to systemctl, pki-tomcatd is running:

root@mat-ipa-master-1:~$ systemctl status [e-mail]
● [e-mail] - PKI Tomcat Server pki-tomcat
    Loaded: loaded (/lib/systemd/system/pki-tomcatd@.service; enabled; 
vendor preset: disabled)
    Active: active (running) since Wed 2018-01-10 10:08:25 CET; 3min 55s ago
(...)

But, ipactl still complains that it is stopped:

root@mat-ipa-master-1:~$ ipactl start --ignore-service-failures
Existing service file detected!
Assuming stale, cleaning and proceeding
Starting Directory Service
(...)
Failed to start pki-tomcatd Service
Forced start, ignoring pki-tomcatd Service, continuing normal operation
(...)

As you suggested, I've checked the debug log 
/var/log/pki/pki-tomcat/ca/debug:
(...)
Internal Database Error encountered: Could not connect to LDAP server 
host mat-ipa-master-1.materna-com.de port 636 Error 
netscape.ldap.LDAPException: Authentication failed (48)
(...)

So, I've examined the config and the certificates, with the commands 
from the blog post:

root@mat-ipa-master-1:~$ grep internaldb.ldapauth 
/etc/pki/pki-tomcat/ca/CS.cfg
internaldb.ldapauth.authtype=SslClientAuth
internaldb.ldapauth.bindDN=cn=Directory Manager
internaldb.ldapauth.bindPWPrompt=internaldb
internaldb.ldapauth.clientCertNickname=subsystemCert cert-pki-ca
root@mat-ipa-master-1:~$ grep internaldb.ldapconn 
/etc/pki/pki-tomcat/ca/CS.cfg
internaldb.ldapconn.host=mat-ipa-master-1.materna-com.de
internaldb.ldapconn.port=636
internaldb.ldapconn.secureConn=true

Ok, we're using LDAPS.

root@mat-ipa-master-1:~$ certutil -L -d /etc/pki/pki-tomcat/alias -n 
'subsystemCert cert-pki-ca'
Certificate:
     Data:
         Version: 3 (0x2)
         Serial Number: 33 (0x21)
         Signature Algorithm: PKCS #1 SHA-256 With RSA Encryption
         Issuer: "CN=Certificate Authority,O=MATERNA-COM.DE"
         Validity:
             Not Before: Fri Jan 12 23:56:02 2018
             Not After : Sat Jan 13 14:45:00 2018
(...)

Interesting, I've reset the date to Jan 10th:

root@mat-ipa-master-1:~$ date
Wed Jan 10 10:05:49 CET 2018

So, the certificate is not expired, but invalid since it's too new?!
Never mind, going further:

root@mat-ipa-master-1:~$ grep internal 
/var/lib/pki/pki-tomcat/conf/password.conf | cut -d= -f2 > /tmp/pwdfile.txt
root@mat-ipa-master-1:~$ certutil -K -d /etc/pki/pki-tomcat/alias -f 
/tmp/pwdfile.txt -n 'subsystemCert cert-pki-ca'
certutil: Checking token "NSS Certificate DB" in slot "NSS User Private 
Key and Certificate Services"
certutil: problem listing keys: SEC_ERROR_UNRECOGNIZED_OID: Unrecognized 
Object Identifier.

Do you have an idea why the key for this OID cannot be found?

Thanks, and best regards from Germany,

Harald

On 30.01.2018 14:05, Florence Blanc-Renaud wrote:
> On 01/24/2018 07:35 PM, Harald.Husemann--- via FreeIPA-users wrote:
> > Hello Flo,
> >
> > thanks for your answer, and for the explanation of the certutil 
> > output. I have tried your suggestion, first with sudo:
> >
> > hhuseman@mat-ipa-master-1:~$ sudo kinit -kt /etc/krb5.keytab
> > [sudo] password for hhuseman:
> > Sorry, try again.
> > [sudo] password for hhuseman:
> > Sorry, try again.
> > [sudo] password for hhuseman:
> > sudo: 2 incorrect password attempts
> >
> > I'm quite sure my password is correct, so it seems there's something 
> > broken here also, since sudo worked before the certificate update. My 
> > next try was running the command as root:
> >
> > hhuseman@mat-ipa-master-1:~$ su -
> > Password:
> > root@mat-ipa-master-1:~$ kinit -kt /etc/krb5.keytab
> > root@mat-ipa-master-1:~$ exit
> > logout
> >
> > As you see, there is no output at all, so I tried it again with -V:
> >
> > root@mat-ipa-master-1:~$ kinit -V -kt /etc/krb5.keytab
> > Using existing cache: persistent:0:krb_ccache_VPUg94b
> > Using principal: host/[e-mail]
> > Using keytab: /etc/krb5.keytab
> > Authenticated to Kerberos v5
> > root@mat-ipa-master-1:~$
> >
> > I have also re-checked the certificate which is issued by the 
> > HTTPS-Server in my browser, it is still the old one.
> > And, I've tried to get the list of certificates with ipa-getcert:
> >
> > root@mat-ipa-master-1:~$ ipa-getcert list
> > Number of certificates and requests being tracked: 5.
> > Request ID '20170303080146':
> >          status: CA_UNREACHABLE
> >          ca-error: Server at 
> > https://mat-ipa-master-1.materna-com.de/ipa/xml failed request, will 
> > retry: -504 (libcurl failed to execute the HTTP POST transaction, 
> > explaining:  Peer's Certificate has expired.).
> >          stuck: no
> >          key pair storage: 
> > type=NSSDB,location='/etc/dirsrv/slapd-MATERNA-COM-DE',nickname='Server-Cert',token='NSS 
> > Certificate DB',pinfile='/etc/dirsrv/slapd-MATERNA-COM-DE/pwdfile.txt'
> >          certificate: 
> > type=NSSDB,location='/etc/dirsrv/slapd-MATERNA-COM-DE',nickname='Server-Cert',token='NSS 
> > Certificate DB'
> >          CA: IPA
> >          issuer: CN=Certificate Authority,O=MATERNA-COM.DE
> >          subject: CN=mat-ipa-master-1.materna-com.de,O=MATERNA-COM.DE
> >          expires: 2018-01-13 14:45:00 UTC
> >          key usage: 
> > digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment
> >          eku: id-kp-serverAuth,id-kp-clientAuth
> >          pre-save command:
> >          post-save command: 
> > /usr/libexec/ipa/certmonger/restart_dirsrv MATERNA-COM-DE
> >          track: yes
> >          auto-renew: yes
> >
> > Interesting, since the date was still reset to January 11th, so, even 
> > the old certificate should be valid:
> > root@mat-ipa-master-1:~$ date
> > Thu Jan 11 05:22:21 CET 2018
> >
> > Nevertheless, I've set the date to actual time by sync'ing it to our 
> > NTP-Server:
> >
> > root@mat-ipa-master-1:~$ ntpdate omcix
> > 24 Jan 19:09:00 ntpdate[32699]: step time server 172.30.96.6 offset 
> > 1172766.789568 sec
> > root@mat-ipa-master-1:~$ date
> > Wed Jan 24 19:09:06 CET 2018
> >
> > But, ipa-getcert list is still falling:
> >
> > root@mat-ipa-master-1:~$ ipa-getcert list
> > Number of certificates and requests being tracked: 5.
> > Request ID '20170303080146':
> >          status: NEED_TO_SUBMIT
> >          ca-error: Server at 
> > https://mat-ipa-master-1.materna-com.de/ipa/xml failed request, will 
> > retry: -504 (libcurl failed to execute the HTTP POST transaction, 
> > explaining:  Peer's Certificate has expired.).
> >          stuck: no
> >          key pair storage: 
> > type=NSSDB,location='/etc/dirsrv/slapd-MATERNA-COM-DE',nickname='Server-Cert',token='NSS 
> > Certificate DB',pinfile='/etc/dirsrv/slapd-MATERNA-COM-DE/pwdfile.txt'
> >          certificate: 
> > type=NSSDB,location='/etc/dirsrv/slapd-MATERNA-COM-DE',nickname='Server-Cert',token='NSS 
> > Certificate DB'
> >          CA: IPA
> >          issuer: CN=Certificate Authority,O=MATERNA-COM.DE
> >          subject: CN=mat-ipa-master-1.materna-com.de,O=MATERNA-COM.DE
> >          expires: 2018-01-13 14:45:00 UTC
> >          key usage: 
> > digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment
> >          eku: id-kp-serverAuth,id-kp-clientAuth
> >          pre-save command:
> >          post-save command: 
> > /usr/libexec/ipa/certmonger/restart_dirsrv MATERNA-COM-DE
> >          track: yes
> >          auto-renew: yes
> > root@mat-ipa-master-1:~$
> >
> > To ensure everything's running I've issued an ipactl:
> >
> > root@mat-ipa-master-1:~$ ipactl status
> > Directory Service: RUNNING
> > krb5kdc Service: RUNNING
> > kadmin Service: RUNNING
> > ipa_memcached Service: RUNNING
> > httpd Service: RUNNING
> > ipa-custodia Service: RUNNING
> > pki-tomcatd Service: STOPPED
> > ipa-otpd Service: RUNNING
> > ipa: INFO: The ipactl command was successful
> > root@mat-ipa-master-1:~$
> >
> > So it seems everything's ok except of the PKI, I've tried to restart 
> > it, but it fails:
> >
> > root@mat-ipa-master-1:~$ ipactl start pki-tomcatd
> > You must specify one action
> > root@mat-ipa-master-1:~$ ipactl start
> > Existing service file detected!
> > Assuming stale, cleaning and proceeding
> > Starting Directory Service
> > Starting krb5kdc Service
> > Starting kadmin Service
> > Starting ipa_memcached Service
> > Starting httpd Service
> > Starting ipa-custodia Service
> > Starting pki-tomcatd Service
> > Failed to start pki-tomcatd Service
> > Shutting down
> > Hint: You can use --ignore-service-failure option for forced start in 
> > case that a non-critical service failed
> > Aborting ipactl
> > root@mat-ipa-master-1:~$
> >
> > I hope this helps to track down the problem a bit...
>
> pki-tomcat may fail to start if it's unable to authenticate to the 
> LDAP server (LDAP is used as data store by pki-tomcat, and 
> authentication is done with the cert 'subsystemCert cert-pki-ca' that 
> is stored in /etc/pki/pki-tomcat/alias).
>
> You will need to check if this cert is still valid. Sometimes during 
> renewal, the cert properly gets downloaded to the NSS db 
> /etc/pki/pki-tomcat/alias but is not propagated to the LDAP server. 
> You need to compare the cert in the NSS DB and the value for the ldap 
> entry uid=pkidbuser,ou=people,o=ipaca. More information can be found 
> in this 
> blog:https://floblanc.wordpress.com/2017/09/11/troubleshooting-freeipa-pki-tomcat...
>
> But before jumping to the conclusion, please read pki-tomcat logs in 
> /var/log/pki/pki-tomcat/ca/debug and check if the issue is indeed 
> coming from an expired subsystemCert cert-pki-ca certificate.
>
> Flo
> > Many thanks and regards from Germany,
> >
> > Harald

[Florence Blanc-Renaud]

On 01/30/2018 05:17 PM, Harald Husemann via FreeIPA-users wrote:
> Hello Flo,
>
> and thanks again for your response. First of all, I've figured out that 
> the package "pki-symkey" was missing, so I've installed it with yum. 
> Now, according to systemctl, pki-tomcatd is running:
>
> root@mat-ipa-master-1:~$ systemctl status [e-mail]
> ● [e-mail] - PKI Tomcat Server pki-tomcat
>     Loaded: loaded (/lib/systemd/system/pki-tomcatd@.service; enabled; 
> vendor preset: disabled)
>     Active: active (running) since Wed 2018-01-10 10:08:25 CET; 3min 55s 
> ago
> (...)
>
> But, ipactl still complains that it is stopped:
>
> root@mat-ipa-master-1:~$ ipactl start --ignore-service-failures
> Existing service file detected!
> Assuming stale, cleaning and proceeding
> Starting Directory Service
> (...)
> Failed to start pki-tomcatd Service
> Forced start, ignoring pki-tomcatd Service, continuing normal operation
> (...)
>
> As you suggested, I've checked the debug log 
> /var/log/pki/pki-tomcat/ca/debug:
> (...)
> Internal Database Error encountered: Could not connect to LDAP server 
> host mat-ipa-master-1.materna-com.de port 636 Error 
> netscape.ldap.LDAPException: Authentication failed (48)
> (...)
>
> So, I've examined the config and the certificates, with the commands 
> from the blog post:
>
> root@mat-ipa-master-1:~$ grep internaldb.ldapauth 
> /etc/pki/pki-tomcat/ca/CS.cfg
> internaldb.ldapauth.authtype=SslClientAuth
> internaldb.ldapauth.bindDN=cn=Directory Manager
> internaldb.ldapauth.bindPWPrompt=internaldb
> internaldb.ldapauth.clientCertNickname=subsystemCert cert-pki-ca
> root@mat-ipa-master-1:~$ grep internaldb.ldapconn 
> /etc/pki/pki-tomcat/ca/CS.cfg
> internaldb.ldapconn.host=mat-ipa-master-1.materna-com.de
> internaldb.ldapconn.port=636
> internaldb.ldapconn.secureConn=true
>
> Ok, we're using LDAPS.
>
> root@mat-ipa-master-1:~$ certutil -L -d /etc/pki/pki-tomcat/alias -n 
> 'subsystemCert cert-pki-ca'
> Certificate:
>      Data:
>          Version: 3 (0x2)
>          Serial Number: 33 (0x21)
>          Signature Algorithm: PKCS #1 SHA-256 With RSA Encryption
>          Issuer: "CN=Certificate Authority,O=MATERNA-COM.DE"
>          Validity:
>              Not Before: Fri Jan 12 23:56:02 2018
>              Not After : Sat Jan 13 14:45:00 2018
> (...)
>
> Interesting, I've reset the date to Jan 10th:
>
> root@mat-ipa-master-1:~$ date
> Wed Jan 10 10:05:49 CET 2018
>
> So, the certificate is not expired, but invalid since it's too new?!
> Never mind, going further:
>
> root@mat-ipa-master-1:~$ grep internal 
> /var/lib/pki/pki-tomcat/conf/password.conf | cut -d= -f2 > /tmp/pwdfile.txt
> root@mat-ipa-master-1:~$ certutil -K -d /etc/pki/pki-tomcat/alias -f 
> /tmp/pwdfile.txt -n 'subsystemCert cert-pki-ca'
> certutil: Checking token "NSS Certificate DB" in slot "NSS User Private 
> Key and Certificate Services"
> certutil: problem listing keys: SEC_ERROR_UNRECOGNIZED_OID: Unrecognized 
> Object Identifier.
>
> Do you have an idea why the key for this OID cannot be found?

Hi,

let's not focus on this issue and try to check what went wrong. Can you 
compare the content of the certificate field in 
uid=pkidbuser,ou=people,o=ipaca and the blob that can be seen with:
certutil -L -d /etc/pki/pki-tomcat/alias -n 'subsystemCert cert-pki-ca' -a

This one is often the culprit, linked to SElinux issues. By the way, did 
you check if there were any AVC?

Flo
> Thanks, and best regards from Germany,
>
> Harald
>
> On 30.01.2018 14:05, Florence Blanc-Renaud wrote:
> > On 01/24/2018 07:35 PM, Harald.Husemann--- via FreeIPA-users wrote:
> > > Hello Flo,
> > >
> > > thanks for your answer, and for the explanation of the certutil 
> > > output. I have tried your suggestion, first with sudo:
> > >
> > > hhuseman@mat-ipa-master-1:~$ sudo kinit -kt /etc/krb5.keytab
> > > [sudo] password for hhuseman:
> > > Sorry, try again.
> > > [sudo] password for hhuseman:
> > > Sorry, try again.
> > > [sudo] password for hhuseman:
> > > sudo: 2 incorrect password attempts
> > >
> > > I'm quite sure my password is correct, so it seems there's something 
> > > broken here also, since sudo worked before the certificate update. My 
> > > next try was running the command as root:
> > >
> > > hhuseman@mat-ipa-master-1:~$ su -
> > > Password:
> > > root@mat-ipa-master-1:~$ kinit -kt /etc/krb5.keytab
> > > root@mat-ipa-master-1:~$ exit
> > > logout
> > >
> > > As you see, there is no output at all, so I tried it again with -V:
> > >
> > > root@mat-ipa-master-1:~$ kinit -V -kt /etc/krb5.keytab
> > > Using existing cache: persistent:0:krb_ccache_VPUg94b
> > > Using principal: host/[e-mail]
> > > Using keytab: /etc/krb5.keytab
> > > Authenticated to Kerberos v5
> > > root@mat-ipa-master-1:~$
> > >
> > > I have also re-checked the certificate which is issued by the 
> > > HTTPS-Server in my browser, it is still the old one.
> > > And, I've tried to get the list of certificates with ipa-getcert:
> > >
> > > root@mat-ipa-master-1:~$ ipa-getcert list
> > > Number of certificates and requests being tracked: 5.
> > > Request ID '20170303080146':
> > >          status: CA_UNREACHABLE
> > >          ca-error: Server at 
> > > https://mat-ipa-master-1.materna-com.de/ipa/xml failed request, will 
> > > retry: -504 (libcurl failed to execute the HTTP POST transaction, 
> > > explaining:  Peer's Certificate has expired.).
> > >          stuck: no
> > >          key pair storage: 
> > > type=NSSDB,location='/etc/dirsrv/slapd-MATERNA-COM-DE',nickname='Server-Cert',token='NSS 
> > > Certificate DB',pinfile='/etc/dirsrv/slapd-MATERNA-COM-DE/pwdfile.txt'
> > >          certificate: 
> > > type=NSSDB,location='/etc/dirsrv/slapd-MATERNA-COM-DE',nickname='Server-Cert',token='NSS 
> > > Certificate DB'
> > >          CA: IPA
> > >          issuer: CN=Certificate Authority,O=MATERNA-COM.DE
> > >          subject: CN=mat-ipa-master-1.materna-com.de,O=MATERNA-COM.DE
> > >          expires: 2018-01-13 14:45:00 UTC
> > >          key usage: 
> > > digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment
> > >          eku: id-kp-serverAuth,id-kp-clientAuth
> > >          pre-save command:
> > >          post-save command: 
> > > /usr/libexec/ipa/certmonger/restart_dirsrv MATERNA-COM-DE
> > >          track: yes
> > >          auto-renew: yes
> > >
> > > Interesting, since the date was still reset to January 11th, so, even 
> > > the old certificate should be valid:
> > > root@mat-ipa-master-1:~$ date
> > > Thu Jan 11 05:22:21 CET 2018
> > >
> > > Nevertheless, I've set the date to actual time by sync'ing it to our 
> > > NTP-Server:
> > >
> > > root@mat-ipa-master-1:~$ ntpdate omcix
> > > 24 Jan 19:09:00 ntpdate[32699]: step time server 172.30.96.6 offset 
> > > 1172766.789568 sec
> > > root@mat-ipa-master-1:~$ date
> > > Wed Jan 24 19:09:06 CET 2018
> > >
> > > But, ipa-getcert list is still falling:
> > >
> > > root@mat-ipa-master-1:~$ ipa-getcert list
> > > Number of certificates and requests being tracked: 5.
> > > Request ID '20170303080146':
> > >          status: NEED_TO_SUBMIT
> > >          ca-error: Server at 
> > > https://mat-ipa-master-1.materna-com.de/ipa/xml failed request, will 
> > > retry: -504 (libcurl failed to execute the HTTP POST transaction, 
> > > explaining:  Peer's Certificate has expired.).
> > >          stuck: no
> > >          key pair storage: 
> > > type=NSSDB,location='/etc/dirsrv/slapd-MATERNA-COM-DE',nickname='Server-Cert',token='NSS 
> > > Certificate DB',pinfile='/etc/dirsrv/slapd-MATERNA-COM-DE/pwdfile.txt'
> > >          certificate: 
> > > type=NSSDB,location='/etc/dirsrv/slapd-MATERNA-COM-DE',nickname='Server-Cert',token='NSS 
> > > Certificate DB'
> > >          CA: IPA
> > >          issuer: CN=Certificate Authority,O=MATERNA-COM.DE
> > >          subject: CN=mat-ipa-master-1.materna-com.de,O=MATERNA-COM.DE
> > >          expires: 2018-01-13 14:45:00 UTC
> > >          key usage: 
> > > digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment
> > >          eku: id-kp-serverAuth,id-kp-clientAuth
> > >          pre-save command:
> > >          post-save command: 
> > > /usr/libexec/ipa/certmonger/restart_dirsrv MATERNA-COM-DE
> > >          track: yes
> > >          auto-renew: yes
> > > root@mat-ipa-master-1:~$
> > >
> > > To ensure everything's running I've issued an ipactl:
> > >
> > > root@mat-ipa-master-1:~$ ipactl status
> > > Directory Service: RUNNING
> > > krb5kdc Service: RUNNING
> > > kadmin Service: RUNNING
> > > ipa_memcached Service: RUNNING
> > > httpd Service: RUNNING
> > > ipa-custodia Service: RUNNING
> > > pki-tomcatd Service: STOPPED
> > > ipa-otpd Service: RUNNING
> > > ipa: INFO: The ipactl command was successful
> > > root@mat-ipa-master-1:~$
> > >
> > > So it seems everything's ok except of the PKI, I've tried to restart 
> > > it, but it fails:
> > >
> > > root@mat-ipa-master-1:~$ ipactl start pki-tomcatd
> > > You must specify one action
> > > root@mat-ipa-master-1:~$ ipactl start
> > > Existing service file detected!
> > > Assuming stale, cleaning and proceeding
> > > Starting Directory Service
> > > Starting krb5kdc Service
> > > Starting kadmin Service
> > > Starting ipa_memcached Service
> > > Starting httpd Service
> > > Starting ipa-custodia Service
> > > Starting pki-tomcatd Service
> > > Failed to start pki-tomcatd Service
> > > Shutting down
> > > Hint: You can use --ignore-service-failure option for forced start in 
> > > case that a non-critical service failed
> > > Aborting ipactl
> > > root@mat-ipa-master-1:~$
> > >
> > > I hope this helps to track down the problem a bit...
> >
> > pki-tomcat may fail to start if it's unable to authenticate to the 
> > LDAP server (LDAP is used as data store by pki-tomcat, and 
> > authentication is done with the cert 'subsystemCert cert-pki-ca' that 
> > is stored in /etc/pki/pki-tomcat/alias).
> >
> > You will need to check if this cert is still valid. Sometimes during 
> > renewal, the cert properly gets downloaded to the NSS db 
> > /etc/pki/pki-tomcat/alias but is not propagated to the LDAP server. 
> > You need to compare the cert in the NSS DB and the value for the ldap 
> > entry uid=pkidbuser,ou=people,o=ipaca. More information can be found 
> > in this 
> > blog:https://floblanc.wordpress.com/2017/09/11/troubleshooting-freeipa-pki-tomcat...
> >
> > But before jumping to the conclusion, please read pki-tomcat logs in 
> > /var/log/pki/pki-tomcat/ca/debug and check if the issue is indeed 
> > coming from an expired subsystemCert cert-pki-ca certificate.
> >
> > Flo
> > > Many thanks and regards from Germany,
> > >
> > > Harald
>
> FreeIPA-users mailing list -- [e-mail]
> To unsubscribe send an email to [e-mail]

Hello Flo,

I've checked the certificates, there are several ones in the LDAP databases (got them with "ldapsearch -x -D "cn=Directory Manager" -W -b "uid=pkiuser,ou=people,o=ipaca", hope that's correct?) and one of them is identical to the one which I've got with certutil.
I've also checked AVC, but as far as I see there are no issues, avcstat gives "ERROR: open '(null)/avc/cache_stats': No such file or directory". That’s what I would expect, since SELinux is disabled in /etc/selinux/config.

Many thanks for your support and your quick responses,

kind regards from Germany,

Harald

Hello Flo,

I'm resending the mail since my first response was rejected because of SPF, and only found its way to the mailing list...

Many thanks again for your response. First of all, I've figured out that 
the package "pki-symkey" was missing, so I've installed it with yum. 
Now, according to systemctl, pki-tomcatd is running:

root@mat-ipa-master-1:~$ systemctl status [e-mail]
● [e-mail] - PKI Tomcat Server pki-tomcat
    Loaded: loaded (/lib/systemd/system/pki-tomcatd@.service; enabled; 
vendor preset: disabled)
    Active: active (running) since Wed 2018-01-10 10:08:25 CET; 3min 55s 
ago
(...)

But, ipactl still complains that it is stopped:

root@mat-ipa-master-1:~$ ipactl start --ignore-service-failures
Existing service file detected!
Assuming stale, cleaning and proceeding
Starting Directory Service
(...)
Failed to start pki-tomcatd Service
Forced start, ignoring pki-tomcatd Service, continuing normal operation
(...)

As you suggested, I've checked the debug log 
/var/log/pki/pki-tomcat/ca/debug:
(...)
Internal Database Error encountered: Could not connect to LDAP server 
host mat-ipa-master-1.materna-com.de port 636 Error 
netscape.ldap.LDAPException: Authentication failed (48)
(...)

So, I've examined the config and the certificates, with the commands 
from the blog post:

root@mat-ipa-master-1:~$ grep internaldb.ldapauth 
/etc/pki/pki-tomcat/ca/CS.cfg
internaldb.ldapauth.authtype=SslClientAuth
internaldb.ldapauth.bindDN=cn=Directory Manager
internaldb.ldapauth.bindPWPrompt=internaldb
internaldb.ldapauth.clientCertNickname=subsystemCert cert-pki-ca
root@mat-ipa-master-1:~$ grep internaldb.ldapconn 
/etc/pki/pki-tomcat/ca/CS.cfg
internaldb.ldapconn.host=mat-ipa-master-1.materna-com.de
internaldb.ldapconn.port=636
internaldb.ldapconn.secureConn=true

Ok, we're using LDAPS.

root@mat-ipa-master-1:~$ certutil -L -d /etc/pki/pki-tomcat/alias -n 
'subsystemCert cert-pki-ca'
Certificate:
     Data:
         Version: 3 (0x2)
         Serial Number: 33 (0x21)
         Signature Algorithm: PKCS #1 SHA-256 With RSA Encryption
         Issuer: "CN=Certificate Authority,O=MATERNA-COM.DE"
         Validity:
             Not Before: Fri Jan 12 23:56:02 2018
             Not After : Sat Jan 13 14:45:00 2018
(...)

Interesting, I've reset the date to Jan 10th:

root@mat-ipa-master-1:~$ date
Wed Jan 10 10:05:49 CET 2018

So, the certificate is not expired, but invalid since it's too new?!
Never mind, going further:

root@mat-ipa-master-1:~$ grep internal 
/var/lib/pki/pki-tomcat/conf/password.conf | cut -d= -f2 > /tmp/pwdfile.txt
root@mat-ipa-master-1:~$ certutil -K -d /etc/pki/pki-tomcat/alias -f 
/tmp/pwdfile.txt -n 'subsystemCert cert-pki-ca'
certutil: Checking token "NSS Certificate DB" in slot "NSS User Private 
Key and Certificate Services"
certutil: problem listing keys: SEC_ERROR_UNRECOGNIZED_OID: Unrecognized 
Object Identifier.

Do you have an idea why the key for this OID cannot be found?

Thanks, and best regards from Germany,

Harald