Hello Flo,
and thanks again for your response. First of all, I've figured out that
the package "pki-symkey" was missing, so I've installed it with yum.
Now, according to systemctl, pki-tomcatd is running:
root@mat-ipa-master-1:~$ systemctl status pki-tomcatd(a)pki-tomcat.service
● pki-tomcatd(a)pki-tomcat.service - PKI Tomcat Server pki-tomcat
Loaded: loaded (/lib/systemd/system/pki-tomcatd@.service; enabled;
vendor preset: disabled)
Active: active (running) since Wed 2018-01-10 10:08:25 CET; 3min 55s ago
(...)
But, ipactl still complains that it is stopped:
root@mat-ipa-master-1:~$ ipactl start --ignore-service-failures
Existing service file detected!
Assuming stale, cleaning and proceeding
Starting Directory Service
(...)
Failed to start pki-tomcatd Service
Forced start, ignoring pki-tomcatd Service, continuing normal operation
(...)
As you suggested, I've checked the debug log
/var/log/pki/pki-tomcat/ca/debug:
(...)
Internal Database Error encountered: Could not connect to LDAP server
host mat-ipa-master-1.materna-com.de port 636 Error
netscape.ldap.LDAPException: Authentication failed (48)
(...)
So, I've examined the config and the certificates, with the commands
from the blog post:
root@mat-ipa-master-1:~$ grep internaldb.ldapauth
/etc/pki/pki-tomcat/ca/CS.cfg
internaldb.ldapauth.authtype=SslClientAuth
internaldb.ldapauth.bindDN=cn=Directory Manager
internaldb.ldapauth.bindPWPrompt=internaldb
internaldb.ldapauth.clientCertNickname=subsystemCert cert-pki-ca
root@mat-ipa-master-1:~$ grep internaldb.ldapconn
/etc/pki/pki-tomcat/ca/CS.cfg
internaldb.ldapconn.host=mat-ipa-master-1.materna-com.de
internaldb.ldapconn.port=636
internaldb.ldapconn.secureConn=true
Ok, we're using LDAPS.
root@mat-ipa-master-1:~$ certutil -L -d /etc/pki/pki-tomcat/alias -n
'subsystemCert cert-pki-ca'
Certificate:
Data:
Version: 3 (0x2)
Serial Number: 33 (0x21)
Signature Algorithm: PKCS #1 SHA-256 With RSA Encryption
Issuer: "CN=Certificate Authority,O=MATERNA-COM.DE"
Validity:
Not Before: Fri Jan 12 23:56:02 2018
Not After : Sat Jan 13 14:45:00 2018
(...)
Interesting, I've reset the date to Jan 10th:
root@mat-ipa-master-1:~$ date
Wed Jan 10 10:05:49 CET 2018
So, the certificate is not expired, but invalid since it's too new?!
Never mind, going further:
root@mat-ipa-master-1:~$ grep internal
/var/lib/pki/pki-tomcat/conf/password.conf | cut -d= -f2 > /tmp/pwdfile.txt
root@mat-ipa-master-1:~$ certutil -K -d /etc/pki/pki-tomcat/alias -f
/tmp/pwdfile.txt -n 'subsystemCert cert-pki-ca'
certutil: Checking token "NSS Certificate DB" in slot "NSS User Private
Key and Certificate Services"
certutil: problem listing keys: SEC_ERROR_UNRECOGNIZED_OID: Unrecognized
Object Identifier.
Do you have an idea why the key for this OID cannot be found?
Thanks, and best regards from Germany,
Harald
On 30.01.2018 14:05, Florence Blanc-Renaud wrote:
On 01/24/2018 07:35 PM, Harald.Husemann--- via FreeIPA-users wrote:
> Hello Flo,
>
> thanks for your answer, and for the explanation of the certutil
> output. I have tried your suggestion, first with sudo:
>
> hhuseman@mat-ipa-master-1:~$ sudo kinit -kt /etc/krb5.keytab
> [sudo] password for hhuseman:
> Sorry, try again.
> [sudo] password for hhuseman:
> Sorry, try again.
> [sudo] password for hhuseman:
> sudo: 2 incorrect password attempts
>
> I'm quite sure my password is correct, so it seems there's something
> broken here also, since sudo worked before the certificate update. My
> next try was running the command as root:
>
> hhuseman@mat-ipa-master-1:~$ su -
> Password:
> root@mat-ipa-master-1:~$ kinit -kt /etc/krb5.keytab
> root@mat-ipa-master-1:~$ exit
> logout
>
> As you see, there is no output at all, so I tried it again with -V:
>
> root@mat-ipa-master-1:~$ kinit -V -kt /etc/krb5.keytab
> Using existing cache: persistent:0:krb_ccache_VPUg94b
> Using principal: host/mat-ipa-master-1.materna-com.de(a)MATERNA-COM.DE
> Using keytab: /etc/krb5.keytab
> Authenticated to Kerberos v5
> root@mat-ipa-master-1:~$
>
> I have also re-checked the certificate which is issued by the
> HTTPS-Server in my browser, it is still the old one.
> And, I've tried to get the list of certificates with ipa-getcert:
>
> root@mat-ipa-master-1:~$ ipa-getcert list
> Number of certificates and requests being tracked: 5.
> Request ID '20170303080146':
> status: CA_UNREACHABLE
> ca-error: Server at
>
https://mat-ipa-master-1.materna-com.de/ipa/xml failed request, will
> retry: -504 (libcurl failed to execute the HTTP POST transaction,
> explaining: Peer's Certificate has expired.).
> stuck: no
> key pair storage:
>
type=NSSDB,location='/etc/dirsrv/slapd-MATERNA-COM-DE',nickname='Server-Cert',token='NSS
> Certificate DB',pinfile='/etc/dirsrv/slapd-MATERNA-COM-DE/pwdfile.txt'
> certificate:
>
type=NSSDB,location='/etc/dirsrv/slapd-MATERNA-COM-DE',nickname='Server-Cert',token='NSS
> Certificate DB'
> CA: IPA
> issuer: CN=Certificate Authority,O=MATERNA-COM.DE
> subject: CN=mat-ipa-master-1.materna-com.de,O=MATERNA-COM.DE
> expires: 2018-01-13 14:45:00 UTC
> key usage:
> digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment
> eku: id-kp-serverAuth,id-kp-clientAuth
> pre-save command:
> post-save command:
> /usr/libexec/ipa/certmonger/restart_dirsrv MATERNA-COM-DE
> track: yes
> auto-renew: yes
>
> Interesting, since the date was still reset to January 11th, so, even
> the old certificate should be valid:
> root@mat-ipa-master-1:~$ date
> Thu Jan 11 05:22:21 CET 2018
>
> Nevertheless, I've set the date to actual time by sync'ing it to our
> NTP-Server:
>
> root@mat-ipa-master-1:~$ ntpdate omcix
> 24 Jan 19:09:00 ntpdate[32699]: step time server 172.30.96.6 offset
> 1172766.789568 sec
> root@mat-ipa-master-1:~$ date
> Wed Jan 24 19:09:06 CET 2018
>
> But, ipa-getcert list is still falling:
>
> root@mat-ipa-master-1:~$ ipa-getcert list
> Number of certificates and requests being tracked: 5.
> Request ID '20170303080146':
> status: NEED_TO_SUBMIT
> ca-error: Server at
>
https://mat-ipa-master-1.materna-com.de/ipa/xml failed request, will
> retry: -504 (libcurl failed to execute the HTTP POST transaction,
> explaining: Peer's Certificate has expired.).
> stuck: no
> key pair storage:
>
type=NSSDB,location='/etc/dirsrv/slapd-MATERNA-COM-DE',nickname='Server-Cert',token='NSS
> Certificate DB',pinfile='/etc/dirsrv/slapd-MATERNA-COM-DE/pwdfile.txt'
> certificate:
>
type=NSSDB,location='/etc/dirsrv/slapd-MATERNA-COM-DE',nickname='Server-Cert',token='NSS
> Certificate DB'
> CA: IPA
> issuer: CN=Certificate Authority,O=MATERNA-COM.DE
> subject: CN=mat-ipa-master-1.materna-com.de,O=MATERNA-COM.DE
> expires: 2018-01-13 14:45:00 UTC
> key usage:
> digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment
> eku: id-kp-serverAuth,id-kp-clientAuth
> pre-save command:
> post-save command:
> /usr/libexec/ipa/certmonger/restart_dirsrv MATERNA-COM-DE
> track: yes
> auto-renew: yes
> root@mat-ipa-master-1:~$
>
> To ensure everything's running I've issued an ipactl:
>
> root@mat-ipa-master-1:~$ ipactl status
> Directory Service: RUNNING
> krb5kdc Service: RUNNING
> kadmin Service: RUNNING
> ipa_memcached Service: RUNNING
> httpd Service: RUNNING
> ipa-custodia Service: RUNNING
> pki-tomcatd Service: STOPPED
> ipa-otpd Service: RUNNING
> ipa: INFO: The ipactl command was successful
> root@mat-ipa-master-1:~$
>
> So it seems everything's ok except of the PKI, I've tried to restart
> it, but it fails:
>
> root@mat-ipa-master-1:~$ ipactl start pki-tomcatd
> You must specify one action
> root@mat-ipa-master-1:~$ ipactl start
> Existing service file detected!
> Assuming stale, cleaning and proceeding
> Starting Directory Service
> Starting krb5kdc Service
> Starting kadmin Service
> Starting ipa_memcached Service
> Starting httpd Service
> Starting ipa-custodia Service
> Starting pki-tomcatd Service
> Failed to start pki-tomcatd Service
> Shutting down
> Hint: You can use --ignore-service-failure option for forced start in
> case that a non-critical service failed
> Aborting ipactl
> root@mat-ipa-master-1:~$
>
> I hope this helps to track down the problem a bit...
>
pki-tomcat may fail to start if it's unable to authenticate to the
LDAP server (LDAP is used as data store by pki-tomcat, and
authentication is done with the cert 'subsystemCert cert-pki-ca' that
is stored in /etc/pki/pki-tomcat/alias).
You will need to check if this cert is still valid. Sometimes during
renewal, the cert properly gets downloaded to the NSS db
/etc/pki/pki-tomcat/alias but is not propagated to the LDAP server.
You need to compare the cert in the NSS DB and the value for the ldap
entry uid=pkidbuser,ou=people,o=ipaca. More information can be found
in this
blog:https://floblanc.wordpress.com/2017/09/11/troubleshooting-freeipa-pk...
But before jumping to the conclusion, please read pki-tomcat logs in
/var/log/pki/pki-tomcat/ca/debug and check if the issue is indeed
coming from an expired subsystemCert cert-pki-ca certificate.
Flo
> Many thanks and regards from Germany,
>
> Harald
>