Hello everyone,
I'm trying to fix an issue with our freeipa setup where multiple servers in the domain are failing to start the pki-tomcatd service. At present we have used "pactl start --ignore-service-failure" just to get the rest of the services up and running and now trying to figure out how to fix the overall problem.
Below is a summary of the current state from what I've managed to find so far:
- We have 6 servers in our freeipa domain running version 4.5.4 - Three servers including the "IPA CA renewal master" is having an issue when starting up the pki-tomcatd service, with slightly different observations. - Problem started at different times, other two servers starting the problem many weeks ago, which we haven't been able to fix
On the IPA CA renewal master server (this server only started having the problem few days ago following a crash > reboot ) - - All the certificates listed in "getcert list" are valid (one closest to expiry has about 2 years left) - Cerfificate in /etc/pki/pki-tomcat/alias/ and LDAP match along with the "description" field that has the correct serial - Logs contain this error : Internal Database Error encountered: Could not connect to LDAP server host <FQDN> port 636 Error netscape.ldap.LDAPException: Authentication failed (49)
On the other two servers (where the problem started weeks apart following a server reboot or a ipactl restart ) - - Most of the certificates listed "getcert list" are already expired - Logs contain this error : Internal Database Error encountered: Could not connect to LDAP server host <FQDN> port 636 Error netscape.ldap.LDAPException: Authentication failed (49) - Cerfificate in /etc/pki/pki-tomcat/alias/ and LDAP does NOT match
Other than that there are many other errors on the logs on all the servers related to replication : ERR - NSMMReplicationPlugin - send_updates" and messages like : Certificate in file "/var/kerberos/krb5kdc/kdc.crt" is no longer valid.
To try to come up with some method to fix the servers, I've taken a clone(disconnected from the network) of a non IPA CA renewal master. Below steps I've found on various articles were all done within the clone.
Some things I've tried within the clone out so far along with the errors: https://floblanc.wordpress.com/2017/09/11/troubleshooting-freeipa-pki-tomcat...
$ sudo grep internal /var/lib/pki/pki-tomcat/conf/password.conf | cut -d= -f2 > /tmp/pwdfile.txt $ sudo certutil -K -d /etc/pki/pki-tomcat/alias -f /tmp/pwdfile.txt -n 'subsystemCert cert-pki-ca' certutil: Checking token "NSS Certificate DB" in slot "NSS User Private Key and Certificate Services" certutil: problem listing keys: SEC_ERROR_INVALID_ARGS: security library: invalid arguments.
I did notice is that the message says "NSS Certificate DB" but on the "getcert list" this certificate shows with token='NSS FIPS 140-2 Certificate DB', not sure if there is an actual problem with the password or if I'm not using the command correctly.
Tried resetting the time back to a point where the expired certs are still valid as mentioned in https://access.redhat.com/solutions/3357261 and to get the pki-tomcatd to come up, tried to update the cert. in LDAP to match what is in /etc/pki/pki-tomcat/alias/ using this method https://access.redhat.com/solutions/3614001 fails for me.
It updates the certificate serial in "description" field but never changes the certificate blob. ldapmodify command and the contents of the .ldif file I used are below.
# ldapmodify -x -h localhost -p 389 -D “cn=directory manager” -w -f updatecert.ldif -v ldap_initializer( ldap://localhost:389 ) replace usercertificate: NOT ASCII (894 bytes) replace description: 2;32;CN=Certificate Authority,O=;CN=CA Subsystem,O=<OUR.DOMAIN> modifying entry “uid=pkidbuser,ou=people,o=ipaca” modify complete
# cat updatecert.ldif dn: uid=pkidbuser,ou=people,o=ipaca changetype: modify replace: usercertificate usercertificate::MII.. – replace: description description: 2;32;CN=Certificate Authority,O=;CN=CA Subsystem,O=<OUR.DOMAIN>
I've used various combinations here even a delete of "userCertificate" field to add the correct one later but none of which worked for me.
I'm hoping someone can point me in the right direction.
Thanks in advance, yajith
Hi, I don't know if it's a copy/paste issue but the separator in your ldif file renders badly on my screen. It should be a simple dash ("-") but it looks different.
flo
On Fri, Apr 8, 2022 at 2:28 AM Yajith Dayarathna via FreeIPA-users < freeipa-users@lists.fedorahosted.org> wrote:
Hello everyone,
I'm trying to fix an issue with our freeipa setup where multiple servers in the domain are failing to start the pki-tomcatd service. At present we have used "pactl start --ignore-service-failure" just to get the rest of the services up and running and now trying to figure out how to fix the overall problem.
Below is a summary of the current state from what I've managed to find so far:
- We have 6 servers in our freeipa domain running version 4.5.4
- Three servers including the "IPA CA renewal master" is having an issue
when starting up the pki-tomcatd service, with slightly different observations.
- Problem started at different times, other two servers starting the
problem many weeks ago, which we haven't been able to fix
On the IPA CA renewal master server (this server only started having the problem few days ago following a crash > reboot ) -
- All the certificates listed in "getcert list" are valid (one closest to
expiry has about 2 years left)
- Cerfificate in /etc/pki/pki-tomcat/alias/ and LDAP match along with the
"description" field that has the correct serial
- Logs contain this error : Internal Database Error encountered: Could not
connect to LDAP server host <FQDN> port 636 Error netscape.ldap.LDAPException: Authentication failed (49)
On the other two servers (where the problem started weeks apart following a server reboot or a ipactl restart ) -
- Most of the certificates listed "getcert list" are already expired
- Logs contain this error : Internal Database Error encountered: Could not
connect to LDAP server host <FQDN> port 636 Error netscape.ldap.LDAPException: Authentication failed (49)
- Cerfificate in /etc/pki/pki-tomcat/alias/ and LDAP does NOT match
Other than that there are many other errors on the logs on all the servers related to replication : ERR - NSMMReplicationPlugin - send_updates" and messages like : Certificate in file "/var/kerberos/krb5kdc/kdc.crt" is no longer valid.
To try to come up with some method to fix the servers, I've taken a clone(disconnected from the network) of a non IPA CA renewal master. Below steps I've found on various articles were all done within the clone.
Some things I've tried within the clone out so far along with the errors:
https://floblanc.wordpress.com/2017/09/11/troubleshooting-freeipa-pki-tomcat...
$ sudo grep internal /var/lib/pki/pki-tomcat/conf/password.conf | cut -d= -f2 > /tmp/pwdfile.txt $ sudo certutil -K -d /etc/pki/pki-tomcat/alias -f /tmp/pwdfile.txt -n 'subsystemCert cert-pki-ca' certutil: Checking token "NSS Certificate DB" in slot "NSS User Private Key and Certificate Services" certutil: problem listing keys: SEC_ERROR_INVALID_ARGS: security library: invalid arguments.
I did notice is that the message says "NSS Certificate DB" but on the "getcert list" this certificate shows with token='NSS FIPS 140-2 Certificate DB', not sure if there is an actual problem with the password or if I'm not using the command correctly.
Tried resetting the time back to a point where the expired certs are still valid as mentioned in https://access.redhat.com/solutions/3357261 and to get the pki-tomcatd to come up, tried to update the cert. in LDAP to match what is in /etc/pki/pki-tomcat/alias/ using this method https://access.redhat.com/solutions/3614001 fails for me.
It updates the certificate serial in "description" field but never changes the certificate blob. ldapmodify command and the contents of the .ldif file I used are below.
# ldapmodify -x -h localhost -p 389 -D “cn=directory manager” -w -f updatecert.ldif -v ldap_initializer( ldap://localhost:389 ) replace usercertificate: NOT ASCII (894 bytes) replace description: 2;32;CN=Certificate Authority,O=;CN=CA Subsystem,O=<OUR.DOMAIN> modifying entry “uid=pkidbuser,ou=people,o=ipaca” modify complete
# cat updatecert.ldif dn: uid=pkidbuser,ou=people,o=ipaca changetype: modify replace: usercertificate usercertificate::MII.. – replace: description description: 2;32;CN=Certificate Authority,O=;CN=CA Subsystem,O=<OUR.DOMAIN>
I've used various combinations here even a delete of "userCertificate" field to add the correct one later but none of which worked for me.
I'm hoping someone can point me in the right direction.
Thanks in advance, yajith _______________________________________________ FreeIPA-users mailing list -- freeipa-users@lists.fedorahosted.org To unsubscribe send an email to freeipa-users-leave@lists.fedorahosted.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedorahoste... Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Hi flo,
It was a copy-paste issue. I've also tested it again without the "-" at all where I'm only attempting to update the certificate blob and that doesn't seem to work either even though the ldapmodify doesn't error.
ldap_initializer( ldap://localhost:389 ) replace usercertificate: NOT ASCII (894 bytes) modifying entry “uid=pkidbuser,ou=people,o=ipaca” modify complete
Thanks, yajith
Yajith Dayarathna via FreeIPA-users wrote:
Hi flo,
It was a copy-paste issue. I've also tested it again without the "-" at all where I'm only attempting to update the certificate blob and that doesn't seem to work either even though the ldapmodify doesn't error.
ldap_initializer( ldap://localhost:389 ) replace usercertificate: NOT ASCII (894 bytes) modifying entry “uid=pkidbuser,ou=people,o=ipaca” modify complete
Please define not working. I'd suggest looking at the value stored and see if it begins with LS0 which would imply it is double base64-encoded.
I prefer the file option when loading certs myself. Convert the PEM file to DER:
openssl x509 -in /path/to/cert.pem -out /path/to/cert.der -outform der
In the LDIF use this form:
userCertificate:< file:///path/to/cert.der
To a previous comment, it appears that you're running in FIPS mode which is why the NSS token is different, 'NSS FIPS 140-2 Certificate DB' vs 'NSS Certificate DB'. You can try prefixing the nickname with NSS FIPS 140-2 Certificate DB which will more specifically target that nick.
rob
freeipa-users@lists.fedorahosted.org
-
Florence Blanc-Renaud -
Rob Crittenden -
Yajith Dayarathna