On 11/20/19 8:13 PM, Natxo Asenjo via FreeIPA-users wrote:
hi,
after patching our centos 7 hosts to the latest version today, one of the two replicas is having trouble.
[root@kdc2 ~]# ipactl status Directory Service: RUNNING krb5kdc Service: STOPPED kadmin Service: STOPPED named Service: STOPPED httpd Service: RUNNING ipa-custodia Service: STOPPED ntpd Service: STOPPED pki-tomcatd Service: RUNNING smb Service: STOPPED winbind Service: STOPPED ipa-otpd Service: STOPPED ipa-dnskeysyncd Service: STOPPED ipa: INFO: The ipactl command was successful
and after digging in the logs I come across this in /var/log/ipaupgrade.log:
2019-11-20T18:18:29Z DEBUG stderr= 2019-11-20T18:18:31Z INFO Certmonger certificate renewal configuration already up-to-date 2019-11-20T18:18:31Z INFO [Enable PKIX certificate path discovery and validation] 2019-11-20T18:18:31Z DEBUG Loading StateFile from '/var/lib/ipa/sysupgrade/sysupgrade.state' 2019-11-20T18:18:31Z INFO PKIX already enabled 2019-11-20T18:18:31Z INFO [Authorizing RA Agent to modify profiles] 2019-11-20T18:18:31Z INFO [Authorizing RA Agent to manage lightweight CAs] 2019-11-20T18:18:31Z INFO [Ensuring Lightweight CAs container exists in Dogtag database] 2019-11-20T18:18:31Z DEBUG Created connection context.ldap2_139740162547472 2019-11-20T18:18:31Z DEBUG flushing ldapi://%2fvar%2frun%2fslapd-L-DOMAIN-IT.socket from SchemaCache 2019-11-20T18:18:31Z DEBUG retrieving schema for SchemaCache url=ldapi://%2fvar%2frun%2fslapd-L-DOMAIN-IT.socket conn=<ldap.ldapobject.SimpleLDAPObject instance at 0x7f17cc24b638> 2019-11-20T18:18:31Z DEBUG Destroyed connection context.ldap2_139740162547472 2019-11-20T18:18:31Z INFO [Adding default OCSP URI configuration] 2019-11-20T18:18:31Z INFO [Ensuring CA is using LDAPProfileSubsystem] 2019-11-20T18:18:31Z INFO [Migrating certificate profiles to LDAP] 2019-11-20T18:18:31Z DEBUG Created connection context.ldap2_139740160021648 2019-11-20T18:18:31Z DEBUG flushing ldapi://%2fvar%2frun%2fslapd-L-DOMAIN-IT.socket from SchemaCache 2019-11-20T18:18:31Z DEBUG retrieving schema for SchemaCache url=ldapi://%2fvar%2frun%2fslapd-L-DOMAIN-IT.socket conn=<ldap.ldapobject.SimpleLDAPObject instance at 0x7f17cc289b00> 2019-11-20T18:18:31Z DEBUG Destroyed connection context.ldap2_139740160021648 2019-11-20T18:18:31Z DEBUG request GET https://kdc2.l.domain.it:8443/ca/rest/account/login 2019-11-20T18:18:31Z DEBUG request body '' 2019-11-20T18:18:31Z DEBUG response status 401 2019-11-20T18:18:31Z DEBUG response headers Server: Apache-Coyote/1.1 Cache-Control: private Expires: Thu, 01 Jan 1970 01:00:00 CET WWW-Authenticate: Basic realm="Certificate Authority" Content-Type: text/html;charset=utf-8 Content-Language: en Content-Length: 951 Date: Wed, 20 Nov 2019 18:18:31 GMT
2019-11-20T18:18:31Z DEBUG response body '<html><head><title>Apache Tomcat/7.0.76 - Error report</title><style><!--H1 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:22px;} H2 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:16px;} H3 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:14px;} BODY {font-family:Tahoma,Arial,sans-serif;color:black;background-color:white;} B {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;} P {font-family:Tahoma,Arial,sans-serif;background:white;color:black;font-size:12px;}A {color : black;}A.name {color : black;}HR {color : #525D76;}--></style>
</head><body><h1>HTTP Status 401 - </h1><HR size="1" noshade="noshade"><p><b>type</b> Status report</p><p><b>message</b> <u></u></p><p><b>description</b> <u>This request requires HTTP authentication.</u></p><HR size="1" noshade="noshade"><h3>Apache Tomcat/7.0.76</h3></body></html>' 2019-11-20T18:18:31Z ERROR IPA server upgrade failed: Inspect /var/log/ipaupgrade.log and run command ipa-server-upgrade manually. 2019-11-20T18:18:31Z DEBUG File "/usr/lib/python2.7/site-packages/ipapython/admintool.py", line 178, in execute return_value = self.run() File "/usr/lib/python2.7/site-packages/ipaserver/install/ipa_server_upgrade.py", line 54, in run server.upgrade() File "/usr/lib/python2.7/site-packages/ipaserver/install/server/upgrade.py", line 2146, in upgrade upgrade_configuration() File "/usr/lib/python2.7/site-packages/ipaserver/install/server/upgrade.py", line 2018, in upgrade_configuration ca_enable_ldap_profile_subsystem(ca) File "/usr/lib/python2.7/site-packages/ipaserver/install/server/upgrade.py", line 406, in ca_enable_ldap_profile_subsystem cainstance.migrate_profiles_to_ldap() File "/usr/lib/python2.7/site-packages/ipaserver/install/cainstance.py", line 2027, in migrate_profiles_to_ldap _create_dogtag_profile(profile_id, profile_data, overwrite=False) File "/usr/lib/python2.7/site-packages/ipaserver/install/cainstance.py", line 2033, in _create_dogtag_profile with api.Backend.ra_certprofile as profile_api: File "/usr/lib/python2.7/site-packages/ipaserver/plugins/dogtag.py", line 1315, in __enter__ raise errors.RemoteRetrieveError(reason=_('Failed to authenticate to CA REST API'))
2019-11-20T18:18:31Z DEBUG The ipa-server-upgrade command failed, exception: RemoteRetrieveError: Failed to authenticate to CA REST API 2019-11-20T18:18:31Z ERROR Unexpected error - see /var/log/ipaupgrade.log for details: RemoteRetrieveError: Failed to authenticate to CA REST API
The authentication between IPA and dogtag is done using the ra-agent cert located in /var/lib/ipa/ra-agent.pem. As its expiration date is near, it's possible that the renewal process for this cert started but did not complete successfully.
You need to check the following: - note the serial ID of the cert, its subject and issuer: $ openssl x509 -noout -text -in /var/lib/ipa/ra-agent.pem You can also check if it has already been renewed (look at the date Not Before / Not After).
If it has been renewed, check the content of the entry uid=ipara,ou=people,o=ipaca: $ ldapsearch -D cn=directory\ manager -W -b uid=ipara,ou=people,o=ipaca
There are 2 things to check: - The userCertificate attribute must contain the cert (same value as in ra-agent.pem, in a single line and without the -----BEGIN CERTIFICATE----- and -----END CERTIFICATE---- lines). - The description attribute must have the foollowing value: description: 2;<serial number retrieved in previous step>;<issuer>;<subject>
If it's not the case, it's likely that the renewal failed to update the entry and that may be causing your issue. You will need to manually fix the entry using ldapmodify.
After that, restart ipa with ipactl stop / ipactl start and check if certmonger is able to renew the other certs that will expire soon.
HTH, flo
In this kdc I see these errors in getcert list:
Request ID '20190220182014': status: MONITORING ca-error: Invalid cookie: u'' stuck: no key pair storage: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='auditSigningCert cert-pki-ca',token='NSS Certificate DB',pin set certificate: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='auditSigningCert cert-pki-ca',token='NSS Certificate DB' CA: dogtag-ipa-ca-renew-agent issuer: CN=Certificate Authority,O=L.DOMAIN.IT http://L.DOMAIN.IT subject: CN=CA Audit,O=L.DOMAIN.IT http://L.DOMAIN.IT expires: 2019-12-05 13:58:24 UTC key usage: digitalSignature,nonRepudiation pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert "auditSigningCert cert-pki-ca" track: yes auto-renew: yes Request ID '20190220182015': status: MONITORING ca-error: Invalid cookie: u'' stuck: no key pair storage: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='ocspSigningCert cert-pki-ca',token='NSS Certificate DB',pin set certificate: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='ocspSigningCert cert-pki-ca',token='NSS Certificate DB' CA: dogtag-ipa-ca-renew-agent issuer: CN=Certificate Authority,O=L.DOMAIN.IT http://L.DOMAIN.IT subject: CN=OCSP Subsystem,O=L.DOMAIN.IT http://L.DOMAIN.IT expires: 2019-12-05 13:58:24 UTC key usage: digitalSignature,nonRepudiation,keyCertSign,cRLSign eku: id-kp-OCSPSigning pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert "ocspSigningCert cert-pki-ca" track: yes auto-renew: yes Request ID '20190220182016': status: MONITORING ca-error: Invalid cookie: u'' stuck: no key pair storage: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='subsystemCert cert-pki-ca',token='NSS Certificate DB',pin set certificate: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='subsystemCert cert-pki-ca',token='NSS Certificate DB' CA: dogtag-ipa-ca-renew-agent issuer: CN=Certificate Authority,O=L.DOMAIN.IT http://L.DOMAIN.IT subject: CN=CA Subsystem,O=L.DOMAIN.IT http://L.DOMAIN.IT expires: 2019-12-05 13:58:24 UTC key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment eku: id-kp-serverAuth,id-kp-clientAuth pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert "subsystemCert cert-pki-ca" track: yes auto-renew: yes
Request ID '20190220182018': status: MONITORING ca-error: Invalid cookie: u'' stuck: no key pair storage: type=FILE,location='/var/lib/ipa/ra-agent.key' certificate: type=FILE,location='/var/lib/ipa/ra-agent.pem' CA: dogtag-ipa-ca-renew-agent issuer: CN=Certificate Authority,O=L.DOMAIN.IT http://L.DOMAIN.IT subject: CN=IPA RA,O=L.DOMAIN.IT http://L.DOMAIN.IT expires: 2019-12-05 13:58:44 UTC key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment eku: id-kp-serverAuth,id-kp-clientAuth pre-save command: /usr/libexec/ipa/certmonger/renew_ra_cert_pre post-save command: /usr/libexec/ipa/certmonger/renew_ra_cert track: yes auto-renew: yes Request ID '20190220182019': status: MONITORING ca-error: Server at "https://kdc2.l.domain.it:8443/ca/agent/ca/profileProcess" replied: 1: Invalid Credential. stuck: no key pair storage: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='Server-Cert cert-pki-ca',token='NSS Certificate DB',pin set certificate: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='Server-Cert cert-pki-ca',token='NSS Certificate DB' CA: dogtag-ipa-ca-renew-agent issuer: CN=Certificate Authority,O=L.DOMAIN.IT http://L.DOMAIN.IT subject: CN=kdc2.l.domain.it http://kdc2.l.domain.it,O=L.DOMAIN.IT http://L.DOMAIN.IT expires: 2019-12-10 10:57:52 UTC key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment eku: id-kp-serverAuth,id-kp-clientAuth,id-kp-emailProtection pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert "Server-Cert cert-pki-ca" track: yes auto-renew: yes
I still have a working replica, so I could just reinstall and have a working set in a couple of minutes, but I would like to find out what has gone wrong.
The systems are running ipa-server-4.6.5-11.el7.centos.3.x86_64
Any help welcome ;-)
Thanks,
-- Groeten, natxo
FreeIPA-users mailing list -- freeipa-users@lists.fedorahosted.org To unsubscribe send an email to freeipa-users-leave@lists.fedorahosted.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedorahoste...