Hi,
my IPA system consists of 2 masters with their own self-signed CAs, one of them being the certificate renewal master (ipa1). The system has been running for years and has been migrated from an IPA 3 system.
Since a while, the Web UI logins on ipa1 don't work anymore ("Login failed due to an unknown reason.").
Web UI logins on the other server (ipa2) work and everything else is working fine, too, ipactl status reports all services running.
On login attempt:
--- httpd log [...] [:error] [pid 15551] [remote 141.51.X.X:0] mod_wsgi (pid=15551): Exception occurred processing WSGI script '/usr/share/ipa/wsgi.py'. [...] [:error] [pid 15551] [remote 141.51.X.X:0] CalledProcessError: Command '/usr/bin/kinit -n -c /var/run/ipa/ccaches/armor_15551 -X X509_anchors=FILE:/var/kerberos/krb5kdc/kdc.crt -X X509_anchors=FILE:/var/lib/ipa-client/pki/kdc-ca-bundle.pem' returned non-zero exit status 1 ---
--- krb5kdc.log [...] Dec 20 16:06:54 ipa1.example.com krb5kdc[15517](info): AS_REQ (8 etypes {18 17 20 19 16 23 25 26}) 141.51.X.Y: NEEDED_PREAUTH: WELLKNOWN/ANONYMOUS@EXAMPLE.COM for krbtgt/EXAMPLE.COM@EXAMPLE.COM, Additional pre-authentication required Dec 20 16:06:54 ipa1.example.com krb5kdc[15517](info): closing down fd 11 Dec 20 16:06:54 ipa1.example.com krb5kdc[15518](info): AS_REQ (8 etypes {18 17 20 19 16 23 25 26}) 141.51.X.Y: KDC_RETURN_PADATA: WELLKNOWN/ANONYMOUS@EXAMPLE.COM for krbtgt/EXAMPLE.COM@EXAMPLE.COM, Failed to verify own certificate (depth 0): certificate has expired Dec 20 16:06:54 ipa1.example.com krb5kdc[15518](info): closing down fd 11 ---
--- ipa-checkcerts.py IPA version 4.5.4-10.el7.centos.3 Check CA status Check tracking Check NSS trust Check dates Checking certificates in CS.cfg Comparing certificates to requests in LDAP Checking RA certificate Checking authorities Checking host keytab Validating certificates Checking renewal master End-to-end cert API test Checking permissions and ownership Failures: Unable to find request for serial 268304391 Unable to find request for serial 268304394 Unable to find request for serial 268304393 Unable to find request for serial 268304392 Subject O=EXAMPLE.COM,CN=ipa2.example.com and template subject CN=ipa1.example.com,O=EXAMPLE.COM do not match for serial 57 ---
--- ipa pkinit-status --all ----------------- 2 servers matched ----------------- Server name: ipa2.example.com PKINIT status: enabled
Server name: ipa1.example.com PKINIT status: enabled ---------------------------- Number of entries returned 2 ----------------------------
To my understanding, proper certificate exchange between my two servers ceased working at some point. How do i track this down and fix it?
Mit freundlichen Gruessen/With best regards,
--Daniel.
On 12/20/18 4:22 PM, dbischof--- via FreeIPA-users wrote:
Hi,
my IPA system consists of 2 masters with their own self-signed CAs, one of them being the certificate renewal master (ipa1). The system has been running for years and has been migrated from an IPA 3 system.
Since a while, the Web UI logins on ipa1 don't work anymore ("Login failed due to an unknown reason.").
Web UI logins on the other server (ipa2) work and everything else is working fine, too, ipactl status reports all services running.
On login attempt:
--- httpd log [...] [:error] [pid 15551] [remote 141.51.X.X:0] mod_wsgi (pid=15551): Exception occurred processing WSGI script '/usr/share/ipa/wsgi.py'. [...] [:error] [pid 15551] [remote 141.51.X.X:0] CalledProcessError: Command '/usr/bin/kinit -n -c /var/run/ipa/ccaches/armor_15551 -X X509_anchors=FILE:/var/kerberos/krb5kdc/kdc.crt -X X509_anchors=FILE:/var/lib/ipa-client/pki/kdc-ca-bundle.pem' returned non-zero exit status 1
--- krb5kdc.log [...] Dec 20 16:06:54 ipa1.example.com krb5kdc[15517](info): AS_REQ (8 etypes {18 17 20 19 16 23 25 26}) 141.51.X.Y: NEEDED_PREAUTH: WELLKNOWN/ANONYMOUS@EXAMPLE.COM for krbtgt/EXAMPLE.COM@EXAMPLE.COM, Additional pre-authentication required Dec 20 16:06:54 ipa1.example.com krb5kdc[15517](info): closing down fd 11 Dec 20 16:06:54 ipa1.example.com krb5kdc[15518](info): AS_REQ (8 etypes {18 17 20 19 16 23 25 26}) 141.51.X.Y: KDC_RETURN_PADATA: WELLKNOWN/ANONYMOUS@EXAMPLE.COM for krbtgt/EXAMPLE.COM@EXAMPLE.COM, Failed to verify own certificate (depth 0): certificate has expired Dec 20 16:06:54 ipa1.example.com krb5kdc[15518](info): closing down fd 11
--- ipa-checkcerts.py IPA version 4.5.4-10.el7.centos.3 Check CA status Check tracking Check NSS trust Check dates Checking certificates in CS.cfg Comparing certificates to requests in LDAP Checking RA certificate Checking authorities Checking host keytab Validating certificates Checking renewal master End-to-end cert API test Checking permissions and ownership Failures: Unable to find request for serial 268304391 Unable to find request for serial 268304394 Unable to find request for serial 268304393 Unable to find request for serial 268304392 Subject O=EXAMPLE.COM,CN=ipa2.example.com and template subject CN=ipa1.example.com,O=EXAMPLE.COM do not match for serial 57
--- ipa pkinit-status --all
2 servers matched
Server name: ipa2.example.com PKINIT status: enabled
Server name: ipa1.example.com PKINIT status: enabled
Number of entries returned 2
To my understanding, proper certificate exchange between my two servers ceased working at some point. How do i track this down and fix it?
Hi,
your issue looks similar to ticket #6792 [1]. Can you check the result of upgrade in /var/log/ipaupgrade.log? Also check the output of $ ipa-pkinit-manage status and if the files /var/lib/ipa-client/pki/kdc-ca-bundle.pem and /var/lib/ipa-client/pki/ca-bundle.pem exist, with -rw-r--r-- permissions.
Regarding the certificates, does getcert list show expired certs? flo
[1] https://pagure.io/freeipa/issue/6792
Mit freundlichen Gruessen/With best regards,
--Daniel. _______________________________________________ FreeIPA-users mailing list -- freeipa-users@lists.fedorahosted.org To unsubscribe send an email to freeipa-users-leave@lists.fedorahosted.org Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedorahoste...
Hi Florence,
On Thu, 20 Dec 2018, Florence Blanc-Renaud via FreeIPA-users wrote:
On 12/20/18 4:22 PM, dbischof--- via FreeIPA-users wrote:
my IPA system consists of 2 masters with their own self-signed CAs, one of them being the certificate renewal master (ipa1). The system has been running for years and has been migrated from an IPA 3 system.
Since a while, the Web UI logins on ipa1 don't work anymore ("Login failed due to an unknown reason.").
Web UI logins on the other server (ipa2) work and everything else is working fine, too, ipactl status reports all services running.
On login attempt:
--- httpd log [...] [:error] [pid 15551] [remote 141.51.X.X:0] mod_wsgi (pid=15551): Exception occurred processing WSGI script '/usr/share/ipa/wsgi.py'. [...] [:error] [pid 15551] [remote 141.51.X.X:0] CalledProcessError: Command '/usr/bin/kinit -n -c /var/run/ipa/ccaches/armor_15551 -X X509_anchors=FILE:/var/kerberos/krb5kdc/kdc.crt -X X509_anchors=FILE:/var/lib/ipa-client/pki/kdc-ca-bundle.pem' returned non-zero exit status 1
--- krb5kdc.log [...] Dec 20 16:06:54 ipa1.example.com krb5kdc[15517](info): AS_REQ (8 etypes {18 17 20 19 16 23 25 26}) 141.51.X.Y: NEEDED_PREAUTH: WELLKNOWN/ANONYMOUS@EXAMPLE.COM for krbtgt/EXAMPLE.COM@EXAMPLE.COM, Additional pre-authentication required Dec 20 16:06:54 ipa1.example.com krb5kdc[15517](info): closing down fd 11 Dec 20 16:06:54 ipa1.example.com krb5kdc[15518](info): AS_REQ (8 etypes {18 17 20 19 16 23 25 26}) 141.51.X.Y: KDC_RETURN_PADATA: WELLKNOWN/ANONYMOUS@EXAMPLE.COM for krbtgt/EXAMPLE.COM@EXAMPLE.COM, Failed to verify own certificate (depth 0): certificate has expired Dec 20 16:06:54 ipa1.example.com krb5kdc[15518](info): closing down fd 11
--- ipa-checkcerts.py IPA version 4.5.4-10.el7.centos.3 Check CA status Check tracking Check NSS trust Check dates Checking certificates in CS.cfg Comparing certificates to requests in LDAP Checking RA certificate Checking authorities Checking host keytab Validating certificates Checking renewal master End-to-end cert API test Checking permissions and ownership Failures: Unable to find request for serial 268304391 Unable to find request for serial 268304394 Unable to find request for serial 268304393 Unable to find request for serial 268304392 Subject O=EXAMPLE.COM,CN=ipa2.example.com and template subject CN=ipa1.example.com,O=EXAMPLE.COM do not match for serial 57
--- ipa pkinit-status --all
2 servers matched
Server name: ipa2.example.com PKINIT status: enabled
Server name: ipa1.example.com PKINIT status: enabled
Number of entries returned 2
To my understanding, proper certificate exchange between my two servers ceased working at some point. How do i track this down and fix it?
your issue looks similar to ticket #6792 [1]. Can you check the result of upgrade in /var/log/ipaupgrade.log? Also check the output of $ ipa-pkinit-manage status and if the files /var/lib/ipa-client/pki/kdc-ca-bundle.pem and /var/lib/ipa-client/pki/ca-bundle.pem exist, with -rw-r--r-- permissions.
Regarding the certificates, does getcert list show expired certs? flo
--- $ ipa-pkinit-manage status PKINIT is enabled ---
There are no expired certificates, kdc-ca-bundle.pem and ca-bundle.pem exist with proper permissions, but I found something in ipaupgrade.log:
--- 2018-09-12T13:37:18Z DEBUG args=/usr/bin/certutil -d /etc/httpd/alias -L -f /etc/httpd/alias/pwdfile.txt 2018-09-12T13:37:19Z DEBUG Process finished, return code=0 2018-09-12T13:37:19Z DEBUG stdout= Certificate Nickname Trust Attributes
SSL,S/MIME,JAR/XPI
Server-Cert u,u,u EXAMPLE.COM IPA CA CT,C,C
2018-09-12T13:37:19Z DEBUG stderr= 2018-09-12T13:37:19Z DEBUG Starting external process 2018-09-12T13:37:19Z DEBUG args=/usr/bin/certutil -d /etc/httpd/alias -L -n EXAMPLE.COM IPA CA -a -f /etc/httpd/alias/pwdfile.txt 2018-09-12T13:37:19Z DEBUG Process finished, return code=0 2018-09-12T13:37:19Z DEBUG stdout= -----BEGIN CERTIFICATE----- [...] -----END CERTIFICATE-----
2018-09-12T13:37:19Z DEBUG stderr= 2018-09-12T13:37:19Z DEBUG Executing upgrade plugin: update_ra_cert_store 2018-09-12T13:37:19Z DEBUG raw: update_ra_cert_store 2018-09-12T13:37:19Z DEBUG raw: ca_is_enabled(version=u'2.228') 2018-09-12T13:37:19Z DEBUG ca_is_enabled(version=u'2.228') 2018-09-12T13:37:19Z DEBUG Starting external process 2018-09-12T13:37:19Z DEBUG args=/usr/bin/certutil -d /etc/httpd/alias -L -n ipaCert -a -f /etc/httpd/alias/pwdfile.txt 2018-09-12T13:37:19Z DEBUG Process finished, return code=255 2018-09-12T13:37:19Z DEBUG stdout= 2018-09-12T13:37:19Z DEBUG stderr=certutil: Could not find cert: ipaCert : PR_FILE_NOT_FOUND_ERROR: File not found [...] ---
Mit freundlichen Gruessen/With best regards,
--Daniel.
On 12/20/18 6:52 PM, dbischof--- via FreeIPA-users wrote:
Hi Florence,
On Thu, 20 Dec 2018, Florence Blanc-Renaud via FreeIPA-users wrote:
On 12/20/18 4:22 PM, dbischof--- via FreeIPA-users wrote:
my IPA system consists of 2 masters with their own self-signed CAs, one of them being the certificate renewal master (ipa1). The system has been running for years and has been migrated from an IPA 3 system.
Since a while, the Web UI logins on ipa1 don't work anymore ("Login failed due to an unknown reason.").
Web UI logins on the other server (ipa2) work and everything else is working fine, too, ipactl status reports all services running.
On login attempt:
--- httpd log [...] [:error] [pid 15551] [remote 141.51.X.X:0] mod_wsgi (pid=15551): Exception occurred processing WSGI script '/usr/share/ipa/wsgi.py'. [...] [:error] [pid 15551] [remote 141.51.X.X:0] CalledProcessError: Command '/usr/bin/kinit -n -c /var/run/ipa/ccaches/armor_15551 -X X509_anchors=FILE:/var/kerberos/krb5kdc/kdc.crt -X X509_anchors=FILE:/var/lib/ipa-client/pki/kdc-ca-bundle.pem' returned non-zero exit status 1 ---
--- krb5kdc.log [...] Dec 20 16:06:54 ipa1.example.com krb5kdc[15517](info): AS_REQ (8 etypes {18 17 20 19 16 23 25 26}) 141.51.X.Y: NEEDED_PREAUTH: WELLKNOWN/ANONYMOUS@EXAMPLE.COM for krbtgt/EXAMPLE.COM@EXAMPLE.COM, Additional pre-authentication required Dec 20 16:06:54 ipa1.example.com krb5kdc[15517](info): closing down fd 11 Dec 20 16:06:54 ipa1.example.com krb5kdc[15518](info): AS_REQ (8 etypes {18 17 20 19 16 23 25 26}) 141.51.X.Y: KDC_RETURN_PADATA: WELLKNOWN/ANONYMOUS@EXAMPLE.COM for krbtgt/EXAMPLE.COM@EXAMPLE.COM, Failed to verify own certificate (depth 0): certificate has expired Dec 20 16:06:54 ipa1.example.com krb5kdc[15518](info): closing down fd 11 ---
--- ipa-checkcerts.py IPA version 4.5.4-10.el7.centos.3 Check CA status Check tracking Check NSS trust Check dates Checking certificates in CS.cfg Comparing certificates to requests in LDAP Checking RA certificate Checking authorities Checking host keytab Validating certificates Checking renewal master End-to-end cert API test Checking permissions and ownership Failures: Unable to find request for serial 268304391 Unable to find request for serial 268304394 Unable to find request for serial 268304393 Unable to find request for serial 268304392 Subject O=EXAMPLE.COM,CN=ipa2.example.com and template subject CN=ipa1.example.com,O=EXAMPLE.COM do not match for serial 57 ---
--- ipa pkinit-status --all ----------------- 2 servers matched ----------------- Server name: ipa2.example.com PKINIT status: enabled
Server name: ipa1.example.com PKINIT status: enabled ---------------------------- Number of entries returned 2 ----------------------------
To my understanding, proper certificate exchange between my two servers ceased working at some point. How do i track this down and fix it?
your issue looks similar to ticket #6792 [1]. Can you check the result of upgrade in /var/log/ipaupgrade.log? Also check the output of $ ipa-pkinit-manage status and if the files /var/lib/ipa-client/pki/kdc-ca-bundle.pem and /var/lib/ipa-client/pki/ca-bundle.pem exist, with -rw-r--r-- permissions.
Regarding the certificates, does getcert list show expired certs? flo
$ ipa-pkinit-manage status PKINIT is enabled
There are no expired certificates, kdc-ca-bundle.pem and ca-bundle.pem exist with proper permissions, but I found something in ipaupgrade.log:
2018-09-12T13:37:18Z DEBUG args=/usr/bin/certutil -d /etc/httpd/alias -L -f /etc/httpd/alias/pwdfile.txt 2018-09-12T13:37:19Z DEBUG Process finished, return code=0 2018-09-12T13:37:19Z DEBUG stdout= Certificate Nickname Trust Attributes
SSL,S/MIME,JAR/XPI
Server-Cert u,u,u EXAMPLE.COM IPA CA CT,C,C
2018-09-12T13:37:19Z DEBUG stderr= 2018-09-12T13:37:19Z DEBUG Starting external process 2018-09-12T13:37:19Z DEBUG args=/usr/bin/certutil -d /etc/httpd/alias -L -n EXAMPLE.COM IPA CA -a -f /etc/httpd/alias/pwdfile.txt 2018-09-12T13:37:19Z DEBUG Process finished, return code=0 2018-09-12T13:37:19Z DEBUG stdout= -----BEGIN CERTIFICATE----- [...] -----END CERTIFICATE-----
2018-09-12T13:37:19Z DEBUG stderr= 2018-09-12T13:37:19Z DEBUG Executing upgrade plugin: update_ra_cert_store 2018-09-12T13:37:19Z DEBUG raw: update_ra_cert_store 2018-09-12T13:37:19Z DEBUG raw: ca_is_enabled(version=u'2.228') 2018-09-12T13:37:19Z DEBUG ca_is_enabled(version=u'2.228') 2018-09-12T13:37:19Z DEBUG Starting external process 2018-09-12T13:37:19Z DEBUG args=/usr/bin/certutil -d /etc/httpd/alias -L -n ipaCert -a -f /etc/httpd/alias/pwdfile.txt 2018-09-12T13:37:19Z DEBUG Process finished, return code=255 2018-09-12T13:37:19Z DEBUG stdout= 2018-09-12T13:37:19Z DEBUG stderr=certutil: Could not find cert: ipaCert : PR_FILE_NOT_FOUND_ERROR: File not found [...]
Hi, this error can be ignored in most of the cases. The upgrade is trying to move ipaCert (cert+key) from the NSS database /etc/httpd/alias to the files /var/lib/ipa/ra-agent.pem and /var/lib/ipa/ra-agent.key. So if the upgrade is run a second time, he won't find ipaCert in the NSS database. To be sure, you can check if /var/lib/ipa/ra-agent.{pem|key} are present and contain a certificate with Subject CN=IPA RA,O=DOMAIN.COM. The files must be readable by root and ipaapi group, and must contain the same cert as the other masters.
What is the content of /var/lib/ipa-client/pki/kdc-ca-bundle.pem and ca-bundle.pem? both must contain IPA CA certificate.
What are the permissions of /var/kerberos/krb5kdc/kdc.crt? It needs to be readable by everyone. And what is the content of this cert? It should be issued by IPA CA.
About the errors spotted by ipa-checkcerts.py, what are the certificates with errors reported? You can find them with ipa cert-show <serial>.
flo
Mit freundlichen Gruessen/With best regards,
--Daniel. _______________________________________________ FreeIPA-users mailing list -- freeipa-users@lists.fedorahosted.org To unsubscribe send an email to freeipa-users-leave@lists.fedorahosted.org Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedorahoste...
Hi Florence,
thank you very much for your help.
On Fri, 21 Dec 2018, Florence Blanc-Renaud via FreeIPA-users wrote:
On 12/20/18 6:52 PM, dbischof--- via FreeIPA-users wrote:
On Thu, 20 Dec 2018, Florence Blanc-Renaud via FreeIPA-users wrote:
On 12/20/18 4:22 PM, dbischof--- via FreeIPA-users wrote:
my IPA system consists of 2 masters with their own self-signed CAs, one of them being the certificate renewal master (ipa1). The system has been running for years and has been migrated from an IPA 3 system.
Since a while, the Web UI logins on ipa1 don't work anymore ("Login failed due to an unknown reason.").
Web UI logins on the other server (ipa2) work and everything else is working fine, too, ipactl status reports all services running.
On login attempt:
--- httpd log [...] [:error] [pid 15551] [remote 141.51.X.X:0] mod_wsgi (pid=15551): Exception occurred processing WSGI script '/usr/share/ipa/wsgi.py'. [...] [:error] [pid 15551] [remote 141.51.X.X:0] CalledProcessError: Command '/usr/bin/kinit -n -c /var/run/ipa/ccaches/armor_15551 -X X509_anchors=FILE:/var/kerberos/krb5kdc/kdc.crt -X X509_anchors=FILE:/var/lib/ipa-client/pki/kdc-ca-bundle.pem' returned non-zero exit status 1 ---
--- krb5kdc.log [...] Dec 20 16:06:54 ipa1.example.com krb5kdc[15517](info): AS_REQ (8 etypes {18 17 20 19 16 23 25 26}) 141.51.X.Y: NEEDED_PREAUTH: WELLKNOWN/ANONYMOUS@EXAMPLE.COM for krbtgt/EXAMPLE.COM@EXAMPLE.COM, Additional pre-authentication required Dec 20 16:06:54 ipa1.example.com krb5kdc[15517](info): closing down fd 11 Dec 20 16:06:54 ipa1.example.com krb5kdc[15518](info): AS_REQ (8 etypes {18 17 20 19 16 23 25 26}) 141.51.X.Y: KDC_RETURN_PADATA: WELLKNOWN/ANONYMOUS@EXAMPLE.COM for krbtgt/EXAMPLE.COM@EXAMPLE.COM, Failed to verify own certificate (depth 0): certificate has expired Dec 20 16:06:54 ipa1.example.com krb5kdc[15518](info): closing down fd 11 ---
--- ipa-checkcerts.py IPA version 4.5.4-10.el7.centos.3 Check CA status Check tracking Check NSS trust Check dates Checking certificates in CS.cfg Comparing certificates to requests in LDAP Checking RA certificate Checking authorities Checking host keytab Validating certificates Checking renewal master End-to-end cert API test Checking permissions and ownership Failures: Unable to find request for serial 268304391 Unable to find request for serial 268304394 Unable to find request for serial 268304393 Unable to find request for serial 268304392 Subject O=EXAMPLE.COM,CN=ipa2.example.com and template subject CN=ipa1.example.com,O=EXAMPLE.COM do not match for serial 57 ---
--- ipa pkinit-status --all ----------------- 2 servers matched ----------------- Server name: ipa2.example.com PKINIT status: enabled
Server name: ipa1.example.com PKINIT status: enabled ---------------------------- Number of entries returned 2 ----------------------------
To my understanding, proper certificate exchange between my two servers ceased working at some point. How do i track this down and fix it?
your issue looks similar to ticket #6792 [1]. Can you check the result of upgrade in /var/log/ipaupgrade.log? Also check the output of $ ipa-pkinit-manage status and if the files /var/lib/ipa-client/pki/kdc-ca-bundle.pem and /var/lib/ipa-client/pki/ca-bundle.pem exist, with -rw-r--r-- permissions.
Regarding the certificates, does getcert list show expired certs? flo
$ ipa-pkinit-manage status PKINIT is enabled
There are no expired certificates, kdc-ca-bundle.pem and ca-bundle.pem exist with proper permissions, but I found something in ipaupgrade.log:
2018-09-12T13:37:18Z DEBUG args=/usr/bin/certutil -d /etc/httpd/alias -L -f /etc/httpd/alias/pwdfile.txt 2018-09-12T13:37:19Z DEBUG Process finished, return code=0 2018-09-12T13:37:19Z DEBUG stdout= Certificate Nickname Trust Attributes
SSL,S/MIME,JAR/XPI
Server-Cert u,u,u EXAMPLE.COM IPA CA CT,C,C
2018-09-12T13:37:19Z DEBUG stderr= 2018-09-12T13:37:19Z DEBUG Starting external process 2018-09-12T13:37:19Z DEBUG args=/usr/bin/certutil -d /etc/httpd/alias -L -n EXAMPLE.COM IPA CA -a -f /etc/httpd/alias/pwdfile.txt 2018-09-12T13:37:19Z DEBUG Process finished, return code=0 2018-09-12T13:37:19Z DEBUG stdout= -----BEGIN CERTIFICATE----- [...] -----END CERTIFICATE-----
2018-09-12T13:37:19Z DEBUG stderr= 2018-09-12T13:37:19Z DEBUG Executing upgrade plugin: update_ra_cert_store 2018-09-12T13:37:19Z DEBUG raw: update_ra_cert_store 2018-09-12T13:37:19Z DEBUG raw: ca_is_enabled(version=u'2.228') 2018-09-12T13:37:19Z DEBUG ca_is_enabled(version=u'2.228') 2018-09-12T13:37:19Z DEBUG Starting external process 2018-09-12T13:37:19Z DEBUG args=/usr/bin/certutil -d /etc/httpd/alias -L -n ipaCert -a -f /etc/httpd/alias/pwdfile.txt 2018-09-12T13:37:19Z DEBUG Process finished, return code=255 2018-09-12T13:37:19Z DEBUG stdout= 2018-09-12T13:37:19Z DEBUG stderr=certutil: Could not find cert: ipaCert : PR_FILE_NOT_FOUND_ERROR: File not found [...]
this error can be ignored in most of the cases. The upgrade is trying to move ipaCert (cert+key) from the NSS database /etc/httpd/alias to the files /var/lib/ipa/ra-agent.pem and /var/lib/ipa/ra-agent.key. So if the upgrade is run a second time, he won't find ipaCert in the NSS database. To be sure, you can check if /var/lib/ipa/ra-agent.{pem|key} are present and contain a certificate with Subject CN=IPA RA,O=DOMAIN.COM. The files must be readable by root and ipaapi group, and must contain the same cert as the other masters.
checked this, is ok.
What is the content of /var/lib/ipa-client/pki/kdc-ca-bundle.pem and ca-bundle.pem? both must contain IPA CA certificate.
True on both servers.
What are the permissions of /var/kerberos/krb5kdc/kdc.crt? It needs to be readable by everyone. And what is the content of this cert? It should be issued by IPA CA.
Permissions are ok, contents:
--- ipa1 $ openssl x509 -in /var/kerberos/krb5kdc/kdc.crt -text -noout -fingerprint Certificate: Data: Version: 3 (0x2) Serial Number: 1 (0x1) Signature Algorithm: sha256WithRSAEncryption Issuer: O=EXAMPLE.COM, CN=ipa1.example.com Validity Not Before: Nov 28 12:43:05 2017 GMT Not After : Nov 28 12:43:05 2018 GMT Subject: O=EXAMPLE.COM, CN=ipa1.example.com Subject Public Key Info: Public Key Algorithm: rsaEncryption Public-Key: (2048 bit) Modulus: [...] Exponent: 65537 (0x10001) X509v3 extensions: X509v3 Subject Alternative Name: othername:<unsupported>, othername:<unsupported> X509v3 Basic Constraints: critical CA:FALSE X509v3 Subject Key Identifier: 86:52:EC:A1:C3:FB:EC:CC:6D:F2:09:E7:64:88:D1:80:F4:71:81:AE 1.3.6.1.4.1.311.20.2: .".K.D.C.s._.P.K.I.N.I.T._.C.e.r.t.s [...] ---
--- ipa2 $ openssl x509 -in /var/kerberos/krb5kdc/kdc.crt -text -noout -fingerprint Certificate: Data: Version: 3 (0x2) Serial Number: 805240833 (0x2fff0001) Signature Algorithm: sha256WithRSAEncryption Issuer: O=EXAMPLE.COM, CN=Certificate Authority Validity Not Before: Jan 18 13:04:17 2018 GMT Not After : Jan 19 13:04:17 2020 GMT Subject: O=EXAMPLE.COM, CN=ipa2.example.com Subject Public Key Info: Public Key Algorithm: rsaEncryption Public-Key: (2048 bit) Modulus: [...] Exponent: 65537 (0x10001) X509v3 extensions: X509v3 Authority Key Identifier: keyid:4B:BA:AA:46:F1:29:E4:43:8B:DC:30:B4:90:3E:66:72:DD:F6:C7:FB
Authority Information Access: OCSP - URI:http://ipa-ca.example.com/ca/ocsp
X509v3 Key Usage: critical Digital Signature, Non Repudiation, Key Encipherment, Data Encipherment X509v3 Extended Key Usage: TLS Web Server Authentication, 1.3.6.1.5.2.3.5 X509v3 CRL Distribution Points:
Full Name: URI:http://ipa-ca.example.com/ipa/crl/MasterCRL.bin CRL Issuer: DirName: O = ipaca, CN = Certificate Authority
X509v3 Subject Key Identifier: 96:C3:94:70:7E:46:77:DB:91:F8:DF:D6:27:FE:73:0A:45:F3:78:F3 X509v3 Subject Alternative Name: othername:<unsupported>, othername:<unsupported> [...] ---
Additional info: I have DNS separate from IPA, but i (hopefully) made proper records as IPA would have done it. In particular, i made an A record "ipa-ca" that has IPs of both ipa1 and ipa2 - hope, this is not the root cause of my problems, since DNS is not under my control.
Since /var/kerberos/krb5kdc/kdc.crt on ipa1 appears to be not issued by IPA CA, might this be the actual problem?
About the errors spotted by ipa-checkcerts.py, what are the certificates with errors reported? You can find them with ipa cert-show <serial>.
--- $ipa cert-show 57 Issuing CA: ipa Certificate: [...] Subject: CN=ipa1.example.com,O=EXAMPLE.COM Issuer: CN=Certificate Authority,O=EXAMPLE.COM Not Before: Tue Nov 28 12:42:07 2017 UTC Not After: Mon Nov 18 12:42:07 2019 UTC Serial number: 57 Serial number (hex): 0x39 Revoked: False
$ ipa cert-show 268304391 Issuing CA: ipa Certificate: [...] Subject: CN=IPA RA,O=EXAMPLE.COM Issuer: CN=Certificate Authority,O=EXAMPLE.COM Not Before: Wed Jul 11 14:26:35 2018 UTC Not After: Tue Jun 30 14:26:35 2020 UTC Serial number: 268304391 Serial number (hex): 0xFFE0007 Revoked: False
$ ipa cert-show 268304392 Issuing CA: ipa Certificate: [...] Subject: CN=CA Subsystem,O=EXAMPLE.COM Issuer: CN=Certificate Authority,O=EXAMPLE.COM Not Before: Wed Jul 11 14:26:45 2018 UTC Not After: Tue Jun 30 14:26:45 2020 UTC Serial number: 268304392 Serial number (hex): 0xFFE0008 Revoked: False
$ ipa cert-show 268304393 Issuing CA: ipa Certificate: [...] Subject: CN=OCSP Subsystem,O=EXAMPLE.COM Issuer: CN=Certificate Authority,O=EXAMPLE.COM Not Before: Wed Jul 11 14:27:05 2018 UTC Not After: Tue Jun 30 14:27:05 2020 UTC Serial number: 268304393 Serial number (hex): 0xFFE0009 Revoked: False
$ ipa cert-show 268304394 Issuing CA: ipa Certificate: [...] Subject: CN=CA Audit,O=EXAMPLE.COM Issuer: CN=Certificate Authority,O=EXAMPLE.COM Not Before: Wed Jul 11 14:27:25 2018 UTC Not After: Tue Jun 30 14:27:25 2020 UTC Serial number: 268304394 Serial number (hex): 0xFFE000A Revoked: False ---
ipa cert-show on server ipa2 fails with "ipa: ERROR: Certificate operation cannot be completed: EXCEPTION (Invalid Credential.)" btw.
Mit freundlichen Gruessen/With best regards,
--Daniel.
On 12/21/18 11:58 AM, dbischof--- via FreeIPA-users wrote:
Hi Florence,
thank you very much for your help.
On Fri, 21 Dec 2018, Florence Blanc-Renaud via FreeIPA-users wrote:
On 12/20/18 6:52 PM, dbischof--- via FreeIPA-users wrote:
On Thu, 20 Dec 2018, Florence Blanc-Renaud via FreeIPA-users wrote:
On 12/20/18 4:22 PM, dbischof--- via FreeIPA-users wrote:
my IPA system consists of 2 masters with their own self-signed CAs, one of them being the certificate renewal master (ipa1). The system has been running for years and has been migrated from an IPA 3 system.
Since a while, the Web UI logins on ipa1 don't work anymore ("Login failed due to an unknown reason.").
Web UI logins on the other server (ipa2) work and everything else is working fine, too, ipactl status reports all services running.
On login attempt:
--- httpd log [...] [:error] [pid 15551] [remote 141.51.X.X:0] mod_wsgi (pid=15551): Exception occurred processing WSGI script '/usr/share/ipa/wsgi.py'. [...] [:error] [pid 15551] [remote 141.51.X.X:0] CalledProcessError: Command '/usr/bin/kinit -n -c /var/run/ipa/ccaches/armor_15551 -X X509_anchors=FILE:/var/kerberos/krb5kdc/kdc.crt -X X509_anchors=FILE:/var/lib/ipa-client/pki/kdc-ca-bundle.pem' returned non-zero exit status 1 ---
--- krb5kdc.log [...] Dec 20 16:06:54 ipa1.example.com krb5kdc[15517](info): AS_REQ (8 etypes {18 17 20 19 16 23 25 26}) 141.51.X.Y: NEEDED_PREAUTH: WELLKNOWN/ANONYMOUS@EXAMPLE.COM for krbtgt/EXAMPLE.COM@EXAMPLE.COM, Additional pre-authentication required Dec 20 16:06:54 ipa1.example.com krb5kdc[15517](info): closing down fd 11 Dec 20 16:06:54 ipa1.example.com krb5kdc[15518](info): AS_REQ (8 etypes {18 17 20 19 16 23 25 26}) 141.51.X.Y: KDC_RETURN_PADATA: WELLKNOWN/ANONYMOUS@EXAMPLE.COM for krbtgt/EXAMPLE.COM@EXAMPLE.COM, Failed to verify own certificate (depth 0): certificate has expired Dec 20 16:06:54 ipa1.example.com krb5kdc[15518](info): closing down fd 11 ---
--- ipa-checkcerts.py IPA version 4.5.4-10.el7.centos.3 Check CA status Check tracking Check NSS trust Check dates Checking certificates in CS.cfg Comparing certificates to requests in LDAP Checking RA certificate Checking authorities Checking host keytab Validating certificates Checking renewal master End-to-end cert API test Checking permissions and ownership Failures: Unable to find request for serial 268304391 Unable to find request for serial 268304394 Unable to find request for serial 268304393 Unable to find request for serial 268304392 Subject O=EXAMPLE.COM,CN=ipa2.example.com and template subject CN=ipa1.example.com,O=EXAMPLE.COM do not match for serial 57 ---
--- ipa pkinit-status --all ----------------- 2 servers matched ----------------- Server name: ipa2.example.com PKINIT status: enabled
Server name: ipa1.example.com PKINIT status: enabled ---------------------------- Number of entries returned 2 ----------------------------
To my understanding, proper certificate exchange between my two servers ceased working at some point. How do i track this down and fix it?
your issue looks similar to ticket #6792 [1]. Can you check the result of upgrade in /var/log/ipaupgrade.log? Also check the output of $ ipa-pkinit-manage status and if the files /var/lib/ipa-client/pki/kdc-ca-bundle.pem and /var/lib/ipa-client/pki/ca-bundle.pem exist, with -rw-r--r-- permissions.
Regarding the certificates, does getcert list show expired certs? flo
--- $ ipa-pkinit-manage status PKINIT is enabled ---
There are no expired certificates, kdc-ca-bundle.pem and ca-bundle.pem exist with proper permissions, but I found something in ipaupgrade.log:
--- 2018-09-12T13:37:18Z DEBUG args=/usr/bin/certutil -d /etc/httpd/alias -L -f /etc/httpd/alias/pwdfile.txt 2018-09-12T13:37:19Z DEBUG Process finished, return code=0 2018-09-12T13:37:19Z DEBUG stdout= Certificate Nickname Trust Attributes
SSL,S/MIME,JAR/XPI
Server-Cert u,u,u EXAMPLE.COM IPA CA CT,C,C
2018-09-12T13:37:19Z DEBUG stderr= 2018-09-12T13:37:19Z DEBUG Starting external process 2018-09-12T13:37:19Z DEBUG args=/usr/bin/certutil -d /etc/httpd/alias -L -n EXAMPLE.COM IPA CA -a -f /etc/httpd/alias/pwdfile.txt 2018-09-12T13:37:19Z DEBUG Process finished, return code=0 2018-09-12T13:37:19Z DEBUG stdout= -----BEGIN CERTIFICATE----- [...] -----END CERTIFICATE-----
2018-09-12T13:37:19Z DEBUG stderr= 2018-09-12T13:37:19Z DEBUG Executing upgrade plugin: update_ra_cert_store 2018-09-12T13:37:19Z DEBUG raw: update_ra_cert_store 2018-09-12T13:37:19Z DEBUG raw: ca_is_enabled(version=u'2.228') 2018-09-12T13:37:19Z DEBUG ca_is_enabled(version=u'2.228') 2018-09-12T13:37:19Z DEBUG Starting external process 2018-09-12T13:37:19Z DEBUG args=/usr/bin/certutil -d /etc/httpd/alias -L -n ipaCert -a -f /etc/httpd/alias/pwdfile.txt 2018-09-12T13:37:19Z DEBUG Process finished, return code=255 2018-09-12T13:37:19Z DEBUG stdout= 2018-09-12T13:37:19Z DEBUG stderr=certutil: Could not find cert: ipaCert : PR_FILE_NOT_FOUND_ERROR: File not found [...] ---
this error can be ignored in most of the cases. The upgrade is trying to move ipaCert (cert+key) from the NSS database /etc/httpd/alias to the files /var/lib/ipa/ra-agent.pem and /var/lib/ipa/ra-agent.key. So if the upgrade is run a second time, he won't find ipaCert in the NSS database. To be sure, you can check if /var/lib/ipa/ra-agent.{pem|key} are present and contain a certificate with Subject CN=IPA RA,O=DOMAIN.COM. The files must be readable by root and ipaapi group, and must contain the same cert as the other masters.
checked this, is ok.
What is the content of /var/lib/ipa-client/pki/kdc-ca-bundle.pem and ca-bundle.pem? both must contain IPA CA certificate.
True on both servers.
What are the permissions of /var/kerberos/krb5kdc/kdc.crt? It needs to be readable by everyone. And what is the content of this cert? It should be issued by IPA CA.
Permissions are ok, contents:
--- ipa1 $ openssl x509 -in /var/kerberos/krb5kdc/kdc.crt -text -noout -fingerprint Certificate: Data: Version: 3 (0x2) Serial Number: 1 (0x1) Signature Algorithm: sha256WithRSAEncryption Issuer: O=EXAMPLE.COM, CN=ipa1.example.com Validity Not Before: Nov 28 12:43:05 2017 GMT Not After : Nov 28 12:43:05 2018 GMT Subject: O=EXAMPLE.COM, CN=ipa1.example.com Subject Public Key Info: Public Key Algorithm: rsaEncryption Public-Key: (2048 bit) Modulus: [...] Exponent: 65537 (0x10001) X509v3 extensions: X509v3 Subject Alternative Name: othername:<unsupported>, othername:<unsupported> X509v3 Basic Constraints: critical CA:FALSE X509v3 Subject Key Identifier:
86:52:EC:A1:C3:FB:EC:CC:6D:F2:09:E7:64:88:D1:80:F4:71:81:AE 1.3.6.1.4.1.311.20.2: .".K.D.C.s._.P.K.I.N.I.T._.C.e.r.t.s [...]
--- ipa2 $ openssl x509 -in /var/kerberos/krb5kdc/kdc.crt -text -noout -fingerprint Certificate: Data: Version: 3 (0x2) Serial Number: 805240833 (0x2fff0001) Signature Algorithm: sha256WithRSAEncryption Issuer: O=EXAMPLE.COM, CN=Certificate Authority Validity Not Before: Jan 18 13:04:17 2018 GMT Not After : Jan 19 13:04:17 2020 GMT Subject: O=EXAMPLE.COM, CN=ipa2.example.com Subject Public Key Info: Public Key Algorithm: rsaEncryption Public-Key: (2048 bit) Modulus: [...] Exponent: 65537 (0x10001) X509v3 extensions: X509v3 Authority Key Identifier:
keyid:4B:BA:AA:46:F1:29:E4:43:8B:DC:30:B4:90:3E:66:72:DD:F6:C7:FB
Authority Information Access: OCSP - URI:http://ipa-ca.example.com/ca/ocsp
X509v3 Key Usage: critical Digital Signature, Non Repudiation, Key Encipherment, Data Encipherment X509v3 Extended Key Usage: TLS Web Server Authentication, 1.3.6.1.5.2.3.5 X509v3 CRL Distribution Points:
Full Name: URI:http://ipa-ca.example.com/ipa/crl/MasterCRL.bin CRL Issuer: DirName: O = ipaca, CN = Certificate Authority
X509v3 Subject Key Identifier:
96:C3:94:70:7E:46:77:DB:91:F8:DF:D6:27:FE:73:0A:45:F3:78:F3 X509v3 Subject Alternative Name: othername:<unsupported>, othername:<unsupported> [...]
Additional info: I have DNS separate from IPA, but i (hopefully) made proper records as IPA would have done it. In particular, i made an A record "ipa-ca" that has IPs of both ipa1 and ipa2 - hope, this is not the root cause of my problems, since DNS is not under my control.
Since /var/kerberos/krb5kdc/kdc.crt on ipa1 appears to be not issued by IPA CA, might this be the actual problem?
The cert is expired on ipa1, which is the real root cause. ipa-pkinit-manage status is reporting that PKINIT is enabled but this does not match the actual configuration. This is a known issue [1].
If the host ipa1 is running a CA instance, then you can delete /var/kerberos/krb5kdc/kdc.{key,crt} and run ipa-pkinit-manage enable to re-generate the KDC cert. Note: if the host doesn't run a CA instance, then this won't work because of the issue [2].
To check which hosts are running a CA instance, you can use # ipa server-role-find --role 'CA server'
[1] https://pagure.io/freeipa/issue/7200 [2] https://pagure.io/freeipa/issue/7795
About the errors spotted by ipa-checkcerts.py, what are the certificates with errors reported? You can find them with ipa cert-show <serial>.
$ipa cert-show 57 Issuing CA: ipa Certificate: [...] Subject: CN=ipa1.example.com,O=EXAMPLE.COM Issuer: CN=Certificate Authority,O=EXAMPLE.COM Not Before: Tue Nov 28 12:42:07 2017 UTC Not After: Mon Nov 18 12:42:07 2019 UTC Serial number: 57 Serial number (hex): 0x39 Revoked: False
$ ipa cert-show 268304391 Issuing CA: ipa Certificate: [...] Subject: CN=IPA RA,O=EXAMPLE.COM Issuer: CN=Certificate Authority,O=EXAMPLE.COM Not Before: Wed Jul 11 14:26:35 2018 UTC Not After: Tue Jun 30 14:26:35 2020 UTC Serial number: 268304391 Serial number (hex): 0xFFE0007 Revoked: False
$ ipa cert-show 268304392 Issuing CA: ipa Certificate: [...] Subject: CN=CA Subsystem,O=EXAMPLE.COM Issuer: CN=Certificate Authority,O=EXAMPLE.COM Not Before: Wed Jul 11 14:26:45 2018 UTC Not After: Tue Jun 30 14:26:45 2020 UTC Serial number: 268304392 Serial number (hex): 0xFFE0008 Revoked: False
$ ipa cert-show 268304393 Issuing CA: ipa Certificate: [...] Subject: CN=OCSP Subsystem,O=EXAMPLE.COM Issuer: CN=Certificate Authority,O=EXAMPLE.COM Not Before: Wed Jul 11 14:27:05 2018 UTC Not After: Tue Jun 30 14:27:05 2020 UTC Serial number: 268304393 Serial number (hex): 0xFFE0009 Revoked: False
$ ipa cert-show 268304394 Issuing CA: ipa Certificate: [...] Subject: CN=CA Audit,O=EXAMPLE.COM Issuer: CN=Certificate Authority,O=EXAMPLE.COM Not Before: Wed Jul 11 14:27:25 2018 UTC Not After: Tue Jun 30 14:27:25 2020 UTC Serial number: 268304394 Serial number (hex): 0xFFE000A Revoked: False
ipa cert-show on server ipa2 fails with "ipa: ERROR: Certificate operation cannot be completed: EXCEPTION (Invalid Credential.)" btw.
The errors related to "Unable to find request for serial xxx" mean that the cert is tracked by certmonger, but there is no corresponding LDAP entry cn=xxx,ou=ca,ou=requests,o=ipaca on the local LDAP server. Is the replication still working between your two masters?
"ipa cert-show" broken on ipa2 often points to an out-of-date certificate in /var/lib/ipa/ra-agent.{pem|key}. The content should be the same as on the renewal master, and also the same as in the entry uid=ipara,ou=People,o=ipaca (which should be replicated and identical on all the masters, except if you have replication issues).
HTH, flo
Mit freundlichen Gruessen/With best regards,
--Daniel. _______________________________________________ FreeIPA-users mailing list -- freeipa-users@lists.fedorahosted.org To unsubscribe send an email to freeipa-users-leave@lists.fedorahosted.org Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedorahoste...
Hi Florence,
On Fri, 21 Dec 2018, Florence Blanc-Renaud via FreeIPA-users wrote:
On 12/21/18 11:58 AM, dbischof--- via FreeIPA-users wrote:
thank you very much for your help.
On Fri, 21 Dec 2018, Florence Blanc-Renaud via FreeIPA-users wrote:
On 12/20/18 6:52 PM, dbischof--- via FreeIPA-users wrote:
On Thu, 20 Dec 2018, Florence Blanc-Renaud via FreeIPA-users wrote:
On 12/20/18 4:22 PM, dbischof--- via FreeIPA-users wrote:
my IPA system consists of 2 masters with their own self-signed CAs, one of them being the certificate renewal master (ipa1). The system has been running for years and has been migrated from an IPA 3 system.
Since a while, the Web UI logins on ipa1 don't work anymore ("Login failed due to an unknown reason.").
Web UI logins on the other server (ipa2) work and everything else is working fine, too, ipactl status reports all services running.
On login attempt:
--- httpd log [...] [:error] [pid 15551] [remote 141.51.X.X:0] mod_wsgi (pid=15551): Exception occurred processing WSGI script '/usr/share/ipa/wsgi.py'. [...] [:error] [pid 15551] [remote 141.51.X.X:0] CalledProcessError: Command '/usr/bin/kinit -n -c /var/run/ipa/ccaches/armor_15551 -X X509_anchors=FILE:/var/kerberos/krb5kdc/kdc.crt -X X509_anchors=FILE:/var/lib/ipa-client/pki/kdc-ca-bundle.pem' returned non-zero exit status 1 ---
--- krb5kdc.log [...] Dec 20 16:06:54 ipa1.example.com krb5kdc[15517](info): AS_REQ (8 etypes {18 17 20 19 16 23 25 26}) 141.51.X.Y: NEEDED_PREAUTH: WELLKNOWN/ANONYMOUS@EXAMPLE.COM for krbtgt/EXAMPLE.COM@EXAMPLE.COM, Additional pre-authentication required Dec 20 16:06:54 ipa1.example.com krb5kdc[15517](info): closing down fd 11 Dec 20 16:06:54 ipa1.example.com krb5kdc[15518](info): AS_REQ (8 etypes {18 17 20 19 16 23 25 26}) 141.51.X.Y: KDC_RETURN_PADATA: WELLKNOWN/ANONYMOUS@EXAMPLE.COM for krbtgt/EXAMPLE.COM@EXAMPLE.COM, Failed to verify own certificate (depth 0): certificate has expired Dec 20 16:06:54 ipa1.example.com krb5kdc[15518](info): closing down fd 11 ---
--- ipa-checkcerts.py IPA version 4.5.4-10.el7.centos.3 Check CA status Check tracking Check NSS trust Check dates Checking certificates in CS.cfg Comparing certificates to requests in LDAP Checking RA certificate Checking authorities Checking host keytab Validating certificates Checking renewal master End-to-end cert API test Checking permissions and ownership Failures: Unable to find request for serial 268304391 Unable to find request for serial 268304394 Unable to find request for serial 268304393 Unable to find request for serial 268304392 Subject O=EXAMPLE.COM,CN=ipa2.example.com and template subject CN=ipa1.example.com,O=EXAMPLE.COM do not match for serial 57 ---
--- ipa pkinit-status --all ----------------- 2 servers matched ----------------- Server name: ipa2.example.com PKINIT status: enabled
Server name: ipa1.example.com PKINIT status: enabled ---------------------------- Number of entries returned 2 ----------------------------
To my understanding, proper certificate exchange between my two servers ceased working at some point. How do i track this down and fix it?
your issue looks similar to ticket #6792 [1]. Can you check the result of upgrade in /var/log/ipaupgrade.log? Also check the output of $ ipa-pkinit-manage status and if the files /var/lib/ipa-client/pki/kdc-ca-bundle.pem and /var/lib/ipa-client/pki/ca-bundle.pem exist, with -rw-r--r-- permissions.
Regarding the certificates, does getcert list show expired certs? flo
--- $ ipa-pkinit-manage status PKINIT is enabled ---
There are no expired certificates, kdc-ca-bundle.pem and ca-bundle.pem exist with proper permissions, but I found something in ipaupgrade.log:
--- 2018-09-12T13:37:18Z DEBUG args=/usr/bin/certutil -d /etc/httpd/alias -L -f /etc/httpd/alias/pwdfile.txt 2018-09-12T13:37:19Z DEBUG Process finished, return code=0 2018-09-12T13:37:19Z DEBUG stdout= Certificate Nickname Trust Attributes
SSL,S/MIME,JAR/XPI
Server-Cert u,u,u EXAMPLE.COM IPA CA CT,C,C
2018-09-12T13:37:19Z DEBUG stderr= 2018-09-12T13:37:19Z DEBUG Starting external process 2018-09-12T13:37:19Z DEBUG args=/usr/bin/certutil -d /etc/httpd/alias -L -n EXAMPLE.COM IPA CA -a -f /etc/httpd/alias/pwdfile.txt 2018-09-12T13:37:19Z DEBUG Process finished, return code=0 2018-09-12T13:37:19Z DEBUG stdout= -----BEGIN CERTIFICATE----- [...] -----END CERTIFICATE-----
2018-09-12T13:37:19Z DEBUG stderr= 2018-09-12T13:37:19Z DEBUG Executing upgrade plugin: update_ra_cert_store 2018-09-12T13:37:19Z DEBUG raw: update_ra_cert_store 2018-09-12T13:37:19Z DEBUG raw: ca_is_enabled(version=u'2.228') 2018-09-12T13:37:19Z DEBUG ca_is_enabled(version=u'2.228') 2018-09-12T13:37:19Z DEBUG Starting external process 2018-09-12T13:37:19Z DEBUG args=/usr/bin/certutil -d /etc/httpd/alias -L -n ipaCert -a -f /etc/httpd/alias/pwdfile.txt 2018-09-12T13:37:19Z DEBUG Process finished, return code=255 2018-09-12T13:37:19Z DEBUG stdout= 2018-09-12T13:37:19Z DEBUG stderr=certutil: Could not find cert: ipaCert : PR_FILE_NOT_FOUND_ERROR: File not found [...] ---
this error can be ignored in most of the cases. The upgrade is trying to move ipaCert (cert+key) from the NSS database /etc/httpd/alias to the files /var/lib/ipa/ra-agent.pem and /var/lib/ipa/ra-agent.key. So if the upgrade is run a second time, he won't find ipaCert in the NSS database. To be sure, you can check if /var/lib/ipa/ra-agent.{pem|key} are present and contain a certificate with Subject CN=IPA RA,O=DOMAIN.COM. The files must be readable by root and ipaapi group, and must contain the same cert as the other masters.
checked this, is ok.
What is the content of /var/lib/ipa-client/pki/kdc-ca-bundle.pem and ca-bundle.pem? both must contain IPA CA certificate.
True on both servers.
What are the permissions of /var/kerberos/krb5kdc/kdc.crt? It needs to be readable by everyone. And what is the content of this cert? It should be issued by IPA CA.
Permissions are ok, contents:
--- ipa1 $ openssl x509 -in /var/kerberos/krb5kdc/kdc.crt -text -noout -fingerprint Certificate: Data: Version: 3 (0x2) Serial Number: 1 (0x1) Signature Algorithm: sha256WithRSAEncryption Issuer: O=EXAMPLE.COM, CN=ipa1.example.com Validity Not Before: Nov 28 12:43:05 2017 GMT Not After : Nov 28 12:43:05 2018 GMT Subject: O=EXAMPLE.COM, CN=ipa1.example.com Subject Public Key Info: Public Key Algorithm: rsaEncryption Public-Key: (2048 bit) Modulus: [...] Exponent: 65537 (0x10001) X509v3 extensions: X509v3 Subject Alternative Name: othername:<unsupported>, othername:<unsupported> X509v3 Basic Constraints: critical CA:FALSE X509v3 Subject Key Identifier: 86:52:EC:A1:C3:FB:EC:CC:6D:F2:09:E7:64:88:D1:80:F4:71:81:AE 1.3.6.1.4.1.311.20.2: .".K.D.C.s._.P.K.I.N.I.T._.C.e.r.t.s [...]
--- ipa2 $ openssl x509 -in /var/kerberos/krb5kdc/kdc.crt -text -noout -fingerprint Certificate: Data: Version: 3 (0x2) Serial Number: 805240833 (0x2fff0001) Signature Algorithm: sha256WithRSAEncryption Issuer: O=EXAMPLE.COM, CN=Certificate Authority Validity Not Before: Jan 18 13:04:17 2018 GMT Not After : Jan 19 13:04:17 2020 GMT Subject: O=EXAMPLE.COM, CN=ipa2.example.com Subject Public Key Info: Public Key Algorithm: rsaEncryption Public-Key: (2048 bit) Modulus: [...] Exponent: 65537 (0x10001) X509v3 extensions: X509v3 Authority Key Identifier: keyid:4B:BA:AA:46:F1:29:E4:43:8B:DC:30:B4:90:3E:66:72:DD:F6:C7:FB
Authority Information Access: OCSP - URI:http://ipa-ca.example.com/ca/ocsp
X509v3 Key Usage: critical Digital Signature, Non Repudiation, Key Encipherment, Data Encipherment X509v3 Extended Key Usage: TLS Web Server Authentication, 1.3.6.1.5.2.3.5 X509v3 CRL Distribution Points:
Full Name: URI:http://ipa-ca.example.com/ipa/crl/MasterCRL.bin CRL Issuer: DirName: O = ipaca, CN = Certificate Authority
X509v3 Subject Key Identifier: 96:C3:94:70:7E:46:77:DB:91:F8:DF:D6:27:FE:73:0A:45:F3:78:F3 X509v3 Subject Alternative Name: othername:<unsupported>, othername:<unsupported> [...]
Additional info: I have DNS separate from IPA, but i (hopefully) made proper records as IPA would have done it. In particular, i made an A record "ipa-ca" that has IPs of both ipa1 and ipa2 - hope, this is not the root cause of my problems, since DNS is not under my control.
Since /var/kerberos/krb5kdc/kdc.crt on ipa1 appears to be not issued by IPA CA, might this be the actual problem?
The cert is expired on ipa1, which is the real root cause. ipa-pkinit-manage status is reporting that PKINIT is enabled but this does not match the actual configuration. This is a known issue [1].
If the host ipa1 is running a CA instance, then you can delete /var/kerberos/krb5kdc/kdc.{key,crt} and run ipa-pkinit-manage enable to re-generate the KDC cert. Note: if the host doesn't run a CA instance, then this won't work because of the issue [2].
To check which hosts are running a CA instance, you can use # ipa server-role-find --role 'CA server'
[1] https://pagure.io/freeipa/issue/7200 [2] https://pagure.io/freeipa/issue/7795
removed the expired certificates, re-enabled PKINIT, certificates were regenerated, Web UI logins working again. Thank you very much.
About the errors spotted by ipa-checkcerts.py, what are the certificates with errors reported? You can find them with ipa cert-show <serial>.
$ipa cert-show 57 Issuing CA: ipa Certificate: [...] Subject: CN=ipa1.example.com,O=EXAMPLE.COM Issuer: CN=Certificate Authority,O=EXAMPLE.COM Not Before: Tue Nov 28 12:42:07 2017 UTC Not After: Mon Nov 18 12:42:07 2019 UTC Serial number: 57 Serial number (hex): 0x39 Revoked: False
$ ipa cert-show 268304391 Issuing CA: ipa Certificate: [...] Subject: CN=IPA RA,O=EXAMPLE.COM Issuer: CN=Certificate Authority,O=EXAMPLE.COM Not Before: Wed Jul 11 14:26:35 2018 UTC Not After: Tue Jun 30 14:26:35 2020 UTC Serial number: 268304391 Serial number (hex): 0xFFE0007 Revoked: False
$ ipa cert-show 268304392 Issuing CA: ipa Certificate: [...] Subject: CN=CA Subsystem,O=EXAMPLE.COM Issuer: CN=Certificate Authority,O=EXAMPLE.COM Not Before: Wed Jul 11 14:26:45 2018 UTC Not After: Tue Jun 30 14:26:45 2020 UTC Serial number: 268304392 Serial number (hex): 0xFFE0008 Revoked: False
$ ipa cert-show 268304393 Issuing CA: ipa Certificate: [...] Subject: CN=OCSP Subsystem,O=EXAMPLE.COM Issuer: CN=Certificate Authority,O=EXAMPLE.COM Not Before: Wed Jul 11 14:27:05 2018 UTC Not After: Tue Jun 30 14:27:05 2020 UTC Serial number: 268304393 Serial number (hex): 0xFFE0009 Revoked: False
$ ipa cert-show 268304394 Issuing CA: ipa Certificate: [...] Subject: CN=CA Audit,O=EXAMPLE.COM Issuer: CN=Certificate Authority,O=EXAMPLE.COM Not Before: Wed Jul 11 14:27:25 2018 UTC Not After: Tue Jun 30 14:27:25 2020 UTC Serial number: 268304394 Serial number (hex): 0xFFE000A Revoked: False
ipa cert-show on server ipa2 fails with "ipa: ERROR: Certificate operation cannot be completed: EXCEPTION (Invalid Credential.)" btw.
The errors related to "Unable to find request for serial xxx" mean that the cert is tracked by certmonger, but there is no corresponding LDAP entry cn=xxx,ou=ca,ou=requests,o=ipaca on the local LDAP server. Is the replication still working between your two masters?
"ipa cert-show" broken on ipa2 often points to an out-of-date certificate in /var/lib/ipa/ra-agent.{pem|key}. The content should be the same as on the renewal master, and also the same as in the entry uid=ipara,ou=People,o=ipaca (which should be replicated and identical on all the masters, except if you have replication issues).
Replication appeared to be working, however, I did server maintenance today, ipa1 and ipa2 got upgraded and rebooted. ipa1 works fine, ipa2 does not come up properly:
--- ipa2 $ ipactl status Directory Service: RUNNING krb5kdc Service: STOPPED kadmin Service: STOPPED httpd Service: RUNNING ipa-custodia Service: STOPPED ntpd Service: RUNNING pki-tomcatd Service: RUNNING ipa-otpd Service: STOPPED ipa: INFO: The ipactl command was successful ---
--- ipa1 $ openssl x509 -in /var/lib/ipa/ra-agent.pem -text -noout | grep Serial Serial Number: 268304391 (0xffe0007)
$ ldapsearch -x -b uid=ipara,ou=People,o=ipaca [...] description: 2;268304391;CN=Certificate Authority,O=EXAMPLE.COM;CN=IPA RA,O=EXAMPLE.COM ---
Looks good.
--- ipa2 $ openssl x509 -in /var/lib/ipa/ra-agent.pem -text -noout | grep Serial Serial Number: 268304391 (0xffe0007)
$ ldapsearch -x -b uid=ipara,ou=People,o=ipaca [...] description: 2;44;CN=Certificate Authority,O=EXAMPLE.COM;CN=IPA RA,O=EXAMPLE.COM ---
Looks bad.
--- /var/log/ipaupgrade.log [...] 2018-12-28T10:55:53Z DEBUG The ipa-server-upgrade command failed, exception: RemoteRetrieveError: Failed to authenticate to CA REST API ---
--- /var/log/dirsrv/slapd-EXAMPLE-COM/errors [...] [28/Dec/2018:11:55:55.559648469 +0100] - ERR - set_krb5_creds - Could not get initial credentials for principal [ldap/ipa2.example.com@EXAMPLE.COM] in keytab [FILE:/etc/dirsrv/ds.keytab]: -1765328228 (Cannot contact any KDC for requested realm) ---
--- ipa-checkcerts.py [...] ipa: ERROR: ra.get_certificate(): EXCEPTION (Invalid Credential.) ra.get_certificate(): EXCEPTION (Invalid Credential.) ipa: INFO: Checking permissions and ownership Checking permissions and ownership ipa: INFO: Failures: Failures: ipa: INFO: Unable to find request for serial 268304391 Unable to find request for serial 268304391 ipa: INFO: Unable to find request for serial 268304394 Unable to find request for serial 268304394 ipa: INFO: Unable to find request for serial 268304393 Unable to find request for serial 268304393 ipa: INFO: Unable to find request for serial 268304392 Unable to find request for serial 268304392 ipa: INFO: Unable to find request for serial 268304390 Unable to find request for serial 268304390 ipa: INFO: RA agent description does not match 2;44;CN=Certificate Authority,O=EXAMPLE.COM;CN=IPA RA,O=EXAMPLE.COM in LDAP and 2;268304391;CN=Certificate Authority,O=EXAMPLE.COM;CN=IPA RA,O=EXAMPLE.COM expected ipa: INFO: RA agent certificate not found in LDAP RA agent certificate not found in LDAP [...] ---
The root cause appears to be the wrong IPA RA certificate in ipa2's LDAP, right? I guess this has to fixed by manually importing the proper certificate using ldapmodify, similar to the procedure described in [1]?
[1] https://floblanc.wordpress.com/2017/09/11/troubleshooting-freeipa-pki-tomcat...
Mit freundlichen Gruessen/With best regards,
--Daniel.
On 12/28/18 1:03 PM, dbischof--- via FreeIPA-users wrote:
Hi Florence,
On Fri, 21 Dec 2018, Florence Blanc-Renaud via FreeIPA-users wrote:
On 12/21/18 11:58 AM, dbischof--- via FreeIPA-users wrote:
thank you very much for your help.
On Fri, 21 Dec 2018, Florence Blanc-Renaud via FreeIPA-users wrote:
On 12/20/18 6:52 PM, dbischof--- via FreeIPA-users wrote:
On Thu, 20 Dec 2018, Florence Blanc-Renaud via FreeIPA-users wrote:
On 12/20/18 4:22 PM, dbischof--- via FreeIPA-users wrote: > my IPA system consists of 2 masters with their own self-signed > CAs, > one of > them being the certificate renewal master (ipa1). The system has > been > running for years and has been migrated from an IPA 3 system. > > Since a while, the Web UI logins on ipa1 don't work anymore > ("Login > failed > due to an unknown reason."). > > Web UI logins on the other server (ipa2) work and everything > else is > working fine, too, ipactl status reports all services running. > > On login attempt: > > --- httpd log > [...] > [:error] [pid 15551] [remote 141.51.X.X:0] mod_wsgi (pid=15551): > Exception > occurred processing WSGI script '/usr/share/ipa/wsgi.py'. > [...] > [:error] [pid 15551] [remote 141.51.X.X:0] CalledProcessError: > Command > '/usr/bin/kinit -n -c /var/run/ipa/ccaches/armor_15551 -X > X509_anchors=FILE:/var/kerberos/krb5kdc/kdc.crt -X > X509_anchors=FILE:/var/lib/ipa-client/pki/kdc-ca-bundle.pem' > returned > non-zero exit status 1 > --- > > --- krb5kdc.log > [...] > Dec 20 16:06:54 ipa1.example.com krb5kdc[15517](info): AS_REQ (8 > etypes > {18 17 20 19 16 23 25 26}) 141.51.X.Y: NEEDED_PREAUTH: > WELLKNOWN/ANONYMOUS@EXAMPLE.COM for > krbtgt/EXAMPLE.COM@EXAMPLE.COM, > Additional pre-authentication required > Dec 20 16:06:54 ipa1.example.com krb5kdc[15517](info): closing > down > fd > 11 > Dec 20 16:06:54 ipa1.example.com krb5kdc[15518](info): AS_REQ (8 > etypes > {18 17 20 19 16 23 25 26}) 141.51.X.Y: KDC_RETURN_PADATA: > WELLKNOWN/ANONYMOUS@EXAMPLE.COM for > krbtgt/EXAMPLE.COM@EXAMPLE.COM, > Failed > to verify own certificate (depth 0): certificate has expired > Dec 20 16:06:54 ipa1.example.com krb5kdc[15518](info): closing > down > fd > 11 > --- > > --- ipa-checkcerts.py > IPA version 4.5.4-10.el7.centos.3 > Check CA status > Check tracking > Check NSS trust > Check dates > Checking certificates in CS.cfg > Comparing certificates to requests in LDAP > Checking RA certificate > Checking authorities > Checking host keytab > Validating certificates > Checking renewal master > End-to-end cert API test > Checking permissions and ownership > Failures: > Unable to find request for serial 268304391 > Unable to find request for serial 268304394 > Unable to find request for serial 268304393 > Unable to find request for serial 268304392 > Subject O=EXAMPLE.COM,CN=ipa2.example.com and template subject > CN=ipa1.example.com,O=EXAMPLE.COM do not match for serial 57 > --- > > --- ipa pkinit-status --all > ----------------- > 2 servers matched > ----------------- > Server name: ipa2.example.com > PKINIT status: enabled > > Server name: ipa1.example.com > PKINIT status: enabled > ---------------------------- > Number of entries returned 2 > ---------------------------- > > To my understanding, proper certificate exchange between my two > servers > ceased working at some point. How do i track this down and fix > it? > your issue looks similar to ticket #6792 [1]. Can you check the result of upgrade in /var/log/ipaupgrade.log? Also check the output of $ ipa-pkinit-manage status and if the files /var/lib/ipa-client/pki/kdc-ca-bundle.pem and /var/lib/ipa-client/pki/ca-bundle.pem exist, with -rw-r--r-- permissions.
Regarding the certificates, does getcert list show expired certs? flo
--- $ ipa-pkinit-manage status PKINIT is enabled ---
There are no expired certificates, kdc-ca-bundle.pem and ca-bundle.pem exist with proper permissions, but I found something in ipaupgrade.log:
--- 2018-09-12T13:37:18Z DEBUG args=/usr/bin/certutil -d /etc/httpd/alias -L -f /etc/httpd/alias/pwdfile.txt 2018-09-12T13:37:19Z DEBUG Process finished, return code=0 2018-09-12T13:37:19Z DEBUG stdout= Certificate Nickname Trust Attributes
SSL,S/MIME,JAR/XPI
Server-Cert u,u,u EXAMPLE.COM IPA CA CT,C,C
2018-09-12T13:37:19Z DEBUG stderr= 2018-09-12T13:37:19Z DEBUG Starting external process 2018-09-12T13:37:19Z DEBUG args=/usr/bin/certutil -d /etc/httpd/alias -L -n EXAMPLE.COM IPA CA -a -f /etc/httpd/alias/pwdfile.txt 2018-09-12T13:37:19Z DEBUG Process finished, return code=0 2018-09-12T13:37:19Z DEBUG stdout= -----BEGIN CERTIFICATE----- [...] -----END CERTIFICATE-----
2018-09-12T13:37:19Z DEBUG stderr= 2018-09-12T13:37:19Z DEBUG Executing upgrade plugin: update_ra_cert_store 2018-09-12T13:37:19Z DEBUG raw: update_ra_cert_store 2018-09-12T13:37:19Z DEBUG raw: ca_is_enabled(version=u'2.228') 2018-09-12T13:37:19Z DEBUG ca_is_enabled(version=u'2.228') 2018-09-12T13:37:19Z DEBUG Starting external process 2018-09-12T13:37:19Z DEBUG args=/usr/bin/certutil -d /etc/httpd/alias -L -n ipaCert -a -f /etc/httpd/alias/pwdfile.txt 2018-09-12T13:37:19Z DEBUG Process finished, return code=255 2018-09-12T13:37:19Z DEBUG stdout= 2018-09-12T13:37:19Z DEBUG stderr=certutil: Could not find cert: ipaCert : PR_FILE_NOT_FOUND_ERROR: File not found [...] ---
this error can be ignored in most of the cases. The upgrade is trying to move ipaCert (cert+key) from the NSS database /etc/httpd/alias to the files /var/lib/ipa/ra-agent.pem and /var/lib/ipa/ra-agent.key. So if the upgrade is run a second time, he won't find ipaCert in the NSS database. To be sure, you can check if /var/lib/ipa/ra-agent.{pem|key} are present and contain a certificate with Subject CN=IPA RA,O=DOMAIN.COM. The files must be readable by root and ipaapi group, and must contain the same cert as the other masters.
checked this, is ok.
What is the content of /var/lib/ipa-client/pki/kdc-ca-bundle.pem and ca-bundle.pem? both must contain IPA CA certificate.
True on both servers.
What are the permissions of /var/kerberos/krb5kdc/kdc.crt? It needs to be readable by everyone. And what is the content of this cert? It should be issued by IPA CA.
Permissions are ok, contents:
--- ipa1 $ openssl x509 -in /var/kerberos/krb5kdc/kdc.crt -text -noout -fingerprint Certificate: Data: Version: 3 (0x2) Serial Number: 1 (0x1) Signature Algorithm: sha256WithRSAEncryption Issuer: O=EXAMPLE.COM, CN=ipa1.example.com Validity Not Before: Nov 28 12:43:05 2017 GMT Not After : Nov 28 12:43:05 2018 GMT Subject: O=EXAMPLE.COM, CN=ipa1.example.com Subject Public Key Info: Public Key Algorithm: rsaEncryption Public-Key: (2048 bit) Modulus: [...] Exponent: 65537 (0x10001) X509v3 extensions: X509v3 Subject Alternative Name: othername:<unsupported>, othername:<unsupported> X509v3 Basic Constraints: critical CA:FALSE X509v3 Subject Key Identifier:
86:52:EC:A1:C3:FB:EC:CC:6D:F2:09:E7:64:88:D1:80:F4:71:81:AE 1.3.6.1.4.1.311.20.2: .".K.D.C.s._.P.K.I.N.I.T._.C.e.r.t.s [...] ---
--- ipa2 $ openssl x509 -in /var/kerberos/krb5kdc/kdc.crt -text -noout -fingerprint Certificate: Data: Version: 3 (0x2) Serial Number: 805240833 (0x2fff0001) Signature Algorithm: sha256WithRSAEncryption Issuer: O=EXAMPLE.COM, CN=Certificate Authority Validity Not Before: Jan 18 13:04:17 2018 GMT Not After : Jan 19 13:04:17 2020 GMT Subject: O=EXAMPLE.COM, CN=ipa2.example.com Subject Public Key Info: Public Key Algorithm: rsaEncryption Public-Key: (2048 bit) Modulus: [...] Exponent: 65537 (0x10001) X509v3 extensions: X509v3 Authority Key Identifier:
keyid:4B:BA:AA:46:F1:29:E4:43:8B:DC:30:B4:90:3E:66:72:DD:F6:C7:FB
Authority Information Access: OCSP - URI:http://ipa-ca.example.com/ca/ocsp
X509v3 Key Usage: critical Digital Signature, Non Repudiation, Key Encipherment, Data Encipherment X509v3 Extended Key Usage: TLS Web Server Authentication, 1.3.6.1.5.2.3.5 X509v3 CRL Distribution Points:
Full Name: URI:http://ipa-ca.example.com/ipa/crl/MasterCRL.bin CRL Issuer: DirName: O = ipaca, CN = Certificate Authority
X509v3 Subject Key Identifier:
96:C3:94:70:7E:46:77:DB:91:F8:DF:D6:27:FE:73:0A:45:F3:78:F3 X509v3 Subject Alternative Name: othername:<unsupported>, othername:<unsupported> [...] ---
Additional info: I have DNS separate from IPA, but i (hopefully) made proper records as IPA would have done it. In particular, i made an A record "ipa-ca" that has IPs of both ipa1 and ipa2 - hope, this is not the root cause of my problems, since DNS is not under my control.
Since /var/kerberos/krb5kdc/kdc.crt on ipa1 appears to be not issued by IPA CA, might this be the actual problem?
The cert is expired on ipa1, which is the real root cause. ipa-pkinit-manage status is reporting that PKINIT is enabled but this does not match the actual configuration. This is a known issue [1].
If the host ipa1 is running a CA instance, then you can delete /var/kerberos/krb5kdc/kdc.{key,crt} and run ipa-pkinit-manage enable to re-generate the KDC cert. Note: if the host doesn't run a CA instance, then this won't work because of the issue [2].
To check which hosts are running a CA instance, you can use # ipa server-role-find --role 'CA server'
[1] https://pagure.io/freeipa/issue/7200 [2] https://pagure.io/freeipa/issue/7795
removed the expired certificates, re-enabled PKINIT, certificates were regenerated, Web UI logins working again. Thank you very much.
About the errors spotted by ipa-checkcerts.py, what are the certificates with errors reported? You can find them with ipa cert-show <serial>.
--- $ipa cert-show 57 Issuing CA: ipa Certificate: [...] Subject: CN=ipa1.example.com,O=EXAMPLE.COM Issuer: CN=Certificate Authority,O=EXAMPLE.COM Not Before: Tue Nov 28 12:42:07 2017 UTC Not After: Mon Nov 18 12:42:07 2019 UTC Serial number: 57 Serial number (hex): 0x39 Revoked: False
$ ipa cert-show 268304391 Issuing CA: ipa Certificate: [...] Subject: CN=IPA RA,O=EXAMPLE.COM Issuer: CN=Certificate Authority,O=EXAMPLE.COM Not Before: Wed Jul 11 14:26:35 2018 UTC Not After: Tue Jun 30 14:26:35 2020 UTC Serial number: 268304391 Serial number (hex): 0xFFE0007 Revoked: False
$ ipa cert-show 268304392 Issuing CA: ipa Certificate: [...] Subject: CN=CA Subsystem,O=EXAMPLE.COM Issuer: CN=Certificate Authority,O=EXAMPLE.COM Not Before: Wed Jul 11 14:26:45 2018 UTC Not After: Tue Jun 30 14:26:45 2020 UTC Serial number: 268304392 Serial number (hex): 0xFFE0008 Revoked: False
$ ipa cert-show 268304393 Issuing CA: ipa Certificate: [...] Subject: CN=OCSP Subsystem,O=EXAMPLE.COM Issuer: CN=Certificate Authority,O=EXAMPLE.COM Not Before: Wed Jul 11 14:27:05 2018 UTC Not After: Tue Jun 30 14:27:05 2020 UTC Serial number: 268304393 Serial number (hex): 0xFFE0009 Revoked: False
$ ipa cert-show 268304394 Issuing CA: ipa Certificate: [...] Subject: CN=CA Audit,O=EXAMPLE.COM Issuer: CN=Certificate Authority,O=EXAMPLE.COM Not Before: Wed Jul 11 14:27:25 2018 UTC Not After: Tue Jun 30 14:27:25 2020 UTC Serial number: 268304394 Serial number (hex): 0xFFE000A Revoked: False ---
ipa cert-show on server ipa2 fails with "ipa: ERROR: Certificate operation cannot be completed: EXCEPTION (Invalid Credential.)" btw.
The errors related to "Unable to find request for serial xxx" mean that the cert is tracked by certmonger, but there is no corresponding LDAP entry cn=xxx,ou=ca,ou=requests,o=ipaca on the local LDAP server. Is the replication still working between your two masters?
"ipa cert-show" broken on ipa2 often points to an out-of-date certificate in /var/lib/ipa/ra-agent.{pem|key}. The content should be the same as on the renewal master, and also the same as in the entry uid=ipara,ou=People,o=ipaca (which should be replicated and identical on all the masters, except if you have replication issues).
Replication appeared to be working, however, I did server maintenance today, ipa1 and ipa2 got upgraded and rebooted. ipa1 works fine, ipa2 does not come up properly:
--- ipa2 $ ipactl status Directory Service: RUNNING krb5kdc Service: STOPPED kadmin Service: STOPPED
You need to have a look at krb5 logs to understand why kerberos failed to start. Please check https://www.freeipa.org/page/Troubleshooting/Kerberos for more information.
httpd Service: RUNNING ipa-custodia Service: STOPPED ntpd Service: RUNNING pki-tomcatd Service: RUNNING ipa-otpd Service: STOPPED ipa: INFO: The ipactl command was successful
--- ipa1 $ openssl x509 -in /var/lib/ipa/ra-agent.pem -text -noout | grep Serial Serial Number: 268304391 (0xffe0007)
$ ldapsearch -x -b uid=ipara,ou=People,o=ipaca [...] description: 2;268304391;CN=Certificate Authority,O=EXAMPLE.COM;CN=IPA RA,O=EXAMPLE.COM
Looks good.
--- ipa2 $ openssl x509 -in /var/lib/ipa/ra-agent.pem -text -noout | grep Serial Serial Number: 268304391 (0xffe0007)
$ ldapsearch -x -b uid=ipara,ou=People,o=ipaca [...] description: 2;44;CN=Certificate Authority,O=EXAMPLE.COM;CN=IPA RA,O=EXAMPLE.COM
Looks bad.
So yes, the replication of the suffix o=ipaca seems broken. In a deployment with CA, the ldap server contains 2 different suffixes, one for the IdM data (your base DN as defined in /etc/ipa/default.conf) and another one for the CA, below o=ipaca.
You can have a look at https://www.freeipa.org/page/Troubleshooting/Directory_Server for replication troubleshooting, but the repl issue could be a consequence of kerberos server not starting.
--- /var/log/ipaupgrade.log [...] 2018-12-28T10:55:53Z DEBUG The ipa-server-upgrade command failed, exception: RemoteRetrieveError: Failed to authenticate to CA REST API
--- /var/log/dirsrv/slapd-EXAMPLE-COM/errors [...] [28/Dec/2018:11:55:55.559648469 +0100] - ERR - set_krb5_creds - Could not get initial credentials for principal [ldap/ipa2.example.com@EXAMPLE.COM] in keytab [FILE:/etc/dirsrv/ds.keytab]: -1765328228 (Cannot contact any KDC for requested realm)
--- ipa-checkcerts.py [...] ipa: ERROR: ra.get_certificate(): EXCEPTION (Invalid Credential.) ra.get_certificate(): EXCEPTION (Invalid Credential.) ipa: INFO: Checking permissions and ownership Checking permissions and ownership ipa: INFO: Failures: Failures: ipa: INFO: Unable to find request for serial 268304391 Unable to find request for serial 268304391 ipa: INFO: Unable to find request for serial 268304394 Unable to find request for serial 268304394 ipa: INFO: Unable to find request for serial 268304393 Unable to find request for serial 268304393 ipa: INFO: Unable to find request for serial 268304392 Unable to find request for serial 268304392 ipa: INFO: Unable to find request for serial 268304390 Unable to find request for serial 268304390 ipa: INFO: RA agent description does not match 2;44;CN=Certificate Authority,O=EXAMPLE.COM;CN=IPA RA,O=EXAMPLE.COM in LDAP and 2;268304391;CN=Certificate Authority,O=EXAMPLE.COM;CN=IPA RA,O=EXAMPLE.COM expected ipa: INFO: RA agent certificate not found in LDAP RA agent certificate not found in LDAP [...]
The root cause appears to be the wrong IPA RA certificate in ipa2's LDAP, right? I guess this has to fixed by manually importing the proper certificate using ldapmodify, similar to the procedure described in [1]?
It is possible to manually modify the IPA RA cert in ipa2's ldap server but this won't solve the replication issue. It will however allow you to run ipa cert-show.
flo
[1] https://floblanc.wordpress.com/2017/09/11/troubleshooting-freeipa-pki-tomcat...
Mit freundlichen Gruessen/With best regards,
--Daniel. _______________________________________________ FreeIPA-users mailing list -- freeipa-users@lists.fedorahosted.org To unsubscribe send an email to freeipa-users-leave@lists.fedorahosted.org Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedorahoste...
Hi Florence,
On Mon, 7 Jan 2019, Florence Blanc-Renaud wrote:
[...]
i shaved this thread a little, since it gets confusing. Hope i kept the interesting bits.
The errors related to "Unable to find request for serial xxx" mean that the cert is tracked by certmonger, but there is no corresponding LDAP entry cn=xxx,ou=ca,ou=requests,o=ipaca on the local LDAP server. Is the replication still working between your two masters?
"ipa cert-show" broken on ipa2 often points to an out-of-date certificate in /var/lib/ipa/ra-agent.{pem|key}. The content should be the same as on the renewal master, and also the same as in the entry uid=ipara,ou=People,o=ipaca (which should be replicated and identical on all the masters, except if you have replication issues).
Replication appeared to be working, however, I did server maintenance today, ipa1 and ipa2 got upgraded and rebooted. ipa1 works fine, ipa2 does not come up properly:
--- ipa2 $ ipactl status Directory Service: RUNNING krb5kdc Service: STOPPED kadmin Service: STOPPED
You need to have a look at krb5 logs to understand why kerberos failed to start. Please check https://www.freeipa.org/page/Troubleshooting/Kerberos for more information.
Turns out that kerberos starts when started manually with systemctl:
--- $ ipactl status Directory Service: RUNNING krb5kdc Service: RUNNING kadmin Service: RUNNING httpd Service: RUNNING ipa-custodia Service: RUNNING ntpd Service: RUNNING pki-tomcatd Service: RUNNING ipa-otpd Service: STOPPED ipa: INFO: The ipactl command was successful ---
Kerberos log:
--- [...] : NEEDED_PREAUTH: admin@EXAMPLE.COM for krbtgt/EXAMPLE.COM@EXAMPLE.COM, Additional pre-authentication required Jan 08 10:39:22 ipa2.example.com krb5kdc[21691](info): closing down fd 11 Jan 08 10:39:39 ipa2.example.com krb5kdc[21692](info): AS_REQ (8 etypes {18 17 20 19 16 23 25 26}) 141.51.x.y: ISSUE: authtime 1546940379, etypes {rep=18 tkt=18 ses=18}, admin@EXAMPLE.COM for krbtgt/EXAMPLE.COM@EXAMPLE.COM Jan 08 10:39:39 ipa2.example.com krb5kdc[21692](info): closing down fd 11 Jan 08 10:40:32 ipa2.example.com krb5kdc[21692](info): TGS_REQ (8 etypes {18 17 20 19 16 23 25 26}) 141.51.x.y: ISSUE: authtime 1546940379, etypes {rep=18 tkt=18 ses=18}, admin@EXAMPLE.COM for HTTP/ipa2.example.com@EXAMPLE.COM Jan 08 10:40:32 ipa2.example.com krb5kdc[21692](info): closing down fd 11 ---
httpd Service: RUNNING ipa-custodia Service: STOPPED ntpd Service: RUNNING pki-tomcatd Service: RUNNING ipa-otpd Service: STOPPED ipa: INFO: The ipactl command was successful
--- ipa1 $ openssl x509 -in /var/lib/ipa/ra-agent.pem -text -noout | grep Serial Serial Number: 268304391 (0xffe0007)
$ ldapsearch -x -b uid=ipara,ou=People,o=ipaca [...] description: 2;268304391;CN=Certificate Authority,O=EXAMPLE.COM;CN=IPA RA,O=EXAMPLE.COM
Looks good.
--- ipa2 $ openssl x509 -in /var/lib/ipa/ra-agent.pem -text -noout | grep Serial Serial Number: 268304391 (0xffe0007)
$ ldapsearch -x -b uid=ipara,ou=People,o=ipaca [...] description: 2;44;CN=Certificate Authority,O=EXAMPLE.COM;CN=IPA RA,O=EXAMPLE.COM
Looks bad.
So yes, the replication of the suffix o=ipaca seems broken. In a deployment with CA, the ldap server contains 2 different suffixes, one for the IdM data (your base DN as defined in /etc/ipa/default.conf) and another one for the CA, below o=ipaca.
You can have a look at https://www.freeipa.org/page/Troubleshooting/Directory_Server for replication troubleshooting, but the repl issue could be a consequence of kerberos server not starting.
I checked the troubleshooting pages and carried out the steps described in there, but my system (ipa2) appears to be ok as far as the steps go.
--- /var/log/ipaupgrade.log [...] 2018-12-28T10:55:53Z DEBUG The ipa-server-upgrade command failed, exception: RemoteRetrieveError: Failed to authenticate to CA REST API
--- /var/log/dirsrv/slapd-EXAMPLE-COM/errors [...] [28/Dec/2018:11:55:55.559648469 +0100] - ERR - set_krb5_creds - Could not get initial credentials for principal [ldap/ipa2.example.com@EXAMPLE.COM] in keytab [FILE:/etc/dirsrv/ds.keytab]: -1765328228 (Cannot contact any KDC for requested realm)
--- ipa-checkcerts.py [...] ipa: ERROR: ra.get_certificate(): EXCEPTION (Invalid Credential.) ra.get_certificate(): EXCEPTION (Invalid Credential.) ipa: INFO: Checking permissions and ownership Checking permissions and ownership ipa: INFO: Failures: Failures: ipa: INFO: Unable to find request for serial 268304391 Unable to find request for serial 268304391 ipa: INFO: Unable to find request for serial 268304394 Unable to find request for serial 268304394 ipa: INFO: Unable to find request for serial 268304393 Unable to find request for serial 268304393 ipa: INFO: Unable to find request for serial 268304392 Unable to find request for serial 268304392 ipa: INFO: Unable to find request for serial 268304390 Unable to find request for serial 268304390 ipa: INFO: RA agent description does not match 2;44;CN=Certificate Authority,O=EXAMPLE.COM;CN=IPA RA,O=EXAMPLE.COM in LDAP and 2;268304391;CN=Certificate Authority,O=EXAMPLE.COM;CN=IPA RA,O=EXAMPLE.COM expected ipa: INFO: RA agent certificate not found in LDAP RA agent certificate not found in LDAP [...]
The root cause appears to be the wrong IPA RA certificate in ipa2's LDAP, right? I guess this has to fixed by manually importing the proper certificate using ldapmodify, similar to the procedure described in [1]?
It is possible to manually modify the IPA RA cert in ipa2's ldap server but this won't solve the replication issue. It will however allow you to run ipa cert-show.
Unfortunately, it doesn't (ipa2):
--- ipa: ERROR: Certificate operation cannot be completed: EXCEPTION (Invalid Credential.) ---
Kerberos log:
--- Jan 08 15:42:45 ipa2.example.com krb5kdc[23520](info): AS_REQ (8 etypes {18 17 16 23 20 19 25 26}) 141.51.124.20: NEEDED_PREAUTH: host/ipa2.example.com@EXAMPLE.COM for krbtgt/EXAMPLE.COM@EXAMPLE.COM, Additional pre-authentication required ---
Works ok on ipa1.
I'm unsure how to proceed here. My obvious problem is ipa2 not getting upgraded properly:
--- [...] 2019-01-08T09:46:07Z INFO [Updating HTTPD service IPA configuration] 2019-01-08T09:46:07Z DEBUG Starting external process 2019-01-08T09:46:07Z DEBUG args=/usr/sbin/selinuxenabled 2019-01-08T09:46:07Z DEBUG Process finished, return code=1 2019-01-08T09:46:07Z DEBUG stdout= 2019-01-08T09:46:07Z DEBUG stderr= 2019-01-08T09:46:07Z DEBUG Starting external process 2019-01-08T09:46:07Z DEBUG args=/bin/systemctl --system daemon-reload 2019-01-08T09:46:07Z DEBUG Process finished, return code=0 2019-01-08T09:46:07Z DEBUG stdout= 2019-01-08T09:46:07Z DEBUG stderr= 2019-01-08T09:46:07Z INFO [Updating HTTPD service IPA WSGI configuration] 2019-01-08T09:46:07Z INFO Nothing to do for configure_httpd_wsgi_conf 2019-01-08T09:46:07Z INFO [Updating mod_nss protocol versions] 2019-01-08T09:46:07Z DEBUG Loading StateFile from '/var/lib/ipa/sysupgrade/sysupgrade.state' 2019-01-08T09:46:07Z INFO Protocol versions already updated 2019-01-08T09:46:07Z INFO [Updating mod_nss cipher suite] 2019-01-08T09:46:07Z DEBUG Loading StateFile from '/var/lib/ipa/sysupgrade/sysupgrade.state' 2019-01-08T09:46:07Z DEBUG Cipher suite already updated 2019-01-08T09:46:07Z INFO [Updating mod_nss enabling OCSP] 2019-01-08T09:46:07Z DEBUG Loading StateFile from '/var/lib/ipa/sysupgrade/sysupgrade.state' 2019-01-08T09:46:07Z INFO [Fixing trust flags in /etc/httpd/alias] 2019-01-08T09:46:07Z DEBUG Loading StateFile from '/var/lib/ipa/sysupgrade/sysupgrade.state' 2019-01-08T09:46:07Z INFO Trust flags already processed 2019-01-08T09:46:07Z INFO [Moving HTTPD service keytab to gssproxy] 2019-01-08T09:46:07Z DEBUG Starting external process 2019-01-08T09:46:07Z DEBUG args=/usr/sbin/selinuxenabled 2019-01-08T09:46:07Z DEBUG Process finished, return code=1 2019-01-08T09:46:07Z DEBUG stdout= 2019-01-08T09:46:07Z DEBUG stderr= 2019-01-08T09:46:07Z DEBUG Starting external process 2019-01-08T09:46:07Z DEBUG args=/bin/systemctl restart gssproxy.service 2019-01-08T09:46:07Z DEBUG Process finished, return code=0 2019-01-08T09:46:07Z DEBUG stdout= 2019-01-08T09:46:07Z DEBUG stderr= 2019-01-08T09:46:07Z DEBUG Starting external process 2019-01-08T09:46:07Z DEBUG args=/bin/systemctl is-active gssproxy.service 2019-01-08T09:46:07Z DEBUG Process finished, return code=0 2019-01-08T09:46:07Z DEBUG stdout=active
2019-01-08T09:46:07Z DEBUG stderr= 2019-01-08T09:46:07Z DEBUG Restart of gssproxy.service complete 2019-01-08T09:46:07Z DEBUG Starting external process 2019-01-08T09:46:07Z DEBUG args=/bin/systemctl start httpd.service 2019-01-08T09:46:09Z DEBUG Process finished, return code=0 2019-01-08T09:46:09Z DEBUG stdout= 2019-01-08T09:46:09Z DEBUG stderr= 2019-01-08T09:46:09Z DEBUG Starting external process 2019-01-08T09:46:09Z DEBUG args=/bin/systemctl is-active httpd.service 2019-01-08T09:46:09Z DEBUG Process finished, return code=0 2019-01-08T09:46:09Z DEBUG stdout=active
2019-01-08T09:46:09Z DEBUG stderr= 2019-01-08T09:46:09Z DEBUG Start of httpd.service complete 2019-01-08T09:46:09Z INFO [Removing self-signed CA] 2019-01-08T09:46:09Z DEBUG Self-signed CA is not installed 2019-01-08T09:46:09Z INFO [Removing Dogtag 9 CA] 2019-01-08T09:46:09Z DEBUG Dogtag is version 10 or above 2019-01-08T09:46:09Z DEBUG Loading StateFile from '/var/lib/ipa/sysrestore/sysrestore.state' 2019-01-08T09:46:09Z DEBUG Loading Index file from '/var/lib/ipa/sysrestore/sysrestore.index' 2019-01-08T09:46:09Z INFO [Checking for deprecated KDC configuration files] 2019-01-08T09:46:09Z INFO [Checking for deprecated backups of Samba configuration files] 2019-01-08T09:46:09Z DEBUG raw: ca_is_enabled(version=u'2.229') 2019-01-08T09:46:09Z DEBUG ca_is_enabled(version=u'2.229') 2019-01-08T09:46:09Z DEBUG flushing ldapi://%2fvar%2frun%2fslapd-EXAMPLE-COM.socket from SchemaCache 2019-01-08T09:46:09Z DEBUG retrieving schema for SchemaCache url=ldapi://%2fvar%2frun%2fslapd-EXAMPLE-COM.socket conn=<ldap.ldapobject.SimpleLDAPObject instance at 0x7f8d6ccff5a8> 2019-01-08T09:46:09Z DEBUG raw: kra_is_enabled(version=u'2.229') 2019-01-08T09:46:09Z DEBUG kra_is_enabled(version=u'2.229') 2019-01-08T09:46:09Z DEBUG Cleaning up after pkispawn for the CA subsystem 2019-01-08T09:46:09Z DEBUG Loading StateFile from '/var/lib/ipa/sysrestore/sysrestore.state' 2019-01-08T09:46:09Z DEBUG Loading Index file from '/var/lib/ipa/sysrestore/sysrestore.index' 2019-01-08T09:46:09Z DEBUG Loading StateFile from '/var/lib/ipa/sysrestore/sysrestore.state' 2019-01-08T09:46:09Z INFO [Add missing CA DNS records] 2019-01-08T09:46:09Z DEBUG Loading StateFile from '/var/lib/ipa/sysupgrade/sysupgrade.state' 2019-01-08T09:46:09Z INFO IPA CA DNS records already processed 2019-01-08T09:46:09Z INFO [Removing deprecated DNS configuration options] 2019-01-08T09:46:09Z INFO DNS is not configured 2019-01-08T09:46:09Z INFO [Ensuring minimal number of connections] 2019-01-08T09:46:09Z INFO DNS is not configured 2019-01-08T09:46:09Z INFO [Updating GSSAPI configuration in DNS] 2019-01-08T09:46:09Z INFO DNS is not configured 2019-01-08T09:46:09Z INFO [Updating pid-file configuration in DNS] 2019-01-08T09:46:09Z INFO DNS is not configured 2019-01-08T09:46:09Z INFO DNS is not configured 2019-01-08T09:46:09Z INFO DNS is not configured 2019-01-08T09:46:09Z INFO DNS is not configured 2019-01-08T09:46:09Z INFO DNS is not configured 2019-01-08T09:46:09Z INFO DNS is not configured 2019-01-08T09:46:09Z DEBUG Loading StateFile from '/var/lib/ipa/sysrestore/sysrestore.state' 2019-01-08T09:46:09Z DEBUG Loading Index file from '/var/lib/ipa/sysrestore/sysrestore.index' 2019-01-08T09:46:09Z INFO DNS is not configured 2019-01-08T09:46:09Z DEBUG Loading StateFile from '/var/lib/ipa/sysupgrade/sysupgrade.state' 2019-01-08T09:46:09Z DEBUG Loading StateFile from '/var/lib/ipa/sysupgrade/sysupgrade.state' 2019-01-08T09:46:09Z DEBUG Loading StateFile from '/var/lib/ipa/sysrestore/sysrestore.state' 2019-01-08T09:46:09Z DEBUG Loading Index file from '/var/lib/ipa/sysrestore/sysrestore.index' 2019-01-08T09:46:09Z INFO DNS is not configured 2019-01-08T09:46:09Z DEBUG Loading StateFile from '/var/lib/ipa/sysrestore/sysrestore.state' 2019-01-08T09:46:09Z DEBUG Loading Index file from '/var/lib/ipa/sysrestore/sysrestore.index' 2019-01-08T09:46:09Z DEBUG Loading StateFile from '/var/lib/ipa/sysupgrade/sysupgrade.state' 2019-01-08T09:46:09Z INFO [Upgrading CA schema] 2019-01-08T09:46:10Z DEBUG Processing schema LDIF file /usr/share/pki/server/conf/schema-certProfile.ldif 2019-01-08T09:46:10Z DEBUG Processing schema LDIF file /usr/share/pki/server/conf/schema-authority.ldif 2019-01-08T09:46:10Z DEBUG Not updating schema 2019-01-08T09:46:10Z INFO CA schema update complete (no changes) 2019-01-08T09:46:10Z INFO [Verifying that CA audit signing cert has 2 year validity] 2019-01-08T09:46:10Z DEBUG caSignedLogCert.cfg profile validity range is 720 2019-01-08T09:46:10Z INFO [Update certmonger certificate renewal configuration] 2019-01-08T09:46:10Z DEBUG Loading Index file from '/var/lib/ipa/sysrestore/sysrestore.index' 2019-01-08T09:46:10Z DEBUG Starting external process 2019-01-08T09:46:10Z DEBUG args=/usr/bin/certutil -d dbm:/etc/httpd/alias -L -n Server-Cert -a -f /etc/httpd/alias/pwdfile.txt 2019-01-08T09:46:10Z DEBUG Process finished, return code=0 2019-01-08T09:46:10Z DEBUG stdout=-----BEGIN CERTIFICATE----- MIIFYjCCBEqgAwIBAgIED/4ABTANBgkqhkiG9w0BAQsFADBHMSUwIwYDVQQKExxE [...] Serial Number: 268304389 (0xffe0005) [...] n7aRLXFxWB8va5AxibMHYUJb08nWJRqHHwufF/YQnJdskFuDyAM= -----END CERTIFICATE-----
2019-01-08T09:46:10Z DEBUG stderr= 2019-01-08T09:46:10Z DEBUG Loading Index file from '/var/lib/ipa/sysrestore/sysrestore.index' 2019-01-08T09:46:10Z DEBUG Starting external process 2019-01-08T09:46:10Z DEBUG args=/usr/bin/certutil -d dbm:/etc/dirsrv/slapd-EXAMPLE-COM/ -L -n Server-Cert -a -f /etc/dirsrv/slapd-EXAMPLE-COM/pwdfile.txt 2019-01-08T09:46:10Z DEBUG Process finished, return code=0 2019-01-08T09:46:10Z DEBUG stdout=-----BEGIN CERTIFICATE----- MIIFYjCCBEqgAwIBAgIED/4ABDANBgkqhkiG9w0BAQsFADBHMSUwIwYDVQQKExxE [...] Serial Number: 268304388 (0xffe0004) [...] 7D6pk95Rgq6g73cZmogJBfJicVF0odkf2VkvV/aJCxTtZ4xkVds= -----END CERTIFICATE-----
2019-01-08T09:46:10Z DEBUG stderr= 2019-01-08T09:46:10Z DEBUG Loading Index file from '/var/lib/ipa/sysrestore/sysrestore.index' 2019-01-08T09:46:10Z DEBUG Starting external process 2019-01-08T09:46:10Z DEBUG args=/usr/bin/certutil -d dbm:/etc/pki/pki-tomcat/alias -L -f /etc/pki/pki-tomcat/alias/pwdfile.txt 2019-01-08T09:46:10Z DEBUG Process finished, return code=0 2019-01-08T09:46:10Z DEBUG stdout= Certificate Nickname Trust Attributes SSL,S/MIME,JAR/XPI
ocspSigningCert cert-pki-ca u,u,u caSigningCert cert-pki-ca CTu,Cu,Cu Server-Cert cert-pki-ca u,u,u auditSigningCert cert-pki-ca u,u,Pu subsystemCert cert-pki-ca u,u,u
2019-01-08T09:46:10Z DEBUG stderr= 2019-01-08T09:46:11Z INFO Certmonger certificate renewal configuration already up-to-date 2019-01-08T09:46:11Z INFO [Enable PKIX certificate path discovery and validation] 2019-01-08T09:46:11Z DEBUG Loading StateFile from '/var/lib/ipa/sysupgrade/sysupgrade.state' 2019-01-08T09:46:11Z INFO PKIX already enabled 2019-01-08T09:46:11Z INFO [Authorizing RA Agent to modify profiles] 2019-01-08T09:46:11Z INFO [Authorizing RA Agent to manage lightweight CAs] 2019-01-08T09:46:11Z INFO [Ensuring Lightweight CAs container exists in Dogtag database] 2019-01-08T09:46:11Z DEBUG Created connection context.ldap2_140245392902928 2019-01-08T09:46:11Z DEBUG flushing ldapi://%2fvar%2frun%2fslapd-EXAMPLE-COM.socket from SchemaCache 2019-01-08T09:46:11Z DEBUG retrieving schema for SchemaCache url=ldapi://%2fvar%2frun%2fslapd-EXAMPLE-COM.socket conn=<ldap.ldapobject.SimpleLDAPObject instance at 0x7f8d6cea4fc8> 2019-01-08T09:46:12Z DEBUG Destroyed connection context.ldap2_140245392902928 2019-01-08T09:46:12Z INFO [Adding default OCSP URI configuration] 2019-01-08T09:46:12Z INFO [Ensuring CA is using LDAPProfileSubsystem] 2019-01-08T09:46:12Z INFO [Migrating certificate profiles to LDAP] 2019-01-08T09:46:12Z DEBUG Created connection context.ldap2_140245366754832 2019-01-08T09:46:12Z DEBUG flushing ldapi://%2fvar%2frun%2fslapd-EXAMPLE-COM.socket from SchemaCache 2019-01-08T09:46:12Z DEBUG retrieving schema for SchemaCache url=ldapi://%2fvar%2frun%2fslapd-EXAMPLE-COM.socket conn=<ldap.ldapobject.SimpleLDAPObject instance at 0x7f8d6b5db128> 2019-01-08T09:46:12Z DEBUG Destroyed connection context.ldap2_140245366754832 2019-01-08T09:46:12Z DEBUG request GET https://ipa2.example.com:8443/ca/rest/account/login 2019-01-08T09:46:12Z DEBUG request body '' 2019-01-08T09:46:12Z DEBUG response status 401 2019-01-08T09:46:12Z DEBUG response headers Server: Apache-Coyote/1.1 Cache-Control: private Expires: Thu, 01 Jan 1970 01:00:00 CET WWW-Authenticate: Basic realm="Certificate Authority" Content-Type: text/html;charset=utf-8 Content-Language: en Content-Length: 951 Date: Tue, 08 Jan 2019 09:46:12 GMT
2019-01-08T09:46:12Z DEBUG response body '<html> [...] HTTP Status 401 [...] message [...] This request requires HTTP authentication. [...] </html>' 2019-01-08T09:46:12Z ERROR IPA server upgrade failed: Inspect /var/log/ipaupgrade.log and run command ipa-server-upgrade manually. 2019-01-08T09:46:12Z DEBUG File "/usr/lib/python2.7/site-packages/ipapython/admintool.py", line 178, in execute return_value = self.run() File "/usr/lib/python2.7/site-packages/ipaserver/install/ipa_server_upgrade.py", line 54, in run server.upgrade() File "/usr/lib/python2.7/site-packages/ipaserver/install/server/upgrade.py", line 2085, in upgrade upgrade_configuration() File "/usr/lib/python2.7/site-packages/ipaserver/install/server/upgrade.py", line 1952, in upgrade_configuration ca_enable_ldap_profile_subsystem(ca) File "/usr/lib/python2.7/site-packages/ipaserver/install/server/upgrade.py", line 396, in ca_enable_ldap_profile_subsystem cainstance.migrate_profiles_to_ldap() File "/usr/lib/python2.7/site-packages/ipaserver/install/cainstance.py", line 1814, in migrate_profiles_to_ldap _create_dogtag_profile(profile_id, profile_data, overwrite=False) File "/usr/lib/python2.7/site-packages/ipaserver/install/cainstance.py", line 1820, in _create_dogtag_profile with api.Backend.ra_certprofile as profile_api: File "/usr/lib/python2.7/site-packages/ipaserver/plugins/dogtag.py", line 1302, in __enter__ raise errors.RemoteRetrieveError(reason=_('Failed to authenticate to CA REST API'))
2019-01-08T09:46:12Z DEBUG The ipa-server-upgrade command failed, exception: RemoteRetrieveError: Failed to authenticate to CA REST API 2019-01-08T09:46:12Z ERROR Unexpected error - see /var/log/ipaupgrade.log for details: RemoteRetrieveError: Failed to authenticate to CA REST API 2019-01-08T09:46:12Z ERROR The ipa-server-upgrade command failed. See /var/log/ipaupgrade.log for more information ---
Mit freundlichen Gruessen/With best regards,
--Daniel.
On 1/8/19 3:51 PM, dbischof--- via FreeIPA-users wrote:
Hi Florence,
On Mon, 7 Jan 2019, Florence Blanc-Renaud wrote:
[...]
i shaved this thread a little, since it gets confusing. Hope i kept the interesting bits.
The errors related to "Unable to find request for serial xxx" mean that the cert is tracked by certmonger, but there is no corresponding LDAP entry cn=xxx,ou=ca,ou=requests,o=ipaca on the local LDAP server. Is the replication still working between your two masters?
"ipa cert-show" broken on ipa2 often points to an out-of-date certificate in /var/lib/ipa/ra-agent.{pem|key}. The content should be the same as on the renewal master, and also the same as in the entry uid=ipara,ou=People,o=ipaca (which should be replicated and identical on all the masters, except if you have replication issues).
Replication appeared to be working, however, I did server maintenance today, ipa1 and ipa2 got upgraded and rebooted. ipa1 works fine, ipa2 does not come up properly:
--- ipa2 $ ipactl status Directory Service: RUNNING krb5kdc Service: STOPPED kadmin Service: STOPPED
You need to have a look at krb5 logs to understand why kerberos failed to start. Please check https://www.freeipa.org/page/Troubleshooting/Kerberos for more information.
Turns out that kerberos starts when started manually with systemctl:
$ ipactl status Directory Service: RUNNING krb5kdc Service: RUNNING kadmin Service: RUNNING httpd Service: RUNNING ipa-custodia Service: RUNNING ntpd Service: RUNNING pki-tomcatd Service: RUNNING ipa-otpd Service: STOPPED ipa: INFO: The ipactl command was successful
Kerberos log:
[...] : NEEDED_PREAUTH: admin@EXAMPLE.COM for krbtgt/EXAMPLE.COM@EXAMPLE.COM, Additional pre-authentication required Jan 08 10:39:22 ipa2.example.com krb5kdc[21691](info): closing down fd 11 Jan 08 10:39:39 ipa2.example.com krb5kdc[21692](info): AS_REQ (8 etypes {18 17 20 19 16 23 25 26}) 141.51.x.y: ISSUE: authtime 1546940379, etypes {rep=18 tkt=18 ses=18}, admin@EXAMPLE.COM for krbtgt/EXAMPLE.COM@EXAMPLE.COM Jan 08 10:39:39 ipa2.example.com krb5kdc[21692](info): closing down fd 11 Jan 08 10:40:32 ipa2.example.com krb5kdc[21692](info): TGS_REQ (8 etypes {18 17 20 19 16 23 25 26}) 141.51.x.y: ISSUE: authtime 1546940379, etypes {rep=18 tkt=18 ses=18}, admin@EXAMPLE.COM for HTTP/ipa2.example.com@EXAMPLE.COM Jan 08 10:40:32 ipa2.example.com krb5kdc[21692](info): closing down fd 11
httpd Service: RUNNING ipa-custodia Service: STOPPED ntpd Service: RUNNING pki-tomcatd Service: RUNNING ipa-otpd Service: STOPPED ipa: INFO: The ipactl command was successful ---
--- ipa1 $ openssl x509 -in /var/lib/ipa/ra-agent.pem -text -noout | grep Serial Serial Number: 268304391 (0xffe0007)
$ ldapsearch -x -b uid=ipara,ou=People,o=ipaca [...] description: 2;268304391;CN=Certificate Authority,O=EXAMPLE.COM;CN=IPA RA,O=EXAMPLE.COM ---
Looks good.
--- ipa2 $ openssl x509 -in /var/lib/ipa/ra-agent.pem -text -noout | grep Serial Serial Number: 268304391 (0xffe0007)
$ ldapsearch -x -b uid=ipara,ou=People,o=ipaca [...] description: 2;44;CN=Certificate Authority,O=EXAMPLE.COM;CN=IPA RA,O=EXAMPLE.COM ---
Looks bad.
So yes, the replication of the suffix o=ipaca seems broken. In a deployment with CA, the ldap server contains 2 different suffixes, one for the IdM data (your base DN as defined in /etc/ipa/default.conf) and another one for the CA, below o=ipaca.
You can have a look at https://www.freeipa.org/page/Troubleshooting/Directory_Server for replication troubleshooting, but the repl issue could be a consequence of kerberos server not starting.
I checked the troubleshooting pages and carried out the steps described in there, but my system (ipa2) appears to be ok as far as the steps go.
--- /var/log/ipaupgrade.log [...] 2018-12-28T10:55:53Z DEBUG The ipa-server-upgrade command failed, exception: RemoteRetrieveError: Failed to authenticate to CA REST API ---
--- /var/log/dirsrv/slapd-EXAMPLE-COM/errors [...] [28/Dec/2018:11:55:55.559648469 +0100] - ERR - set_krb5_creds - Could not get initial credentials for principal [ldap/ipa2.example.com@EXAMPLE.COM] in keytab [FILE:/etc/dirsrv/ds.keytab]: -1765328228 (Cannot contact any KDC for requested realm) ---
--- ipa-checkcerts.py [...] ipa: ERROR: ra.get_certificate(): EXCEPTION (Invalid Credential.) ra.get_certificate(): EXCEPTION (Invalid Credential.) ipa: INFO: Checking permissions and ownership Checking permissions and ownership ipa: INFO: Failures: Failures: ipa: INFO: Unable to find request for serial 268304391 Unable to find request for serial 268304391 ipa: INFO: Unable to find request for serial 268304394 Unable to find request for serial 268304394 ipa: INFO: Unable to find request for serial 268304393 Unable to find request for serial 268304393 ipa: INFO: Unable to find request for serial 268304392 Unable to find request for serial 268304392 ipa: INFO: Unable to find request for serial 268304390 Unable to find request for serial 268304390 ipa: INFO: RA agent description does not match 2;44;CN=Certificate Authority,O=EXAMPLE.COM;CN=IPA RA,O=EXAMPLE.COM in LDAP and 2;268304391;CN=Certificate Authority,O=EXAMPLE.COM;CN=IPA RA,O=EXAMPLE.COM expected ipa: INFO: RA agent certificate not found in LDAP RA agent certificate not found in LDAP [...] ---
The root cause appears to be the wrong IPA RA certificate in ipa2's LDAP, right? I guess this has to fixed by manually importing the proper certificate using ldapmodify, similar to the procedure described in [1]?
It is possible to manually modify the IPA RA cert in ipa2's ldap server but this won't solve the replication issue. It will however allow you to run ipa cert-show.
Unfortunately, it doesn't (ipa2):
ipa: ERROR: Certificate operation cannot be completed: EXCEPTION (Invalid Credential.)
Kerberos log:
Jan 08 15:42:45 ipa2.example.com krb5kdc[23520](info): AS_REQ (8 etypes {18 17 16 23 20 19 25 26}) 141.51.124.20: NEEDED_PREAUTH: host/ipa2.example.com@EXAMPLE.COM for krbtgt/EXAMPLE.COM@EXAMPLE.COM, Additional pre-authentication required
Works ok on ipa1.
I'm unsure how to proceed here. My obvious problem is ipa2 not getting upgraded properly:
[...] 2019-01-08T09:46:07Z INFO [Updating HTTPD service IPA configuration] 2019-01-08T09:46:07Z DEBUG Starting external process 2019-01-08T09:46:07Z DEBUG args=/usr/sbin/selinuxenabled 2019-01-08T09:46:07Z DEBUG Process finished, return code=1 2019-01-08T09:46:07Z DEBUG stdout= 2019-01-08T09:46:07Z DEBUG stderr= 2019-01-08T09:46:07Z DEBUG Starting external process 2019-01-08T09:46:07Z DEBUG args=/bin/systemctl --system daemon-reload 2019-01-08T09:46:07Z DEBUG Process finished, return code=0 2019-01-08T09:46:07Z DEBUG stdout= 2019-01-08T09:46:07Z DEBUG stderr= 2019-01-08T09:46:07Z INFO [Updating HTTPD service IPA WSGI configuration] 2019-01-08T09:46:07Z INFO Nothing to do for configure_httpd_wsgi_conf 2019-01-08T09:46:07Z INFO [Updating mod_nss protocol versions] 2019-01-08T09:46:07Z DEBUG Loading StateFile from '/var/lib/ipa/sysupgrade/sysupgrade.state' 2019-01-08T09:46:07Z INFO Protocol versions already updated 2019-01-08T09:46:07Z INFO [Updating mod_nss cipher suite] 2019-01-08T09:46:07Z DEBUG Loading StateFile from '/var/lib/ipa/sysupgrade/sysupgrade.state' 2019-01-08T09:46:07Z DEBUG Cipher suite already updated 2019-01-08T09:46:07Z INFO [Updating mod_nss enabling OCSP] 2019-01-08T09:46:07Z DEBUG Loading StateFile from '/var/lib/ipa/sysupgrade/sysupgrade.state' 2019-01-08T09:46:07Z INFO [Fixing trust flags in /etc/httpd/alias] 2019-01-08T09:46:07Z DEBUG Loading StateFile from '/var/lib/ipa/sysupgrade/sysupgrade.state' 2019-01-08T09:46:07Z INFO Trust flags already processed 2019-01-08T09:46:07Z INFO [Moving HTTPD service keytab to gssproxy] 2019-01-08T09:46:07Z DEBUG Starting external process 2019-01-08T09:46:07Z DEBUG args=/usr/sbin/selinuxenabled 2019-01-08T09:46:07Z DEBUG Process finished, return code=1 2019-01-08T09:46:07Z DEBUG stdout= 2019-01-08T09:46:07Z DEBUG stderr= 2019-01-08T09:46:07Z DEBUG Starting external process 2019-01-08T09:46:07Z DEBUG args=/bin/systemctl restart gssproxy.service 2019-01-08T09:46:07Z DEBUG Process finished, return code=0 2019-01-08T09:46:07Z DEBUG stdout= 2019-01-08T09:46:07Z DEBUG stderr= 2019-01-08T09:46:07Z DEBUG Starting external process 2019-01-08T09:46:07Z DEBUG args=/bin/systemctl is-active gssproxy.service 2019-01-08T09:46:07Z DEBUG Process finished, return code=0 2019-01-08T09:46:07Z DEBUG stdout=active
2019-01-08T09:46:07Z DEBUG stderr= 2019-01-08T09:46:07Z DEBUG Restart of gssproxy.service complete 2019-01-08T09:46:07Z DEBUG Starting external process 2019-01-08T09:46:07Z DEBUG args=/bin/systemctl start httpd.service 2019-01-08T09:46:09Z DEBUG Process finished, return code=0 2019-01-08T09:46:09Z DEBUG stdout= 2019-01-08T09:46:09Z DEBUG stderr= 2019-01-08T09:46:09Z DEBUG Starting external process 2019-01-08T09:46:09Z DEBUG args=/bin/systemctl is-active httpd.service 2019-01-08T09:46:09Z DEBUG Process finished, return code=0 2019-01-08T09:46:09Z DEBUG stdout=active
2019-01-08T09:46:09Z DEBUG stderr= 2019-01-08T09:46:09Z DEBUG Start of httpd.service complete 2019-01-08T09:46:09Z INFO [Removing self-signed CA] 2019-01-08T09:46:09Z DEBUG Self-signed CA is not installed 2019-01-08T09:46:09Z INFO [Removing Dogtag 9 CA] 2019-01-08T09:46:09Z DEBUG Dogtag is version 10 or above 2019-01-08T09:46:09Z DEBUG Loading StateFile from '/var/lib/ipa/sysrestore/sysrestore.state' 2019-01-08T09:46:09Z DEBUG Loading Index file from '/var/lib/ipa/sysrestore/sysrestore.index' 2019-01-08T09:46:09Z INFO [Checking for deprecated KDC configuration files] 2019-01-08T09:46:09Z INFO [Checking for deprecated backups of Samba configuration files] 2019-01-08T09:46:09Z DEBUG raw: ca_is_enabled(version=u'2.229') 2019-01-08T09:46:09Z DEBUG ca_is_enabled(version=u'2.229') 2019-01-08T09:46:09Z DEBUG flushing ldapi://%2fvar%2frun%2fslapd-EXAMPLE-COM.socket from SchemaCache 2019-01-08T09:46:09Z DEBUG retrieving schema for SchemaCache url=ldapi://%2fvar%2frun%2fslapd-EXAMPLE-COM.socket conn=<ldap.ldapobject.SimpleLDAPObject instance at 0x7f8d6ccff5a8> 2019-01-08T09:46:09Z DEBUG raw: kra_is_enabled(version=u'2.229') 2019-01-08T09:46:09Z DEBUG kra_is_enabled(version=u'2.229') 2019-01-08T09:46:09Z DEBUG Cleaning up after pkispawn for the CA subsystem 2019-01-08T09:46:09Z DEBUG Loading StateFile from '/var/lib/ipa/sysrestore/sysrestore.state' 2019-01-08T09:46:09Z DEBUG Loading Index file from '/var/lib/ipa/sysrestore/sysrestore.index' 2019-01-08T09:46:09Z DEBUG Loading StateFile from '/var/lib/ipa/sysrestore/sysrestore.state' 2019-01-08T09:46:09Z INFO [Add missing CA DNS records] 2019-01-08T09:46:09Z DEBUG Loading StateFile from '/var/lib/ipa/sysupgrade/sysupgrade.state' 2019-01-08T09:46:09Z INFO IPA CA DNS records already processed 2019-01-08T09:46:09Z INFO [Removing deprecated DNS configuration options] 2019-01-08T09:46:09Z INFO DNS is not configured 2019-01-08T09:46:09Z INFO [Ensuring minimal number of connections] 2019-01-08T09:46:09Z INFO DNS is not configured 2019-01-08T09:46:09Z INFO [Updating GSSAPI configuration in DNS] 2019-01-08T09:46:09Z INFO DNS is not configured 2019-01-08T09:46:09Z INFO [Updating pid-file configuration in DNS] 2019-01-08T09:46:09Z INFO DNS is not configured 2019-01-08T09:46:09Z INFO DNS is not configured 2019-01-08T09:46:09Z INFO DNS is not configured 2019-01-08T09:46:09Z INFO DNS is not configured 2019-01-08T09:46:09Z INFO DNS is not configured 2019-01-08T09:46:09Z INFO DNS is not configured 2019-01-08T09:46:09Z DEBUG Loading StateFile from '/var/lib/ipa/sysrestore/sysrestore.state' 2019-01-08T09:46:09Z DEBUG Loading Index file from '/var/lib/ipa/sysrestore/sysrestore.index' 2019-01-08T09:46:09Z INFO DNS is not configured 2019-01-08T09:46:09Z DEBUG Loading StateFile from '/var/lib/ipa/sysupgrade/sysupgrade.state' 2019-01-08T09:46:09Z DEBUG Loading StateFile from '/var/lib/ipa/sysupgrade/sysupgrade.state' 2019-01-08T09:46:09Z DEBUG Loading StateFile from '/var/lib/ipa/sysrestore/sysrestore.state' 2019-01-08T09:46:09Z DEBUG Loading Index file from '/var/lib/ipa/sysrestore/sysrestore.index' 2019-01-08T09:46:09Z INFO DNS is not configured 2019-01-08T09:46:09Z DEBUG Loading StateFile from '/var/lib/ipa/sysrestore/sysrestore.state' 2019-01-08T09:46:09Z DEBUG Loading Index file from '/var/lib/ipa/sysrestore/sysrestore.index' 2019-01-08T09:46:09Z DEBUG Loading StateFile from '/var/lib/ipa/sysupgrade/sysupgrade.state' 2019-01-08T09:46:09Z INFO [Upgrading CA schema] 2019-01-08T09:46:10Z DEBUG Processing schema LDIF file /usr/share/pki/server/conf/schema-certProfile.ldif 2019-01-08T09:46:10Z DEBUG Processing schema LDIF file /usr/share/pki/server/conf/schema-authority.ldif 2019-01-08T09:46:10Z DEBUG Not updating schema 2019-01-08T09:46:10Z INFO CA schema update complete (no changes) 2019-01-08T09:46:10Z INFO [Verifying that CA audit signing cert has 2 year validity] 2019-01-08T09:46:10Z DEBUG caSignedLogCert.cfg profile validity range is 720 2019-01-08T09:46:10Z INFO [Update certmonger certificate renewal configuration] 2019-01-08T09:46:10Z DEBUG Loading Index file from '/var/lib/ipa/sysrestore/sysrestore.index' 2019-01-08T09:46:10Z DEBUG Starting external process 2019-01-08T09:46:10Z DEBUG args=/usr/bin/certutil -d dbm:/etc/httpd/alias -L -n Server-Cert -a -f /etc/httpd/alias/pwdfile.txt 2019-01-08T09:46:10Z DEBUG Process finished, return code=0 2019-01-08T09:46:10Z DEBUG stdout=-----BEGIN CERTIFICATE----- MIIFYjCCBEqgAwIBAgIED/4ABTANBgkqhkiG9w0BAQsFADBHMSUwIwYDVQQKExxE [...] Serial Number: 268304389 (0xffe0005) [...] n7aRLXFxWB8va5AxibMHYUJb08nWJRqHHwufF/YQnJdskFuDyAM= -----END CERTIFICATE-----
2019-01-08T09:46:10Z DEBUG stderr= 2019-01-08T09:46:10Z DEBUG Loading Index file from '/var/lib/ipa/sysrestore/sysrestore.index' 2019-01-08T09:46:10Z DEBUG Starting external process 2019-01-08T09:46:10Z DEBUG args=/usr/bin/certutil -d dbm:/etc/dirsrv/slapd-EXAMPLE-COM/ -L -n Server-Cert -a -f /etc/dirsrv/slapd-EXAMPLE-COM/pwdfile.txt 2019-01-08T09:46:10Z DEBUG Process finished, return code=0 2019-01-08T09:46:10Z DEBUG stdout=-----BEGIN CERTIFICATE----- MIIFYjCCBEqgAwIBAgIED/4ABDANBgkqhkiG9w0BAQsFADBHMSUwIwYDVQQKExxE [...] Serial Number: 268304388 (0xffe0004) [...] 7D6pk95Rgq6g73cZmogJBfJicVF0odkf2VkvV/aJCxTtZ4xkVds= -----END CERTIFICATE-----
2019-01-08T09:46:10Z DEBUG stderr= 2019-01-08T09:46:10Z DEBUG Loading Index file from '/var/lib/ipa/sysrestore/sysrestore.index' 2019-01-08T09:46:10Z DEBUG Starting external process 2019-01-08T09:46:10Z DEBUG args=/usr/bin/certutil -d dbm:/etc/pki/pki-tomcat/alias -L -f /etc/pki/pki-tomcat/alias/pwdfile.txt 2019-01-08T09:46:10Z DEBUG Process finished, return code=0 2019-01-08T09:46:10Z DEBUG stdout= Certificate Nickname Trust Attributes
SSL,S/MIME,JAR/XPI
ocspSigningCert cert-pki-ca u,u,u caSigningCert cert-pki-ca CTu,Cu,Cu Server-Cert cert-pki-ca u,u,u auditSigningCert cert-pki-ca u,u,Pu subsystemCert cert-pki-ca u,u,u
2019-01-08T09:46:10Z DEBUG stderr= 2019-01-08T09:46:11Z INFO Certmonger certificate renewal configuration already up-to-date 2019-01-08T09:46:11Z INFO [Enable PKIX certificate path discovery and validation] 2019-01-08T09:46:11Z DEBUG Loading StateFile from '/var/lib/ipa/sysupgrade/sysupgrade.state' 2019-01-08T09:46:11Z INFO PKIX already enabled 2019-01-08T09:46:11Z INFO [Authorizing RA Agent to modify profiles] 2019-01-08T09:46:11Z INFO [Authorizing RA Agent to manage lightweight CAs] 2019-01-08T09:46:11Z INFO [Ensuring Lightweight CAs container exists in Dogtag database] 2019-01-08T09:46:11Z DEBUG Created connection context.ldap2_140245392902928 2019-01-08T09:46:11Z DEBUG flushing ldapi://%2fvar%2frun%2fslapd-EXAMPLE-COM.socket from SchemaCache 2019-01-08T09:46:11Z DEBUG retrieving schema for SchemaCache url=ldapi://%2fvar%2frun%2fslapd-EXAMPLE-COM.socket conn=<ldap.ldapobject.SimpleLDAPObject instance at 0x7f8d6cea4fc8> 2019-01-08T09:46:12Z DEBUG Destroyed connection context.ldap2_140245392902928 2019-01-08T09:46:12Z INFO [Adding default OCSP URI configuration] 2019-01-08T09:46:12Z INFO [Ensuring CA is using LDAPProfileSubsystem] 2019-01-08T09:46:12Z INFO [Migrating certificate profiles to LDAP] 2019-01-08T09:46:12Z DEBUG Created connection context.ldap2_140245366754832 2019-01-08T09:46:12Z DEBUG flushing ldapi://%2fvar%2frun%2fslapd-EXAMPLE-COM.socket from SchemaCache 2019-01-08T09:46:12Z DEBUG retrieving schema for SchemaCache url=ldapi://%2fvar%2frun%2fslapd-EXAMPLE-COM.socket conn=<ldap.ldapobject.SimpleLDAPObject instance at 0x7f8d6b5db128> 2019-01-08T09:46:12Z DEBUG Destroyed connection context.ldap2_140245366754832 2019-01-08T09:46:12Z DEBUG request GET https://ipa2.example.com:8443/ca/rest/account/login 2019-01-08T09:46:12Z DEBUG request body '' 2019-01-08T09:46:12Z DEBUG response status 401 2019-01-08T09:46:12Z DEBUG response headers Server: Apache-Coyote/1.1 Cache-Control: private Expires: Thu, 01 Jan 1970 01:00:00 CET WWW-Authenticate: Basic realm="Certificate Authority" Content-Type: text/html;charset=utf-8 Content-Language: en Content-Length: 951 Date: Tue, 08 Jan 2019 09:46:12 GMT
2019-01-08T09:46:12Z DEBUG response body '<html> [...] HTTP Status 401 [...] message [...] This request requires HTTP authentication. [...]
</html>' 2019-01-08T09:46:12Z ERROR IPA server upgrade failed: Inspect /var/log/ipaupgrade.log and run command ipa-server-upgrade manually. 2019-01-08T09:46:12Z DEBUG File "/usr/lib/python2.7/site-packages/ipapython/admintool.py", line 178, in execute return_value = self.run() File "/usr/lib/python2.7/site-packages/ipaserver/install/ipa_server_upgrade.py", line 54, in run server.upgrade() File "/usr/lib/python2.7/site-packages/ipaserver/install/server/upgrade.py", line 2085, in upgrade upgrade_configuration() File "/usr/lib/python2.7/site-packages/ipaserver/install/server/upgrade.py", line 1952, in upgrade_configuration ca_enable_ldap_profile_subsystem(ca) File "/usr/lib/python2.7/site-packages/ipaserver/install/server/upgrade.py", line 396, in ca_enable_ldap_profile_subsystem cainstance.migrate_profiles_to_ldap() File "/usr/lib/python2.7/site-packages/ipaserver/install/cainstance.py", line 1814, in migrate_profiles_to_ldap _create_dogtag_profile(profile_id, profile_data, overwrite=False) File "/usr/lib/python2.7/site-packages/ipaserver/install/cainstance.py", line 1820, in _create_dogtag_profile with api.Backend.ra_certprofile as profile_api: File "/usr/lib/python2.7/site-packages/ipaserver/plugins/dogtag.py", line 1302, in __enter__ raise errors.RemoteRetrieveError(reason=_('Failed to authenticate to CA REST API'))
2019-01-08T09:46:12Z DEBUG The ipa-server-upgrade command failed, exception: RemoteRetrieveError: Failed to authenticate to CA REST API 2019-01-08T09:46:12Z ERROR Unexpected error - see /var/log/ipaupgrade.log for details: RemoteRetrieveError: Failed to authenticate to CA REST API 2019-01-08T09:46:12Z ERROR The ipa-server-upgrade command failed. See /var/log/ipaupgrade.log for more information
Hi, on ipa2, what is the content of /var/lib/ipa/ra-agent.pem? Is it the same as on ipa1? Can you check the file permissions?
Can you also compare with the ldap entry uid=ipara,ou=People,o=ipaca (description and usercertificate fields are important).
flo
Mit freundlichen Gruessen/With best regards,
--Daniel. _______________________________________________ FreeIPA-users mailing list -- freeipa-users@lists.fedorahosted.org To unsubscribe send an email to freeipa-users-leave@lists.fedorahosted.org Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedorahoste...
Hi Florence,
On Tue, 8 Jan 2019, Florence Blanc-Renaud wrote:
On 1/8/19 3:51 PM, dbischof--- via FreeIPA-users wrote:
Hi Florence,
On Mon, 7 Jan 2019, Florence Blanc-Renaud wrote:
[...]
i shaved this thread a little, since it gets confusing. Hope i kept the interesting bits.
The errors related to "Unable to find request for serial xxx" mean that the cert is tracked by certmonger, but there is no corresponding LDAP entry cn=xxx,ou=ca,ou=requests,o=ipaca on the local LDAP server. Is the replication still working between your two masters?
"ipa cert-show" broken on ipa2 often points to an out-of-date certificate in /var/lib/ipa/ra-agent.{pem|key}. The content should be the same as on the renewal master, and also the same as in the entry uid=ipara,ou=People,o=ipaca (which should be replicated and identical on all the masters, except if you have replication issues).
Replication appeared to be working, however, I did server maintenance today, ipa1 and ipa2 got upgraded and rebooted. ipa1 works fine, ipa2 does not come up properly:
--- ipa2 $ ipactl status Directory Service: RUNNING krb5kdc Service: STOPPED kadmin Service: STOPPED
You need to have a look at krb5 logs to understand why kerberos failed to start. Please check https://www.freeipa.org/page/Troubleshooting/Kerberos for more information.
Turns out that kerberos starts when started manually with systemctl:
$ ipactl status Directory Service: RUNNING krb5kdc Service: RUNNING kadmin Service: RUNNING httpd Service: RUNNING ipa-custodia Service: RUNNING ntpd Service: RUNNING pki-tomcatd Service: RUNNING ipa-otpd Service: STOPPED ipa: INFO: The ipactl command was successful
Kerberos log:
[...] : NEEDED_PREAUTH: admin@EXAMPLE.COM for krbtgt/EXAMPLE.COM@EXAMPLE.COM, Additional pre-authentication required Jan 08 10:39:22 ipa2.example.com krb5kdc[21691](info): closing down fd 11 Jan 08 10:39:39 ipa2.example.com krb5kdc[21692](info): AS_REQ (8 etypes {18 17 20 19 16 23 25 26}) 141.51.x.y: ISSUE: authtime 1546940379, etypes {rep=18 tkt=18 ses=18}, admin@EXAMPLE.COM for krbtgt/EXAMPLE.COM@EXAMPLE.COM Jan 08 10:39:39 ipa2.example.com krb5kdc[21692](info): closing down fd 11 Jan 08 10:40:32 ipa2.example.com krb5kdc[21692](info): TGS_REQ (8 etypes {18 17 20 19 16 23 25 26}) 141.51.x.y: ISSUE: authtime 1546940379, etypes {rep=18 tkt=18 ses=18}, admin@EXAMPLE.COM for HTTP/ipa2.example.com@EXAMPLE.COM Jan 08 10:40:32 ipa2.example.com krb5kdc[21692](info): closing down fd 11
httpd Service: RUNNING ipa-custodia Service: STOPPED ntpd Service: RUNNING pki-tomcatd Service: RUNNING ipa-otpd Service: STOPPED ipa: INFO: The ipactl command was successful ---
--- ipa1 $ openssl x509 -in /var/lib/ipa/ra-agent.pem -text -noout | grep Serial Serial Number: 268304391 (0xffe0007)
$ ldapsearch -x -b uid=ipara,ou=People,o=ipaca [...] description: 2;268304391;CN=Certificate Authority,O=EXAMPLE.COM;CN=IPA RA,O=EXAMPLE.COM ---
Looks good.
--- ipa2 $ openssl x509 -in /var/lib/ipa/ra-agent.pem -text -noout | grep Serial Serial Number: 268304391 (0xffe0007)
$ ldapsearch -x -b uid=ipara,ou=People,o=ipaca [...] description: 2;44;CN=Certificate Authority,O=EXAMPLE.COM;CN=IPA RA,O=EXAMPLE.COM ---
Looks bad.
So yes, the replication of the suffix o=ipaca seems broken. In a deployment with CA, the ldap server contains 2 different suffixes, one for the IdM data (your base DN as defined in /etc/ipa/default.conf) and another one for the CA, below o=ipaca.
You can have a look at https://www.freeipa.org/page/Troubleshooting/Directory_Server for replication troubleshooting, but the repl issue could be a consequence of kerberos server not starting.
I checked the troubleshooting pages and carried out the steps described in there, but my system (ipa2) appears to be ok as far as the steps go.
--- /var/log/ipaupgrade.log [...] 2018-12-28T10:55:53Z DEBUG The ipa-server-upgrade command failed, exception: RemoteRetrieveError: Failed to authenticate to CA REST API ---
--- /var/log/dirsrv/slapd-EXAMPLE-COM/errors [...] [28/Dec/2018:11:55:55.559648469 +0100] - ERR - set_krb5_creds - Could not get initial credentials for principal [ldap/ipa2.example.com@EXAMPLE.COM] in keytab [FILE:/etc/dirsrv/ds.keytab]: -1765328228 (Cannot contact any KDC for requested realm) ---
--- ipa-checkcerts.py [...] ipa: ERROR: ra.get_certificate(): EXCEPTION (Invalid Credential.) ra.get_certificate(): EXCEPTION (Invalid Credential.) ipa: INFO: Checking permissions and ownership Checking permissions and ownership ipa: INFO: Failures: Failures: ipa: INFO: Unable to find request for serial 268304391 Unable to find request for serial 268304391 ipa: INFO: Unable to find request for serial 268304394 Unable to find request for serial 268304394 ipa: INFO: Unable to find request for serial 268304393 Unable to find request for serial 268304393 ipa: INFO: Unable to find request for serial 268304392 Unable to find request for serial 268304392 ipa: INFO: Unable to find request for serial 268304390 Unable to find request for serial 268304390 ipa: INFO: RA agent description does not match 2;44;CN=Certificate Authority,O=EXAMPLE.COM;CN=IPA RA,O=EXAMPLE.COM in LDAP and 2;268304391;CN=Certificate Authority,O=EXAMPLE.COM;CN=IPA RA,O=EXAMPLE.COM expected ipa: INFO: RA agent certificate not found in LDAP RA agent certificate not found in LDAP [...] ---
The root cause appears to be the wrong IPA RA certificate in ipa2's LDAP, right? I guess this has to fixed by manually importing the proper certificate using ldapmodify, similar to the procedure described in [1]?
It is possible to manually modify the IPA RA cert in ipa2's ldap server but this won't solve the replication issue. It will however allow you to run ipa cert-show.
Unfortunately, it doesn't (ipa2):
ipa: ERROR: Certificate operation cannot be completed: EXCEPTION (Invalid Credential.)
Kerberos log:
Jan 08 15:42:45 ipa2.example.com krb5kdc[23520](info): AS_REQ (8 etypes {18 17 16 23 20 19 25 26}) 141.51.124.20: NEEDED_PREAUTH: host/ipa2.example.com@EXAMPLE.COM for krbtgt/EXAMPLE.COM@EXAMPLE.COM, Additional pre-authentication required
Works ok on ipa1.
I'm unsure how to proceed here. My obvious problem is ipa2 not getting upgraded properly:
[...] 2019-01-08T09:46:07Z INFO [Updating HTTPD service IPA configuration] 2019-01-08T09:46:07Z DEBUG Starting external process 2019-01-08T09:46:07Z DEBUG args=/usr/sbin/selinuxenabled 2019-01-08T09:46:07Z DEBUG Process finished, return code=1 2019-01-08T09:46:07Z DEBUG stdout= 2019-01-08T09:46:07Z DEBUG stderr= 2019-01-08T09:46:07Z DEBUG Starting external process 2019-01-08T09:46:07Z DEBUG args=/bin/systemctl --system daemon-reload 2019-01-08T09:46:07Z DEBUG Process finished, return code=0 2019-01-08T09:46:07Z DEBUG stdout= 2019-01-08T09:46:07Z DEBUG stderr= 2019-01-08T09:46:07Z INFO [Updating HTTPD service IPA WSGI configuration] 2019-01-08T09:46:07Z INFO Nothing to do for configure_httpd_wsgi_conf 2019-01-08T09:46:07Z INFO [Updating mod_nss protocol versions] 2019-01-08T09:46:07Z DEBUG Loading StateFile from '/var/lib/ipa/sysupgrade/sysupgrade.state' 2019-01-08T09:46:07Z INFO Protocol versions already updated 2019-01-08T09:46:07Z INFO [Updating mod_nss cipher suite] 2019-01-08T09:46:07Z DEBUG Loading StateFile from '/var/lib/ipa/sysupgrade/sysupgrade.state' 2019-01-08T09:46:07Z DEBUG Cipher suite already updated 2019-01-08T09:46:07Z INFO [Updating mod_nss enabling OCSP] 2019-01-08T09:46:07Z DEBUG Loading StateFile from '/var/lib/ipa/sysupgrade/sysupgrade.state' 2019-01-08T09:46:07Z INFO [Fixing trust flags in /etc/httpd/alias] 2019-01-08T09:46:07Z DEBUG Loading StateFile from '/var/lib/ipa/sysupgrade/sysupgrade.state' 2019-01-08T09:46:07Z INFO Trust flags already processed 2019-01-08T09:46:07Z INFO [Moving HTTPD service keytab to gssproxy] 2019-01-08T09:46:07Z DEBUG Starting external process 2019-01-08T09:46:07Z DEBUG args=/usr/sbin/selinuxenabled 2019-01-08T09:46:07Z DEBUG Process finished, return code=1 2019-01-08T09:46:07Z DEBUG stdout= 2019-01-08T09:46:07Z DEBUG stderr= 2019-01-08T09:46:07Z DEBUG Starting external process 2019-01-08T09:46:07Z DEBUG args=/bin/systemctl restart gssproxy.service 2019-01-08T09:46:07Z DEBUG Process finished, return code=0 2019-01-08T09:46:07Z DEBUG stdout= 2019-01-08T09:46:07Z DEBUG stderr= 2019-01-08T09:46:07Z DEBUG Starting external process 2019-01-08T09:46:07Z DEBUG args=/bin/systemctl is-active gssproxy.service 2019-01-08T09:46:07Z DEBUG Process finished, return code=0 2019-01-08T09:46:07Z DEBUG stdout=active
2019-01-08T09:46:07Z DEBUG stderr= 2019-01-08T09:46:07Z DEBUG Restart of gssproxy.service complete 2019-01-08T09:46:07Z DEBUG Starting external process 2019-01-08T09:46:07Z DEBUG args=/bin/systemctl start httpd.service 2019-01-08T09:46:09Z DEBUG Process finished, return code=0 2019-01-08T09:46:09Z DEBUG stdout= 2019-01-08T09:46:09Z DEBUG stderr= 2019-01-08T09:46:09Z DEBUG Starting external process 2019-01-08T09:46:09Z DEBUG args=/bin/systemctl is-active httpd.service 2019-01-08T09:46:09Z DEBUG Process finished, return code=0 2019-01-08T09:46:09Z DEBUG stdout=active
2019-01-08T09:46:09Z DEBUG stderr= 2019-01-08T09:46:09Z DEBUG Start of httpd.service complete 2019-01-08T09:46:09Z INFO [Removing self-signed CA] 2019-01-08T09:46:09Z DEBUG Self-signed CA is not installed 2019-01-08T09:46:09Z INFO [Removing Dogtag 9 CA] 2019-01-08T09:46:09Z DEBUG Dogtag is version 10 or above 2019-01-08T09:46:09Z DEBUG Loading StateFile from '/var/lib/ipa/sysrestore/sysrestore.state' 2019-01-08T09:46:09Z DEBUG Loading Index file from '/var/lib/ipa/sysrestore/sysrestore.index' 2019-01-08T09:46:09Z INFO [Checking for deprecated KDC configuration files] 2019-01-08T09:46:09Z INFO [Checking for deprecated backups of Samba configuration files] 2019-01-08T09:46:09Z DEBUG raw: ca_is_enabled(version=u'2.229') 2019-01-08T09:46:09Z DEBUG ca_is_enabled(version=u'2.229') 2019-01-08T09:46:09Z DEBUG flushing ldapi://%2fvar%2frun%2fslapd-EXAMPLE-COM.socket from SchemaCache 2019-01-08T09:46:09Z DEBUG retrieving schema for SchemaCache url=ldapi://%2fvar%2frun%2fslapd-EXAMPLE-COM.socket conn=<ldap.ldapobject.SimpleLDAPObject instance at 0x7f8d6ccff5a8> 2019-01-08T09:46:09Z DEBUG raw: kra_is_enabled(version=u'2.229') 2019-01-08T09:46:09Z DEBUG kra_is_enabled(version=u'2.229') 2019-01-08T09:46:09Z DEBUG Cleaning up after pkispawn for the CA subsystem 2019-01-08T09:46:09Z DEBUG Loading StateFile from '/var/lib/ipa/sysrestore/sysrestore.state' 2019-01-08T09:46:09Z DEBUG Loading Index file from '/var/lib/ipa/sysrestore/sysrestore.index' 2019-01-08T09:46:09Z DEBUG Loading StateFile from '/var/lib/ipa/sysrestore/sysrestore.state' 2019-01-08T09:46:09Z INFO [Add missing CA DNS records] 2019-01-08T09:46:09Z DEBUG Loading StateFile from '/var/lib/ipa/sysupgrade/sysupgrade.state' 2019-01-08T09:46:09Z INFO IPA CA DNS records already processed 2019-01-08T09:46:09Z INFO [Removing deprecated DNS configuration options] 2019-01-08T09:46:09Z INFO DNS is not configured 2019-01-08T09:46:09Z INFO [Ensuring minimal number of connections] 2019-01-08T09:46:09Z INFO DNS is not configured 2019-01-08T09:46:09Z INFO [Updating GSSAPI configuration in DNS] 2019-01-08T09:46:09Z INFO DNS is not configured 2019-01-08T09:46:09Z INFO [Updating pid-file configuration in DNS] 2019-01-08T09:46:09Z INFO DNS is not configured 2019-01-08T09:46:09Z INFO DNS is not configured 2019-01-08T09:46:09Z INFO DNS is not configured 2019-01-08T09:46:09Z INFO DNS is not configured 2019-01-08T09:46:09Z INFO DNS is not configured 2019-01-08T09:46:09Z INFO DNS is not configured 2019-01-08T09:46:09Z DEBUG Loading StateFile from '/var/lib/ipa/sysrestore/sysrestore.state' 2019-01-08T09:46:09Z DEBUG Loading Index file from '/var/lib/ipa/sysrestore/sysrestore.index' 2019-01-08T09:46:09Z INFO DNS is not configured 2019-01-08T09:46:09Z DEBUG Loading StateFile from '/var/lib/ipa/sysupgrade/sysupgrade.state' 2019-01-08T09:46:09Z DEBUG Loading StateFile from '/var/lib/ipa/sysupgrade/sysupgrade.state' 2019-01-08T09:46:09Z DEBUG Loading StateFile from '/var/lib/ipa/sysrestore/sysrestore.state' 2019-01-08T09:46:09Z DEBUG Loading Index file from '/var/lib/ipa/sysrestore/sysrestore.index' 2019-01-08T09:46:09Z INFO DNS is not configured 2019-01-08T09:46:09Z DEBUG Loading StateFile from '/var/lib/ipa/sysrestore/sysrestore.state' 2019-01-08T09:46:09Z DEBUG Loading Index file from '/var/lib/ipa/sysrestore/sysrestore.index' 2019-01-08T09:46:09Z DEBUG Loading StateFile from '/var/lib/ipa/sysupgrade/sysupgrade.state' 2019-01-08T09:46:09Z INFO [Upgrading CA schema] 2019-01-08T09:46:10Z DEBUG Processing schema LDIF file /usr/share/pki/server/conf/schema-certProfile.ldif 2019-01-08T09:46:10Z DEBUG Processing schema LDIF file /usr/share/pki/server/conf/schema-authority.ldif 2019-01-08T09:46:10Z DEBUG Not updating schema 2019-01-08T09:46:10Z INFO CA schema update complete (no changes) 2019-01-08T09:46:10Z INFO [Verifying that CA audit signing cert has 2 year validity] 2019-01-08T09:46:10Z DEBUG caSignedLogCert.cfg profile validity range is 720 2019-01-08T09:46:10Z INFO [Update certmonger certificate renewal configuration] 2019-01-08T09:46:10Z DEBUG Loading Index file from '/var/lib/ipa/sysrestore/sysrestore.index' 2019-01-08T09:46:10Z DEBUG Starting external process 2019-01-08T09:46:10Z DEBUG args=/usr/bin/certutil -d dbm:/etc/httpd/alias -L -n Server-Cert -a -f /etc/httpd/alias/pwdfile.txt 2019-01-08T09:46:10Z DEBUG Process finished, return code=0 2019-01-08T09:46:10Z DEBUG stdout=-----BEGIN CERTIFICATE----- MIIFYjCCBEqgAwIBAgIED/4ABTANBgkqhkiG9w0BAQsFADBHMSUwIwYDVQQKExxE [...] Serial Number: 268304389 (0xffe0005) [...] n7aRLXFxWB8va5AxibMHYUJb08nWJRqHHwufF/YQnJdskFuDyAM= -----END CERTIFICATE-----
2019-01-08T09:46:10Z DEBUG stderr= 2019-01-08T09:46:10Z DEBUG Loading Index file from '/var/lib/ipa/sysrestore/sysrestore.index' 2019-01-08T09:46:10Z DEBUG Starting external process 2019-01-08T09:46:10Z DEBUG args=/usr/bin/certutil -d dbm:/etc/dirsrv/slapd-EXAMPLE-COM/ -L -n Server-Cert -a -f /etc/dirsrv/slapd-EXAMPLE-COM/pwdfile.txt 2019-01-08T09:46:10Z DEBUG Process finished, return code=0 2019-01-08T09:46:10Z DEBUG stdout=-----BEGIN CERTIFICATE----- MIIFYjCCBEqgAwIBAgIED/4ABDANBgkqhkiG9w0BAQsFADBHMSUwIwYDVQQKExxE [...] Serial Number: 268304388 (0xffe0004) [...] 7D6pk95Rgq6g73cZmogJBfJicVF0odkf2VkvV/aJCxTtZ4xkVds= -----END CERTIFICATE-----
2019-01-08T09:46:10Z DEBUG stderr= 2019-01-08T09:46:10Z DEBUG Loading Index file from '/var/lib/ipa/sysrestore/sysrestore.index' 2019-01-08T09:46:10Z DEBUG Starting external process 2019-01-08T09:46:10Z DEBUG args=/usr/bin/certutil -d dbm:/etc/pki/pki-tomcat/alias -L -f /etc/pki/pki-tomcat/alias/pwdfile.txt 2019-01-08T09:46:10Z DEBUG Process finished, return code=0 2019-01-08T09:46:10Z DEBUG stdout= Certificate Nickname Trust Attributes SSL,S/MIME,JAR/XPI
ocspSigningCert cert-pki-ca u,u,u caSigningCert cert-pki-ca CTu,Cu,Cu Server-Cert cert-pki-ca u,u,u auditSigningCert cert-pki-ca u,u,Pu subsystemCert cert-pki-ca u,u,u
2019-01-08T09:46:10Z DEBUG stderr= 2019-01-08T09:46:11Z INFO Certmonger certificate renewal configuration already up-to-date 2019-01-08T09:46:11Z INFO [Enable PKIX certificate path discovery and validation] 2019-01-08T09:46:11Z DEBUG Loading StateFile from '/var/lib/ipa/sysupgrade/sysupgrade.state' 2019-01-08T09:46:11Z INFO PKIX already enabled 2019-01-08T09:46:11Z INFO [Authorizing RA Agent to modify profiles] 2019-01-08T09:46:11Z INFO [Authorizing RA Agent to manage lightweight CAs] 2019-01-08T09:46:11Z INFO [Ensuring Lightweight CAs container exists in Dogtag database] 2019-01-08T09:46:11Z DEBUG Created connection context.ldap2_140245392902928 2019-01-08T09:46:11Z DEBUG flushing ldapi://%2fvar%2frun%2fslapd-EXAMPLE-COM.socket from SchemaCache 2019-01-08T09:46:11Z DEBUG retrieving schema for SchemaCache url=ldapi://%2fvar%2frun%2fslapd-EXAMPLE-COM.socket conn=<ldap.ldapobject.SimpleLDAPObject instance at 0x7f8d6cea4fc8> 2019-01-08T09:46:12Z DEBUG Destroyed connection context.ldap2_140245392902928 2019-01-08T09:46:12Z INFO [Adding default OCSP URI configuration] 2019-01-08T09:46:12Z INFO [Ensuring CA is using LDAPProfileSubsystem] 2019-01-08T09:46:12Z INFO [Migrating certificate profiles to LDAP] 2019-01-08T09:46:12Z DEBUG Created connection context.ldap2_140245366754832 2019-01-08T09:46:12Z DEBUG flushing ldapi://%2fvar%2frun%2fslapd-EXAMPLE-COM.socket from SchemaCache 2019-01-08T09:46:12Z DEBUG retrieving schema for SchemaCache url=ldapi://%2fvar%2frun%2fslapd-EXAMPLE-COM.socket conn=<ldap.ldapobject.SimpleLDAPObject instance at 0x7f8d6b5db128> 2019-01-08T09:46:12Z DEBUG Destroyed connection context.ldap2_140245366754832 2019-01-08T09:46:12Z DEBUG request GET https://ipa2.example.com:8443/ca/rest/account/login 2019-01-08T09:46:12Z DEBUG request body '' 2019-01-08T09:46:12Z DEBUG response status 401 2019-01-08T09:46:12Z DEBUG response headers Server: Apache-Coyote/1.1 Cache-Control: private Expires: Thu, 01 Jan 1970 01:00:00 CET WWW-Authenticate: Basic realm="Certificate Authority" Content-Type: text/html;charset=utf-8 Content-Language: en Content-Length: 951 Date: Tue, 08 Jan 2019 09:46:12 GMT
2019-01-08T09:46:12Z DEBUG response body '<html> [...] HTTP Status 401 [...] message [...] This request requires HTTP authentication. [...]
</html>' 2019-01-08T09:46:12Z ERROR IPA server upgrade failed: Inspect /var/log/ipaupgrade.log and run command ipa-server-upgrade manually. 2019-01-08T09:46:12Z DEBUG File "/usr/lib/python2.7/site-packages/ipapython/admintool.py", line 178, in execute return_value = self.run() File "/usr/lib/python2.7/site-packages/ipaserver/install/ipa_server_upgrade.py", line 54, in run server.upgrade() File "/usr/lib/python2.7/site-packages/ipaserver/install/server/upgrade.py", line 2085, in upgrade upgrade_configuration() File "/usr/lib/python2.7/site-packages/ipaserver/install/server/upgrade.py", line 1952, in upgrade_configuration ca_enable_ldap_profile_subsystem(ca) File "/usr/lib/python2.7/site-packages/ipaserver/install/server/upgrade.py", line 396, in ca_enable_ldap_profile_subsystem cainstance.migrate_profiles_to_ldap() File "/usr/lib/python2.7/site-packages/ipaserver/install/cainstance.py", line 1814, in migrate_profiles_to_ldap _create_dogtag_profile(profile_id, profile_data, overwrite=False) File "/usr/lib/python2.7/site-packages/ipaserver/install/cainstance.py", line 1820, in _create_dogtag_profile with api.Backend.ra_certprofile as profile_api: File "/usr/lib/python2.7/site-packages/ipaserver/plugins/dogtag.py", line 1302, in __enter__ raise errors.RemoteRetrieveError(reason=_('Failed to authenticate to CA REST API'))
2019-01-08T09:46:12Z DEBUG The ipa-server-upgrade command failed, exception: RemoteRetrieveError: Failed to authenticate to CA REST API 2019-01-08T09:46:12Z ERROR Unexpected error - see /var/log/ipaupgrade.log for details: RemoteRetrieveError: Failed to authenticate to CA REST API 2019-01-08T09:46:12Z ERROR The ipa-server-upgrade command failed. See /var/log/ipaupgrade.log for more information
on ipa2, what is the content of /var/lib/ipa/ra-agent.pem? Is it the same as on ipa1? Can you check the file permissions?
Can you also compare with the ldap entry uid=ipara,ou=People,o=ipaca (description and usercertificate fields are important).
looked deeper this time, as you requested:
--- ipa1 $ ls -la /var/lib/ipa/ra-agent.* -r--r----- 1 root ipaapi 1854 Jul 11 2018 /var/lib/ipa/ra-agent.key -r--r----- 1 root ipaapi 1318 Jul 11 2018 /var/lib/ipa/ra-agent.pem
-- $ openssl x509 -in /var/lib/ipa/ra-agent.pem -text -noout | grep Serial Serial Number: 268304391 (0xffe0007)
-- $ ldapsearch -x -b uid=ipara,ou=People,o=ipaca # extended LDIF # # LDAPv3 # base <uid=ipara,ou=People,o=ipaca> with scope subtree # filter: (objectclass=*) # requesting: ALL #
# ipara, people, ipaca dn: uid=ipara,ou=people,o=ipaca userCertificate:: MIIDozCCAougAwIBAgIBBzANBgkqhkiG9w0BAQsFADBHMSUwIwYDVQQKExxE [...] DoNVNCVX+P2rYhhKizm5W/+Q77YYMxs4= userCertificate:: MIIDozCCAougAwIBAgIBLDANBgkqhkiG9w0BAQsFADBHMSUwIwYDVQQKExxE [...] DFVH/Om2gwOobX7Rt3L582BOjz97gRSQ= userCertificate:: MIIDoTCCAomgAwIBAgIED/4ABzANBgkqhkiG9w0BAQsFADBHMSUwIwYDVQQK [...] zon51BrP2rvAW9tpXAgPE1XqQuRzn userstate: 1 usertype: agentType cn: ipara sn: ipara uid: ipara objectClass: top objectClass: person objectClass: organizationalPerson objectClass: inetOrgPerson objectClass: cmsuser description: 2;268304391;CN=Certificate Authority,O=EXAMPLE.COM;CN=IPA RA,O=EXAMPLE.COM
# search result search: 2 result: 0 Success
# numResponses: 2 # numEntries: 1
-- $ head -n 5 ra-agent.key Bag Attributes friendlyName: ipaCert localKeyID: 5E 90 B6 ED CA 3E 5B FF 2E 76 F6 B8 73 FD AC 1A A7 84 D7 11 Key Attributes: <No Attributes> -----BEGIN PRIVATE KEY-----
--- ipa2 $ ls -la /var/lib/ipa/ra-agent.* -r--r----- 1 root ipaapi 1828 Jul 11 2018 /var/lib/ipa/ra-agent.key -r--r----- 1 root ipaapi 1318 Jul 11 2018 /var/lib/ipa/ra-agent.pem
-- $ openssl x509 -in /var/lib/ipa/ra-agent.pem -text -noout | grep Serial Serial Number: 268304391 (0xffe0007)
-- $ ldapsearch -x -b uid=ipara,ou=People,o=ipaca # extended LDIF # # LDAPv3 # base <uid=ipara,ou=People,o=ipaca> with scope subtree # filter: (objectclass=*) # requesting: ALL #
# ipara, people, ipaca dn: uid=ipara,ou=people,o=ipaca description: 2;44;CN=Certificate Authority,O=EXAMPLE.COM;CN=IPA RA,O=EXAMPLE.COM objectClass: top objectClass: person objectClass: organizationalPerson objectClass: inetOrgPerson objectClass: cmsuser uid: ipara sn: ipara cn: ipara usertype: agentType userstate: 1 userCertificate:: MIIDozCCAougAwIBAgIBBzANBgkqhkiG9w0BAQsFADBHMSUwIwYDVQQKExxE [...] DoNVNCVX+P2rYhhKizm5W/+Q77YYMxs4= userCertificate:: MIIDozCCAougAwIBAgIBLDANBgkqhkiG9w0BAQsFADBHMSUwIwYDVQQKExxE [...] DFVH/Om2gwOobX7Rt3L582BOjz97gRSQ=
# search result search: 2 result: 0 Success
# numResponses: 2 # numEntries: 1
-- $ head -n 5 ra-agent.key Bag Attributes localKeyID: 5E 90 B6 ED CA 3E 5B FF 2E 76 F6 B8 73 FD AC 1A A7 84 D7 11 Key Attributes: <No Attributes> -----BEGIN PRIVATE KEY----- [...] ---
Two differences:
1. The key file on ipa1 contains a "Bag Attribute" "friendlyName", that the key key file on ipa2 misses - don't know if this is relevant, but probably not.
2. The LDAP entry on ipa1 contains 3 "userCertificate"s, but only the first 2 of them are in ipa2's LDAP. This is probably the interesting part.
Am i right to assume, that adding the missing userCertificate to ipa2's LDAP will solve my problem? If yes, how would I achieve that?
Mit freundlichen Gruessen/With best regards,
--Daniel.
On 1/10/19 1:46 PM, dbischof--- via FreeIPA-users wrote:
Hi Florence,
On Tue, 8 Jan 2019, Florence Blanc-Renaud wrote:
On 1/8/19 3:51 PM, dbischof--- via FreeIPA-users wrote:
Hi Florence,
On Mon, 7 Jan 2019, Florence Blanc-Renaud wrote:
[...]
i shaved this thread a little, since it gets confusing. Hope i kept the interesting bits.
The errors related to "Unable to find request for serial xxx" mean that the cert is tracked by certmonger, but there is no corresponding LDAP entry cn=xxx,ou=ca,ou=requests,o=ipaca on the local LDAP server. Is the replication still working between your two masters?
"ipa cert-show" broken on ipa2 often points to an out-of-date certificate in /var/lib/ipa/ra-agent.{pem|key}. The content should be the same as on the renewal master, and also the same as in the entry uid=ipara,ou=People,o=ipaca (which should be replicated and identical on all the masters, except if you have replication issues).
Replication appeared to be working, however, I did server maintenance today, ipa1 and ipa2 got upgraded and rebooted. ipa1 works fine, ipa2 does not come up properly:
--- ipa2 $ ipactl status Directory Service: RUNNING krb5kdc Service: STOPPED kadmin Service: STOPPED
You need to have a look at krb5 logs to understand why kerberos failed to start. Please check https://www.freeipa.org/page/Troubleshooting/Kerberos for more information.
Turns out that kerberos starts when started manually with systemctl:
--- $ ipactl status Directory Service: RUNNING krb5kdc Service: RUNNING kadmin Service: RUNNING httpd Service: RUNNING ipa-custodia Service: RUNNING ntpd Service: RUNNING pki-tomcatd Service: RUNNING ipa-otpd Service: STOPPED ipa: INFO: The ipactl command was successful ---
Kerberos log:
--- [...] : NEEDED_PREAUTH: admin@EXAMPLE.COM for krbtgt/EXAMPLE.COM@EXAMPLE.COM, Additional pre-authentication required Jan 08 10:39:22 ipa2.example.com krb5kdc[21691](info): closing down fd 11 Jan 08 10:39:39 ipa2.example.com krb5kdc[21692](info): AS_REQ (8 etypes {18 17 20 19 16 23 25 26}) 141.51.x.y: ISSUE: authtime 1546940379, etypes {rep=18 tkt=18 ses=18}, admin@EXAMPLE.COM for krbtgt/EXAMPLE.COM@EXAMPLE.COM Jan 08 10:39:39 ipa2.example.com krb5kdc[21692](info): closing down fd 11 Jan 08 10:40:32 ipa2.example.com krb5kdc[21692](info): TGS_REQ (8 etypes {18 17 20 19 16 23 25 26}) 141.51.x.y: ISSUE: authtime 1546940379, etypes {rep=18 tkt=18 ses=18}, admin@EXAMPLE.COM for HTTP/ipa2.example.com@EXAMPLE.COM Jan 08 10:40:32 ipa2.example.com krb5kdc[21692](info): closing down fd 11 ---
httpd Service: RUNNING ipa-custodia Service: STOPPED ntpd Service: RUNNING pki-tomcatd Service: RUNNING ipa-otpd Service: STOPPED ipa: INFO: The ipactl command was successful ---
--- ipa1 $ openssl x509 -in /var/lib/ipa/ra-agent.pem -text -noout | grep Serial Serial Number: 268304391 (0xffe0007)
$ ldapsearch -x -b uid=ipara,ou=People,o=ipaca [...] description: 2;268304391;CN=Certificate Authority,O=EXAMPLE.COM;CN=IPA RA,O=EXAMPLE.COM ---
Looks good.
--- ipa2 $ openssl x509 -in /var/lib/ipa/ra-agent.pem -text -noout | grep Serial Serial Number: 268304391 (0xffe0007)
$ ldapsearch -x -b uid=ipara,ou=People,o=ipaca [...] description: 2;44;CN=Certificate Authority,O=EXAMPLE.COM;CN=IPA RA,O=EXAMPLE.COM ---
Looks bad.
So yes, the replication of the suffix o=ipaca seems broken. In a deployment with CA, the ldap server contains 2 different suffixes, one for the IdM data (your base DN as defined in /etc/ipa/default.conf) and another one for the CA, below o=ipaca.
You can have a look at https://www.freeipa.org/page/Troubleshooting/Directory_Server for replication troubleshooting, but the repl issue could be a consequence of kerberos server not starting.
I checked the troubleshooting pages and carried out the steps described in there, but my system (ipa2) appears to be ok as far as the steps go.
--- /var/log/ipaupgrade.log [...] 2018-12-28T10:55:53Z DEBUG The ipa-server-upgrade command failed, exception: RemoteRetrieveError: Failed to authenticate to CA REST API ---
--- /var/log/dirsrv/slapd-EXAMPLE-COM/errors [...] [28/Dec/2018:11:55:55.559648469 +0100] - ERR - set_krb5_creds - Could not get initial credentials for principal [ldap/ipa2.example.com@EXAMPLE.COM] in keytab [FILE:/etc/dirsrv/ds.keytab]: -1765328228 (Cannot contact any KDC for requested realm) ---
--- ipa-checkcerts.py [...] ipa: ERROR: ra.get_certificate(): EXCEPTION (Invalid Credential.) ra.get_certificate(): EXCEPTION (Invalid Credential.) ipa: INFO: Checking permissions and ownership Checking permissions and ownership ipa: INFO: Failures: Failures: ipa: INFO: Unable to find request for serial 268304391 Unable to find request for serial 268304391 ipa: INFO: Unable to find request for serial 268304394 Unable to find request for serial 268304394 ipa: INFO: Unable to find request for serial 268304393 Unable to find request for serial 268304393 ipa: INFO: Unable to find request for serial 268304392 Unable to find request for serial 268304392 ipa: INFO: Unable to find request for serial 268304390 Unable to find request for serial 268304390 ipa: INFO: RA agent description does not match 2;44;CN=Certificate Authority,O=EXAMPLE.COM;CN=IPA RA,O=EXAMPLE.COM in LDAP and 2;268304391;CN=Certificate Authority,O=EXAMPLE.COM;CN=IPA RA,O=EXAMPLE.COM expected ipa: INFO: RA agent certificate not found in LDAP RA agent certificate not found in LDAP [...] ---
The root cause appears to be the wrong IPA RA certificate in ipa2's LDAP, right? I guess this has to fixed by manually importing the proper certificate using ldapmodify, similar to the procedure described in [1]?
It is possible to manually modify the IPA RA cert in ipa2's ldap server but this won't solve the replication issue. It will however allow you to run ipa cert-show.
Unfortunately, it doesn't (ipa2):
--- ipa: ERROR: Certificate operation cannot be completed: EXCEPTION (Invalid Credential.) ---
Kerberos log:
--- Jan 08 15:42:45 ipa2.example.com krb5kdc[23520](info): AS_REQ (8 etypes {18 17 16 23 20 19 25 26}) 141.51.124.20: NEEDED_PREAUTH: host/ipa2.example.com@EXAMPLE.COM for krbtgt/EXAMPLE.COM@EXAMPLE.COM, Additional pre-authentication required ---
Works ok on ipa1.
I'm unsure how to proceed here. My obvious problem is ipa2 not getting upgraded properly:
--- [...] 2019-01-08T09:46:07Z INFO [Updating HTTPD service IPA configuration] 2019-01-08T09:46:07Z DEBUG Starting external process 2019-01-08T09:46:07Z DEBUG args=/usr/sbin/selinuxenabled 2019-01-08T09:46:07Z DEBUG Process finished, return code=1 2019-01-08T09:46:07Z DEBUG stdout= 2019-01-08T09:46:07Z DEBUG stderr= 2019-01-08T09:46:07Z DEBUG Starting external process 2019-01-08T09:46:07Z DEBUG args=/bin/systemctl --system daemon-reload 2019-01-08T09:46:07Z DEBUG Process finished, return code=0 2019-01-08T09:46:07Z DEBUG stdout= 2019-01-08T09:46:07Z DEBUG stderr= 2019-01-08T09:46:07Z INFO [Updating HTTPD service IPA WSGI configuration] 2019-01-08T09:46:07Z INFO Nothing to do for configure_httpd_wsgi_conf 2019-01-08T09:46:07Z INFO [Updating mod_nss protocol versions] 2019-01-08T09:46:07Z DEBUG Loading StateFile from '/var/lib/ipa/sysupgrade/sysupgrade.state' 2019-01-08T09:46:07Z INFO Protocol versions already updated 2019-01-08T09:46:07Z INFO [Updating mod_nss cipher suite] 2019-01-08T09:46:07Z DEBUG Loading StateFile from '/var/lib/ipa/sysupgrade/sysupgrade.state' 2019-01-08T09:46:07Z DEBUG Cipher suite already updated 2019-01-08T09:46:07Z INFO [Updating mod_nss enabling OCSP] 2019-01-08T09:46:07Z DEBUG Loading StateFile from '/var/lib/ipa/sysupgrade/sysupgrade.state' 2019-01-08T09:46:07Z INFO [Fixing trust flags in /etc/httpd/alias] 2019-01-08T09:46:07Z DEBUG Loading StateFile from '/var/lib/ipa/sysupgrade/sysupgrade.state' 2019-01-08T09:46:07Z INFO Trust flags already processed 2019-01-08T09:46:07Z INFO [Moving HTTPD service keytab to gssproxy] 2019-01-08T09:46:07Z DEBUG Starting external process 2019-01-08T09:46:07Z DEBUG args=/usr/sbin/selinuxenabled 2019-01-08T09:46:07Z DEBUG Process finished, return code=1 2019-01-08T09:46:07Z DEBUG stdout= 2019-01-08T09:46:07Z DEBUG stderr= 2019-01-08T09:46:07Z DEBUG Starting external process 2019-01-08T09:46:07Z DEBUG args=/bin/systemctl restart gssproxy.service 2019-01-08T09:46:07Z DEBUG Process finished, return code=0 2019-01-08T09:46:07Z DEBUG stdout= 2019-01-08T09:46:07Z DEBUG stderr= 2019-01-08T09:46:07Z DEBUG Starting external process 2019-01-08T09:46:07Z DEBUG args=/bin/systemctl is-active gssproxy.service 2019-01-08T09:46:07Z DEBUG Process finished, return code=0 2019-01-08T09:46:07Z DEBUG stdout=active
2019-01-08T09:46:07Z DEBUG stderr= 2019-01-08T09:46:07Z DEBUG Restart of gssproxy.service complete 2019-01-08T09:46:07Z DEBUG Starting external process 2019-01-08T09:46:07Z DEBUG args=/bin/systemctl start httpd.service 2019-01-08T09:46:09Z DEBUG Process finished, return code=0 2019-01-08T09:46:09Z DEBUG stdout= 2019-01-08T09:46:09Z DEBUG stderr= 2019-01-08T09:46:09Z DEBUG Starting external process 2019-01-08T09:46:09Z DEBUG args=/bin/systemctl is-active httpd.service 2019-01-08T09:46:09Z DEBUG Process finished, return code=0 2019-01-08T09:46:09Z DEBUG stdout=active
2019-01-08T09:46:09Z DEBUG stderr= 2019-01-08T09:46:09Z DEBUG Start of httpd.service complete 2019-01-08T09:46:09Z INFO [Removing self-signed CA] 2019-01-08T09:46:09Z DEBUG Self-signed CA is not installed 2019-01-08T09:46:09Z INFO [Removing Dogtag 9 CA] 2019-01-08T09:46:09Z DEBUG Dogtag is version 10 or above 2019-01-08T09:46:09Z DEBUG Loading StateFile from '/var/lib/ipa/sysrestore/sysrestore.state' 2019-01-08T09:46:09Z DEBUG Loading Index file from '/var/lib/ipa/sysrestore/sysrestore.index' 2019-01-08T09:46:09Z INFO [Checking for deprecated KDC configuration files] 2019-01-08T09:46:09Z INFO [Checking for deprecated backups of Samba configuration files] 2019-01-08T09:46:09Z DEBUG raw: ca_is_enabled(version=u'2.229') 2019-01-08T09:46:09Z DEBUG ca_is_enabled(version=u'2.229') 2019-01-08T09:46:09Z DEBUG flushing ldapi://%2fvar%2frun%2fslapd-EXAMPLE-COM.socket from SchemaCache 2019-01-08T09:46:09Z DEBUG retrieving schema for SchemaCache url=ldapi://%2fvar%2frun%2fslapd-EXAMPLE-COM.socket conn=<ldap.ldapobject.SimpleLDAPObject instance at 0x7f8d6ccff5a8> 2019-01-08T09:46:09Z DEBUG raw: kra_is_enabled(version=u'2.229') 2019-01-08T09:46:09Z DEBUG kra_is_enabled(version=u'2.229') 2019-01-08T09:46:09Z DEBUG Cleaning up after pkispawn for the CA subsystem 2019-01-08T09:46:09Z DEBUG Loading StateFile from '/var/lib/ipa/sysrestore/sysrestore.state' 2019-01-08T09:46:09Z DEBUG Loading Index file from '/var/lib/ipa/sysrestore/sysrestore.index' 2019-01-08T09:46:09Z DEBUG Loading StateFile from '/var/lib/ipa/sysrestore/sysrestore.state' 2019-01-08T09:46:09Z INFO [Add missing CA DNS records] 2019-01-08T09:46:09Z DEBUG Loading StateFile from '/var/lib/ipa/sysupgrade/sysupgrade.state' 2019-01-08T09:46:09Z INFO IPA CA DNS records already processed 2019-01-08T09:46:09Z INFO [Removing deprecated DNS configuration options] 2019-01-08T09:46:09Z INFO DNS is not configured 2019-01-08T09:46:09Z INFO [Ensuring minimal number of connections] 2019-01-08T09:46:09Z INFO DNS is not configured 2019-01-08T09:46:09Z INFO [Updating GSSAPI configuration in DNS] 2019-01-08T09:46:09Z INFO DNS is not configured 2019-01-08T09:46:09Z INFO [Updating pid-file configuration in DNS] 2019-01-08T09:46:09Z INFO DNS is not configured 2019-01-08T09:46:09Z INFO DNS is not configured 2019-01-08T09:46:09Z INFO DNS is not configured 2019-01-08T09:46:09Z INFO DNS is not configured 2019-01-08T09:46:09Z INFO DNS is not configured 2019-01-08T09:46:09Z INFO DNS is not configured 2019-01-08T09:46:09Z DEBUG Loading StateFile from '/var/lib/ipa/sysrestore/sysrestore.state' 2019-01-08T09:46:09Z DEBUG Loading Index file from '/var/lib/ipa/sysrestore/sysrestore.index' 2019-01-08T09:46:09Z INFO DNS is not configured 2019-01-08T09:46:09Z DEBUG Loading StateFile from '/var/lib/ipa/sysupgrade/sysupgrade.state' 2019-01-08T09:46:09Z DEBUG Loading StateFile from '/var/lib/ipa/sysupgrade/sysupgrade.state' 2019-01-08T09:46:09Z DEBUG Loading StateFile from '/var/lib/ipa/sysrestore/sysrestore.state' 2019-01-08T09:46:09Z DEBUG Loading Index file from '/var/lib/ipa/sysrestore/sysrestore.index' 2019-01-08T09:46:09Z INFO DNS is not configured 2019-01-08T09:46:09Z DEBUG Loading StateFile from '/var/lib/ipa/sysrestore/sysrestore.state' 2019-01-08T09:46:09Z DEBUG Loading Index file from '/var/lib/ipa/sysrestore/sysrestore.index' 2019-01-08T09:46:09Z DEBUG Loading StateFile from '/var/lib/ipa/sysupgrade/sysupgrade.state' 2019-01-08T09:46:09Z INFO [Upgrading CA schema] 2019-01-08T09:46:10Z DEBUG Processing schema LDIF file /usr/share/pki/server/conf/schema-certProfile.ldif 2019-01-08T09:46:10Z DEBUG Processing schema LDIF file /usr/share/pki/server/conf/schema-authority.ldif 2019-01-08T09:46:10Z DEBUG Not updating schema 2019-01-08T09:46:10Z INFO CA schema update complete (no changes) 2019-01-08T09:46:10Z INFO [Verifying that CA audit signing cert has 2 year validity] 2019-01-08T09:46:10Z DEBUG caSignedLogCert.cfg profile validity range is 720 2019-01-08T09:46:10Z INFO [Update certmonger certificate renewal configuration] 2019-01-08T09:46:10Z DEBUG Loading Index file from '/var/lib/ipa/sysrestore/sysrestore.index' 2019-01-08T09:46:10Z DEBUG Starting external process 2019-01-08T09:46:10Z DEBUG args=/usr/bin/certutil -d dbm:/etc/httpd/alias -L -n Server-Cert -a -f /etc/httpd/alias/pwdfile.txt 2019-01-08T09:46:10Z DEBUG Process finished, return code=0 2019-01-08T09:46:10Z DEBUG stdout=-----BEGIN CERTIFICATE----- MIIFYjCCBEqgAwIBAgIED/4ABTANBgkqhkiG9w0BAQsFADBHMSUwIwYDVQQKExxE [...] Serial Number: 268304389 (0xffe0005) [...] n7aRLXFxWB8va5AxibMHYUJb08nWJRqHHwufF/YQnJdskFuDyAM= -----END CERTIFICATE-----
2019-01-08T09:46:10Z DEBUG stderr= 2019-01-08T09:46:10Z DEBUG Loading Index file from '/var/lib/ipa/sysrestore/sysrestore.index' 2019-01-08T09:46:10Z DEBUG Starting external process 2019-01-08T09:46:10Z DEBUG args=/usr/bin/certutil -d dbm:/etc/dirsrv/slapd-EXAMPLE-COM/ -L -n Server-Cert -a -f /etc/dirsrv/slapd-EXAMPLE-COM/pwdfile.txt 2019-01-08T09:46:10Z DEBUG Process finished, return code=0 2019-01-08T09:46:10Z DEBUG stdout=-----BEGIN CERTIFICATE----- MIIFYjCCBEqgAwIBAgIED/4ABDANBgkqhkiG9w0BAQsFADBHMSUwIwYDVQQKExxE [...] Serial Number: 268304388 (0xffe0004) [...] 7D6pk95Rgq6g73cZmogJBfJicVF0odkf2VkvV/aJCxTtZ4xkVds= -----END CERTIFICATE-----
2019-01-08T09:46:10Z DEBUG stderr= 2019-01-08T09:46:10Z DEBUG Loading Index file from '/var/lib/ipa/sysrestore/sysrestore.index' 2019-01-08T09:46:10Z DEBUG Starting external process 2019-01-08T09:46:10Z DEBUG args=/usr/bin/certutil -d dbm:/etc/pki/pki-tomcat/alias -L -f /etc/pki/pki-tomcat/alias/pwdfile.txt 2019-01-08T09:46:10Z DEBUG Process finished, return code=0 2019-01-08T09:46:10Z DEBUG stdout= Certificate Nickname Trust Attributes
SSL,S/MIME,JAR/XPI
ocspSigningCert cert-pki-ca u,u,u caSigningCert cert-pki-ca CTu,Cu,Cu Server-Cert cert-pki-ca u,u,u auditSigningCert cert-pki-ca u,u,Pu subsystemCert cert-pki-ca u,u,u
2019-01-08T09:46:10Z DEBUG stderr= 2019-01-08T09:46:11Z INFO Certmonger certificate renewal configuration already up-to-date 2019-01-08T09:46:11Z INFO [Enable PKIX certificate path discovery and validation] 2019-01-08T09:46:11Z DEBUG Loading StateFile from '/var/lib/ipa/sysupgrade/sysupgrade.state' 2019-01-08T09:46:11Z INFO PKIX already enabled 2019-01-08T09:46:11Z INFO [Authorizing RA Agent to modify profiles] 2019-01-08T09:46:11Z INFO [Authorizing RA Agent to manage lightweight CAs] 2019-01-08T09:46:11Z INFO [Ensuring Lightweight CAs container exists in Dogtag database] 2019-01-08T09:46:11Z DEBUG Created connection context.ldap2_140245392902928 2019-01-08T09:46:11Z DEBUG flushing ldapi://%2fvar%2frun%2fslapd-EXAMPLE-COM.socket from SchemaCache 2019-01-08T09:46:11Z DEBUG retrieving schema for SchemaCache url=ldapi://%2fvar%2frun%2fslapd-EXAMPLE-COM.socket conn=<ldap.ldapobject.SimpleLDAPObject instance at 0x7f8d6cea4fc8> 2019-01-08T09:46:12Z DEBUG Destroyed connection context.ldap2_140245392902928 2019-01-08T09:46:12Z INFO [Adding default OCSP URI configuration] 2019-01-08T09:46:12Z INFO [Ensuring CA is using LDAPProfileSubsystem] 2019-01-08T09:46:12Z INFO [Migrating certificate profiles to LDAP] 2019-01-08T09:46:12Z DEBUG Created connection context.ldap2_140245366754832 2019-01-08T09:46:12Z DEBUG flushing ldapi://%2fvar%2frun%2fslapd-EXAMPLE-COM.socket from SchemaCache 2019-01-08T09:46:12Z DEBUG retrieving schema for SchemaCache url=ldapi://%2fvar%2frun%2fslapd-EXAMPLE-COM.socket conn=<ldap.ldapobject.SimpleLDAPObject instance at 0x7f8d6b5db128> 2019-01-08T09:46:12Z DEBUG Destroyed connection context.ldap2_140245366754832 2019-01-08T09:46:12Z DEBUG request GET https://ipa2.example.com:8443/ca/rest/account/login 2019-01-08T09:46:12Z DEBUG request body '' 2019-01-08T09:46:12Z DEBUG response status 401 2019-01-08T09:46:12Z DEBUG response headers Server: Apache-Coyote/1.1 Cache-Control: private Expires: Thu, 01 Jan 1970 01:00:00 CET WWW-Authenticate: Basic realm="Certificate Authority" Content-Type: text/html;charset=utf-8 Content-Language: en Content-Length: 951 Date: Tue, 08 Jan 2019 09:46:12 GMT
2019-01-08T09:46:12Z DEBUG response body '<html> [...] HTTP Status 401 [...] message [...] This request requires HTTP authentication. [...] </html>' 2019-01-08T09:46:12Z ERROR IPA server upgrade failed: Inspect /var/log/ipaupgrade.log and run command ipa-server-upgrade manually. 2019-01-08T09:46:12Z DEBUG File "/usr/lib/python2.7/site-packages/ipapython/admintool.py", line 178, in execute return_value = self.run() File "/usr/lib/python2.7/site-packages/ipaserver/install/ipa_server_upgrade.py",
line 54, in run server.upgrade() File "/usr/lib/python2.7/site-packages/ipaserver/install/server/upgrade.py", line 2085, in upgrade upgrade_configuration() File "/usr/lib/python2.7/site-packages/ipaserver/install/server/upgrade.py", line 1952, in upgrade_configuration ca_enable_ldap_profile_subsystem(ca) File "/usr/lib/python2.7/site-packages/ipaserver/install/server/upgrade.py", line 396, in ca_enable_ldap_profile_subsystem cainstance.migrate_profiles_to_ldap() File "/usr/lib/python2.7/site-packages/ipaserver/install/cainstance.py", line 1814, in migrate_profiles_to_ldap _create_dogtag_profile(profile_id, profile_data, overwrite=False) File "/usr/lib/python2.7/site-packages/ipaserver/install/cainstance.py", line 1820, in _create_dogtag_profile with api.Backend.ra_certprofile as profile_api: File "/usr/lib/python2.7/site-packages/ipaserver/plugins/dogtag.py", line 1302, in __enter__ raise errors.RemoteRetrieveError(reason=_('Failed to authenticate to CA REST API'))
2019-01-08T09:46:12Z DEBUG The ipa-server-upgrade command failed, exception: RemoteRetrieveError: Failed to authenticate to CA REST API 2019-01-08T09:46:12Z ERROR Unexpected error - see /var/log/ipaupgrade.log for details: RemoteRetrieveError: Failed to authenticate to CA REST API 2019-01-08T09:46:12Z ERROR The ipa-server-upgrade command failed. See /var/log/ipaupgrade.log for more information ---
on ipa2, what is the content of /var/lib/ipa/ra-agent.pem? Is it the same as on ipa1? Can you check the file permissions?
Can you also compare with the ldap entry uid=ipara,ou=People,o=ipaca (description and usercertificate fields are important).
looked deeper this time, as you requested:
--- ipa1 $ ls -la /var/lib/ipa/ra-agent.* -r--r----- 1 root ipaapi 1854 Jul 11 2018 /var/lib/ipa/ra-agent.key -r--r----- 1 root ipaapi 1318 Jul 11 2018 /var/lib/ipa/ra-agent.pem
-- $ openssl x509 -in /var/lib/ipa/ra-agent.pem -text -noout | grep Serial Serial Number: 268304391 (0xffe0007)
-- $ ldapsearch -x -b uid=ipara,ou=People,o=ipaca # extended LDIF # # LDAPv3 # base <uid=ipara,ou=People,o=ipaca> with scope subtree # filter: (objectclass=*) # requesting: ALL #
# ipara, people, ipaca dn: uid=ipara,ou=people,o=ipaca userCertificate:: MIIDozCCAougAwIBAgIBBzANBgkqhkiG9w0BAQsFADBHMSUwIwYDVQQKExxE [...] DoNVNCVX+P2rYhhKizm5W/+Q77YYMxs4= userCertificate:: MIIDozCCAougAwIBAgIBLDANBgkqhkiG9w0BAQsFADBHMSUwIwYDVQQKExxE [...] DFVH/Om2gwOobX7Rt3L582BOjz97gRSQ= userCertificate:: MIIDoTCCAomgAwIBAgIED/4ABzANBgkqhkiG9w0BAQsFADBHMSUwIwYDVQQK [...] zon51BrP2rvAW9tpXAgPE1XqQuRzn userstate: 1 usertype: agentType cn: ipara sn: ipara uid: ipara objectClass: top objectClass: person objectClass: organizationalPerson objectClass: inetOrgPerson objectClass: cmsuser description: 2;268304391;CN=Certificate Authority,O=EXAMPLE.COM;CN=IPA RA,O=EXAMPLE.COM
# search result search: 2 result: 0 Success
# numResponses: 2 # numEntries: 1
-- $ head -n 5 ra-agent.key Bag Attributes friendlyName: ipaCert localKeyID: 5E 90 B6 ED CA 3E 5B FF 2E 76 F6 B8 73 FD AC 1A A7 84 D7 11 Key Attributes: <No Attributes> -----BEGIN PRIVATE KEY-----
--- ipa2 $ ls -la /var/lib/ipa/ra-agent.* -r--r----- 1 root ipaapi 1828 Jul 11 2018 /var/lib/ipa/ra-agent.key -r--r----- 1 root ipaapi 1318 Jul 11 2018 /var/lib/ipa/ra-agent.pem
-- $ openssl x509 -in /var/lib/ipa/ra-agent.pem -text -noout | grep Serial Serial Number: 268304391 (0xffe0007)
-- $ ldapsearch -x -b uid=ipara,ou=People,o=ipaca # extended LDIF # # LDAPv3 # base <uid=ipara,ou=People,o=ipaca> with scope subtree # filter: (objectclass=*) # requesting: ALL #
# ipara, people, ipaca dn: uid=ipara,ou=people,o=ipaca description: 2;44;CN=Certificate Authority,O=EXAMPLE.COM;CN=IPA RA,O=EXAMPLE.COM objectClass: top objectClass: person objectClass: organizationalPerson objectClass: inetOrgPerson objectClass: cmsuser uid: ipara sn: ipara cn: ipara usertype: agentType userstate: 1 userCertificate:: MIIDozCCAougAwIBAgIBBzANBgkqhkiG9w0BAQsFADBHMSUwIwYDVQQKExxE [...] DoNVNCVX+P2rYhhKizm5W/+Q77YYMxs4= userCertificate:: MIIDozCCAougAwIBAgIBLDANBgkqhkiG9w0BAQsFADBHMSUwIwYDVQQKExxE [...] DFVH/Om2gwOobX7Rt3L582BOjz97gRSQ=
# search result search: 2 result: 0 Success
# numResponses: 2 # numEntries: 1
-- $ head -n 5 ra-agent.key Bag Attributes localKeyID: 5E 90 B6 ED CA 3E 5B FF 2E 76 F6 B8 73 FD AC 1A A7 84 D7 11 Key Attributes: <No Attributes> -----BEGIN PRIVATE KEY----- [...]
Two differences:
- The key file on ipa1 contains a "Bag Attribute" "friendlyName", that
the key key file on ipa2 misses - don't know if this is relevant, but probably not.
- The LDAP entry on ipa1 contains 3 "userCertificate"s, but only the
first 2 of them are in ipa2's LDAP. This is probably the interesting part.
Am i right to assume, that adding the missing userCertificate to ipa2's LDAP will solve my problem? If yes, how would I achieve that?
Hi,
you can use ldapmodify to manually add the missing certificate:
1. transform the RA agent cert into der format $ openssl x509 -outform der -in /var/lib/ipa/ra-agent.pem -out /tmp/ra-agent.der
2. upload the cert in LDAP $ ldapmodify -h ipa2 -p 389 -D "cn=directory manager" -W Enter LDAP Password: dn: uid=ipara,ou=people,o=ipaca changetype: modify add: usercertificate usercertificate:< file:///tmp/ra-agent.der
modifying entry "uid=ipara,ou=people,o=ipaca"
<Ctrl-D> to exit
After that, you should be able to re-run ipa-server-upgrade. At this point, please make sure that replication could be re-established between the two nodes.
HTH, flo
Mit freundlichen Gruessen/With best regards,
--Daniel. _______________________________________________ FreeIPA-users mailing list -- freeipa-users@lists.fedorahosted.org To unsubscribe send an email to freeipa-users-leave@lists.fedorahosted.org Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedorahoste...
Hi Florence,
On Thu, 10 Jan 2019, Florence Blanc-Renaud wrote:
On 1/10/19 1:46 PM, dbischof--- via FreeIPA-users wrote: [...] you can use ldapmodify to manually add the missing certificate:
transform the RA agent cert into der format $ openssl x509 -outform der -in /var/lib/ipa/ra-agent.pem -out /tmp/ra-agent.der
upload the cert in LDAP
$ ldapmodify -h ipa2 -p 389 -D "cn=directory manager" -W Enter LDAP Password: dn: uid=ipara,ou=people,o=ipaca changetype: modify add: usercertificate usercertificate:< file:///tmp/ra-agent.der
modifying entry "uid=ipara,ou=people,o=ipaca"
<Ctrl-D> to exit
After that, you should be able to re-run ipa-server-upgrade. At this point, please make sure that replication could be re-established between the two nodes.
your help is greatly appreciated.
I had to change the cert serial in "description" additionally the same way via ldapmodify, but now ipa-server-upgrade goes smooth and IPA on ipa2 comes up properly after a reboot. Fine.
Regarding replication: Checking, whether replication works properly is achieved with "ipa-replica-manage -v list <host>", right? Has to work on both IPA servers and "last update ended" must be a reasonable recent timestamp?
Mit freundlichen Gruessen/With best regards,
--Daniel.
On 1/11/19 3:24 PM, dbischof--- via FreeIPA-users wrote:
Hi Florence,
On Thu, 10 Jan 2019, Florence Blanc-Renaud wrote:
On 1/10/19 1:46 PM, dbischof--- via FreeIPA-users wrote: [...] you can use ldapmodify to manually add the missing certificate:
- transform the RA agent cert into der format $ openssl x509 -outform
der -in /var/lib/ipa/ra-agent.pem -out /tmp/ra-agent.der
- upload the cert in LDAP
$ ldapmodify -h ipa2 -p 389 -D "cn=directory manager" -W Enter LDAP Password: dn: uid=ipara,ou=people,o=ipaca changetype: modify add: usercertificate usercertificate:< file:///tmp/ra-agent.der
modifying entry "uid=ipara,ou=people,o=ipaca"
<Ctrl-D> to exit
After that, you should be able to re-run ipa-server-upgrade. At this point, please make sure that replication could be re-established between the two nodes.
your help is greatly appreciated.
I had to change the cert serial in "description" additionally the same way via ldapmodify, but now ipa-server-upgrade goes smooth and IPA on ipa2 comes up properly after a reboot. Fine.
Regarding replication: Checking, whether replication works properly is achieved with "ipa-replica-manage -v list <host>", right? Has to work on both IPA servers and "last update ended" must be a reasonable recent timestamp?
Yes, ipa-replica-manage -v list <host> will display the status of the replication for the domain (user, hosts, ...). The value of "last update status" must be "Replica acquired successfully: Incremental update succeeded".
If the topology includes multiple CA instances, replication is also configured for the CA data, and the status can be found using ipa-csreplica-manage -v list <host>.
HTH, flo
Mit freundlichen Gruessen/With best regards,
--Daniel. _______________________________________________ FreeIPA-users mailing list -- freeipa-users@lists.fedorahosted.org To unsubscribe send an email to freeipa-users-leave@lists.fedorahosted.org Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedorahoste...
Hi Florence,
On Fri, 11 Jan 2019, Florence Blanc-Renaud via FreeIPA-users wrote:
On 1/11/19 3:24 PM, dbischof--- via FreeIPA-users wrote:
On Thu, 10 Jan 2019, Florence Blanc-Renaud wrote:
On 1/10/19 1:46 PM, dbischof--- via FreeIPA-users wrote: [...] you can use ldapmodify to manually add the missing certificate:
- transform the RA agent cert into der format $ openssl x509 -outform
der -in /var/lib/ipa/ra-agent.pem -out /tmp/ra-agent.der
- upload the cert in LDAP
$ ldapmodify -h ipa2 -p 389 -D "cn=directory manager" -W Enter LDAP Password: dn: uid=ipara,ou=people,o=ipaca changetype: modify add: usercertificate usercertificate:< file:///tmp/ra-agent.der
modifying entry "uid=ipara,ou=people,o=ipaca"
<Ctrl-D> to exit
After that, you should be able to re-run ipa-server-upgrade. At this point, please make sure that replication could be re-established between the two nodes.
your help is greatly appreciated.
I had to change the cert serial in "description" additionally the same way via ldapmodify, but now ipa-server-upgrade goes smooth and IPA on ipa2 comes up properly after a reboot. Fine.
Regarding replication: Checking, whether replication works properly is achieved with "ipa-replica-manage -v list <host>", right? Has to work on both IPA servers and "last update ended" must be a reasonable recent timestamp?
Yes, ipa-replica-manage -v list <host> will display the status of the replication for the domain (user, hosts, ...). The value of "last update status" must be "Replica acquired successfully: Incremental update succeeded".
this is working now, thanks again.
If the topology includes multiple CA instances, replication is also configured for the CA data, and the status can be found using ipa-csreplica-manage -v list <host>.
I do have 2 CA instances and it appears that i'm not yet out of the woods here:
---ipa2 $ ipa-csreplica-manage -v list ipa1.example.com ipa2.example.com last init status: None last init ended: 1970-01-01 00:00:00+00:00 last update status: Error (-1) Problem connecting to replica - LDAP error: Can't contact LDAP server (connection error) last update ended: 1970-01-01 00:00:00+00:00 ---
But i will start a new thread for this, if i can't get fixed myself.
Mit freundlichen Gruessen/With best regards,
--Daniel.
freeipa-users@lists.fedorahosted.org
-
dbischof＠hrz.uni-kassel.de -
Florence Blanc-Renaud