9:22 a.m.
Hi,
my IPA system consists of 2 masters with their own self-signed CAs, one of
them being the certificate renewal master (ipa1). The system has been
running for years and has been migrated from an IPA 3 system.
Since a while, the Web UI logins on ipa1 don't work anymore ("Login failed
due to an unknown reason.").
Web UI logins on the other server (ipa2) work and everything else is
working fine, too, ipactl status reports all services running.
On login attempt:
--- httpd log
[...]
[:error] [pid 15551] [remote 141.51.X.X:0] mod_wsgi (pid=15551): Exception occurred
processing WSGI script '/usr/share/ipa/wsgi.py'.
[...]
[:error] [pid 15551] [remote 141.51.X.X:0] CalledProcessError: Command '/usr/bin/kinit
-n -c /var/run/ipa/ccaches/armor_15551 -X X509_anchors=FILE:/var/kerberos/krb5kdc/kdc.crt
-X X509_anchors=FILE:/var/lib/ipa-client/pki/kdc-ca-bundle.pem' returned non-zero exit
status 1
---
--- krb5kdc.log
[...]
Dec 20 16:06:54 ipa1.example.com krb5kdc[15517](info): AS_REQ (8 etypes {18 17 20 19 16 23
25 26}) 141.51.X.Y: NEEDED_PREAUTH: WELLKNOWN/ANONYMOUS(a)EXAMPLE.COM for
krbtgt/EXAMPLE.COM(a)EXAMPLE.COM, Additional pre-authentication required
Dec 20 16:06:54 ipa1.example.com krb5kdc[15517](info): closing down fd 11
Dec 20 16:06:54 ipa1.example.com krb5kdc[15518](info): AS_REQ (8 etypes {18 17 20 19 16 23
25 26}) 141.51.X.Y: KDC_RETURN_PADATA: WELLKNOWN/ANONYMOUS(a)EXAMPLE.COM for
krbtgt/EXAMPLE.COM(a)EXAMPLE.COM, Failed to verify own certificate (depth 0): certificate
has expired
Dec 20 16:06:54 ipa1.example.com krb5kdc[15518](info): closing down fd 11
---
--- ipa-checkcerts.py
IPA version 4.5.4-10.el7.centos.3
Check CA status
Check tracking
Check NSS trust
Check dates
Checking certificates in CS.cfg
Comparing certificates to requests in LDAP
Checking RA certificate
Checking authorities
Checking host keytab
Validating certificates
Checking renewal master
End-to-end cert API test
Checking permissions and ownership
Failures:
Unable to find request for serial 268304391
Unable to find request for serial 268304394
Unable to find request for serial 268304393
Unable to find request for serial 268304392
Subject O=EXAMPLE.COM,CN=ipa2.example.com and template subject
CN=ipa1.example.com,O=EXAMPLE.COM do not match for serial 57
---
--- ipa pkinit-status --all
-----------------
2 servers matched
-----------------
Server name: ipa2.example.com
PKINIT status: enabled
Server name: ipa1.example.com
PKINIT status: enabled
----------------------------
Number of entries returned 2
----------------------------
To my understanding, proper certificate exchange between my two servers
ceased working at some point. How do i track this down and fix it?
Mit freundlichen Gruessen/With best regards,
--Daniel.
10:50 a.m.
On 12/20/18 4:22 PM, dbischof--- via FreeIPA-users wrote:
Hi,
my IPA system consists of 2 masters with their own self-signed CAs, one
of them being the certificate renewal master (ipa1). The system has been
running for years and has been migrated from an IPA 3 system.
Since a while, the Web UI logins on ipa1 don't work anymore ("Login
failed due to an unknown reason.").
Web UI logins on the other server (ipa2) work and everything else is
working fine, too, ipactl status reports all services running.
On login attempt:
--- httpd log
[...]
[:error] [pid 15551] [remote 141.51.X.X:0] mod_wsgi (pid=15551):
Exception occurred processing WSGI script '/usr/share/ipa/wsgi.py'.
[...]
[:error] [pid 15551] [remote 141.51.X.X:0] CalledProcessError: Command
'/usr/bin/kinit -n -c /var/run/ipa/ccaches/armor_15551 -X
X509_anchors=FILE:/var/kerberos/krb5kdc/kdc.crt -X
X509_anchors=FILE:/var/lib/ipa-client/pki/kdc-ca-bundle.pem' returned
non-zero exit status 1
---
--- krb5kdc.log
[...]
Dec 20 16:06:54 ipa1.example.com krb5kdc[15517](info): AS_REQ (8 etypes
{18 17 20 19 16 23 25 26}) 141.51.X.Y: NEEDED_PREAUTH:
WELLKNOWN/ANONYMOUS(a)EXAMPLE.COM for krbtgt/EXAMPLE.COM(a)EXAMPLE.COM,
Additional pre-authentication required
Dec 20 16:06:54 ipa1.example.com krb5kdc[15517](info): closing down fd 11
Dec 20 16:06:54 ipa1.example.com krb5kdc[15518](info): AS_REQ (8 etypes
{18 17 20 19 16 23 25 26}) 141.51.X.Y: KDC_RETURN_PADATA:
WELLKNOWN/ANONYMOUS(a)EXAMPLE.COM for krbtgt/EXAMPLE.COM(a)EXAMPLE.COM,
Failed to verify own certificate (depth 0): certificate has expired
Dec 20 16:06:54 ipa1.example.com krb5kdc[15518](info): closing down fd 11
---
--- ipa-checkcerts.py
IPA version 4.5.4-10.el7.centos.3
Check CA status
Check tracking
Check NSS trust
Check dates
Checking certificates in CS.cfg
Comparing certificates to requests in LDAP
Checking RA certificate
Checking authorities
Checking host keytab
Validating certificates
Checking renewal master
End-to-end cert API test
Checking permissions and ownership
Failures:
Unable to find request for serial 268304391
Unable to find request for serial 268304394
Unable to find request for serial 268304393
Unable to find request for serial 268304392
Subject O=EXAMPLE.COM,CN=ipa2.example.com and template subject
CN=ipa1.example.com,O=EXAMPLE.COM do not match for serial 57
---
--- ipa pkinit-status --all
-----------------
2 servers matched
-----------------
Server name: ipa2.example.com
PKINIT status: enabled
Server name: ipa1.example.com
PKINIT status: enabled
----------------------------
Number of entries returned 2
----------------------------
To my understanding, proper certificate exchange between my two servers
ceased working at some point. How do i track this down and fix it?
Hi,
your issue looks similar to ticket #6792 [1]. Can you check the result
of upgrade in /var/log/ipaupgrade.log?
Also check the output of
$ ipa-pkinit-manage status
and if the files /var/lib/ipa-client/pki/kdc-ca-bundle.pem and
/var/lib/ipa-client/pki/ca-bundle.pem exist, with -rw-r--r-- permissions.
Regarding the certificates, does getcert list show expired certs?
flo
[1] https://pagure.io/freeipa/issue/6792
Mit freundlichen Gruessen/With best regards,
--Daniel.
_______________________________________________
FreeIPA-users mailing list -- freeipa-users(a)lists.fedorahosted.org
To unsubscribe send an email to freeipa-users-leave(a)lists.fedorahosted.org
Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives:
https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedoraho...
11:52 a.m.
Hi Florence,
On Thu, 20 Dec 2018, Florence Blanc-Renaud via FreeIPA-users wrote:
On 12/20/18 4:22 PM, dbischof--- via FreeIPA-users wrote:
> my IPA system consists of 2 masters with their own self-signed CAs, one of
> them being the certificate renewal master (ipa1). The system has been
> running for years and has been migrated from an IPA 3 system.
>
> Since a while, the Web UI logins on ipa1 don't work anymore ("Login failed
> due to an unknown reason.").
>
> Web UI logins on the other server (ipa2) work and everything else is
> working fine, too, ipactl status reports all services running.
>
> On login attempt:
>
> --- httpd log
> [...]
> [:error] [pid 15551] [remote 141.51.X.X:0] mod_wsgi (pid=15551): Exception
> occurred processing WSGI script '/usr/share/ipa/wsgi.py'.
> [...]
> [:error] [pid 15551] [remote 141.51.X.X:0] CalledProcessError: Command
> '/usr/bin/kinit -n -c /var/run/ipa/ccaches/armor_15551 -X
> X509_anchors=FILE:/var/kerberos/krb5kdc/kdc.crt -X
> X509_anchors=FILE:/var/lib/ipa-client/pki/kdc-ca-bundle.pem' returned
> non-zero exit status 1
> ---
>
> --- krb5kdc.log
> [...]
> Dec 20 16:06:54 ipa1.example.com krb5kdc[15517](info): AS_REQ (8 etypes
> {18 17 20 19 16 23 25 26}) 141.51.X.Y: NEEDED_PREAUTH:
> WELLKNOWN/ANONYMOUS(a)EXAMPLE.COM for krbtgt/EXAMPLE.COM(a)EXAMPLE.COM,
> Additional pre-authentication required
> Dec 20 16:06:54 ipa1.example.com krb5kdc[15517](info): closing down fd 11
> Dec 20 16:06:54 ipa1.example.com krb5kdc[15518](info): AS_REQ (8 etypes
> {18 17 20 19 16 23 25 26}) 141.51.X.Y: KDC_RETURN_PADATA:
> WELLKNOWN/ANONYMOUS(a)EXAMPLE.COM for krbtgt/EXAMPLE.COM(a)EXAMPLE.COM, Failed
> to verify own certificate (depth 0): certificate has expired
> Dec 20 16:06:54 ipa1.example.com krb5kdc[15518](info): closing down fd 11
> ---
>
> --- ipa-checkcerts.py
> IPA version 4.5.4-10.el7.centos.3
> Check CA status
> Check tracking
> Check NSS trust
> Check dates
> Checking certificates in CS.cfg
> Comparing certificates to requests in LDAP
> Checking RA certificate
> Checking authorities
> Checking host keytab
> Validating certificates
> Checking renewal master
> End-to-end cert API test
> Checking permissions and ownership
> Failures:
> Unable to find request for serial 268304391
> Unable to find request for serial 268304394
> Unable to find request for serial 268304393
> Unable to find request for serial 268304392
> Subject O=EXAMPLE.COM,CN=ipa2.example.com and template subject
> CN=ipa1.example.com,O=EXAMPLE.COM do not match for serial 57
> ---
>
> --- ipa pkinit-status --all
> -----------------
> 2 servers matched
> -----------------
> Server name: ipa2.example.com
> PKINIT status: enabled
>
> Server name: ipa1.example.com
> PKINIT status: enabled
> ----------------------------
> Number of entries returned 2
> ----------------------------
>
> To my understanding, proper certificate exchange between my two servers
> ceased working at some point. How do i track this down and fix it?
>
your issue looks similar to ticket #6792 [1]. Can you check the result of
upgrade in /var/log/ipaupgrade.log?
Also check the output of
$ ipa-pkinit-manage status
and if the files /var/lib/ipa-client/pki/kdc-ca-bundle.pem and
/var/lib/ipa-client/pki/ca-bundle.pem exist, with -rw-r--r-- permissions.
Regarding the certificates, does getcert list show expired certs?
flo
[1] https://pagure.io/freeipa/issue/6792
---
$ ipa-pkinit-manage status
PKINIT is enabled
---
There are no expired certificates, kdc-ca-bundle.pem and ca-bundle.pem
exist with proper permissions, but I found something in ipaupgrade.log:
---
2018-09-12T13:37:18Z DEBUG args=/usr/bin/certutil -d /etc/httpd/alias -L -f
/etc/httpd/alias/pwdfile.txt
2018-09-12T13:37:19Z DEBUG Process finished, return code=0
2018-09-12T13:37:19Z DEBUG stdout=
Certificate Nickname Trust
Attributes
SSL,S/MIME,JAR/XPI
Server-Cert u,u,u
EXAMPLE.COM IPA CA CT,C,C
2018-09-12T13:37:19Z DEBUG stderr=
2018-09-12T13:37:19Z DEBUG Starting external process
2018-09-12T13:37:19Z DEBUG args=/usr/bin/certutil -d /etc/httpd/alias -L -n EXAMPLE.COM
IPA CA -a -f /etc/httpd/alias/pwdfile.txt
2018-09-12T13:37:19Z DEBUG Process finished, return code=0
2018-09-12T13:37:19Z DEBUG stdout=
-----BEGIN CERTIFICATE-----
[...]
-----END CERTIFICATE-----
2018-09-12T13:37:19Z DEBUG stderr=
2018-09-12T13:37:19Z DEBUG Executing upgrade plugin: update_ra_cert_store
2018-09-12T13:37:19Z DEBUG raw: update_ra_cert_store
2018-09-12T13:37:19Z DEBUG raw: ca_is_enabled(version=u'2.228')
2018-09-12T13:37:19Z DEBUG ca_is_enabled(version=u'2.228')
2018-09-12T13:37:19Z DEBUG Starting external process
2018-09-12T13:37:19Z DEBUG args=/usr/bin/certutil -d /etc/httpd/alias -L -n ipaCert -a -f
/etc/httpd/alias/pwdfile.txt
2018-09-12T13:37:19Z DEBUG Process finished, return code=255
2018-09-12T13:37:19Z DEBUG stdout=
2018-09-12T13:37:19Z DEBUG stderr=certutil: Could not find cert: ipaCert
: PR_FILE_NOT_FOUND_ERROR: File not found
[...]
---
Mit freundlichen Gruessen/With best regards,
--Daniel.
3:28 a.m.
On 12/20/18 6:52 PM, dbischof--- via FreeIPA-users wrote:
Hi Florence,
On Thu, 20 Dec 2018, Florence Blanc-Renaud via FreeIPA-users wrote:
> On 12/20/18 4:22 PM, dbischof--- via FreeIPA-users wrote:
>> my IPA system consists of 2 masters with their own self-signed CAs,
>> one of
>> them being the certificate renewal master (ipa1). The system has been
>> running for years and has been migrated from an IPA 3 system.
>>
>> Since a while, the Web UI logins on ipa1 don't work anymore ("Login
>> failed
>> due to an unknown reason.").
>>
>> Web UI logins on the other server (ipa2) work and everything else is
>> working fine, too, ipactl status reports all services running.
>>
>> On login attempt:
>>
>> --- httpd log
>> [...]
>> [:error] [pid 15551] [remote 141.51.X.X:0] mod_wsgi (pid=15551):
>> Exception
>> occurred processing WSGI script '/usr/share/ipa/wsgi.py'.
>> [...]
>> [:error] [pid 15551] [remote 141.51.X.X:0] CalledProcessError: Command
>> '/usr/bin/kinit -n -c /var/run/ipa/ccaches/armor_15551 -X
>> X509_anchors=FILE:/var/kerberos/krb5kdc/kdc.crt -X
>> X509_anchors=FILE:/var/lib/ipa-client/pki/kdc-ca-bundle.pem' returned
>> non-zero exit status 1
>> ---
>>
>> --- krb5kdc.log
>> [...]
>> Dec 20 16:06:54 ipa1.example.com krb5kdc[15517](info): AS_REQ (8 etypes
>> {18 17 20 19 16 23 25 26}) 141.51.X.Y: NEEDED_PREAUTH:
>> WELLKNOWN/ANONYMOUS(a)EXAMPLE.COM for krbtgt/EXAMPLE.COM(a)EXAMPLE.COM,
>> Additional pre-authentication required
>> Dec 20 16:06:54 ipa1.example.com krb5kdc[15517](info): closing down
>> fd 11
>> Dec 20 16:06:54 ipa1.example.com krb5kdc[15518](info): AS_REQ (8 etypes
>> {18 17 20 19 16 23 25 26}) 141.51.X.Y: KDC_RETURN_PADATA:
>> WELLKNOWN/ANONYMOUS(a)EXAMPLE.COM for krbtgt/EXAMPLE.COM(a)EXAMPLE.COM,
>> Failed
>> to verify own certificate (depth 0): certificate has expired
>> Dec 20 16:06:54 ipa1.example.com krb5kdc[15518](info): closing down
>> fd 11
>> ---
>>
>> --- ipa-checkcerts.py
>> IPA version 4.5.4-10.el7.centos.3
>> Check CA status
>> Check tracking
>> Check NSS trust
>> Check dates
>> Checking certificates in CS.cfg
>> Comparing certificates to requests in LDAP
>> Checking RA certificate
>> Checking authorities
>> Checking host keytab
>> Validating certificates
>> Checking renewal master
>> End-to-end cert API test
>> Checking permissions and ownership
>> Failures:
>> Unable to find request for serial 268304391
>> Unable to find request for serial 268304394
>> Unable to find request for serial 268304393
>> Unable to find request for serial 268304392
>> Subject O=EXAMPLE.COM,CN=ipa2.example.com and template subject
>> CN=ipa1.example.com,O=EXAMPLE.COM do not match for serial 57
>> ---
>>
>> --- ipa pkinit-status --all
>> -----------------
>> 2 servers matched
>> -----------------
>> Server name: ipa2.example.com
>> PKINIT status: enabled
>>
>> Server name: ipa1.example.com
>> PKINIT status: enabled
>> ----------------------------
>> Number of entries returned 2
>> ----------------------------
>>
>> To my understanding, proper certificate exchange between my two servers
>> ceased working at some point. How do i track this down and fix it?
>>
> your issue looks similar to ticket #6792 [1]. Can you check the result
> of upgrade in /var/log/ipaupgrade.log?
> Also check the output of
> $ ipa-pkinit-manage status
> and if the files /var/lib/ipa-client/pki/kdc-ca-bundle.pem and
> /var/lib/ipa-client/pki/ca-bundle.pem exist, with -rw-r--r-- permissions.
>
> Regarding the certificates, does getcert list show expired certs?
> flo
>
> [1] https://pagure.io/freeipa/issue/6792
---
$ ipa-pkinit-manage status
PKINIT is enabled
---
There are no expired certificates, kdc-ca-bundle.pem and ca-bundle.pem
exist with proper permissions, but I found something in ipaupgrade.log:
---
2018-09-12T13:37:18Z DEBUG args=/usr/bin/certutil -d /etc/httpd/alias -L
-f /etc/httpd/alias/pwdfile.txt
2018-09-12T13:37:19Z DEBUG Process finished, return code=0
2018-09-12T13:37:19Z DEBUG stdout=
Certificate Nickname Trust
Attributes
SSL,S/MIME,JAR/XPI
Server-Cert u,u,u
EXAMPLE.COM IPA CA CT,C,C
2018-09-12T13:37:19Z DEBUG stderr=
2018-09-12T13:37:19Z DEBUG Starting external process
2018-09-12T13:37:19Z DEBUG args=/usr/bin/certutil -d /etc/httpd/alias -L
-n EXAMPLE.COM IPA CA -a -f /etc/httpd/alias/pwdfile.txt
2018-09-12T13:37:19Z DEBUG Process finished, return code=0
2018-09-12T13:37:19Z DEBUG stdout=
-----BEGIN CERTIFICATE-----
[...]
-----END CERTIFICATE-----
2018-09-12T13:37:19Z DEBUG stderr=
2018-09-12T13:37:19Z DEBUG Executing upgrade plugin: update_ra_cert_store
2018-09-12T13:37:19Z DEBUG raw: update_ra_cert_store
2018-09-12T13:37:19Z DEBUG raw: ca_is_enabled(version=u'2.228')
2018-09-12T13:37:19Z DEBUG ca_is_enabled(version=u'2.228')
2018-09-12T13:37:19Z DEBUG Starting external process
2018-09-12T13:37:19Z DEBUG args=/usr/bin/certutil -d /etc/httpd/alias -L
-n ipaCert -a -f /etc/httpd/alias/pwdfile.txt
2018-09-12T13:37:19Z DEBUG Process finished, return code=255
2018-09-12T13:37:19Z DEBUG stdout=
2018-09-12T13:37:19Z DEBUG stderr=certutil: Could not find cert: ipaCert
: PR_FILE_NOT_FOUND_ERROR: File not found
[...]
---
Hi,
this error can be ignored in most of the cases. The upgrade is trying to
move ipaCert (cert+key) from the NSS database /etc/httpd/alias to the
files /var/lib/ipa/ra-agent.pem and /var/lib/ipa/ra-agent.key. So if the
upgrade is run a second time, he won't find ipaCert in the NSS database.
To be sure, you can check if /var/lib/ipa/ra-agent.{pem|key} are present
and contain a certificate with Subject CN=IPA RA,O=DOMAIN.COM. The files
must be readable by root and ipaapi group, and must contain the same
cert as the other masters.
What is the content of /var/lib/ipa-client/pki/kdc-ca-bundle.pem and
ca-bundle.pem? both must contain IPA CA certificate.
What are the permissions of /var/kerberos/krb5kdc/kdc.crt? It needs to
be readable by everyone. And what is the content of this cert? It should
be issued by IPA CA.
About the errors spotted by ipa-checkcerts.py, what are the certificates
with errors reported? You can find them with ipa cert-show <serial>.
flo
Mit freundlichen Gruessen/With best regards,
--Daniel.
_______________________________________________
FreeIPA-users mailing list -- freeipa-users(a)lists.fedorahosted.org
To unsubscribe send an email to freeipa-users-leave(a)lists.fedorahosted.org
Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives:
https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedoraho...
4:58 a.m.
Hi Florence,
thank you very much for your help.
On Fri, 21 Dec 2018, Florence Blanc-Renaud via FreeIPA-users wrote:
On 12/20/18 6:52 PM, dbischof--- via FreeIPA-users wrote:
> On Thu, 20 Dec 2018, Florence Blanc-Renaud via FreeIPA-users wrote:
>
>> On 12/20/18 4:22 PM, dbischof--- via FreeIPA-users wrote:
>>> my IPA system consists of 2 masters with their own self-signed CAs,
>>> one of
>>> them being the certificate renewal master (ipa1). The system has been
>>> running for years and has been migrated from an IPA 3 system.
>>>
>>> Since a while, the Web UI logins on ipa1 don't work anymore
("Login
>>> failed
>>> due to an unknown reason.").
>>>
>>> Web UI logins on the other server (ipa2) work and everything else is
>>> working fine, too, ipactl status reports all services running.
>>>
>>> On login attempt:
>>>
>>> --- httpd log
>>> [...]
>>> [:error] [pid 15551] [remote 141.51.X.X:0] mod_wsgi (pid=15551):
>>> Exception
>>> occurred processing WSGI script '/usr/share/ipa/wsgi.py'.
>>> [...]
>>> [:error] [pid 15551] [remote 141.51.X.X:0] CalledProcessError: Command
>>> '/usr/bin/kinit -n -c /var/run/ipa/ccaches/armor_15551 -X
>>> X509_anchors=FILE:/var/kerberos/krb5kdc/kdc.crt -X
>>> X509_anchors=FILE:/var/lib/ipa-client/pki/kdc-ca-bundle.pem' returned
>>> non-zero exit status 1
>>> ---
>>>
>>> --- krb5kdc.log
>>> [...]
>>> Dec 20 16:06:54 ipa1.example.com krb5kdc[15517](info): AS_REQ (8 etypes
>>> {18 17 20 19 16 23 25 26}) 141.51.X.Y: NEEDED_PREAUTH:
>>> WELLKNOWN/ANONYMOUS(a)EXAMPLE.COM for krbtgt/EXAMPLE.COM(a)EXAMPLE.COM,
>>> Additional pre-authentication required
>>> Dec 20 16:06:54 ipa1.example.com krb5kdc[15517](info): closing down fd
>>> 11
>>> Dec 20 16:06:54 ipa1.example.com krb5kdc[15518](info): AS_REQ (8 etypes
>>> {18 17 20 19 16 23 25 26}) 141.51.X.Y: KDC_RETURN_PADATA:
>>> WELLKNOWN/ANONYMOUS(a)EXAMPLE.COM for krbtgt/EXAMPLE.COM(a)EXAMPLE.COM,
>>> Failed
>>> to verify own certificate (depth 0): certificate has expired
>>> Dec 20 16:06:54 ipa1.example.com krb5kdc[15518](info): closing down fd
>>> 11
>>> ---
>>>
>>> --- ipa-checkcerts.py
>>> IPA version 4.5.4-10.el7.centos.3
>>> Check CA status
>>> Check tracking
>>> Check NSS trust
>>> Check dates
>>> Checking certificates in CS.cfg
>>> Comparing certificates to requests in LDAP
>>> Checking RA certificate
>>> Checking authorities
>>> Checking host keytab
>>> Validating certificates
>>> Checking renewal master
>>> End-to-end cert API test
>>> Checking permissions and ownership
>>> Failures:
>>> Unable to find request for serial 268304391
>>> Unable to find request for serial 268304394
>>> Unable to find request for serial 268304393
>>> Unable to find request for serial 268304392
>>> Subject O=EXAMPLE.COM,CN=ipa2.example.com and template subject
>>> CN=ipa1.example.com,O=EXAMPLE.COM do not match for serial 57
>>> ---
>>>
>>> --- ipa pkinit-status --all
>>> -----------------
>>> 2 servers matched
>>> -----------------
>>> Server name: ipa2.example.com
>>> PKINIT status: enabled
>>>
>>> Server name: ipa1.example.com
>>> PKINIT status: enabled
>>> ----------------------------
>>> Number of entries returned 2
>>> ----------------------------
>>>
>>> To my understanding, proper certificate exchange between my two servers
>>> ceased working at some point. How do i track this down and fix it?
>>>
>> your issue looks similar to ticket #6792 [1]. Can you check the result of
>> upgrade in /var/log/ipaupgrade.log?
>> Also check the output of
>> $ ipa-pkinit-manage status
>> and if the files /var/lib/ipa-client/pki/kdc-ca-bundle.pem and
>> /var/lib/ipa-client/pki/ca-bundle.pem exist, with -rw-r--r-- permissions.
>>
>> Regarding the certificates, does getcert list show expired certs?
>> flo
>>
>> [1] https://pagure.io/freeipa/issue/6792
>
> ---
> $ ipa-pkinit-manage status
> PKINIT is enabled
> ---
>
> There are no expired certificates, kdc-ca-bundle.pem and ca-bundle.pem
> exist with proper permissions, but I found something in ipaupgrade.log:
>
> ---
> 2018-09-12T13:37:18Z DEBUG args=/usr/bin/certutil -d /etc/httpd/alias -L
> -f /etc/httpd/alias/pwdfile.txt
> 2018-09-12T13:37:19Z DEBUG Process finished, return code=0
> 2018-09-12T13:37:19Z DEBUG stdout=
> Certificate Nickname Trust
> Attributes
>
> SSL,S/MIME,JAR/XPI
>
> Server-Cert u,u,u
> EXAMPLE.COM IPA CA CT,C,C
>
> 2018-09-12T13:37:19Z DEBUG stderr=
> 2018-09-12T13:37:19Z DEBUG Starting external process
> 2018-09-12T13:37:19Z DEBUG args=/usr/bin/certutil -d /etc/httpd/alias -L
> -n EXAMPLE.COM IPA CA -a -f /etc/httpd/alias/pwdfile.txt
> 2018-09-12T13:37:19Z DEBUG Process finished, return code=0
> 2018-09-12T13:37:19Z DEBUG stdout=
> -----BEGIN CERTIFICATE-----
> [...]
> -----END CERTIFICATE-----
>
> 2018-09-12T13:37:19Z DEBUG stderr=
> 2018-09-12T13:37:19Z DEBUG Executing upgrade plugin: update_ra_cert_store
> 2018-09-12T13:37:19Z DEBUG raw: update_ra_cert_store
> 2018-09-12T13:37:19Z DEBUG raw: ca_is_enabled(version=u'2.228')
> 2018-09-12T13:37:19Z DEBUG ca_is_enabled(version=u'2.228')
> 2018-09-12T13:37:19Z DEBUG Starting external process
> 2018-09-12T13:37:19Z DEBUG args=/usr/bin/certutil -d /etc/httpd/alias -L
> -n ipaCert -a -f /etc/httpd/alias/pwdfile.txt
> 2018-09-12T13:37:19Z DEBUG Process finished, return code=255
> 2018-09-12T13:37:19Z DEBUG stdout=
> 2018-09-12T13:37:19Z DEBUG stderr=certutil: Could not find cert: ipaCert
> : PR_FILE_NOT_FOUND_ERROR: File not found
> [...]
> ---
>
this error can be ignored in most of the cases. The upgrade is trying to
move ipaCert (cert+key) from the NSS database /etc/httpd/alias to the
files /var/lib/ipa/ra-agent.pem and /var/lib/ipa/ra-agent.key. So if the
upgrade is run a second time, he won't find ipaCert in the NSS database.
To be sure, you can check if /var/lib/ipa/ra-agent.{pem|key} are present
and contain a certificate with Subject CN=IPA RA,O=DOMAIN.COM. The files
must be readable by root and ipaapi group, and must contain the same
cert as the other masters.
checked this, is ok.
What is the content of /var/lib/ipa-client/pki/kdc-ca-bundle.pem and
ca-bundle.pem? both must contain IPA CA certificate.
True on both servers.
What are the permissions of /var/kerberos/krb5kdc/kdc.crt? It needs
to
be readable by everyone. And what is the content of this cert? It should
be issued by IPA CA.
Permissions are ok, contents:
--- ipa1
$ openssl x509 -in /var/kerberos/krb5kdc/kdc.crt -text -noout -fingerprint
Certificate:
Data:
Version: 3 (0x2)
Serial Number: 1 (0x1)
Signature Algorithm: sha256WithRSAEncryption
Issuer: O=EXAMPLE.COM, CN=ipa1.example.com
Validity
Not Before: Nov 28 12:43:05 2017 GMT
Not After : Nov 28 12:43:05 2018 GMT
Subject: O=EXAMPLE.COM, CN=ipa1.example.com
Subject Public Key Info:
Public Key Algorithm: rsaEncryption
Public-Key: (2048 bit)
Modulus:
[...]
Exponent: 65537 (0x10001)
X509v3 extensions:
X509v3 Subject Alternative Name:
othername:<unsupported>, othername:<unsupported>
X509v3 Basic Constraints: critical
CA:FALSE
X509v3 Subject Key Identifier:
86:52:EC:A1:C3:FB:EC:CC:6D:F2:09:E7:64:88:D1:80:F4:71:81:AE
1.3.6.1.4.1.311.20.2:
.".K.D.C.s._.P.K.I.N.I.T._.C.e.r.t.s
[...]
---
--- ipa2
$ openssl x509 -in /var/kerberos/krb5kdc/kdc.crt -text -noout -fingerprint
Certificate:
Data:
Version: 3 (0x2)
Serial Number: 805240833 (0x2fff0001)
Signature Algorithm: sha256WithRSAEncryption
Issuer: O=EXAMPLE.COM, CN=Certificate Authority
Validity
Not Before: Jan 18 13:04:17 2018 GMT
Not After : Jan 19 13:04:17 2020 GMT
Subject: O=EXAMPLE.COM, CN=ipa2.example.com
Subject Public Key Info:
Public Key Algorithm: rsaEncryption
Public-Key: (2048 bit)
Modulus:
[...]
Exponent: 65537 (0x10001)
X509v3 extensions:
X509v3 Authority Key Identifier:
keyid:4B:BA:AA:46:F1:29:E4:43:8B:DC:30:B4:90:3E:66:72:DD:F6:C7:FB
Authority Information Access:
OCSP - URI:http://ipa-ca.example.com/ca/ocsp
X509v3 Key Usage: critical
Digital Signature, Non Repudiation, Key Encipherment, Data Encipherment
X509v3 Extended Key Usage:
TLS Web Server Authentication, 1.3.6.1.5.2.3.5
X509v3 CRL Distribution Points:
Full Name:
URI:http://ipa-ca.example.com/ipa/crl/MasterCRL.bin
CRL Issuer:
DirName: O = ipaca, CN = Certificate Authority
X509v3 Subject Key Identifier:
96:C3:94:70:7E:46:77:DB:91:F8:DF:D6:27:FE:73:0A:45:F3:78:F3
X509v3 Subject Alternative Name:
othername:<unsupported>, othername:<unsupported>
[...]
---
Additional info: I have DNS separate from IPA, but i (hopefully) made
proper records as IPA would have done it. In particular, i made an A
record "ipa-ca" that has IPs of both ipa1 and ipa2 - hope, this is not the
root cause of my problems, since DNS is not under my control.
Since /var/kerberos/krb5kdc/kdc.crt on ipa1 appears to be not issued by
IPA CA, might this be the actual problem?
About the errors spotted by ipa-checkcerts.py, what are the
certificates
with errors reported? You can find them with ipa cert-show <serial>.
---
$ipa cert-show 57
Issuing CA: ipa
Certificate: [...]
Subject: CN=ipa1.example.com,O=EXAMPLE.COM
Issuer: CN=Certificate Authority,O=EXAMPLE.COM
Not Before: Tue Nov 28 12:42:07 2017 UTC
Not After: Mon Nov 18 12:42:07 2019 UTC
Serial number: 57
Serial number (hex): 0x39
Revoked: False
$ ipa cert-show 268304391
Issuing CA: ipa
Certificate: [...]
Subject: CN=IPA RA,O=EXAMPLE.COM
Issuer: CN=Certificate Authority,O=EXAMPLE.COM
Not Before: Wed Jul 11 14:26:35 2018 UTC
Not After: Tue Jun 30 14:26:35 2020 UTC
Serial number: 268304391
Serial number (hex): 0xFFE0007
Revoked: False
$ ipa cert-show 268304392
Issuing CA: ipa
Certificate: [...]
Subject: CN=CA Subsystem,O=EXAMPLE.COM
Issuer: CN=Certificate Authority,O=EXAMPLE.COM
Not Before: Wed Jul 11 14:26:45 2018 UTC
Not After: Tue Jun 30 14:26:45 2020 UTC
Serial number: 268304392
Serial number (hex): 0xFFE0008
Revoked: False
$ ipa cert-show 268304393
Issuing CA: ipa
Certificate: [...]
Subject: CN=OCSP Subsystem,O=EXAMPLE.COM
Issuer: CN=Certificate Authority,O=EXAMPLE.COM
Not Before: Wed Jul 11 14:27:05 2018 UTC
Not After: Tue Jun 30 14:27:05 2020 UTC
Serial number: 268304393
Serial number (hex): 0xFFE0009
Revoked: False
$ ipa cert-show 268304394
Issuing CA: ipa
Certificate: [...]
Subject: CN=CA Audit,O=EXAMPLE.COM
Issuer: CN=Certificate Authority,O=EXAMPLE.COM
Not Before: Wed Jul 11 14:27:25 2018 UTC
Not After: Tue Jun 30 14:27:25 2020 UTC
Serial number: 268304394
Serial number (hex): 0xFFE000A
Revoked: False
---
ipa cert-show on server ipa2 fails with "ipa: ERROR: Certificate operation
cannot be completed: EXCEPTION (Invalid Credential.)" btw.
Mit freundlichen Gruessen/With best regards,
--Daniel.
5:30 a.m.
On 12/21/18 11:58 AM, dbischof--- via FreeIPA-users wrote:
Hi Florence,
thank you very much for your help.
On Fri, 21 Dec 2018, Florence Blanc-Renaud via FreeIPA-users wrote:
> On 12/20/18 6:52 PM, dbischof--- via FreeIPA-users wrote:
>> On Thu, 20 Dec 2018, Florence Blanc-Renaud via FreeIPA-users wrote:
>>
>>> On 12/20/18 4:22 PM, dbischof--- via FreeIPA-users wrote:
>>>> my IPA system consists of 2 masters with their own self-signed CAs,
>>>> one of
>>>> them being the certificate renewal master (ipa1). The system has
>>>> been
>>>> running for years and has been migrated from an IPA 3 system.
>>>>
>>>> Since a while, the Web UI logins on ipa1 don't work anymore
("Login
>>>> failed
>>>> due to an unknown reason.").
>>>>
>>>> Web UI logins on the other server (ipa2) work and everything else is
>>>> working fine, too, ipactl status reports all services running.
>>>>
>>>> On login attempt:
>>>>
>>>> --- httpd log
>>>> [...]
>>>> [:error] [pid 15551] [remote 141.51.X.X:0] mod_wsgi (pid=15551):
>>>> Exception
>>>> occurred processing WSGI script '/usr/share/ipa/wsgi.py'.
>>>> [...]
>>>> [:error] [pid 15551] [remote 141.51.X.X:0] CalledProcessError:
>>>> Command
>>>> '/usr/bin/kinit -n -c /var/run/ipa/ccaches/armor_15551 -X
>>>> X509_anchors=FILE:/var/kerberos/krb5kdc/kdc.crt -X
>>>> X509_anchors=FILE:/var/lib/ipa-client/pki/kdc-ca-bundle.pem'
>>>> returned
>>>> non-zero exit status 1
>>>> ---
>>>>
>>>> --- krb5kdc.log
>>>> [...]
>>>> Dec 20 16:06:54 ipa1.example.com krb5kdc[15517](info): AS_REQ (8
>>>> etypes
>>>> {18 17 20 19 16 23 25 26}) 141.51.X.Y: NEEDED_PREAUTH:
>>>> WELLKNOWN/ANONYMOUS(a)EXAMPLE.COM for krbtgt/EXAMPLE.COM(a)EXAMPLE.COM,
>>>> Additional pre-authentication required
>>>> Dec 20 16:06:54 ipa1.example.com krb5kdc[15517](info): closing
>>>> down fd
>>>> 11
>>>> Dec 20 16:06:54 ipa1.example.com krb5kdc[15518](info): AS_REQ (8
>>>> etypes
>>>> {18 17 20 19 16 23 25 26}) 141.51.X.Y: KDC_RETURN_PADATA:
>>>> WELLKNOWN/ANONYMOUS(a)EXAMPLE.COM for krbtgt/EXAMPLE.COM(a)EXAMPLE.COM,
>>>> Failed
>>>> to verify own certificate (depth 0): certificate has expired
>>>> Dec 20 16:06:54 ipa1.example.com krb5kdc[15518](info): closing
>>>> down fd
>>>> 11
>>>> ---
>>>>
>>>> --- ipa-checkcerts.py
>>>> IPA version 4.5.4-10.el7.centos.3
>>>> Check CA status
>>>> Check tracking
>>>> Check NSS trust
>>>> Check dates
>>>> Checking certificates in CS.cfg
>>>> Comparing certificates to requests in LDAP
>>>> Checking RA certificate
>>>> Checking authorities
>>>> Checking host keytab
>>>> Validating certificates
>>>> Checking renewal master
>>>> End-to-end cert API test
>>>> Checking permissions and ownership
>>>> Failures:
>>>> Unable to find request for serial 268304391
>>>> Unable to find request for serial 268304394
>>>> Unable to find request for serial 268304393
>>>> Unable to find request for serial 268304392
>>>> Subject O=EXAMPLE.COM,CN=ipa2.example.com and template subject
>>>> CN=ipa1.example.com,O=EXAMPLE.COM do not match for serial 57
>>>> ---
>>>>
>>>> --- ipa pkinit-status --all
>>>> -----------------
>>>> 2 servers matched
>>>> -----------------
>>>> Server name: ipa2.example.com
>>>> PKINIT status: enabled
>>>>
>>>> Server name: ipa1.example.com
>>>> PKINIT status: enabled
>>>> ----------------------------
>>>> Number of entries returned 2
>>>> ----------------------------
>>>>
>>>> To my understanding, proper certificate exchange between my two
>>>> servers
>>>> ceased working at some point. How do i track this down and fix it?
>>>>
>>> your issue looks similar to ticket #6792 [1]. Can you check the
>>> result of
>>> upgrade in /var/log/ipaupgrade.log?
>>> Also check the output of
>>> $ ipa-pkinit-manage status
>>> and if the files /var/lib/ipa-client/pki/kdc-ca-bundle.pem and
>>> /var/lib/ipa-client/pki/ca-bundle.pem exist, with -rw-r--r--
>>> permissions.
>>>
>>> Regarding the certificates, does getcert list show expired certs?
>>> flo
>>>
>>> [1] https://pagure.io/freeipa/issue/6792
>>
>> ---
>> $ ipa-pkinit-manage status
>> PKINIT is enabled
>> ---
>>
>> There are no expired certificates, kdc-ca-bundle.pem and ca-bundle.pem
>> exist with proper permissions, but I found something in ipaupgrade.log:
>>
>> ---
>> 2018-09-12T13:37:18Z DEBUG args=/usr/bin/certutil -d
>> /etc/httpd/alias -L
>> -f /etc/httpd/alias/pwdfile.txt
>> 2018-09-12T13:37:19Z DEBUG Process finished, return code=0
>> 2018-09-12T13:37:19Z DEBUG stdout=
>> Certificate Nickname Trust
>> Attributes
>>
>> SSL,S/MIME,JAR/XPI
>>
>> Server-Cert u,u,u
>> EXAMPLE.COM IPA CA CT,C,C
>>
>> 2018-09-12T13:37:19Z DEBUG stderr=
>> 2018-09-12T13:37:19Z DEBUG Starting external process
>> 2018-09-12T13:37:19Z DEBUG args=/usr/bin/certutil -d
>> /etc/httpd/alias -L
>> -n EXAMPLE.COM IPA CA -a -f /etc/httpd/alias/pwdfile.txt
>> 2018-09-12T13:37:19Z DEBUG Process finished, return code=0
>> 2018-09-12T13:37:19Z DEBUG stdout=
>> -----BEGIN CERTIFICATE-----
>> [...]
>> -----END CERTIFICATE-----
>>
>> 2018-09-12T13:37:19Z DEBUG stderr=
>> 2018-09-12T13:37:19Z DEBUG Executing upgrade plugin:
>> update_ra_cert_store
>> 2018-09-12T13:37:19Z DEBUG raw: update_ra_cert_store
>> 2018-09-12T13:37:19Z DEBUG raw: ca_is_enabled(version=u'2.228')
>> 2018-09-12T13:37:19Z DEBUG ca_is_enabled(version=u'2.228')
>> 2018-09-12T13:37:19Z DEBUG Starting external process
>> 2018-09-12T13:37:19Z DEBUG args=/usr/bin/certutil -d
>> /etc/httpd/alias -L
>> -n ipaCert -a -f /etc/httpd/alias/pwdfile.txt
>> 2018-09-12T13:37:19Z DEBUG Process finished, return code=255
>> 2018-09-12T13:37:19Z DEBUG stdout=
>> 2018-09-12T13:37:19Z DEBUG stderr=certutil: Could not find cert:
>> ipaCert
>> : PR_FILE_NOT_FOUND_ERROR: File not found
>> [...]
>> ---
>>
> this error can be ignored in most of the cases. The upgrade is trying
> to move ipaCert (cert+key) from the NSS database /etc/httpd/alias to
> the files /var/lib/ipa/ra-agent.pem and /var/lib/ipa/ra-agent.key. So
> if the upgrade is run a second time, he won't find ipaCert in the NSS
> database. To be sure, you can check if /var/lib/ipa/ra-agent.{pem|key}
> are present and contain a certificate with Subject CN=IPA
> RA,O=DOMAIN.COM. The files must be readable by root and ipaapi group,
> and must contain the same cert as the other masters.
checked this, is ok.
> What is the content of /var/lib/ipa-client/pki/kdc-ca-bundle.pem and
> ca-bundle.pem? both must contain IPA CA certificate.
True on both servers.
> What are the permissions of /var/kerberos/krb5kdc/kdc.crt? It needs to
> be readable by everyone. And what is the content of this cert? It
> should be issued by IPA CA.
Permissions are ok, contents:
--- ipa1
$ openssl x509 -in /var/kerberos/krb5kdc/kdc.crt -text -noout -fingerprint
Certificate:
Data:
Version: 3 (0x2)
Serial Number: 1 (0x1)
Signature Algorithm: sha256WithRSAEncryption
Issuer: O=EXAMPLE.COM, CN=ipa1.example.com
Validity
Not Before: Nov 28 12:43:05 2017 GMT
Not After : Nov 28 12:43:05 2018 GMT
Subject: O=EXAMPLE.COM, CN=ipa1.example.com
Subject Public Key Info:
Public Key Algorithm: rsaEncryption
Public-Key: (2048 bit)
Modulus:
[...]
Exponent: 65537 (0x10001)
X509v3 extensions:
X509v3 Subject Alternative Name:
othername:<unsupported>, othername:<unsupported>
X509v3 Basic Constraints: critical
CA:FALSE
X509v3 Subject Key Identifier:
86:52:EC:A1:C3:FB:EC:CC:6D:F2:09:E7:64:88:D1:80:F4:71:81:AE
1.3.6.1.4.1.311.20.2:
.".K.D.C.s._.P.K.I.N.I.T._.C.e.r.t.s
[...]
---
--- ipa2
$ openssl x509 -in /var/kerberos/krb5kdc/kdc.crt -text -noout -fingerprint
Certificate:
Data:
Version: 3 (0x2)
Serial Number: 805240833 (0x2fff0001)
Signature Algorithm: sha256WithRSAEncryption
Issuer: O=EXAMPLE.COM, CN=Certificate Authority
Validity
Not Before: Jan 18 13:04:17 2018 GMT
Not After : Jan 19 13:04:17 2020 GMT
Subject: O=EXAMPLE.COM, CN=ipa2.example.com
Subject Public Key Info:
Public Key Algorithm: rsaEncryption
Public-Key: (2048 bit)
Modulus:
[...]
Exponent: 65537 (0x10001)
X509v3 extensions:
X509v3 Authority Key Identifier:
keyid:4B:BA:AA:46:F1:29:E4:43:8B:DC:30:B4:90:3E:66:72:DD:F6:C7:FB
Authority Information Access:
OCSP - URI:http://ipa-ca.example.com/ca/ocsp
X509v3 Key Usage: critical
Digital Signature, Non Repudiation, Key Encipherment,
Data Encipherment
X509v3 Extended Key Usage:
TLS Web Server Authentication, 1.3.6.1.5.2.3.5
X509v3 CRL Distribution Points:
Full Name:
URI:http://ipa-ca.example.com/ipa/crl/MasterCRL.bin
CRL Issuer:
DirName: O = ipaca, CN = Certificate Authority
X509v3 Subject Key Identifier:
96:C3:94:70:7E:46:77:DB:91:F8:DF:D6:27:FE:73:0A:45:F3:78:F3
X509v3 Subject Alternative Name:
othername:<unsupported>, othername:<unsupported>
[...]
---
Additional info: I have DNS separate from IPA, but i (hopefully) made
proper records as IPA would have done it. In particular, i made an A
record "ipa-ca" that has IPs of both ipa1 and ipa2 - hope, this is not
the root cause of my problems, since DNS is not under my control.
Since /var/kerberos/krb5kdc/kdc.crt on ipa1 appears to be not issued by
IPA CA, might this be the actual problem?
The cert is expired on ipa1, which is the real root cause.
ipa-pkinit-manage status is reporting that PKINIT is enabled but this
does not match the actual configuration. This is a known issue [1].
If the host ipa1 is running a CA instance, then you can delete
/var/kerberos/krb5kdc/kdc.{key,crt} and run ipa-pkinit-manage enable to
re-generate the KDC cert.
Note: if the host doesn't run a CA instance, then this won't work
because of the issue [2].
To check which hosts are running a CA instance, you can use
# ipa server-role-find --role 'CA server'
[1] https://pagure.io/freeipa/issue/7200
[2] https://pagure.io/freeipa/issue/7795
> About the errors spotted by ipa-checkcerts.py, what are the
> certificates with errors reported? You can find them with ipa
> cert-show <serial>.
---
$ipa cert-show 57
Issuing CA: ipa
Certificate: [...]
Subject: CN=ipa1.example.com,O=EXAMPLE.COM
Issuer: CN=Certificate Authority,O=EXAMPLE.COM
Not Before: Tue Nov 28 12:42:07 2017 UTC
Not After: Mon Nov 18 12:42:07 2019 UTC
Serial number: 57
Serial number (hex): 0x39
Revoked: False
$ ipa cert-show 268304391
Issuing CA: ipa
Certificate: [...]
Subject: CN=IPA RA,O=EXAMPLE.COM
Issuer: CN=Certificate Authority,O=EXAMPLE.COM
Not Before: Wed Jul 11 14:26:35 2018 UTC
Not After: Tue Jun 30 14:26:35 2020 UTC
Serial number: 268304391
Serial number (hex): 0xFFE0007
Revoked: False
$ ipa cert-show 268304392
Issuing CA: ipa
Certificate: [...]
Subject: CN=CA Subsystem,O=EXAMPLE.COM
Issuer: CN=Certificate Authority,O=EXAMPLE.COM
Not Before: Wed Jul 11 14:26:45 2018 UTC
Not After: Tue Jun 30 14:26:45 2020 UTC
Serial number: 268304392
Serial number (hex): 0xFFE0008
Revoked: False
$ ipa cert-show 268304393
Issuing CA: ipa
Certificate: [...]
Subject: CN=OCSP Subsystem,O=EXAMPLE.COM
Issuer: CN=Certificate Authority,O=EXAMPLE.COM
Not Before: Wed Jul 11 14:27:05 2018 UTC
Not After: Tue Jun 30 14:27:05 2020 UTC
Serial number: 268304393
Serial number (hex): 0xFFE0009
Revoked: False
$ ipa cert-show 268304394
Issuing CA: ipa
Certificate: [...]
Subject: CN=CA Audit,O=EXAMPLE.COM
Issuer: CN=Certificate Authority,O=EXAMPLE.COM
Not Before: Wed Jul 11 14:27:25 2018 UTC
Not After: Tue Jun 30 14:27:25 2020 UTC
Serial number: 268304394
Serial number (hex): 0xFFE000A
Revoked: False
---
ipa cert-show on server ipa2 fails with "ipa: ERROR: Certificate
operation cannot be completed: EXCEPTION (Invalid Credential.)" btw.
The errors related to "Unable to find request for serial xxx" mean that
the cert is tracked by certmonger, but there is no corresponding LDAP
entry cn=xxx,ou=ca,ou=requests,o=ipaca on the local LDAP server.
Is the replication still working between your two masters?
"ipa cert-show" broken on ipa2 often points to an out-of-date
certificate in /var/lib/ipa/ra-agent.{pem|key}. The content should be
the same as on the renewal master, and also the same as in the entry
uid=ipara,ou=People,o=ipaca (which should be replicated and identical on
all the masters, except if you have replication issues).
HTH,
flo
Mit freundlichen Gruessen/With best regards,
--Daniel.
_______________________________________________
FreeIPA-users mailing list -- freeipa-users(a)lists.fedorahosted.org
To unsubscribe send an email to freeipa-users-leave(a)lists.fedorahosted.org
Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives:
https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedoraho...
6:03 a.m.
Hi Florence,
On Fri, 21 Dec 2018, Florence Blanc-Renaud via FreeIPA-users wrote:
On 12/21/18 11:58 AM, dbischof--- via FreeIPA-users wrote:
> thank you very much for your help.
>
> On Fri, 21 Dec 2018, Florence Blanc-Renaud via FreeIPA-users wrote:
>
>> On 12/20/18 6:52 PM, dbischof--- via FreeIPA-users wrote:
>>> On Thu, 20 Dec 2018, Florence Blanc-Renaud via FreeIPA-users wrote:
>>>
>>>> On 12/20/18 4:22 PM, dbischof--- via FreeIPA-users wrote:
>>>>> my IPA system consists of 2 masters with their own self-signed
CAs,
>>>>> one of
>>>>> them being the certificate renewal master (ipa1). The system has
>>>>> been
>>>>> running for years and has been migrated from an IPA 3 system.
>>>>>
>>>>> Since a while, the Web UI logins on ipa1 don't work anymore
("Login
>>>>> failed
>>>>> due to an unknown reason.").
>>>>>
>>>>> Web UI logins on the other server (ipa2) work and everything else
is
>>>>> working fine, too, ipactl status reports all services running.
>>>>>
>>>>> On login attempt:
>>>>>
>>>>> --- httpd log
>>>>> [...]
>>>>> [:error] [pid 15551] [remote 141.51.X.X:0] mod_wsgi (pid=15551):
>>>>> Exception
>>>>> occurred processing WSGI script '/usr/share/ipa/wsgi.py'.
>>>>> [...]
>>>>> [:error] [pid 15551] [remote 141.51.X.X:0] CalledProcessError:
>>>>> Command
>>>>> '/usr/bin/kinit -n -c /var/run/ipa/ccaches/armor_15551 -X
>>>>> X509_anchors=FILE:/var/kerberos/krb5kdc/kdc.crt -X
>>>>> X509_anchors=FILE:/var/lib/ipa-client/pki/kdc-ca-bundle.pem'
>>>>> returned
>>>>> non-zero exit status 1
>>>>> ---
>>>>>
>>>>> --- krb5kdc.log
>>>>> [...]
>>>>> Dec 20 16:06:54 ipa1.example.com krb5kdc[15517](info): AS_REQ (8
>>>>> etypes
>>>>> {18 17 20 19 16 23 25 26}) 141.51.X.Y: NEEDED_PREAUTH:
>>>>> WELLKNOWN/ANONYMOUS(a)EXAMPLE.COM for
krbtgt/EXAMPLE.COM(a)EXAMPLE.COM,
>>>>> Additional pre-authentication required
>>>>> Dec 20 16:06:54 ipa1.example.com krb5kdc[15517](info): closing
down
>>>>> fd
>>>>> 11
>>>>> Dec 20 16:06:54 ipa1.example.com krb5kdc[15518](info): AS_REQ (8
>>>>> etypes
>>>>> {18 17 20 19 16 23 25 26}) 141.51.X.Y: KDC_RETURN_PADATA:
>>>>> WELLKNOWN/ANONYMOUS(a)EXAMPLE.COM for
krbtgt/EXAMPLE.COM(a)EXAMPLE.COM,
>>>>> Failed
>>>>> to verify own certificate (depth 0): certificate has expired
>>>>> Dec 20 16:06:54 ipa1.example.com krb5kdc[15518](info): closing
down
>>>>> fd
>>>>> 11
>>>>> ---
>>>>>
>>>>> --- ipa-checkcerts.py
>>>>> IPA version 4.5.4-10.el7.centos.3
>>>>> Check CA status
>>>>> Check tracking
>>>>> Check NSS trust
>>>>> Check dates
>>>>> Checking certificates in CS.cfg
>>>>> Comparing certificates to requests in LDAP
>>>>> Checking RA certificate
>>>>> Checking authorities
>>>>> Checking host keytab
>>>>> Validating certificates
>>>>> Checking renewal master
>>>>> End-to-end cert API test
>>>>> Checking permissions and ownership
>>>>> Failures:
>>>>> Unable to find request for serial 268304391
>>>>> Unable to find request for serial 268304394
>>>>> Unable to find request for serial 268304393
>>>>> Unable to find request for serial 268304392
>>>>> Subject O=EXAMPLE.COM,CN=ipa2.example.com and template subject
>>>>> CN=ipa1.example.com,O=EXAMPLE.COM do not match for serial 57
>>>>> ---
>>>>>
>>>>> --- ipa pkinit-status --all
>>>>> -----------------
>>>>> 2 servers matched
>>>>> -----------------
>>>>> Server name: ipa2.example.com
>>>>> PKINIT status: enabled
>>>>>
>>>>> Server name: ipa1.example.com
>>>>> PKINIT status: enabled
>>>>> ----------------------------
>>>>> Number of entries returned 2
>>>>> ----------------------------
>>>>>
>>>>> To my understanding, proper certificate exchange between my two
>>>>> servers
>>>>> ceased working at some point. How do i track this down and fix
it?
>>>>>
>>>> your issue looks similar to ticket #6792 [1]. Can you check the result
>>>> of
>>>> upgrade in /var/log/ipaupgrade.log?
>>>> Also check the output of
>>>> $ ipa-pkinit-manage status
>>>> and if the files /var/lib/ipa-client/pki/kdc-ca-bundle.pem and
>>>> /var/lib/ipa-client/pki/ca-bundle.pem exist, with -rw-r--r--
>>>> permissions.
>>>>
>>>> Regarding the certificates, does getcert list show expired certs?
>>>> flo
>>>>
>>>> [1] https://pagure.io/freeipa/issue/6792
>>>
>>> ---
>>> $ ipa-pkinit-manage status
>>> PKINIT is enabled
>>> ---
>>>
>>> There are no expired certificates, kdc-ca-bundle.pem and ca-bundle.pem
>>> exist with proper permissions, but I found something in ipaupgrade.log:
>>>
>>> ---
>>> 2018-09-12T13:37:18Z DEBUG args=/usr/bin/certutil -d /etc/httpd/alias
>>> -L
>>> -f /etc/httpd/alias/pwdfile.txt
>>> 2018-09-12T13:37:19Z DEBUG Process finished, return code=0
>>> 2018-09-12T13:37:19Z DEBUG stdout=
>>> Certificate Nickname Trust
>>> Attributes
>>>
>>> SSL,S/MIME,JAR/XPI
>>>
>>> Server-Cert u,u,u
>>> EXAMPLE.COM IPA CA CT,C,C
>>>
>>> 2018-09-12T13:37:19Z DEBUG stderr=
>>> 2018-09-12T13:37:19Z DEBUG Starting external process
>>> 2018-09-12T13:37:19Z DEBUG args=/usr/bin/certutil -d /etc/httpd/alias
>>> -L
>>> -n EXAMPLE.COM IPA CA -a -f /etc/httpd/alias/pwdfile.txt
>>> 2018-09-12T13:37:19Z DEBUG Process finished, return code=0
>>> 2018-09-12T13:37:19Z DEBUG stdout=
>>> -----BEGIN CERTIFICATE-----
>>> [...]
>>> -----END CERTIFICATE-----
>>>
>>> 2018-09-12T13:37:19Z DEBUG stderr=
>>> 2018-09-12T13:37:19Z DEBUG Executing upgrade plugin:
>>> update_ra_cert_store
>>> 2018-09-12T13:37:19Z DEBUG raw: update_ra_cert_store
>>> 2018-09-12T13:37:19Z DEBUG raw: ca_is_enabled(version=u'2.228')
>>> 2018-09-12T13:37:19Z DEBUG ca_is_enabled(version=u'2.228')
>>> 2018-09-12T13:37:19Z DEBUG Starting external process
>>> 2018-09-12T13:37:19Z DEBUG args=/usr/bin/certutil -d /etc/httpd/alias
>>> -L
>>> -n ipaCert -a -f /etc/httpd/alias/pwdfile.txt
>>> 2018-09-12T13:37:19Z DEBUG Process finished, return code=255
>>> 2018-09-12T13:37:19Z DEBUG stdout=
>>> 2018-09-12T13:37:19Z DEBUG stderr=certutil: Could not find cert:
>>> ipaCert
>>> : PR_FILE_NOT_FOUND_ERROR: File not found
>>> [...]
>>> ---
>>>
>> this error can be ignored in most of the cases. The upgrade is trying to
>> move ipaCert (cert+key) from the NSS database /etc/httpd/alias to the
>> files /var/lib/ipa/ra-agent.pem and /var/lib/ipa/ra-agent.key. So if the
>> upgrade is run a second time, he won't find ipaCert in the NSS database.
>> To be sure, you can check if /var/lib/ipa/ra-agent.{pem|key} are present
>> and contain a certificate with Subject CN=IPA RA,O=DOMAIN.COM. The files
>> must be readable by root and ipaapi group, and must contain the same cert
>> as the other masters.
>
> checked this, is ok.
>
>> What is the content of /var/lib/ipa-client/pki/kdc-ca-bundle.pem and
>> ca-bundle.pem? both must contain IPA CA certificate.
>
> True on both servers.
>
>> What are the permissions of /var/kerberos/krb5kdc/kdc.crt? It needs to be
>> readable by everyone. And what is the content of this cert? It should be
>> issued by IPA CA.
>
> Permissions are ok, contents:
>
> --- ipa1
> $ openssl x509 -in /var/kerberos/krb5kdc/kdc.crt -text -noout -fingerprint
> Certificate:
> Data:
> Version: 3 (0x2)
> Serial Number: 1 (0x1)
> Signature Algorithm: sha256WithRSAEncryption
> Issuer: O=EXAMPLE.COM, CN=ipa1.example.com
> Validity
> Not Before: Nov 28 12:43:05 2017 GMT
> Not After : Nov 28 12:43:05 2018 GMT
> Subject: O=EXAMPLE.COM, CN=ipa1.example.com
> Subject Public Key Info:
> Public Key Algorithm: rsaEncryption
> Public-Key: (2048 bit)
> Modulus:
> [...]
> Exponent: 65537 (0x10001)
> X509v3 extensions:
> X509v3 Subject Alternative Name:
> othername:<unsupported>, othername:<unsupported>
> X509v3 Basic Constraints: critical
> CA:FALSE
> X509v3 Subject Key Identifier:
> 86:52:EC:A1:C3:FB:EC:CC:6D:F2:09:E7:64:88:D1:80:F4:71:81:AE
> 1.3.6.1.4.1.311.20.2:
> .".K.D.C.s._.P.K.I.N.I.T._.C.e.r.t.s
> [...]
> ---
>
> --- ipa2
> $ openssl x509 -in /var/kerberos/krb5kdc/kdc.crt -text -noout -fingerprint
> Certificate:
> Data:
> Version: 3 (0x2)
> Serial Number: 805240833 (0x2fff0001)
> Signature Algorithm: sha256WithRSAEncryption
> Issuer: O=EXAMPLE.COM, CN=Certificate Authority
> Validity
> Not Before: Jan 18 13:04:17 2018 GMT
> Not After : Jan 19 13:04:17 2020 GMT
> Subject: O=EXAMPLE.COM, CN=ipa2.example.com
> Subject Public Key Info:
> Public Key Algorithm: rsaEncryption
> Public-Key: (2048 bit)
> Modulus:
> [...]
> Exponent: 65537 (0x10001)
> X509v3 extensions:
> X509v3 Authority Key Identifier:
> keyid:4B:BA:AA:46:F1:29:E4:43:8B:DC:30:B4:90:3E:66:72:DD:F6:C7:FB
>
> Authority Information Access:
> OCSP - URI:http://ipa-ca.example.com/ca/ocsp
>
> X509v3 Key Usage: critical
> Digital Signature, Non Repudiation, Key Encipherment,
> Data Encipherment
> X509v3 Extended Key Usage:
> TLS Web Server Authentication, 1.3.6.1.5.2.3.5
> X509v3 CRL Distribution Points:
>
> Full Name:
> URI:http://ipa-ca.example.com/ipa/crl/MasterCRL.bin
> CRL Issuer:
> DirName: O = ipaca, CN = Certificate Authority
>
> X509v3 Subject Key Identifier:
> 96:C3:94:70:7E:46:77:DB:91:F8:DF:D6:27:FE:73:0A:45:F3:78:F3
> X509v3 Subject Alternative Name:
> othername:<unsupported>, othername:<unsupported>
> [...]
> ---
>
> Additional info: I have DNS separate from IPA, but i (hopefully) made
> proper records as IPA would have done it. In particular, i made an A
> record "ipa-ca" that has IPs of both ipa1 and ipa2 - hope, this is not
the
> root cause of my problems, since DNS is not under my control.
>
> Since /var/kerberos/krb5kdc/kdc.crt on ipa1 appears to be not issued by
> IPA CA, might this be the actual problem?
>
The cert is expired on ipa1, which is the real root cause.
ipa-pkinit-manage status is reporting that PKINIT is enabled but this
does not match the actual configuration. This is a known issue [1].
If the host ipa1 is running a CA instance, then you can delete
/var/kerberos/krb5kdc/kdc.{key,crt} and run ipa-pkinit-manage enable to
re-generate the KDC cert. Note: if the host doesn't run a CA instance,
then this won't work because of the issue [2].
To check which hosts are running a CA instance, you can use
# ipa server-role-find --role 'CA server'
[1] https://pagure.io/freeipa/issue/7200
[2] https://pagure.io/freeipa/issue/7795
removed the expired certificates, re-enabled PKINIT, certificates were
regenerated, Web UI logins working again. Thank you very much.
>> About the errors spotted by ipa-checkcerts.py, what are the
certificates
>> with errors reported? You can find them with ipa cert-show <serial>.
>
> ---
> $ipa cert-show 57
> Issuing CA: ipa
> Certificate: [...]
> Subject: CN=ipa1.example.com,O=EXAMPLE.COM
> Issuer: CN=Certificate Authority,O=EXAMPLE.COM
> Not Before: Tue Nov 28 12:42:07 2017 UTC
> Not After: Mon Nov 18 12:42:07 2019 UTC
> Serial number: 57
> Serial number (hex): 0x39
> Revoked: False
>
> $ ipa cert-show 268304391
> Issuing CA: ipa
> Certificate: [...]
> Subject: CN=IPA RA,O=EXAMPLE.COM
> Issuer: CN=Certificate Authority,O=EXAMPLE.COM
> Not Before: Wed Jul 11 14:26:35 2018 UTC
> Not After: Tue Jun 30 14:26:35 2020 UTC
> Serial number: 268304391
> Serial number (hex): 0xFFE0007
> Revoked: False
>
> $ ipa cert-show 268304392
> Issuing CA: ipa
> Certificate: [...]
> Subject: CN=CA Subsystem,O=EXAMPLE.COM
> Issuer: CN=Certificate Authority,O=EXAMPLE.COM
> Not Before: Wed Jul 11 14:26:45 2018 UTC
> Not After: Tue Jun 30 14:26:45 2020 UTC
> Serial number: 268304392
> Serial number (hex): 0xFFE0008
> Revoked: False
>
> $ ipa cert-show 268304393
> Issuing CA: ipa
> Certificate: [...]
> Subject: CN=OCSP Subsystem,O=EXAMPLE.COM
> Issuer: CN=Certificate Authority,O=EXAMPLE.COM
> Not Before: Wed Jul 11 14:27:05 2018 UTC
> Not After: Tue Jun 30 14:27:05 2020 UTC
> Serial number: 268304393
> Serial number (hex): 0xFFE0009
> Revoked: False
>
> $ ipa cert-show 268304394
> Issuing CA: ipa
> Certificate: [...]
> Subject: CN=CA Audit,O=EXAMPLE.COM
> Issuer: CN=Certificate Authority,O=EXAMPLE.COM
> Not Before: Wed Jul 11 14:27:25 2018 UTC
> Not After: Tue Jun 30 14:27:25 2020 UTC
> Serial number: 268304394
> Serial number (hex): 0xFFE000A
> Revoked: False
> ---
>
> ipa cert-show on server ipa2 fails with "ipa: ERROR: Certificate operation
> cannot be completed: EXCEPTION (Invalid Credential.)" btw.
>
The errors related to "Unable to find request for serial xxx" mean that the
cert is tracked by certmonger, but there is no corresponding LDAP entry
cn=xxx,ou=ca,ou=requests,o=ipaca on the local LDAP server.
Is the replication still working between your two masters?
"ipa cert-show" broken on ipa2 often points to an out-of-date certificate in
/var/lib/ipa/ra-agent.{pem|key}. The content should be the same as on the
renewal master, and also the same as in the entry uid=ipara,ou=People,o=ipaca
(which should be replicated and identical on all the masters, except if you
have replication issues).
Replication appeared to be working, however, I did server maintenance
today, ipa1 and ipa2 got upgraded and rebooted. ipa1 works fine, ipa2 does
not come up properly:
--- ipa2
$ ipactl status
Directory Service: RUNNING
krb5kdc Service: STOPPED
kadmin Service: STOPPED
httpd Service: RUNNING
ipa-custodia Service: STOPPED
ntpd Service: RUNNING
pki-tomcatd Service: RUNNING
ipa-otpd Service: STOPPED
ipa: INFO: The ipactl command was successful
---
--- ipa1
$ openssl x509 -in /var/lib/ipa/ra-agent.pem -text -noout | grep Serial
Serial Number: 268304391 (0xffe0007)
$ ldapsearch -x -b uid=ipara,ou=People,o=ipaca
[...]
description: 2;268304391;CN=Certificate Authority,O=EXAMPLE.COM;CN=IPA RA,O=EXAMPLE.COM
---
Looks good.
--- ipa2
$ openssl x509 -in /var/lib/ipa/ra-agent.pem -text -noout | grep Serial
Serial Number: 268304391 (0xffe0007)
$ ldapsearch -x -b uid=ipara,ou=People,o=ipaca
[...]
description: 2;44;CN=Certificate Authority,O=EXAMPLE.COM;CN=IPA RA,O=EXAMPLE.COM
---
Looks bad.
--- /var/log/ipaupgrade.log
[...]
2018-12-28T10:55:53Z DEBUG The ipa-server-upgrade command failed, exception:
RemoteRetrieveError: Failed to authenticate to CA REST API
---
--- /var/log/dirsrv/slapd-EXAMPLE-COM/errors
[...]
[28/Dec/2018:11:55:55.559648469 +0100] - ERR - set_krb5_creds - Could not get initial
credentials for principal [ldap/ipa2.example.com(a)EXAMPLE.COM] in keytab
[FILE:/etc/dirsrv/ds.keytab]: -1765328228 (Cannot contact any KDC for requested realm)
---
--- ipa-checkcerts.py
[...]
ipa: ERROR: ra.get_certificate(): EXCEPTION (Invalid Credential.)
ra.get_certificate(): EXCEPTION (Invalid Credential.)
ipa: INFO: Checking permissions and ownership
Checking permissions and ownership
ipa: INFO: Failures:
Failures:
ipa: INFO: Unable to find request for serial 268304391
Unable to find request for serial 268304391
ipa: INFO: Unable to find request for serial 268304394
Unable to find request for serial 268304394
ipa: INFO: Unable to find request for serial 268304393
Unable to find request for serial 268304393
ipa: INFO: Unable to find request for serial 268304392
Unable to find request for serial 268304392
ipa: INFO: Unable to find request for serial 268304390
Unable to find request for serial 268304390
ipa: INFO: RA agent description does not match 2;44;CN=Certificate
Authority,O=EXAMPLE.COM;CN=IPA RA,O=EXAMPLE.COM in LDAP and 2;268304391;CN=Certificate
Authority,O=EXAMPLE.COM;CN=IPA RA,O=EXAMPLE.COM expected
ipa: INFO: RA agent certificate not found in LDAP RA agent certificate not found in LDAP
[...]
---
The root cause appears to be the wrong IPA RA certificate in ipa2's LDAP,
right? I guess this has to fixed by manually importing the proper
certificate using ldapmodify, similar to the procedure described in [1]?
[1]
https://floblanc.wordpress.com/2017/09/11/troubleshooting-freeipa-pki-tom...
Mit freundlichen Gruessen/With best regards,
--Daniel.
9:44 a.m.
On 12/28/18 1:03 PM, dbischof--- via FreeIPA-users wrote:
Hi Florence,
On Fri, 21 Dec 2018, Florence Blanc-Renaud via FreeIPA-users wrote:
> On 12/21/18 11:58 AM, dbischof--- via FreeIPA-users wrote:
>> thank you very much for your help.
>>
>> On Fri, 21 Dec 2018, Florence Blanc-Renaud via FreeIPA-users wrote:
>>
>>> On 12/20/18 6:52 PM, dbischof--- via FreeIPA-users wrote:
>>>> On Thu, 20 Dec 2018, Florence Blanc-Renaud via FreeIPA-users wrote:
>>>>
>>>>> On 12/20/18 4:22 PM, dbischof--- via FreeIPA-users wrote:
>>>>>> my IPA system consists of 2 masters with their own self-signed
>>>>>> CAs,
>>>>>> one of
>>>>>> them being the certificate renewal master (ipa1). The system
has
>>>>>> been
>>>>>> running for years and has been migrated from an IPA 3 system.
>>>>>>
>>>>>> Since a while, the Web UI logins on ipa1 don't work
anymore
>>>>>> ("Login
>>>>>> failed
>>>>>> due to an unknown reason.").
>>>>>>
>>>>>> Web UI logins on the other server (ipa2) work and everything
>>>>>> else is
>>>>>> working fine, too, ipactl status reports all services
running.
>>>>>>
>>>>>> On login attempt:
>>>>>>
>>>>>> --- httpd log
>>>>>> [...]
>>>>>> [:error] [pid 15551] [remote 141.51.X.X:0] mod_wsgi
(pid=15551):
>>>>>> Exception
>>>>>> occurred processing WSGI script
'/usr/share/ipa/wsgi.py'.
>>>>>> [...]
>>>>>> [:error] [pid 15551] [remote 141.51.X.X:0]
CalledProcessError:
>>>>>> Command
>>>>>> '/usr/bin/kinit -n -c /var/run/ipa/ccaches/armor_15551 -X
>>>>>> X509_anchors=FILE:/var/kerberos/krb5kdc/kdc.crt -X
>>>>>>
X509_anchors=FILE:/var/lib/ipa-client/pki/kdc-ca-bundle.pem'
>>>>>> returned
>>>>>> non-zero exit status 1
>>>>>> ---
>>>>>>
>>>>>> --- krb5kdc.log
>>>>>> [...]
>>>>>> Dec 20 16:06:54 ipa1.example.com krb5kdc[15517](info): AS_REQ
(8
>>>>>> etypes
>>>>>> {18 17 20 19 16 23 25 26}) 141.51.X.Y: NEEDED_PREAUTH:
>>>>>> WELLKNOWN/ANONYMOUS(a)EXAMPLE.COM for
>>>>>> krbtgt/EXAMPLE.COM(a)EXAMPLE.COM,
>>>>>> Additional pre-authentication required
>>>>>> Dec 20 16:06:54 ipa1.example.com krb5kdc[15517](info): closing
>>>>>> down
>>>>>> fd
>>>>>> 11
>>>>>> Dec 20 16:06:54 ipa1.example.com krb5kdc[15518](info): AS_REQ
(8
>>>>>> etypes
>>>>>> {18 17 20 19 16 23 25 26}) 141.51.X.Y: KDC_RETURN_PADATA:
>>>>>> WELLKNOWN/ANONYMOUS(a)EXAMPLE.COM for
>>>>>> krbtgt/EXAMPLE.COM(a)EXAMPLE.COM,
>>>>>> Failed
>>>>>> to verify own certificate (depth 0): certificate has expired
>>>>>> Dec 20 16:06:54 ipa1.example.com krb5kdc[15518](info): closing
>>>>>> down
>>>>>> fd
>>>>>> 11
>>>>>> ---
>>>>>>
>>>>>> --- ipa-checkcerts.py
>>>>>> IPA version 4.5.4-10.el7.centos.3
>>>>>> Check CA status
>>>>>> Check tracking
>>>>>> Check NSS trust
>>>>>> Check dates
>>>>>> Checking certificates in CS.cfg
>>>>>> Comparing certificates to requests in LDAP
>>>>>> Checking RA certificate
>>>>>> Checking authorities
>>>>>> Checking host keytab
>>>>>> Validating certificates
>>>>>> Checking renewal master
>>>>>> End-to-end cert API test
>>>>>> Checking permissions and ownership
>>>>>> Failures:
>>>>>> Unable to find request for serial 268304391
>>>>>> Unable to find request for serial 268304394
>>>>>> Unable to find request for serial 268304393
>>>>>> Unable to find request for serial 268304392
>>>>>> Subject O=EXAMPLE.COM,CN=ipa2.example.com and template
subject
>>>>>> CN=ipa1.example.com,O=EXAMPLE.COM do not match for serial 57
>>>>>> ---
>>>>>>
>>>>>> --- ipa pkinit-status --all
>>>>>> -----------------
>>>>>> 2 servers matched
>>>>>> -----------------
>>>>>> Server name: ipa2.example.com
>>>>>> PKINIT status: enabled
>>>>>>
>>>>>> Server name: ipa1.example.com
>>>>>> PKINIT status: enabled
>>>>>> ----------------------------
>>>>>> Number of entries returned 2
>>>>>> ----------------------------
>>>>>>
>>>>>> To my understanding, proper certificate exchange between my
two
>>>>>> servers
>>>>>> ceased working at some point. How do i track this down and fix
>>>>>> it?
>>>>>>
>>>>> your issue looks similar to ticket #6792 [1]. Can you check the
>>>>> result
>>>>> of
>>>>> upgrade in /var/log/ipaupgrade.log?
>>>>> Also check the output of
>>>>> $ ipa-pkinit-manage status
>>>>> and if the files /var/lib/ipa-client/pki/kdc-ca-bundle.pem and
>>>>> /var/lib/ipa-client/pki/ca-bundle.pem exist, with -rw-r--r--
>>>>> permissions.
>>>>>
>>>>> Regarding the certificates, does getcert list show expired certs?
>>>>> flo
>>>>>
>>>>> [1] https://pagure.io/freeipa/issue/6792
>>>>
>>>> ---
>>>> $ ipa-pkinit-manage status
>>>> PKINIT is enabled
>>>> ---
>>>>
>>>> There are no expired certificates, kdc-ca-bundle.pem and
>>>> ca-bundle.pem
>>>> exist with proper permissions, but I found something in
>>>> ipaupgrade.log:
>>>>
>>>> ---
>>>> 2018-09-12T13:37:18Z DEBUG args=/usr/bin/certutil -d
>>>> /etc/httpd/alias
>>>> -L
>>>> -f /etc/httpd/alias/pwdfile.txt
>>>> 2018-09-12T13:37:19Z DEBUG Process finished, return code=0
>>>> 2018-09-12T13:37:19Z DEBUG stdout=
>>>> Certificate Nickname Trust
>>>> Attributes
>>>>
>>>> SSL,S/MIME,JAR/XPI
>>>>
>>>> Server-Cert u,u,u
>>>> EXAMPLE.COM IPA CA CT,C,C
>>>>
>>>> 2018-09-12T13:37:19Z DEBUG stderr=
>>>> 2018-09-12T13:37:19Z DEBUG Starting external process
>>>> 2018-09-12T13:37:19Z DEBUG args=/usr/bin/certutil -d
>>>> /etc/httpd/alias
>>>> -L
>>>> -n EXAMPLE.COM IPA CA -a -f /etc/httpd/alias/pwdfile.txt
>>>> 2018-09-12T13:37:19Z DEBUG Process finished, return code=0
>>>> 2018-09-12T13:37:19Z DEBUG stdout=
>>>> -----BEGIN CERTIFICATE-----
>>>> [...]
>>>> -----END CERTIFICATE-----
>>>>
>>>> 2018-09-12T13:37:19Z DEBUG stderr=
>>>> 2018-09-12T13:37:19Z DEBUG Executing upgrade plugin:
>>>> update_ra_cert_store
>>>> 2018-09-12T13:37:19Z DEBUG raw: update_ra_cert_store
>>>> 2018-09-12T13:37:19Z DEBUG raw:
ca_is_enabled(version=u'2.228')
>>>> 2018-09-12T13:37:19Z DEBUG ca_is_enabled(version=u'2.228')
>>>> 2018-09-12T13:37:19Z DEBUG Starting external process
>>>> 2018-09-12T13:37:19Z DEBUG args=/usr/bin/certutil -d
>>>> /etc/httpd/alias
>>>> -L
>>>> -n ipaCert -a -f /etc/httpd/alias/pwdfile.txt
>>>> 2018-09-12T13:37:19Z DEBUG Process finished, return code=255
>>>> 2018-09-12T13:37:19Z DEBUG stdout=
>>>> 2018-09-12T13:37:19Z DEBUG stderr=certutil: Could not find cert:
>>>> ipaCert
>>>> : PR_FILE_NOT_FOUND_ERROR: File not found
>>>> [...]
>>>> ---
>>>>
>>> this error can be ignored in most of the cases. The upgrade is
>>> trying to
>>> move ipaCert (cert+key) from the NSS database /etc/httpd/alias to the
>>> files /var/lib/ipa/ra-agent.pem and /var/lib/ipa/ra-agent.key. So
>>> if the
>>> upgrade is run a second time, he won't find ipaCert in the NSS
>>> database.
>>> To be sure, you can check if /var/lib/ipa/ra-agent.{pem|key} are
>>> present
>>> and contain a certificate with Subject CN=IPA RA,O=DOMAIN.COM. The
>>> files
>>> must be readable by root and ipaapi group, and must contain the
>>> same cert
>>> as the other masters.
>>
>> checked this, is ok.
>>
>>> What is the content of /var/lib/ipa-client/pki/kdc-ca-bundle.pem and
>>> ca-bundle.pem? both must contain IPA CA certificate.
>>
>> True on both servers.
>>
>>> What are the permissions of /var/kerberos/krb5kdc/kdc.crt? It needs
>>> to be
>>> readable by everyone. And what is the content of this cert? It
>>> should be
>>> issued by IPA CA.
>>
>> Permissions are ok, contents:
>>
>> --- ipa1
>> $ openssl x509 -in /var/kerberos/krb5kdc/kdc.crt -text -noout
>> -fingerprint
>> Certificate:
>> Data:
>> Version: 3 (0x2)
>> Serial Number: 1 (0x1)
>> Signature Algorithm: sha256WithRSAEncryption
>> Issuer: O=EXAMPLE.COM, CN=ipa1.example.com
>> Validity
>> Not Before: Nov 28 12:43:05 2017 GMT
>> Not After : Nov 28 12:43:05 2018 GMT
>> Subject: O=EXAMPLE.COM, CN=ipa1.example.com
>> Subject Public Key Info:
>> Public Key Algorithm: rsaEncryption
>> Public-Key: (2048 bit)
>> Modulus:
>> [...]
>> Exponent: 65537 (0x10001)
>> X509v3 extensions:
>> X509v3 Subject Alternative Name:
>> othername:<unsupported>, othername:<unsupported>
>> X509v3 Basic Constraints: critical
>> CA:FALSE
>> X509v3 Subject Key Identifier:
>>
>> 86:52:EC:A1:C3:FB:EC:CC:6D:F2:09:E7:64:88:D1:80:F4:71:81:AE
>> 1.3.6.1.4.1.311.20.2:
>> .".K.D.C.s._.P.K.I.N.I.T._.C.e.r.t.s
>> [...]
>> ---
>>
>> --- ipa2
>> $ openssl x509 -in /var/kerberos/krb5kdc/kdc.crt -text -noout
>> -fingerprint
>> Certificate:
>> Data:
>> Version: 3 (0x2)
>> Serial Number: 805240833 (0x2fff0001)
>> Signature Algorithm: sha256WithRSAEncryption
>> Issuer: O=EXAMPLE.COM, CN=Certificate Authority
>> Validity
>> Not Before: Jan 18 13:04:17 2018 GMT
>> Not After : Jan 19 13:04:17 2020 GMT
>> Subject: O=EXAMPLE.COM, CN=ipa2.example.com
>> Subject Public Key Info:
>> Public Key Algorithm: rsaEncryption
>> Public-Key: (2048 bit)
>> Modulus:
>> [...]
>> Exponent: 65537 (0x10001)
>> X509v3 extensions:
>> X509v3 Authority Key Identifier:
>>
>> keyid:4B:BA:AA:46:F1:29:E4:43:8B:DC:30:B4:90:3E:66:72:DD:F6:C7:FB
>>
>> Authority Information Access:
>> OCSP - URI:http://ipa-ca.example.com/ca/ocsp
>>
>> X509v3 Key Usage: critical
>> Digital Signature, Non Repudiation, Key Encipherment,
>> Data Encipherment
>> X509v3 Extended Key Usage:
>> TLS Web Server Authentication, 1.3.6.1.5.2.3.5
>> X509v3 CRL Distribution Points:
>>
>> Full Name:
>> URI:http://ipa-ca.example.com/ipa/crl/MasterCRL.bin
>> CRL Issuer:
>> DirName: O = ipaca, CN = Certificate Authority
>>
>> X509v3 Subject Key Identifier:
>>
>> 96:C3:94:70:7E:46:77:DB:91:F8:DF:D6:27:FE:73:0A:45:F3:78:F3
>> X509v3 Subject Alternative Name:
>> othername:<unsupported>, othername:<unsupported>
>> [...]
>> ---
>>
>> Additional info: I have DNS separate from IPA, but i (hopefully) made
>> proper records as IPA would have done it. In particular, i made an A
>> record "ipa-ca" that has IPs of both ipa1 and ipa2 - hope, this is
>> not the
>> root cause of my problems, since DNS is not under my control.
>>
>> Since /var/kerberos/krb5kdc/kdc.crt on ipa1 appears to be not issued by
>> IPA CA, might this be the actual problem?
>>
> The cert is expired on ipa1, which is the real root cause.
> ipa-pkinit-manage status is reporting that PKINIT is enabled but this
> does not match the actual configuration. This is a known issue [1].
>
> If the host ipa1 is running a CA instance, then you can delete
> /var/kerberos/krb5kdc/kdc.{key,crt} and run ipa-pkinit-manage enable
> to re-generate the KDC cert. Note: if the host doesn't run a CA
> instance, then this won't work because of the issue [2].
>
> To check which hosts are running a CA instance, you can use
> # ipa server-role-find --role 'CA server'
>
> [1] https://pagure.io/freeipa/issue/7200
> [2] https://pagure.io/freeipa/issue/7795
removed the expired certificates, re-enabled PKINIT, certificates were
regenerated, Web UI logins working again. Thank you very much.
>>> About the errors spotted by ipa-checkcerts.py, what are the
>>> certificates
>>> with errors reported? You can find them with ipa cert-show <serial>.
>>
>> ---
>> $ipa cert-show 57
>> Issuing CA: ipa
>> Certificate: [...]
>> Subject: CN=ipa1.example.com,O=EXAMPLE.COM
>> Issuer: CN=Certificate Authority,O=EXAMPLE.COM
>> Not Before: Tue Nov 28 12:42:07 2017 UTC
>> Not After: Mon Nov 18 12:42:07 2019 UTC
>> Serial number: 57
>> Serial number (hex): 0x39
>> Revoked: False
>>
>> $ ipa cert-show 268304391
>> Issuing CA: ipa
>> Certificate: [...]
>> Subject: CN=IPA RA,O=EXAMPLE.COM
>> Issuer: CN=Certificate Authority,O=EXAMPLE.COM
>> Not Before: Wed Jul 11 14:26:35 2018 UTC
>> Not After: Tue Jun 30 14:26:35 2020 UTC
>> Serial number: 268304391
>> Serial number (hex): 0xFFE0007
>> Revoked: False
>>
>> $ ipa cert-show 268304392
>> Issuing CA: ipa
>> Certificate: [...]
>> Subject: CN=CA Subsystem,O=EXAMPLE.COM
>> Issuer: CN=Certificate Authority,O=EXAMPLE.COM
>> Not Before: Wed Jul 11 14:26:45 2018 UTC
>> Not After: Tue Jun 30 14:26:45 2020 UTC
>> Serial number: 268304392
>> Serial number (hex): 0xFFE0008
>> Revoked: False
>>
>> $ ipa cert-show 268304393
>> Issuing CA: ipa
>> Certificate: [...]
>> Subject: CN=OCSP Subsystem,O=EXAMPLE.COM
>> Issuer: CN=Certificate Authority,O=EXAMPLE.COM
>> Not Before: Wed Jul 11 14:27:05 2018 UTC
>> Not After: Tue Jun 30 14:27:05 2020 UTC
>> Serial number: 268304393
>> Serial number (hex): 0xFFE0009
>> Revoked: False
>>
>> $ ipa cert-show 268304394
>> Issuing CA: ipa
>> Certificate: [...]
>> Subject: CN=CA Audit,O=EXAMPLE.COM
>> Issuer: CN=Certificate Authority,O=EXAMPLE.COM
>> Not Before: Wed Jul 11 14:27:25 2018 UTC
>> Not After: Tue Jun 30 14:27:25 2020 UTC
>> Serial number: 268304394
>> Serial number (hex): 0xFFE000A
>> Revoked: False
>> ---
>>
>> ipa cert-show on server ipa2 fails with "ipa: ERROR: Certificate
>> operation
>> cannot be completed: EXCEPTION (Invalid Credential.)" btw.
>>
> The errors related to "Unable to find request for serial xxx" mean
> that the cert is tracked by certmonger, but there is no corresponding
> LDAP entry cn=xxx,ou=ca,ou=requests,o=ipaca on the local LDAP server.
> Is the replication still working between your two masters?
>
> "ipa cert-show" broken on ipa2 often points to an out-of-date
> certificate in /var/lib/ipa/ra-agent.{pem|key}. The content should be
> the same as on the renewal master, and also the same as in the entry
> uid=ipara,ou=People,o=ipaca (which should be replicated and identical
> on all the masters, except if you have replication issues).
Replication appeared to be working, however, I did server maintenance
today, ipa1 and ipa2 got upgraded and rebooted. ipa1 works fine, ipa2
does not come up properly:
--- ipa2
$ ipactl status
Directory Service: RUNNING
krb5kdc Service: STOPPED
kadmin Service: STOPPED
You need to have a look at krb5 logs to understand why
kerberos failed
to start. Please check
https://www.freeipa.org/page/Troubleshooting/Kerberos for more information.
httpd Service: RUNNING
ipa-custodia Service: STOPPED
ntpd Service: RUNNING
pki-tomcatd Service: RUNNING
ipa-otpd Service: STOPPED
ipa: INFO: The ipactl command was successful
---
--- ipa1
$ openssl x509 -in /var/lib/ipa/ra-agent.pem -text -noout | grep Serial
Serial Number: 268304391 (0xffe0007)
$ ldapsearch -x -b uid=ipara,ou=People,o=ipaca
[...]
description: 2;268304391;CN=Certificate Authority,O=EXAMPLE.COM;CN=IPA
RA,O=EXAMPLE.COM
---
Looks good.
--- ipa2
$ openssl x509 -in /var/lib/ipa/ra-agent.pem -text -noout | grep Serial
Serial Number: 268304391 (0xffe0007)
$ ldapsearch -x -b uid=ipara,ou=People,o=ipaca
[...]
description: 2;44;CN=Certificate Authority,O=EXAMPLE.COM;CN=IPA
RA,O=EXAMPLE.COM
---
Looks bad.
So yes, the replication of the suffix o=ipaca seems broken. In a
deployment with CA, the ldap server contains 2 different suffixes, one
for the IdM data (your base DN as defined in /etc/ipa/default.conf) and
another one for the CA, below o=ipaca.
You can have a look at
https://www.freeipa.org/page/Troubleshooting/Directory_Server for
replication troubleshooting, but the repl issue could be a consequence
of kerberos server not starting.
--- /var/log/ipaupgrade.log
[...]
2018-12-28T10:55:53Z DEBUG The ipa-server-upgrade command failed,
exception: RemoteRetrieveError: Failed to authenticate to CA REST API
---
--- /var/log/dirsrv/slapd-EXAMPLE-COM/errors
[...]
[28/Dec/2018:11:55:55.559648469 +0100] - ERR - set_krb5_creds - Could
not get initial credentials for principal
[ldap/ipa2.example.com(a)EXAMPLE.COM] in keytab
[FILE:/etc/dirsrv/ds.keytab]: -1765328228 (Cannot contact any KDC for
requested realm)
---
--- ipa-checkcerts.py
[...]
ipa: ERROR: ra.get_certificate(): EXCEPTION (Invalid Credential.)
ra.get_certificate(): EXCEPTION (Invalid Credential.)
ipa: INFO: Checking permissions and ownership
Checking permissions and ownership
ipa: INFO: Failures:
Failures:
ipa: INFO: Unable to find request for serial 268304391
Unable to find request for serial 268304391
ipa: INFO: Unable to find request for serial 268304394
Unable to find request for serial 268304394
ipa: INFO: Unable to find request for serial 268304393
Unable to find request for serial 268304393
ipa: INFO: Unable to find request for serial 268304392
Unable to find request for serial 268304392
ipa: INFO: Unable to find request for serial 268304390
Unable to find request for serial 268304390
ipa: INFO: RA agent description does not match 2;44;CN=Certificate
Authority,O=EXAMPLE.COM;CN=IPA RA,O=EXAMPLE.COM in LDAP and
2;268304391;CN=Certificate Authority,O=EXAMPLE.COM;CN=IPA
RA,O=EXAMPLE.COM expected
ipa: INFO: RA agent certificate not found in LDAP RA agent certificate
not found in LDAP
[...]
---
The root cause appears to be the wrong IPA RA certificate in ipa2's
LDAP, right? I guess this has to fixed by manually importing the proper
certificate using ldapmodify, similar to the procedure described in [1]?
It is possible to manually modify the IPA RA cert in ipa2's ldap server
but this won't solve the replication issue. It will however allow you to
run ipa cert-show.
flo
[1]
https://floblanc.wordpress.com/2017/09/11/troubleshooting-freeipa-pki-tom...
Mit freundlichen Gruessen/With best regards,
--Daniel.
_______________________________________________
FreeIPA-users mailing list -- freeipa-users(a)lists.fedorahosted.org
To unsubscribe send an email to freeipa-users-leave(a)lists.fedorahosted.org
Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives:
https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedoraho...
8:51 a.m.
Hi Florence,
On Mon, 7 Jan 2019, Florence Blanc-Renaud wrote:
>> [...]
i shaved this thread a little, since it gets confusing. Hope i kept the
interesting bits.
>> The errors related to "Unable to find request for
serial xxx" mean
>> that the cert is tracked by certmonger, but there is no corresponding
>> LDAP entry cn=xxx,ou=ca,ou=requests,o=ipaca on the local LDAP server.
>> Is the replication still working between your two masters?
>>
>> "ipa cert-show" broken on ipa2 often points to an out-of-date
>> certificate in /var/lib/ipa/ra-agent.{pem|key}. The content should be
>> the same as on the renewal master, and also the same as in the entry
>> uid=ipara,ou=People,o=ipaca (which should be replicated and identical
>> on all the masters, except if you have replication issues).
>
> Replication appeared to be working, however, I did server maintenance
> today, ipa1 and ipa2 got upgraded and rebooted. ipa1 works fine, ipa2
> does not come up properly:
>
> --- ipa2
> $ ipactl status
> Directory Service: RUNNING
> krb5kdc Service: STOPPED
> kadmin Service: STOPPED
>
You need to have a look at krb5 logs to understand why kerberos failed
to start. Please check
https://www.freeipa.org/page/Troubleshooting/Kerberos for more
information.
Turns out that kerberos starts when started manually with systemctl:
---
$ ipactl status
Directory Service: RUNNING
krb5kdc Service: RUNNING
kadmin Service: RUNNING
httpd Service: RUNNING
ipa-custodia Service: RUNNING
ntpd Service: RUNNING
pki-tomcatd Service: RUNNING
ipa-otpd Service: STOPPED
ipa: INFO: The ipactl command was successful
---
Kerberos log:
---
[...]
: NEEDED_PREAUTH: admin(a)EXAMPLE.COM for krbtgt/EXAMPLE.COM(a)EXAMPLE.COM, Additional
pre-authentication required
Jan 08 10:39:22 ipa2.example.com krb5kdc[21691](info): closing down fd 11
Jan 08 10:39:39 ipa2.example.com krb5kdc[21692](info): AS_REQ (8 etypes {18 17 20 19 16 23
25 26}) 141.51.x.y: ISSUE: authtime 1546940379, etypes {rep=18 tkt=18 ses=18},
admin(a)EXAMPLE.COM for krbtgt/EXAMPLE.COM(a)EXAMPLE.COM
Jan 08 10:39:39 ipa2.example.com krb5kdc[21692](info): closing down fd 11
Jan 08 10:40:32 ipa2.example.com krb5kdc[21692](info): TGS_REQ (8 etypes {18 17 20 19 16
23 25 26}) 141.51.x.y: ISSUE: authtime 1546940379, etypes {rep=18 tkt=18 ses=18},
admin(a)EXAMPLE.COM for HTTP/ipa2.example.com(a)EXAMPLE.COM
Jan 08 10:40:32 ipa2.example.com krb5kdc[21692](info): closing down fd 11
---
> httpd Service: RUNNING
> ipa-custodia Service: STOPPED
> ntpd Service: RUNNING
> pki-tomcatd Service: RUNNING
> ipa-otpd Service: STOPPED
> ipa: INFO: The ipactl command was successful
> ---
>
> --- ipa1
> $ openssl x509 -in /var/lib/ipa/ra-agent.pem -text -noout | grep Serial
> Serial Number: 268304391 (0xffe0007)
>
> $ ldapsearch -x -b uid=ipara,ou=People,o=ipaca
> [...]
> description: 2;268304391;CN=Certificate Authority,O=EXAMPLE.COM;CN=IPA
> RA,O=EXAMPLE.COM
> ---
>
> Looks good.
>
> --- ipa2
> $ openssl x509 -in /var/lib/ipa/ra-agent.pem -text -noout | grep Serial
> Serial Number: 268304391 (0xffe0007)
>
> $ ldapsearch -x -b uid=ipara,ou=People,o=ipaca
> [...]
> description: 2;44;CN=Certificate Authority,O=EXAMPLE.COM;CN=IPA
> RA,O=EXAMPLE.COM
> ---
>
> Looks bad.
>
So yes, the replication of the suffix o=ipaca seems broken. In a
deployment with CA, the ldap server contains 2 different suffixes, one
for the IdM data (your base DN as defined in /etc/ipa/default.conf) and
another one for the CA, below o=ipaca.
You can have a look at
https://www.freeipa.org/page/Troubleshooting/Directory_Server for
replication troubleshooting, but the repl issue could be a consequence
of kerberos server not starting.
I checked the troubleshooting pages and carried out the steps described in
there, but my system (ipa2) appears to be ok as far as the steps go.
> --- /var/log/ipaupgrade.log
> [...]
> 2018-12-28T10:55:53Z DEBUG The ipa-server-upgrade command failed,
> exception: RemoteRetrieveError: Failed to authenticate to CA REST API
> ---
>
> --- /var/log/dirsrv/slapd-EXAMPLE-COM/errors
> [...]
> [28/Dec/2018:11:55:55.559648469 +0100] - ERR - set_krb5_creds - Could not
> get initial credentials for principal [ldap/ipa2.example.com(a)EXAMPLE.COM]
> in keytab [FILE:/etc/dirsrv/ds.keytab]: -1765328228 (Cannot contact any
> KDC for requested realm)
> ---
>
> --- ipa-checkcerts.py
> [...]
> ipa: ERROR: ra.get_certificate(): EXCEPTION (Invalid Credential.)
> ra.get_certificate(): EXCEPTION (Invalid Credential.)
> ipa: INFO: Checking permissions and ownership
> Checking permissions and ownership
> ipa: INFO: Failures:
> Failures:
> ipa: INFO: Unable to find request for serial 268304391
> Unable to find request for serial 268304391
> ipa: INFO: Unable to find request for serial 268304394
> Unable to find request for serial 268304394
> ipa: INFO: Unable to find request for serial 268304393
> Unable to find request for serial 268304393
> ipa: INFO: Unable to find request for serial 268304392
> Unable to find request for serial 268304392
> ipa: INFO: Unable to find request for serial 268304390
> Unable to find request for serial 268304390
> ipa: INFO: RA agent description does not match 2;44;CN=Certificate
> Authority,O=EXAMPLE.COM;CN=IPA RA,O=EXAMPLE.COM in LDAP and
> 2;268304391;CN=Certificate Authority,O=EXAMPLE.COM;CN=IPA RA,O=EXAMPLE.COM
> expected
> ipa: INFO: RA agent certificate not found in LDAP RA agent certificate not
> found in LDAP
> [...]
> ---
>
> The root cause appears to be the wrong IPA RA certificate in ipa2's
> LDAP, right? I guess this has to fixed by manually importing the
> proper certificate using ldapmodify, similar to the procedure
> described in [1]?
>
It is possible to manually modify the IPA RA cert in ipa2's ldap server
but this won't solve the replication issue. It will however allow you to
run ipa cert-show.
Unfortunately, it doesn't (ipa2):
---
ipa: ERROR: Certificate operation cannot be completed: EXCEPTION (Invalid Credential.)
---
Kerberos log:
---
Jan 08 15:42:45 ipa2.example.com krb5kdc[23520](info): AS_REQ (8 etypes {18 17 16 23 20 19
25 26}) 141.51.124.20: NEEDED_PREAUTH: host/ipa2.example.com(a)EXAMPLE.COM for
krbtgt/EXAMPLE.COM(a)EXAMPLE.COM, Additional pre-authentication required
---
Works ok on ipa1.
I'm unsure how to proceed here. My obvious problem is ipa2 not getting
upgraded properly:
---
[...]
2019-01-08T09:46:07Z INFO [Updating HTTPD service IPA configuration]
2019-01-08T09:46:07Z DEBUG Starting external process
2019-01-08T09:46:07Z DEBUG args=/usr/sbin/selinuxenabled
2019-01-08T09:46:07Z DEBUG Process finished, return code=1
2019-01-08T09:46:07Z DEBUG stdout=
2019-01-08T09:46:07Z DEBUG stderr=
2019-01-08T09:46:07Z DEBUG Starting external process
2019-01-08T09:46:07Z DEBUG args=/bin/systemctl --system daemon-reload
2019-01-08T09:46:07Z DEBUG Process finished, return code=0
2019-01-08T09:46:07Z DEBUG stdout=
2019-01-08T09:46:07Z DEBUG stderr=
2019-01-08T09:46:07Z INFO [Updating HTTPD service IPA WSGI configuration]
2019-01-08T09:46:07Z INFO Nothing to do for configure_httpd_wsgi_conf
2019-01-08T09:46:07Z INFO [Updating mod_nss protocol versions]
2019-01-08T09:46:07Z DEBUG Loading StateFile from
'/var/lib/ipa/sysupgrade/sysupgrade.state'
2019-01-08T09:46:07Z INFO Protocol versions already updated
2019-01-08T09:46:07Z INFO [Updating mod_nss cipher suite]
2019-01-08T09:46:07Z DEBUG Loading StateFile from
'/var/lib/ipa/sysupgrade/sysupgrade.state'
2019-01-08T09:46:07Z DEBUG Cipher suite already updated
2019-01-08T09:46:07Z INFO [Updating mod_nss enabling OCSP]
2019-01-08T09:46:07Z DEBUG Loading StateFile from
'/var/lib/ipa/sysupgrade/sysupgrade.state'
2019-01-08T09:46:07Z INFO [Fixing trust flags in /etc/httpd/alias]
2019-01-08T09:46:07Z DEBUG Loading StateFile from
'/var/lib/ipa/sysupgrade/sysupgrade.state'
2019-01-08T09:46:07Z INFO Trust flags already processed
2019-01-08T09:46:07Z INFO [Moving HTTPD service keytab to gssproxy]
2019-01-08T09:46:07Z DEBUG Starting external process
2019-01-08T09:46:07Z DEBUG args=/usr/sbin/selinuxenabled
2019-01-08T09:46:07Z DEBUG Process finished, return code=1
2019-01-08T09:46:07Z DEBUG stdout=
2019-01-08T09:46:07Z DEBUG stderr=
2019-01-08T09:46:07Z DEBUG Starting external process
2019-01-08T09:46:07Z DEBUG args=/bin/systemctl restart gssproxy.service
2019-01-08T09:46:07Z DEBUG Process finished, return code=0
2019-01-08T09:46:07Z DEBUG stdout=
2019-01-08T09:46:07Z DEBUG stderr=
2019-01-08T09:46:07Z DEBUG Starting external process
2019-01-08T09:46:07Z DEBUG args=/bin/systemctl is-active gssproxy.service
2019-01-08T09:46:07Z DEBUG Process finished, return code=0
2019-01-08T09:46:07Z DEBUG stdout=active
2019-01-08T09:46:07Z DEBUG stderr=
2019-01-08T09:46:07Z DEBUG Restart of gssproxy.service complete
2019-01-08T09:46:07Z DEBUG Starting external process
2019-01-08T09:46:07Z DEBUG args=/bin/systemctl start httpd.service
2019-01-08T09:46:09Z DEBUG Process finished, return code=0
2019-01-08T09:46:09Z DEBUG stdout=
2019-01-08T09:46:09Z DEBUG stderr=
2019-01-08T09:46:09Z DEBUG Starting external process
2019-01-08T09:46:09Z DEBUG args=/bin/systemctl is-active httpd.service
2019-01-08T09:46:09Z DEBUG Process finished, return code=0
2019-01-08T09:46:09Z DEBUG stdout=active
2019-01-08T09:46:09Z DEBUG stderr=
2019-01-08T09:46:09Z DEBUG Start of httpd.service complete
2019-01-08T09:46:09Z INFO [Removing self-signed CA]
2019-01-08T09:46:09Z DEBUG Self-signed CA is not installed
2019-01-08T09:46:09Z INFO [Removing Dogtag 9 CA]
2019-01-08T09:46:09Z DEBUG Dogtag is version 10 or above
2019-01-08T09:46:09Z DEBUG Loading StateFile from
'/var/lib/ipa/sysrestore/sysrestore.state'
2019-01-08T09:46:09Z DEBUG Loading Index file from
'/var/lib/ipa/sysrestore/sysrestore.index'
2019-01-08T09:46:09Z INFO [Checking for deprecated KDC configuration files]
2019-01-08T09:46:09Z INFO [Checking for deprecated backups of Samba configuration files]
2019-01-08T09:46:09Z DEBUG raw: ca_is_enabled(version=u'2.229')
2019-01-08T09:46:09Z DEBUG ca_is_enabled(version=u'2.229')
2019-01-08T09:46:09Z DEBUG flushing ldapi://%2fvar%2frun%2fslapd-EXAMPLE-COM.socket from
SchemaCache
2019-01-08T09:46:09Z DEBUG retrieving schema for SchemaCache
url=ldapi://%2fvar%2frun%2fslapd-EXAMPLE-COM.socket
conn=<ldap.ldapobject.SimpleLDAPObject instance at 0x7f8d6ccff5a8>
2019-01-08T09:46:09Z DEBUG raw: kra_is_enabled(version=u'2.229')
2019-01-08T09:46:09Z DEBUG kra_is_enabled(version=u'2.229')
2019-01-08T09:46:09Z DEBUG Cleaning up after pkispawn for the CA subsystem
2019-01-08T09:46:09Z DEBUG Loading StateFile from
'/var/lib/ipa/sysrestore/sysrestore.state'
2019-01-08T09:46:09Z DEBUG Loading Index file from
'/var/lib/ipa/sysrestore/sysrestore.index'
2019-01-08T09:46:09Z DEBUG Loading StateFile from
'/var/lib/ipa/sysrestore/sysrestore.state'
2019-01-08T09:46:09Z INFO [Add missing CA DNS records]
2019-01-08T09:46:09Z DEBUG Loading StateFile from
'/var/lib/ipa/sysupgrade/sysupgrade.state'
2019-01-08T09:46:09Z INFO IPA CA DNS records already processed
2019-01-08T09:46:09Z INFO [Removing deprecated DNS configuration options]
2019-01-08T09:46:09Z INFO DNS is not configured
2019-01-08T09:46:09Z INFO [Ensuring minimal number of connections]
2019-01-08T09:46:09Z INFO DNS is not configured
2019-01-08T09:46:09Z INFO [Updating GSSAPI configuration in DNS]
2019-01-08T09:46:09Z INFO DNS is not configured
2019-01-08T09:46:09Z INFO [Updating pid-file configuration in DNS]
2019-01-08T09:46:09Z INFO DNS is not configured
2019-01-08T09:46:09Z INFO DNS is not configured
2019-01-08T09:46:09Z INFO DNS is not configured
2019-01-08T09:46:09Z INFO DNS is not configured
2019-01-08T09:46:09Z INFO DNS is not configured
2019-01-08T09:46:09Z INFO DNS is not configured
2019-01-08T09:46:09Z DEBUG Loading StateFile from
'/var/lib/ipa/sysrestore/sysrestore.state'
2019-01-08T09:46:09Z DEBUG Loading Index file from
'/var/lib/ipa/sysrestore/sysrestore.index'
2019-01-08T09:46:09Z INFO DNS is not configured
2019-01-08T09:46:09Z DEBUG Loading StateFile from
'/var/lib/ipa/sysupgrade/sysupgrade.state'
2019-01-08T09:46:09Z DEBUG Loading StateFile from
'/var/lib/ipa/sysupgrade/sysupgrade.state'
2019-01-08T09:46:09Z DEBUG Loading StateFile from
'/var/lib/ipa/sysrestore/sysrestore.state'
2019-01-08T09:46:09Z DEBUG Loading Index file from
'/var/lib/ipa/sysrestore/sysrestore.index'
2019-01-08T09:46:09Z INFO DNS is not configured
2019-01-08T09:46:09Z DEBUG Loading StateFile from
'/var/lib/ipa/sysrestore/sysrestore.state'
2019-01-08T09:46:09Z DEBUG Loading Index file from
'/var/lib/ipa/sysrestore/sysrestore.index'
2019-01-08T09:46:09Z DEBUG Loading StateFile from
'/var/lib/ipa/sysupgrade/sysupgrade.state'
2019-01-08T09:46:09Z INFO [Upgrading CA schema]
2019-01-08T09:46:10Z DEBUG Processing schema LDIF file
/usr/share/pki/server/conf/schema-certProfile.ldif
2019-01-08T09:46:10Z DEBUG Processing schema LDIF file
/usr/share/pki/server/conf/schema-authority.ldif
2019-01-08T09:46:10Z DEBUG Not updating schema
2019-01-08T09:46:10Z INFO CA schema update complete (no changes)
2019-01-08T09:46:10Z INFO [Verifying that CA audit signing cert has 2 year validity]
2019-01-08T09:46:10Z DEBUG caSignedLogCert.cfg profile validity range is 720
2019-01-08T09:46:10Z INFO [Update certmonger certificate renewal configuration]
2019-01-08T09:46:10Z DEBUG Loading Index file from
'/var/lib/ipa/sysrestore/sysrestore.index'
2019-01-08T09:46:10Z DEBUG Starting external process
2019-01-08T09:46:10Z DEBUG args=/usr/bin/certutil -d dbm:/etc/httpd/alias -L -n
Server-Cert -a -f /etc/httpd/alias/pwdfile.txt
2019-01-08T09:46:10Z DEBUG Process finished, return code=0
2019-01-08T09:46:10Z DEBUG stdout=-----BEGIN CERTIFICATE-----
MIIFYjCCBEqgAwIBAgIED/4ABTANBgkqhkiG9w0BAQsFADBHMSUwIwYDVQQKExxE
[...]
Serial Number: 268304389 (0xffe0005)
[...]
n7aRLXFxWB8va5AxibMHYUJb08nWJRqHHwufF/YQnJdskFuDyAM=
-----END CERTIFICATE-----
2019-01-08T09:46:10Z DEBUG stderr=
2019-01-08T09:46:10Z DEBUG Loading Index file from
'/var/lib/ipa/sysrestore/sysrestore.index'
2019-01-08T09:46:10Z DEBUG Starting external process
2019-01-08T09:46:10Z DEBUG args=/usr/bin/certutil -d dbm:/etc/dirsrv/slapd-EXAMPLE-COM/ -L
-n Server-Cert -a -f /etc/dirsrv/slapd-EXAMPLE-COM/pwdfile.txt
2019-01-08T09:46:10Z DEBUG Process finished, return code=0
2019-01-08T09:46:10Z DEBUG stdout=-----BEGIN CERTIFICATE-----
MIIFYjCCBEqgAwIBAgIED/4ABDANBgkqhkiG9w0BAQsFADBHMSUwIwYDVQQKExxE
[...]
Serial Number: 268304388 (0xffe0004)
[...]
7D6pk95Rgq6g73cZmogJBfJicVF0odkf2VkvV/aJCxTtZ4xkVds=
-----END CERTIFICATE-----
2019-01-08T09:46:10Z DEBUG stderr=
2019-01-08T09:46:10Z DEBUG Loading Index file from
'/var/lib/ipa/sysrestore/sysrestore.index'
2019-01-08T09:46:10Z DEBUG Starting external process
2019-01-08T09:46:10Z DEBUG args=/usr/bin/certutil -d dbm:/etc/pki/pki-tomcat/alias -L -f
/etc/pki/pki-tomcat/alias/pwdfile.txt
2019-01-08T09:46:10Z DEBUG Process finished, return code=0
2019-01-08T09:46:10Z DEBUG stdout=
Certificate Nickname Trust Attributes
SSL,S/MIME,JAR/XPI
ocspSigningCert cert-pki-ca u,u,u
caSigningCert cert-pki-ca CTu,Cu,Cu
Server-Cert cert-pki-ca u,u,u
auditSigningCert cert-pki-ca u,u,Pu
subsystemCert cert-pki-ca u,u,u
2019-01-08T09:46:10Z DEBUG stderr=
2019-01-08T09:46:11Z INFO Certmonger certificate renewal configuration already up-to-date
2019-01-08T09:46:11Z INFO [Enable PKIX certificate path discovery and validation]
2019-01-08T09:46:11Z DEBUG Loading StateFile from
'/var/lib/ipa/sysupgrade/sysupgrade.state'
2019-01-08T09:46:11Z INFO PKIX already enabled
2019-01-08T09:46:11Z INFO [Authorizing RA Agent to modify profiles]
2019-01-08T09:46:11Z INFO [Authorizing RA Agent to manage lightweight CAs]
2019-01-08T09:46:11Z INFO [Ensuring Lightweight CAs container exists in Dogtag database]
2019-01-08T09:46:11Z DEBUG Created connection context.ldap2_140245392902928
2019-01-08T09:46:11Z DEBUG flushing ldapi://%2fvar%2frun%2fslapd-EXAMPLE-COM.socket from
SchemaCache
2019-01-08T09:46:11Z DEBUG retrieving schema for SchemaCache
url=ldapi://%2fvar%2frun%2fslapd-EXAMPLE-COM.socket
conn=<ldap.ldapobject.SimpleLDAPObject instance at 0x7f8d6cea4fc8>
2019-01-08T09:46:12Z DEBUG Destroyed connection context.ldap2_140245392902928
2019-01-08T09:46:12Z INFO [Adding default OCSP URI configuration]
2019-01-08T09:46:12Z INFO [Ensuring CA is using LDAPProfileSubsystem]
2019-01-08T09:46:12Z INFO [Migrating certificate profiles to LDAP]
2019-01-08T09:46:12Z DEBUG Created connection context.ldap2_140245366754832
2019-01-08T09:46:12Z DEBUG flushing ldapi://%2fvar%2frun%2fslapd-EXAMPLE-COM.socket from
SchemaCache
2019-01-08T09:46:12Z DEBUG retrieving schema for SchemaCache
url=ldapi://%2fvar%2frun%2fslapd-EXAMPLE-COM.socket
conn=<ldap.ldapobject.SimpleLDAPObject instance at 0x7f8d6b5db128>
2019-01-08T09:46:12Z DEBUG Destroyed connection context.ldap2_140245366754832
2019-01-08T09:46:12Z DEBUG request GET
https://ipa2.example.com:8443/ca/rest/account/login
2019-01-08T09:46:12Z DEBUG request body ''
2019-01-08T09:46:12Z DEBUG response status 401
2019-01-08T09:46:12Z DEBUG response headers Server: Apache-Coyote/1.1
Cache-Control: private
Expires: Thu, 01 Jan 1970 01:00:00 CET
WWW-Authenticate: Basic realm="Certificate Authority"
Content-Type: text/html;charset=utf-8
Content-Language: en
Content-Length: 951
Date: Tue, 08 Jan 2019 09:46:12 GMT
2019-01-08T09:46:12Z DEBUG response body '<html> [...] HTTP Status 401 [...]
message [...] This request requires HTTP authentication. [...] </html>'
2019-01-08T09:46:12Z ERROR IPA server upgrade failed: Inspect /var/log/ipaupgrade.log and
run command ipa-server-upgrade manually.
2019-01-08T09:46:12Z DEBUG File
"/usr/lib/python2.7/site-packages/ipapython/admintool.py", line 178, in execute
return_value = self.run()
File
"/usr/lib/python2.7/site-packages/ipaserver/install/ipa_server_upgrade.py", line
54, in run
server.upgrade()
File "/usr/lib/python2.7/site-packages/ipaserver/install/server/upgrade.py",
line 2085, in upgrade
upgrade_configuration()
File "/usr/lib/python2.7/site-packages/ipaserver/install/server/upgrade.py",
line 1952, in upgrade_configuration
ca_enable_ldap_profile_subsystem(ca)
File "/usr/lib/python2.7/site-packages/ipaserver/install/server/upgrade.py",
line 396, in ca_enable_ldap_profile_subsystem
cainstance.migrate_profiles_to_ldap()
File "/usr/lib/python2.7/site-packages/ipaserver/install/cainstance.py", line
1814, in migrate_profiles_to_ldap
_create_dogtag_profile(profile_id, profile_data, overwrite=False)
File "/usr/lib/python2.7/site-packages/ipaserver/install/cainstance.py", line
1820, in _create_dogtag_profile
with api.Backend.ra_certprofile as profile_api:
File "/usr/lib/python2.7/site-packages/ipaserver/plugins/dogtag.py", line
1302, in __enter__
raise errors.RemoteRetrieveError(reason=_('Failed to authenticate to CA REST
API'))
2019-01-08T09:46:12Z DEBUG The ipa-server-upgrade command failed, exception:
RemoteRetrieveError: Failed to authenticate to CA REST API
2019-01-08T09:46:12Z ERROR Unexpected error - see /var/log/ipaupgrade.log for details:
RemoteRetrieveError: Failed to authenticate to CA REST API
2019-01-08T09:46:12Z ERROR The ipa-server-upgrade command failed. See
/var/log/ipaupgrade.log for more information
---
Mit freundlichen Gruessen/With best regards,
--Daniel.
9:42 a.m.
On 1/8/19 3:51 PM, dbischof--- via FreeIPA-users wrote:
Hi Florence,
On Mon, 7 Jan 2019, Florence Blanc-Renaud wrote:
>>> [...]
i shaved this thread a little, since it gets confusing. Hope i kept the
interesting bits.
>>> The errors related to "Unable to find request for serial xxx"
mean
>>> that the cert is tracked by certmonger, but there is no corresponding
>>> LDAP entry cn=xxx,ou=ca,ou=requests,o=ipaca on the local LDAP server.
>>> Is the replication still working between your two masters?
>>>
>>> "ipa cert-show" broken on ipa2 often points to an out-of-date
>>> certificate in /var/lib/ipa/ra-agent.{pem|key}. The content should be
>>> the same as on the renewal master, and also the same as in the entry
>>> uid=ipara,ou=People,o=ipaca (which should be replicated and identical
>>> on all the masters, except if you have replication issues).
>>
>> Replication appeared to be working, however, I did server maintenance
>> today, ipa1 and ipa2 got upgraded and rebooted. ipa1 works fine, ipa2
>> does not come up properly:
>>
>> --- ipa2
>> $ ipactl status
>> Directory Service: RUNNING
>> krb5kdc Service: STOPPED
>> kadmin Service: STOPPED
>>
> You need to have a look at krb5 logs to understand why kerberos failed
> to start. Please check
> https://www.freeipa.org/page/Troubleshooting/Kerberos for more
> information.
Turns out that kerberos starts when started manually with systemctl:
---
$ ipactl status
Directory Service: RUNNING
krb5kdc Service: RUNNING
kadmin Service: RUNNING
httpd Service: RUNNING
ipa-custodia Service: RUNNING
ntpd Service: RUNNING
pki-tomcatd Service: RUNNING
ipa-otpd Service: STOPPED
ipa: INFO: The ipactl command was successful
---
Kerberos log:
---
[...]
: NEEDED_PREAUTH: admin(a)EXAMPLE.COM for krbtgt/EXAMPLE.COM(a)EXAMPLE.COM,
Additional pre-authentication required
Jan 08 10:39:22 ipa2.example.com krb5kdc[21691](info): closing down fd 11
Jan 08 10:39:39 ipa2.example.com krb5kdc[21692](info): AS_REQ (8 etypes
{18 17 20 19 16 23 25 26}) 141.51.x.y: ISSUE: authtime 1546940379,
etypes {rep=18 tkt=18 ses=18}, admin(a)EXAMPLE.COM for
krbtgt/EXAMPLE.COM(a)EXAMPLE.COM
Jan 08 10:39:39 ipa2.example.com krb5kdc[21692](info): closing down fd 11
Jan 08 10:40:32 ipa2.example.com krb5kdc[21692](info): TGS_REQ (8 etypes
{18 17 20 19 16 23 25 26}) 141.51.x.y: ISSUE: authtime 1546940379,
etypes {rep=18 tkt=18 ses=18}, admin(a)EXAMPLE.COM for
HTTP/ipa2.example.com(a)EXAMPLE.COM
Jan 08 10:40:32 ipa2.example.com krb5kdc[21692](info): closing down fd 11
---
>> httpd Service: RUNNING
>> ipa-custodia Service: STOPPED
>> ntpd Service: RUNNING
>> pki-tomcatd Service: RUNNING
>> ipa-otpd Service: STOPPED
>> ipa: INFO: The ipactl command was successful
>> ---
>>
>> --- ipa1
>> $ openssl x509 -in /var/lib/ipa/ra-agent.pem -text -noout | grep Serial
>> Serial Number: 268304391 (0xffe0007)
>>
>> $ ldapsearch -x -b uid=ipara,ou=People,o=ipaca
>> [...]
>> description: 2;268304391;CN=Certificate Authority,O=EXAMPLE.COM;CN=IPA
>> RA,O=EXAMPLE.COM
>> ---
>>
>> Looks good.
>>
>> --- ipa2
>> $ openssl x509 -in /var/lib/ipa/ra-agent.pem -text -noout | grep Serial
>> Serial Number: 268304391 (0xffe0007)
>>
>> $ ldapsearch -x -b uid=ipara,ou=People,o=ipaca
>> [...]
>> description: 2;44;CN=Certificate Authority,O=EXAMPLE.COM;CN=IPA
>> RA,O=EXAMPLE.COM
>> ---
>>
>> Looks bad.
>>
> So yes, the replication of the suffix o=ipaca seems broken. In a
> deployment with CA, the ldap server contains 2 different suffixes, one
> for the IdM data (your base DN as defined in /etc/ipa/default.conf)
> and another one for the CA, below o=ipaca.
>
> You can have a look at
> https://www.freeipa.org/page/Troubleshooting/Directory_Server for
> replication troubleshooting, but the repl issue could be a consequence
> of kerberos server not starting.
I checked the troubleshooting pages and carried out the steps described
in there, but my system (ipa2) appears to be ok as far as the steps go.
>> --- /var/log/ipaupgrade.log
>> [...]
>> 2018-12-28T10:55:53Z DEBUG The ipa-server-upgrade command failed,
>> exception: RemoteRetrieveError: Failed to authenticate to CA REST API
>> ---
>>
>> --- /var/log/dirsrv/slapd-EXAMPLE-COM/errors
>> [...]
>> [28/Dec/2018:11:55:55.559648469 +0100] - ERR - set_krb5_creds -
>> Could not
>> get initial credentials for principal
>> [ldap/ipa2.example.com(a)EXAMPLE.COM]
>> in keytab [FILE:/etc/dirsrv/ds.keytab]: -1765328228 (Cannot contact any
>> KDC for requested realm)
>> ---
>>
>> --- ipa-checkcerts.py
>> [...]
>> ipa: ERROR: ra.get_certificate(): EXCEPTION (Invalid Credential.)
>> ra.get_certificate(): EXCEPTION (Invalid Credential.)
>> ipa: INFO: Checking permissions and ownership
>> Checking permissions and ownership
>> ipa: INFO: Failures:
>> Failures:
>> ipa: INFO: Unable to find request for serial 268304391
>> Unable to find request for serial 268304391
>> ipa: INFO: Unable to find request for serial 268304394
>> Unable to find request for serial 268304394
>> ipa: INFO: Unable to find request for serial 268304393
>> Unable to find request for serial 268304393
>> ipa: INFO: Unable to find request for serial 268304392
>> Unable to find request for serial 268304392
>> ipa: INFO: Unable to find request for serial 268304390
>> Unable to find request for serial 268304390
>> ipa: INFO: RA agent description does not match 2;44;CN=Certificate
>> Authority,O=EXAMPLE.COM;CN=IPA RA,O=EXAMPLE.COM in LDAP and
>> 2;268304391;CN=Certificate Authority,O=EXAMPLE.COM;CN=IPA
>> RA,O=EXAMPLE.COM
>> expected
>> ipa: INFO: RA agent certificate not found in LDAP RA agent
>> certificate not
>> found in LDAP
>> [...]
>> ---
>>
>> The root cause appears to be the wrong IPA RA certificate in ipa2's
>> LDAP, right? I guess this has to fixed by manually importing the
>> proper certificate using ldapmodify, similar to the procedure
>> described in [1]?
>>
> It is possible to manually modify the IPA RA cert in ipa2's ldap
> server but this won't solve the replication issue. It will however
> allow you to run ipa cert-show.
Unfortunately, it doesn't (ipa2):
---
ipa: ERROR: Certificate operation cannot be completed: EXCEPTION
(Invalid Credential.)
---
Kerberos log:
---
Jan 08 15:42:45 ipa2.example.com krb5kdc[23520](info): AS_REQ (8 etypes
{18 17 16 23 20 19 25 26}) 141.51.124.20: NEEDED_PREAUTH:
host/ipa2.example.com(a)EXAMPLE.COM for krbtgt/EXAMPLE.COM(a)EXAMPLE.COM,
Additional pre-authentication required
---
Works ok on ipa1.
I'm unsure how to proceed here. My obvious problem is ipa2 not getting
upgraded properly:
---
[...]
2019-01-08T09:46:07Z INFO [Updating HTTPD service IPA configuration]
2019-01-08T09:46:07Z DEBUG Starting external process
2019-01-08T09:46:07Z DEBUG args=/usr/sbin/selinuxenabled
2019-01-08T09:46:07Z DEBUG Process finished, return code=1
2019-01-08T09:46:07Z DEBUG stdout=
2019-01-08T09:46:07Z DEBUG stderr=
2019-01-08T09:46:07Z DEBUG Starting external process
2019-01-08T09:46:07Z DEBUG args=/bin/systemctl --system daemon-reload
2019-01-08T09:46:07Z DEBUG Process finished, return code=0
2019-01-08T09:46:07Z DEBUG stdout=
2019-01-08T09:46:07Z DEBUG stderr=
2019-01-08T09:46:07Z INFO [Updating HTTPD service IPA WSGI configuration]
2019-01-08T09:46:07Z INFO Nothing to do for configure_httpd_wsgi_conf
2019-01-08T09:46:07Z INFO [Updating mod_nss protocol versions]
2019-01-08T09:46:07Z DEBUG Loading StateFile from
'/var/lib/ipa/sysupgrade/sysupgrade.state'
2019-01-08T09:46:07Z INFO Protocol versions already updated
2019-01-08T09:46:07Z INFO [Updating mod_nss cipher suite]
2019-01-08T09:46:07Z DEBUG Loading StateFile from
'/var/lib/ipa/sysupgrade/sysupgrade.state'
2019-01-08T09:46:07Z DEBUG Cipher suite already updated
2019-01-08T09:46:07Z INFO [Updating mod_nss enabling OCSP]
2019-01-08T09:46:07Z DEBUG Loading StateFile from
'/var/lib/ipa/sysupgrade/sysupgrade.state'
2019-01-08T09:46:07Z INFO [Fixing trust flags in /etc/httpd/alias]
2019-01-08T09:46:07Z DEBUG Loading StateFile from
'/var/lib/ipa/sysupgrade/sysupgrade.state'
2019-01-08T09:46:07Z INFO Trust flags already processed
2019-01-08T09:46:07Z INFO [Moving HTTPD service keytab to gssproxy]
2019-01-08T09:46:07Z DEBUG Starting external process
2019-01-08T09:46:07Z DEBUG args=/usr/sbin/selinuxenabled
2019-01-08T09:46:07Z DEBUG Process finished, return code=1
2019-01-08T09:46:07Z DEBUG stdout=
2019-01-08T09:46:07Z DEBUG stderr=
2019-01-08T09:46:07Z DEBUG Starting external process
2019-01-08T09:46:07Z DEBUG args=/bin/systemctl restart gssproxy.service
2019-01-08T09:46:07Z DEBUG Process finished, return code=0
2019-01-08T09:46:07Z DEBUG stdout=
2019-01-08T09:46:07Z DEBUG stderr=
2019-01-08T09:46:07Z DEBUG Starting external process
2019-01-08T09:46:07Z DEBUG args=/bin/systemctl is-active gssproxy.service
2019-01-08T09:46:07Z DEBUG Process finished, return code=0
2019-01-08T09:46:07Z DEBUG stdout=active
2019-01-08T09:46:07Z DEBUG stderr=
2019-01-08T09:46:07Z DEBUG Restart of gssproxy.service complete
2019-01-08T09:46:07Z DEBUG Starting external process
2019-01-08T09:46:07Z DEBUG args=/bin/systemctl start httpd.service
2019-01-08T09:46:09Z DEBUG Process finished, return code=0
2019-01-08T09:46:09Z DEBUG stdout=
2019-01-08T09:46:09Z DEBUG stderr=
2019-01-08T09:46:09Z DEBUG Starting external process
2019-01-08T09:46:09Z DEBUG args=/bin/systemctl is-active httpd.service
2019-01-08T09:46:09Z DEBUG Process finished, return code=0
2019-01-08T09:46:09Z DEBUG stdout=active
2019-01-08T09:46:09Z DEBUG stderr=
2019-01-08T09:46:09Z DEBUG Start of httpd.service complete
2019-01-08T09:46:09Z INFO [Removing self-signed CA]
2019-01-08T09:46:09Z DEBUG Self-signed CA is not installed
2019-01-08T09:46:09Z INFO [Removing Dogtag 9 CA]
2019-01-08T09:46:09Z DEBUG Dogtag is version 10 or above
2019-01-08T09:46:09Z DEBUG Loading StateFile from
'/var/lib/ipa/sysrestore/sysrestore.state'
2019-01-08T09:46:09Z DEBUG Loading Index file from
'/var/lib/ipa/sysrestore/sysrestore.index'
2019-01-08T09:46:09Z INFO [Checking for deprecated KDC configuration files]
2019-01-08T09:46:09Z INFO [Checking for deprecated backups of Samba
configuration files]
2019-01-08T09:46:09Z DEBUG raw: ca_is_enabled(version=u'2.229')
2019-01-08T09:46:09Z DEBUG ca_is_enabled(version=u'2.229')
2019-01-08T09:46:09Z DEBUG flushing
ldapi://%2fvar%2frun%2fslapd-EXAMPLE-COM.socket from SchemaCache
2019-01-08T09:46:09Z DEBUG retrieving schema for SchemaCache
url=ldapi://%2fvar%2frun%2fslapd-EXAMPLE-COM.socket
conn=<ldap.ldapobject.SimpleLDAPObject instance at 0x7f8d6ccff5a8>
2019-01-08T09:46:09Z DEBUG raw: kra_is_enabled(version=u'2.229')
2019-01-08T09:46:09Z DEBUG kra_is_enabled(version=u'2.229')
2019-01-08T09:46:09Z DEBUG Cleaning up after pkispawn for the CA subsystem
2019-01-08T09:46:09Z DEBUG Loading StateFile from
'/var/lib/ipa/sysrestore/sysrestore.state'
2019-01-08T09:46:09Z DEBUG Loading Index file from
'/var/lib/ipa/sysrestore/sysrestore.index'
2019-01-08T09:46:09Z DEBUG Loading StateFile from
'/var/lib/ipa/sysrestore/sysrestore.state'
2019-01-08T09:46:09Z INFO [Add missing CA DNS records]
2019-01-08T09:46:09Z DEBUG Loading StateFile from
'/var/lib/ipa/sysupgrade/sysupgrade.state'
2019-01-08T09:46:09Z INFO IPA CA DNS records already processed
2019-01-08T09:46:09Z INFO [Removing deprecated DNS configuration options]
2019-01-08T09:46:09Z INFO DNS is not configured
2019-01-08T09:46:09Z INFO [Ensuring minimal number of connections]
2019-01-08T09:46:09Z INFO DNS is not configured
2019-01-08T09:46:09Z INFO [Updating GSSAPI configuration in DNS]
2019-01-08T09:46:09Z INFO DNS is not configured
2019-01-08T09:46:09Z INFO [Updating pid-file configuration in DNS]
2019-01-08T09:46:09Z INFO DNS is not configured
2019-01-08T09:46:09Z INFO DNS is not configured
2019-01-08T09:46:09Z INFO DNS is not configured
2019-01-08T09:46:09Z INFO DNS is not configured
2019-01-08T09:46:09Z INFO DNS is not configured
2019-01-08T09:46:09Z INFO DNS is not configured
2019-01-08T09:46:09Z DEBUG Loading StateFile from
'/var/lib/ipa/sysrestore/sysrestore.state'
2019-01-08T09:46:09Z DEBUG Loading Index file from
'/var/lib/ipa/sysrestore/sysrestore.index'
2019-01-08T09:46:09Z INFO DNS is not configured
2019-01-08T09:46:09Z DEBUG Loading StateFile from
'/var/lib/ipa/sysupgrade/sysupgrade.state'
2019-01-08T09:46:09Z DEBUG Loading StateFile from
'/var/lib/ipa/sysupgrade/sysupgrade.state'
2019-01-08T09:46:09Z DEBUG Loading StateFile from
'/var/lib/ipa/sysrestore/sysrestore.state'
2019-01-08T09:46:09Z DEBUG Loading Index file from
'/var/lib/ipa/sysrestore/sysrestore.index'
2019-01-08T09:46:09Z INFO DNS is not configured
2019-01-08T09:46:09Z DEBUG Loading StateFile from
'/var/lib/ipa/sysrestore/sysrestore.state'
2019-01-08T09:46:09Z DEBUG Loading Index file from
'/var/lib/ipa/sysrestore/sysrestore.index'
2019-01-08T09:46:09Z DEBUG Loading StateFile from
'/var/lib/ipa/sysupgrade/sysupgrade.state'
2019-01-08T09:46:09Z INFO [Upgrading CA schema]
2019-01-08T09:46:10Z DEBUG Processing schema LDIF file
/usr/share/pki/server/conf/schema-certProfile.ldif
2019-01-08T09:46:10Z DEBUG Processing schema LDIF file
/usr/share/pki/server/conf/schema-authority.ldif
2019-01-08T09:46:10Z DEBUG Not updating schema
2019-01-08T09:46:10Z INFO CA schema update complete (no changes)
2019-01-08T09:46:10Z INFO [Verifying that CA audit signing cert has 2
year validity]
2019-01-08T09:46:10Z DEBUG caSignedLogCert.cfg profile validity range is
720
2019-01-08T09:46:10Z INFO [Update certmonger certificate renewal
configuration]
2019-01-08T09:46:10Z DEBUG Loading Index file from
'/var/lib/ipa/sysrestore/sysrestore.index'
2019-01-08T09:46:10Z DEBUG Starting external process
2019-01-08T09:46:10Z DEBUG args=/usr/bin/certutil -d
dbm:/etc/httpd/alias -L -n Server-Cert -a -f /etc/httpd/alias/pwdfile.txt
2019-01-08T09:46:10Z DEBUG Process finished, return code=0
2019-01-08T09:46:10Z DEBUG stdout=-----BEGIN CERTIFICATE-----
MIIFYjCCBEqgAwIBAgIED/4ABTANBgkqhkiG9w0BAQsFADBHMSUwIwYDVQQKExxE
[...]
Serial Number: 268304389 (0xffe0005)
[...]
n7aRLXFxWB8va5AxibMHYUJb08nWJRqHHwufF/YQnJdskFuDyAM=
-----END CERTIFICATE-----
2019-01-08T09:46:10Z DEBUG stderr=
2019-01-08T09:46:10Z DEBUG Loading Index file from
'/var/lib/ipa/sysrestore/sysrestore.index'
2019-01-08T09:46:10Z DEBUG Starting external process
2019-01-08T09:46:10Z DEBUG args=/usr/bin/certutil -d
dbm:/etc/dirsrv/slapd-EXAMPLE-COM/ -L -n Server-Cert -a -f
/etc/dirsrv/slapd-EXAMPLE-COM/pwdfile.txt
2019-01-08T09:46:10Z DEBUG Process finished, return code=0
2019-01-08T09:46:10Z DEBUG stdout=-----BEGIN CERTIFICATE-----
MIIFYjCCBEqgAwIBAgIED/4ABDANBgkqhkiG9w0BAQsFADBHMSUwIwYDVQQKExxE
[...]
Serial Number: 268304388 (0xffe0004)
[...]
7D6pk95Rgq6g73cZmogJBfJicVF0odkf2VkvV/aJCxTtZ4xkVds=
-----END CERTIFICATE-----
2019-01-08T09:46:10Z DEBUG stderr=
2019-01-08T09:46:10Z DEBUG Loading Index file from
'/var/lib/ipa/sysrestore/sysrestore.index'
2019-01-08T09:46:10Z DEBUG Starting external process
2019-01-08T09:46:10Z DEBUG args=/usr/bin/certutil -d
dbm:/etc/pki/pki-tomcat/alias -L -f /etc/pki/pki-tomcat/alias/pwdfile.txt
2019-01-08T09:46:10Z DEBUG Process finished, return code=0
2019-01-08T09:46:10Z DEBUG stdout=
Certificate Nickname Trust
Attributes
SSL,S/MIME,JAR/XPI
ocspSigningCert cert-pki-ca u,u,u
caSigningCert cert-pki-ca CTu,Cu,Cu
Server-Cert cert-pki-ca u,u,u
auditSigningCert cert-pki-ca u,u,Pu
subsystemCert cert-pki-ca u,u,u
2019-01-08T09:46:10Z DEBUG stderr=
2019-01-08T09:46:11Z INFO Certmonger certificate renewal configuration
already up-to-date
2019-01-08T09:46:11Z INFO [Enable PKIX certificate path discovery and
validation]
2019-01-08T09:46:11Z DEBUG Loading StateFile from
'/var/lib/ipa/sysupgrade/sysupgrade.state'
2019-01-08T09:46:11Z INFO PKIX already enabled
2019-01-08T09:46:11Z INFO [Authorizing RA Agent to modify profiles]
2019-01-08T09:46:11Z INFO [Authorizing RA Agent to manage lightweight CAs]
2019-01-08T09:46:11Z INFO [Ensuring Lightweight CAs container exists in
Dogtag database]
2019-01-08T09:46:11Z DEBUG Created connection context.ldap2_140245392902928
2019-01-08T09:46:11Z DEBUG flushing
ldapi://%2fvar%2frun%2fslapd-EXAMPLE-COM.socket from SchemaCache
2019-01-08T09:46:11Z DEBUG retrieving schema for SchemaCache
url=ldapi://%2fvar%2frun%2fslapd-EXAMPLE-COM.socket
conn=<ldap.ldapobject.SimpleLDAPObject instance at 0x7f8d6cea4fc8>
2019-01-08T09:46:12Z DEBUG Destroyed connection
context.ldap2_140245392902928
2019-01-08T09:46:12Z INFO [Adding default OCSP URI configuration]
2019-01-08T09:46:12Z INFO [Ensuring CA is using LDAPProfileSubsystem]
2019-01-08T09:46:12Z INFO [Migrating certificate profiles to LDAP]
2019-01-08T09:46:12Z DEBUG Created connection context.ldap2_140245366754832
2019-01-08T09:46:12Z DEBUG flushing
ldapi://%2fvar%2frun%2fslapd-EXAMPLE-COM.socket from SchemaCache
2019-01-08T09:46:12Z DEBUG retrieving schema for SchemaCache
url=ldapi://%2fvar%2frun%2fslapd-EXAMPLE-COM.socket
conn=<ldap.ldapobject.SimpleLDAPObject instance at 0x7f8d6b5db128>
2019-01-08T09:46:12Z DEBUG Destroyed connection
context.ldap2_140245366754832
2019-01-08T09:46:12Z DEBUG request GET
https://ipa2.example.com:8443/ca/rest/account/login
2019-01-08T09:46:12Z DEBUG request body ''
2019-01-08T09:46:12Z DEBUG response status 401
2019-01-08T09:46:12Z DEBUG response headers Server: Apache-Coyote/1.1
Cache-Control: private
Expires: Thu, 01 Jan 1970 01:00:00 CET
WWW-Authenticate: Basic realm="Certificate Authority"
Content-Type: text/html;charset=utf-8
Content-Language: en
Content-Length: 951
Date: Tue, 08 Jan 2019 09:46:12 GMT
2019-01-08T09:46:12Z DEBUG response body '<html> [...] HTTP Status 401
[...] message [...] This request requires HTTP authentication. [...]
</html>'
2019-01-08T09:46:12Z ERROR IPA server upgrade failed: Inspect
/var/log/ipaupgrade.log and run command ipa-server-upgrade manually.
2019-01-08T09:46:12Z DEBUG File
"/usr/lib/python2.7/site-packages/ipapython/admintool.py", line 178, in
execute
return_value = self.run()
File
"/usr/lib/python2.7/site-packages/ipaserver/install/ipa_server_upgrade.py",
line 54, in run
server.upgrade()
File
"/usr/lib/python2.7/site-packages/ipaserver/install/server/upgrade.py",
line 2085, in upgrade
upgrade_configuration()
File
"/usr/lib/python2.7/site-packages/ipaserver/install/server/upgrade.py",
line 1952, in upgrade_configuration
ca_enable_ldap_profile_subsystem(ca)
File
"/usr/lib/python2.7/site-packages/ipaserver/install/server/upgrade.py",
line 396, in ca_enable_ldap_profile_subsystem
cainstance.migrate_profiles_to_ldap()
File
"/usr/lib/python2.7/site-packages/ipaserver/install/cainstance.py", line
1814, in migrate_profiles_to_ldap
_create_dogtag_profile(profile_id, profile_data, overwrite=False)
File
"/usr/lib/python2.7/site-packages/ipaserver/install/cainstance.py", line
1820, in _create_dogtag_profile
with api.Backend.ra_certprofile as profile_api:
File "/usr/lib/python2.7/site-packages/ipaserver/plugins/dogtag.py",
line 1302, in __enter__
raise errors.RemoteRetrieveError(reason=_('Failed to authenticate
to CA REST API'))
2019-01-08T09:46:12Z DEBUG The ipa-server-upgrade command failed,
exception: RemoteRetrieveError: Failed to authenticate to CA REST API
2019-01-08T09:46:12Z ERROR Unexpected error - see
/var/log/ipaupgrade.log for details: RemoteRetrieveError: Failed to
authenticate to CA REST API
2019-01-08T09:46:12Z ERROR The ipa-server-upgrade command failed. See
/var/log/ipaupgrade.log for more information
---
Hi,
on ipa2, what is the content of /var/lib/ipa/ra-agent.pem? Is it the
same as on ipa1? Can you check the file permissions?
Can you also compare with the ldap entry uid=ipara,ou=People,o=ipaca
(description and usercertificate fields are important).
flo
Mit freundlichen Gruessen/With best regards,
--Daniel.
_______________________________________________
FreeIPA-users mailing list -- freeipa-users(a)lists.fedorahosted.org
To unsubscribe send an email to freeipa-users-leave(a)lists.fedorahosted.org
Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives:
https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedoraho...
6:46 a.m.
Hi Florence,
On Tue, 8 Jan 2019, Florence Blanc-Renaud wrote:
On 1/8/19 3:51 PM, dbischof--- via FreeIPA-users wrote:
> Hi Florence,
>
> On Mon, 7 Jan 2019, Florence Blanc-Renaud wrote:
>
>>>> [...]
>
> i shaved this thread a little, since it gets confusing. Hope i kept the
> interesting bits.
>
>>>> The errors related to "Unable to find request for serial xxx"
mean
>>>> that the cert is tracked by certmonger, but there is no corresponding
>>>> LDAP entry cn=xxx,ou=ca,ou=requests,o=ipaca on the local LDAP server.
>>>> Is the replication still working between your two masters?
>>>>
>>>> "ipa cert-show" broken on ipa2 often points to an
out-of-date
>>>> certificate in /var/lib/ipa/ra-agent.{pem|key}. The content should be
>>>> the same as on the renewal master, and also the same as in the entry
>>>> uid=ipara,ou=People,o=ipaca (which should be replicated and identical
>>>> on all the masters, except if you have replication issues).
>>>
>>> Replication appeared to be working, however, I did server maintenance
>>> today, ipa1 and ipa2 got upgraded and rebooted. ipa1 works fine, ipa2
>>> does not come up properly:
>>>
>>> --- ipa2
>>> $ ipactl status
>>> Directory Service: RUNNING
>>> krb5kdc Service: STOPPED
>>> kadmin Service: STOPPED
>>>
>> You need to have a look at krb5 logs to understand why kerberos failed to
>> start. Please check https://www.freeipa.org/page/Troubleshooting/Kerberos
>> for more information.
>
> Turns out that kerberos starts when started manually with systemctl:
>
> ---
> $ ipactl status
> Directory Service: RUNNING
> krb5kdc Service: RUNNING
> kadmin Service: RUNNING
> httpd Service: RUNNING
> ipa-custodia Service: RUNNING
> ntpd Service: RUNNING
> pki-tomcatd Service: RUNNING
> ipa-otpd Service: STOPPED
> ipa: INFO: The ipactl command was successful
> ---
>
> Kerberos log:
>
> ---
> [...]
> : NEEDED_PREAUTH: admin(a)EXAMPLE.COM for krbtgt/EXAMPLE.COM(a)EXAMPLE.COM,
> Additional pre-authentication required
> Jan 08 10:39:22 ipa2.example.com krb5kdc[21691](info): closing down fd 11
> Jan 08 10:39:39 ipa2.example.com krb5kdc[21692](info): AS_REQ (8 etypes
> {18 17 20 19 16 23 25 26}) 141.51.x.y: ISSUE: authtime 1546940379, etypes
> {rep=18 tkt=18 ses=18}, admin(a)EXAMPLE.COM for
> krbtgt/EXAMPLE.COM(a)EXAMPLE.COM
> Jan 08 10:39:39 ipa2.example.com krb5kdc[21692](info): closing down fd 11
> Jan 08 10:40:32 ipa2.example.com krb5kdc[21692](info): TGS_REQ (8 etypes
> {18 17 20 19 16 23 25 26}) 141.51.x.y: ISSUE: authtime 1546940379, etypes
> {rep=18 tkt=18 ses=18}, admin(a)EXAMPLE.COM for
> HTTP/ipa2.example.com(a)EXAMPLE.COM
> Jan 08 10:40:32 ipa2.example.com krb5kdc[21692](info): closing down fd 11
> ---
>
>>> httpd Service: RUNNING
>>> ipa-custodia Service: STOPPED
>>> ntpd Service: RUNNING
>>> pki-tomcatd Service: RUNNING
>>> ipa-otpd Service: STOPPED
>>> ipa: INFO: The ipactl command was successful
>>> ---
>>>
>>> --- ipa1
>>> $ openssl x509 -in /var/lib/ipa/ra-agent.pem -text -noout | grep Serial
>>> Serial Number: 268304391 (0xffe0007)
>>>
>>> $ ldapsearch -x -b uid=ipara,ou=People,o=ipaca
>>> [...]
>>> description: 2;268304391;CN=Certificate Authority,O=EXAMPLE.COM;CN=IPA
>>> RA,O=EXAMPLE.COM
>>> ---
>>>
>>> Looks good.
>>>
>>> --- ipa2
>>> $ openssl x509 -in /var/lib/ipa/ra-agent.pem -text -noout | grep Serial
>>> Serial Number: 268304391 (0xffe0007)
>>>
>>> $ ldapsearch -x -b uid=ipara,ou=People,o=ipaca
>>> [...]
>>> description: 2;44;CN=Certificate Authority,O=EXAMPLE.COM;CN=IPA
>>> RA,O=EXAMPLE.COM
>>> ---
>>>
>>> Looks bad.
>>>
>> So yes, the replication of the suffix o=ipaca seems broken. In a
>> deployment with CA, the ldap server contains 2 different suffixes, one
>> for the IdM data (your base DN as defined in /etc/ipa/default.conf) and
>> another one for the CA, below o=ipaca.
>>
>> You can have a look at
>> https://www.freeipa.org/page/Troubleshooting/Directory_Server for
>> replication troubleshooting, but the repl issue could be a consequence of
>> kerberos server not starting.
>
> I checked the troubleshooting pages and carried out the steps described in
> there, but my system (ipa2) appears to be ok as far as the steps go.
>
>>> --- /var/log/ipaupgrade.log
>>> [...]
>>> 2018-12-28T10:55:53Z DEBUG The ipa-server-upgrade command failed,
>>> exception: RemoteRetrieveError: Failed to authenticate to CA REST API
>>> ---
>>>
>>> --- /var/log/dirsrv/slapd-EXAMPLE-COM/errors
>>> [...]
>>> [28/Dec/2018:11:55:55.559648469 +0100] - ERR - set_krb5_creds - Could
>>> not
>>> get initial credentials for principal
>>> [ldap/ipa2.example.com(a)EXAMPLE.COM]
>>> in keytab [FILE:/etc/dirsrv/ds.keytab]: -1765328228 (Cannot contact any
>>> KDC for requested realm)
>>> ---
>>>
>>> --- ipa-checkcerts.py
>>> [...]
>>> ipa: ERROR: ra.get_certificate(): EXCEPTION (Invalid Credential.)
>>> ra.get_certificate(): EXCEPTION (Invalid Credential.)
>>> ipa: INFO: Checking permissions and ownership
>>> Checking permissions and ownership
>>> ipa: INFO: Failures:
>>> Failures:
>>> ipa: INFO: Unable to find request for serial 268304391
>>> Unable to find request for serial 268304391
>>> ipa: INFO: Unable to find request for serial 268304394
>>> Unable to find request for serial 268304394
>>> ipa: INFO: Unable to find request for serial 268304393
>>> Unable to find request for serial 268304393
>>> ipa: INFO: Unable to find request for serial 268304392
>>> Unable to find request for serial 268304392
>>> ipa: INFO: Unable to find request for serial 268304390
>>> Unable to find request for serial 268304390
>>> ipa: INFO: RA agent description does not match 2;44;CN=Certificate
>>> Authority,O=EXAMPLE.COM;CN=IPA RA,O=EXAMPLE.COM in LDAP and
>>> 2;268304391;CN=Certificate Authority,O=EXAMPLE.COM;CN=IPA
>>> RA,O=EXAMPLE.COM
>>> expected
>>> ipa: INFO: RA agent certificate not found in LDAP RA agent certificate
>>> not
>>> found in LDAP
>>> [...]
>>> ---
>>>
>>> The root cause appears to be the wrong IPA RA certificate in ipa2's
>>> LDAP, right? I guess this has to fixed by manually importing the
>>> proper certificate using ldapmodify, similar to the procedure
>>> described in [1]?
>>>
>> It is possible to manually modify the IPA RA cert in ipa2's ldap server
>> but this won't solve the replication issue. It will however allow you to
>> run ipa cert-show.
>
> Unfortunately, it doesn't (ipa2):
>
> ---
> ipa: ERROR: Certificate operation cannot be completed: EXCEPTION (Invalid
> Credential.)
> ---
>
> Kerberos log:
>
> ---
> Jan 08 15:42:45 ipa2.example.com krb5kdc[23520](info): AS_REQ (8 etypes
> {18 17 16 23 20 19 25 26}) 141.51.124.20: NEEDED_PREAUTH:
> host/ipa2.example.com(a)EXAMPLE.COM for krbtgt/EXAMPLE.COM(a)EXAMPLE.COM,
> Additional pre-authentication required
> ---
>
> Works ok on ipa1.
>
> I'm unsure how to proceed here. My obvious problem is ipa2 not getting
> upgraded properly:
>
> ---
> [...]
> 2019-01-08T09:46:07Z INFO [Updating HTTPD service IPA configuration]
> 2019-01-08T09:46:07Z DEBUG Starting external process
> 2019-01-08T09:46:07Z DEBUG args=/usr/sbin/selinuxenabled
> 2019-01-08T09:46:07Z DEBUG Process finished, return code=1
> 2019-01-08T09:46:07Z DEBUG stdout=
> 2019-01-08T09:46:07Z DEBUG stderr=
> 2019-01-08T09:46:07Z DEBUG Starting external process
> 2019-01-08T09:46:07Z DEBUG args=/bin/systemctl --system daemon-reload
> 2019-01-08T09:46:07Z DEBUG Process finished, return code=0
> 2019-01-08T09:46:07Z DEBUG stdout=
> 2019-01-08T09:46:07Z DEBUG stderr=
> 2019-01-08T09:46:07Z INFO [Updating HTTPD service IPA WSGI configuration]
> 2019-01-08T09:46:07Z INFO Nothing to do for configure_httpd_wsgi_conf
> 2019-01-08T09:46:07Z INFO [Updating mod_nss protocol versions]
> 2019-01-08T09:46:07Z DEBUG Loading StateFile from
> '/var/lib/ipa/sysupgrade/sysupgrade.state'
> 2019-01-08T09:46:07Z INFO Protocol versions already updated
> 2019-01-08T09:46:07Z INFO [Updating mod_nss cipher suite]
> 2019-01-08T09:46:07Z DEBUG Loading StateFile from
> '/var/lib/ipa/sysupgrade/sysupgrade.state'
> 2019-01-08T09:46:07Z DEBUG Cipher suite already updated
> 2019-01-08T09:46:07Z INFO [Updating mod_nss enabling OCSP]
> 2019-01-08T09:46:07Z DEBUG Loading StateFile from
> '/var/lib/ipa/sysupgrade/sysupgrade.state'
> 2019-01-08T09:46:07Z INFO [Fixing trust flags in /etc/httpd/alias]
> 2019-01-08T09:46:07Z DEBUG Loading StateFile from
> '/var/lib/ipa/sysupgrade/sysupgrade.state'
> 2019-01-08T09:46:07Z INFO Trust flags already processed
> 2019-01-08T09:46:07Z INFO [Moving HTTPD service keytab to gssproxy]
> 2019-01-08T09:46:07Z DEBUG Starting external process
> 2019-01-08T09:46:07Z DEBUG args=/usr/sbin/selinuxenabled
> 2019-01-08T09:46:07Z DEBUG Process finished, return code=1
> 2019-01-08T09:46:07Z DEBUG stdout=
> 2019-01-08T09:46:07Z DEBUG stderr=
> 2019-01-08T09:46:07Z DEBUG Starting external process
> 2019-01-08T09:46:07Z DEBUG args=/bin/systemctl restart gssproxy.service
> 2019-01-08T09:46:07Z DEBUG Process finished, return code=0
> 2019-01-08T09:46:07Z DEBUG stdout=
> 2019-01-08T09:46:07Z DEBUG stderr=
> 2019-01-08T09:46:07Z DEBUG Starting external process
> 2019-01-08T09:46:07Z DEBUG args=/bin/systemctl is-active gssproxy.service
> 2019-01-08T09:46:07Z DEBUG Process finished, return code=0
> 2019-01-08T09:46:07Z DEBUG stdout=active
>
> 2019-01-08T09:46:07Z DEBUG stderr=
> 2019-01-08T09:46:07Z DEBUG Restart of gssproxy.service complete
> 2019-01-08T09:46:07Z DEBUG Starting external process
> 2019-01-08T09:46:07Z DEBUG args=/bin/systemctl start httpd.service
> 2019-01-08T09:46:09Z DEBUG Process finished, return code=0
> 2019-01-08T09:46:09Z DEBUG stdout=
> 2019-01-08T09:46:09Z DEBUG stderr=
> 2019-01-08T09:46:09Z DEBUG Starting external process
> 2019-01-08T09:46:09Z DEBUG args=/bin/systemctl is-active httpd.service
> 2019-01-08T09:46:09Z DEBUG Process finished, return code=0
> 2019-01-08T09:46:09Z DEBUG stdout=active
>
> 2019-01-08T09:46:09Z DEBUG stderr=
> 2019-01-08T09:46:09Z DEBUG Start of httpd.service complete
> 2019-01-08T09:46:09Z INFO [Removing self-signed CA]
> 2019-01-08T09:46:09Z DEBUG Self-signed CA is not installed
> 2019-01-08T09:46:09Z INFO [Removing Dogtag 9 CA]
> 2019-01-08T09:46:09Z DEBUG Dogtag is version 10 or above
> 2019-01-08T09:46:09Z DEBUG Loading StateFile from
> '/var/lib/ipa/sysrestore/sysrestore.state'
> 2019-01-08T09:46:09Z DEBUG Loading Index file from
> '/var/lib/ipa/sysrestore/sysrestore.index'
> 2019-01-08T09:46:09Z INFO [Checking for deprecated KDC configuration
> files]
> 2019-01-08T09:46:09Z INFO [Checking for deprecated backups of Samba
> configuration files]
> 2019-01-08T09:46:09Z DEBUG raw: ca_is_enabled(version=u'2.229')
> 2019-01-08T09:46:09Z DEBUG ca_is_enabled(version=u'2.229')
> 2019-01-08T09:46:09Z DEBUG flushing
> ldapi://%2fvar%2frun%2fslapd-EXAMPLE-COM.socket from SchemaCache
> 2019-01-08T09:46:09Z DEBUG retrieving schema for SchemaCache
> url=ldapi://%2fvar%2frun%2fslapd-EXAMPLE-COM.socket
> conn=<ldap.ldapobject.SimpleLDAPObject instance at 0x7f8d6ccff5a8>
> 2019-01-08T09:46:09Z DEBUG raw: kra_is_enabled(version=u'2.229')
> 2019-01-08T09:46:09Z DEBUG kra_is_enabled(version=u'2.229')
> 2019-01-08T09:46:09Z DEBUG Cleaning up after pkispawn for the CA subsystem
> 2019-01-08T09:46:09Z DEBUG Loading StateFile from
> '/var/lib/ipa/sysrestore/sysrestore.state'
> 2019-01-08T09:46:09Z DEBUG Loading Index file from
> '/var/lib/ipa/sysrestore/sysrestore.index'
> 2019-01-08T09:46:09Z DEBUG Loading StateFile from
> '/var/lib/ipa/sysrestore/sysrestore.state'
> 2019-01-08T09:46:09Z INFO [Add missing CA DNS records]
> 2019-01-08T09:46:09Z DEBUG Loading StateFile from
> '/var/lib/ipa/sysupgrade/sysupgrade.state'
> 2019-01-08T09:46:09Z INFO IPA CA DNS records already processed
> 2019-01-08T09:46:09Z INFO [Removing deprecated DNS configuration options]
> 2019-01-08T09:46:09Z INFO DNS is not configured
> 2019-01-08T09:46:09Z INFO [Ensuring minimal number of connections]
> 2019-01-08T09:46:09Z INFO DNS is not configured
> 2019-01-08T09:46:09Z INFO [Updating GSSAPI configuration in DNS]
> 2019-01-08T09:46:09Z INFO DNS is not configured
> 2019-01-08T09:46:09Z INFO [Updating pid-file configuration in DNS]
> 2019-01-08T09:46:09Z INFO DNS is not configured
> 2019-01-08T09:46:09Z INFO DNS is not configured
> 2019-01-08T09:46:09Z INFO DNS is not configured
> 2019-01-08T09:46:09Z INFO DNS is not configured
> 2019-01-08T09:46:09Z INFO DNS is not configured
> 2019-01-08T09:46:09Z INFO DNS is not configured
> 2019-01-08T09:46:09Z DEBUG Loading StateFile from
> '/var/lib/ipa/sysrestore/sysrestore.state'
> 2019-01-08T09:46:09Z DEBUG Loading Index file from
> '/var/lib/ipa/sysrestore/sysrestore.index'
> 2019-01-08T09:46:09Z INFO DNS is not configured
> 2019-01-08T09:46:09Z DEBUG Loading StateFile from
> '/var/lib/ipa/sysupgrade/sysupgrade.state'
> 2019-01-08T09:46:09Z DEBUG Loading StateFile from
> '/var/lib/ipa/sysupgrade/sysupgrade.state'
> 2019-01-08T09:46:09Z DEBUG Loading StateFile from
> '/var/lib/ipa/sysrestore/sysrestore.state'
> 2019-01-08T09:46:09Z DEBUG Loading Index file from
> '/var/lib/ipa/sysrestore/sysrestore.index'
> 2019-01-08T09:46:09Z INFO DNS is not configured
> 2019-01-08T09:46:09Z DEBUG Loading StateFile from
> '/var/lib/ipa/sysrestore/sysrestore.state'
> 2019-01-08T09:46:09Z DEBUG Loading Index file from
> '/var/lib/ipa/sysrestore/sysrestore.index'
> 2019-01-08T09:46:09Z DEBUG Loading StateFile from
> '/var/lib/ipa/sysupgrade/sysupgrade.state'
> 2019-01-08T09:46:09Z INFO [Upgrading CA schema]
> 2019-01-08T09:46:10Z DEBUG Processing schema LDIF file
> /usr/share/pki/server/conf/schema-certProfile.ldif
> 2019-01-08T09:46:10Z DEBUG Processing schema LDIF file
> /usr/share/pki/server/conf/schema-authority.ldif
> 2019-01-08T09:46:10Z DEBUG Not updating schema
> 2019-01-08T09:46:10Z INFO CA schema update complete (no changes)
> 2019-01-08T09:46:10Z INFO [Verifying that CA audit signing cert has 2 year
> validity]
> 2019-01-08T09:46:10Z DEBUG caSignedLogCert.cfg profile validity range is
> 720
> 2019-01-08T09:46:10Z INFO [Update certmonger certificate renewal
> configuration]
> 2019-01-08T09:46:10Z DEBUG Loading Index file from
> '/var/lib/ipa/sysrestore/sysrestore.index'
> 2019-01-08T09:46:10Z DEBUG Starting external process
> 2019-01-08T09:46:10Z DEBUG args=/usr/bin/certutil -d dbm:/etc/httpd/alias
> -L -n Server-Cert -a -f /etc/httpd/alias/pwdfile.txt
> 2019-01-08T09:46:10Z DEBUG Process finished, return code=0
> 2019-01-08T09:46:10Z DEBUG stdout=-----BEGIN CERTIFICATE-----
> MIIFYjCCBEqgAwIBAgIED/4ABTANBgkqhkiG9w0BAQsFADBHMSUwIwYDVQQKExxE
> [...]
> Serial Number: 268304389 (0xffe0005)
> [...]
> n7aRLXFxWB8va5AxibMHYUJb08nWJRqHHwufF/YQnJdskFuDyAM=
> -----END CERTIFICATE-----
>
> 2019-01-08T09:46:10Z DEBUG stderr=
> 2019-01-08T09:46:10Z DEBUG Loading Index file from
> '/var/lib/ipa/sysrestore/sysrestore.index'
> 2019-01-08T09:46:10Z DEBUG Starting external process
> 2019-01-08T09:46:10Z DEBUG args=/usr/bin/certutil -d
> dbm:/etc/dirsrv/slapd-EXAMPLE-COM/ -L -n Server-Cert -a -f
> /etc/dirsrv/slapd-EXAMPLE-COM/pwdfile.txt
> 2019-01-08T09:46:10Z DEBUG Process finished, return code=0
> 2019-01-08T09:46:10Z DEBUG stdout=-----BEGIN CERTIFICATE-----
> MIIFYjCCBEqgAwIBAgIED/4ABDANBgkqhkiG9w0BAQsFADBHMSUwIwYDVQQKExxE
> [...]
> Serial Number: 268304388 (0xffe0004)
> [...]
> 7D6pk95Rgq6g73cZmogJBfJicVF0odkf2VkvV/aJCxTtZ4xkVds=
> -----END CERTIFICATE-----
>
> 2019-01-08T09:46:10Z DEBUG stderr=
> 2019-01-08T09:46:10Z DEBUG Loading Index file from
> '/var/lib/ipa/sysrestore/sysrestore.index'
> 2019-01-08T09:46:10Z DEBUG Starting external process
> 2019-01-08T09:46:10Z DEBUG args=/usr/bin/certutil -d
> dbm:/etc/pki/pki-tomcat/alias -L -f /etc/pki/pki-tomcat/alias/pwdfile.txt
> 2019-01-08T09:46:10Z DEBUG Process finished, return code=0
> 2019-01-08T09:46:10Z DEBUG stdout=
> Certificate Nickname Trust
> Attributes
> SSL,S/MIME,JAR/XPI
>
> ocspSigningCert cert-pki-ca u,u,u
> caSigningCert cert-pki-ca CTu,Cu,Cu
> Server-Cert cert-pki-ca u,u,u
> auditSigningCert cert-pki-ca u,u,Pu
> subsystemCert cert-pki-ca u,u,u
>
> 2019-01-08T09:46:10Z DEBUG stderr=
> 2019-01-08T09:46:11Z INFO Certmonger certificate renewal configuration
> already up-to-date
> 2019-01-08T09:46:11Z INFO [Enable PKIX certificate path discovery and
> validation]
> 2019-01-08T09:46:11Z DEBUG Loading StateFile from
> '/var/lib/ipa/sysupgrade/sysupgrade.state'
> 2019-01-08T09:46:11Z INFO PKIX already enabled
> 2019-01-08T09:46:11Z INFO [Authorizing RA Agent to modify profiles]
> 2019-01-08T09:46:11Z INFO [Authorizing RA Agent to manage lightweight CAs]
> 2019-01-08T09:46:11Z INFO [Ensuring Lightweight CAs container exists in
> Dogtag database]
> 2019-01-08T09:46:11Z DEBUG Created connection
> context.ldap2_140245392902928
> 2019-01-08T09:46:11Z DEBUG flushing
> ldapi://%2fvar%2frun%2fslapd-EXAMPLE-COM.socket from SchemaCache
> 2019-01-08T09:46:11Z DEBUG retrieving schema for SchemaCache
> url=ldapi://%2fvar%2frun%2fslapd-EXAMPLE-COM.socket
> conn=<ldap.ldapobject.SimpleLDAPObject instance at 0x7f8d6cea4fc8>
> 2019-01-08T09:46:12Z DEBUG Destroyed connection
> context.ldap2_140245392902928
> 2019-01-08T09:46:12Z INFO [Adding default OCSP URI configuration]
> 2019-01-08T09:46:12Z INFO [Ensuring CA is using LDAPProfileSubsystem]
> 2019-01-08T09:46:12Z INFO [Migrating certificate profiles to LDAP]
> 2019-01-08T09:46:12Z DEBUG Created connection
> context.ldap2_140245366754832
> 2019-01-08T09:46:12Z DEBUG flushing
> ldapi://%2fvar%2frun%2fslapd-EXAMPLE-COM.socket from SchemaCache
> 2019-01-08T09:46:12Z DEBUG retrieving schema for SchemaCache
> url=ldapi://%2fvar%2frun%2fslapd-EXAMPLE-COM.socket
> conn=<ldap.ldapobject.SimpleLDAPObject instance at 0x7f8d6b5db128>
> 2019-01-08T09:46:12Z DEBUG Destroyed connection
> context.ldap2_140245366754832
> 2019-01-08T09:46:12Z DEBUG request GET
> https://ipa2.example.com:8443/ca/rest/account/login
> 2019-01-08T09:46:12Z DEBUG request body ''
> 2019-01-08T09:46:12Z DEBUG response status 401
> 2019-01-08T09:46:12Z DEBUG response headers Server: Apache-Coyote/1.1
> Cache-Control: private
> Expires: Thu, 01 Jan 1970 01:00:00 CET
> WWW-Authenticate: Basic realm="Certificate Authority"
> Content-Type: text/html;charset=utf-8
> Content-Language: en
> Content-Length: 951
> Date: Tue, 08 Jan 2019 09:46:12 GMT
>
> 2019-01-08T09:46:12Z DEBUG response body '<html> [...] HTTP Status 401
> [...] message [...] This request requires HTTP authentication. [...]
> </html>'
> 2019-01-08T09:46:12Z ERROR IPA server upgrade failed: Inspect
> /var/log/ipaupgrade.log and run command ipa-server-upgrade manually.
> 2019-01-08T09:46:12Z DEBUG File
> "/usr/lib/python2.7/site-packages/ipapython/admintool.py", line 178, in
> execute
> return_value = self.run()
> File
>
"/usr/lib/python2.7/site-packages/ipaserver/install/ipa_server_upgrade.py",
> line 54, in run
> server.upgrade()
> File
> "/usr/lib/python2.7/site-packages/ipaserver/install/server/upgrade.py",
> line 2085, in upgrade
> upgrade_configuration()
> File
> "/usr/lib/python2.7/site-packages/ipaserver/install/server/upgrade.py",
> line 1952, in upgrade_configuration
> ca_enable_ldap_profile_subsystem(ca)
> File
> "/usr/lib/python2.7/site-packages/ipaserver/install/server/upgrade.py",
> line 396, in ca_enable_ldap_profile_subsystem
> cainstance.migrate_profiles_to_ldap()
> File
> "/usr/lib/python2.7/site-packages/ipaserver/install/cainstance.py", line
> 1814, in migrate_profiles_to_ldap
> _create_dogtag_profile(profile_id, profile_data, overwrite=False)
> File
> "/usr/lib/python2.7/site-packages/ipaserver/install/cainstance.py", line
> 1820, in _create_dogtag_profile
> with api.Backend.ra_certprofile as profile_api:
> File "/usr/lib/python2.7/site-packages/ipaserver/plugins/dogtag.py",
> line 1302, in __enter__
> raise errors.RemoteRetrieveError(reason=_('Failed to authenticate to
> CA REST API'))
>
> 2019-01-08T09:46:12Z DEBUG The ipa-server-upgrade command failed,
> exception: RemoteRetrieveError: Failed to authenticate to CA REST API
> 2019-01-08T09:46:12Z ERROR Unexpected error - see /var/log/ipaupgrade.log
> for details: RemoteRetrieveError: Failed to authenticate to CA REST API
> 2019-01-08T09:46:12Z ERROR The ipa-server-upgrade command failed. See
> /var/log/ipaupgrade.log for more information
> ---
>
on ipa2, what is the content of /var/lib/ipa/ra-agent.pem? Is it the
same as on ipa1? Can you check the file permissions?
Can you also compare with the ldap entry uid=ipara,ou=People,o=ipaca
(description and usercertificate fields are important).
looked deeper this time, as you requested:
--- ipa1
$ ls -la /var/lib/ipa/ra-agent.*
-r--r----- 1 root ipaapi 1854 Jul 11 2018 /var/lib/ipa/ra-agent.key
-r--r----- 1 root ipaapi 1318 Jul 11 2018 /var/lib/ipa/ra-agent.pem
--
$ openssl x509 -in /var/lib/ipa/ra-agent.pem -text -noout | grep Serial
Serial Number: 268304391 (0xffe0007)
--
$ ldapsearch -x -b uid=ipara,ou=People,o=ipaca
# extended LDIF
#
# LDAPv3
# base <uid=ipara,ou=People,o=ipaca> with scope subtree
# filter: (objectclass=*)
# requesting: ALL
#
# ipara, people, ipaca
dn: uid=ipara,ou=people,o=ipaca
userCertificate:: MIIDozCCAougAwIBAgIBBzANBgkqhkiG9w0BAQsFADBHMSUwIwYDVQQKExxE
[...]
DoNVNCVX+P2rYhhKizm5W/+Q77YYMxs4=
userCertificate:: MIIDozCCAougAwIBAgIBLDANBgkqhkiG9w0BAQsFADBHMSUwIwYDVQQKExxE
[...]
DFVH/Om2gwOobX7Rt3L582BOjz97gRSQ=
userCertificate:: MIIDoTCCAomgAwIBAgIED/4ABzANBgkqhkiG9w0BAQsFADBHMSUwIwYDVQQK
[...]
zon51BrP2rvAW9tpXAgPE1XqQuRzn
userstate: 1
usertype: agentType
cn: ipara
sn: ipara
uid: ipara
objectClass: top
objectClass: person
objectClass: organizationalPerson
objectClass: inetOrgPerson
objectClass: cmsuser
description: 2;268304391;CN=Certificate Authority,O=EXAMPLE.COM;CN=IPA RA,O=EXAMPLE.COM
# search result
search: 2
result: 0 Success
# numResponses: 2
# numEntries: 1
--
$ head -n 5 ra-agent.key
Bag Attributes
friendlyName: ipaCert
localKeyID: 5E 90 B6 ED CA 3E 5B FF 2E 76 F6 B8 73 FD AC 1A A7 84 D7 11
Key Attributes: <No Attributes>
-----BEGIN PRIVATE KEY-----
--- ipa2
$ ls -la /var/lib/ipa/ra-agent.*
-r--r----- 1 root ipaapi 1828 Jul 11 2018 /var/lib/ipa/ra-agent.key
-r--r----- 1 root ipaapi 1318 Jul 11 2018 /var/lib/ipa/ra-agent.pem
--
$ openssl x509 -in /var/lib/ipa/ra-agent.pem -text -noout | grep Serial
Serial Number: 268304391 (0xffe0007)
--
$ ldapsearch -x -b uid=ipara,ou=People,o=ipaca
# extended LDIF
#
# LDAPv3
# base <uid=ipara,ou=People,o=ipaca> with scope subtree
# filter: (objectclass=*)
# requesting: ALL
#
# ipara, people, ipaca
dn: uid=ipara,ou=people,o=ipaca
description: 2;44;CN=Certificate Authority,O=EXAMPLE.COM;CN=IPA RA,O=EXAMPLE.COM
objectClass: top
objectClass: person
objectClass: organizationalPerson
objectClass: inetOrgPerson
objectClass: cmsuser
uid: ipara
sn: ipara
cn: ipara
usertype: agentType
userstate: 1
userCertificate:: MIIDozCCAougAwIBAgIBBzANBgkqhkiG9w0BAQsFADBHMSUwIwYDVQQKExxE
[...]
DoNVNCVX+P2rYhhKizm5W/+Q77YYMxs4=
userCertificate:: MIIDozCCAougAwIBAgIBLDANBgkqhkiG9w0BAQsFADBHMSUwIwYDVQQKExxE
[...]
DFVH/Om2gwOobX7Rt3L582BOjz97gRSQ=
# search result
search: 2
result: 0 Success
# numResponses: 2
# numEntries: 1
--
$ head -n 5 ra-agent.key
Bag Attributes
localKeyID: 5E 90 B6 ED CA 3E 5B FF 2E 76 F6 B8 73 FD AC 1A A7 84 D7 11
Key Attributes: <No Attributes>
-----BEGIN PRIVATE KEY-----
[...]
---
Two differences:
1. The key file on ipa1 contains a "Bag Attribute" "friendlyName",
that
the key key file on ipa2 misses - don't know if this is relevant, but
probably not.
2. The LDAP entry on ipa1 contains 3 "userCertificate"s, but only the
first 2 of them are in ipa2's LDAP. This is probably the interesting part.
Am i right to assume, that adding the missing userCertificate to ipa2's
LDAP will solve my problem? If yes, how would I achieve that?
Mit freundlichen Gruessen/With best regards,
--Daniel.
9:33 a.m.
On 1/10/19 1:46 PM, dbischof--- via FreeIPA-users wrote:
Hi Florence,
On Tue, 8 Jan 2019, Florence Blanc-Renaud wrote:
> On 1/8/19 3:51 PM, dbischof--- via FreeIPA-users wrote:
>> Hi Florence,
>>
>> On Mon, 7 Jan 2019, Florence Blanc-Renaud wrote:
>>
>>>>> [...]
>>
>> i shaved this thread a little, since it gets confusing. Hope i kept the
>> interesting bits.
>>
>>>>> The errors related to "Unable to find request for serial
xxx" mean
>>>>> that the cert is tracked by certmonger, but there is no
>>>>> corresponding
>>>>> LDAP entry cn=xxx,ou=ca,ou=requests,o=ipaca on the local LDAP
>>>>> server.
>>>>> Is the replication still working between your two masters?
>>>>>
>>>>> "ipa cert-show" broken on ipa2 often points to an
out-of-date
>>>>> certificate in /var/lib/ipa/ra-agent.{pem|key}. The content
>>>>> should be
>>>>> the same as on the renewal master, and also the same as in the
>>>>> entry
>>>>> uid=ipara,ou=People,o=ipaca (which should be replicated and
>>>>> identical
>>>>> on all the masters, except if you have replication issues).
>>>>
>>>> Replication appeared to be working, however, I did server
>>>> maintenance
>>>> today, ipa1 and ipa2 got upgraded and rebooted. ipa1 works fine,
>>>> ipa2
>>>> does not come up properly:
>>>>
>>>> --- ipa2
>>>> $ ipactl status
>>>> Directory Service: RUNNING
>>>> krb5kdc Service: STOPPED
>>>> kadmin Service: STOPPED
>>>>
>>> You need to have a look at krb5 logs to understand why kerberos
>>> failed to
>>> start. Please check
>>> https://www.freeipa.org/page/Troubleshooting/Kerberos
>>> for more information.
>>
>> Turns out that kerberos starts when started manually with systemctl:
>>
>> ---
>> $ ipactl status
>> Directory Service: RUNNING
>> krb5kdc Service: RUNNING
>> kadmin Service: RUNNING
>> httpd Service: RUNNING
>> ipa-custodia Service: RUNNING
>> ntpd Service: RUNNING
>> pki-tomcatd Service: RUNNING
>> ipa-otpd Service: STOPPED
>> ipa: INFO: The ipactl command was successful
>> ---
>>
>> Kerberos log:
>>
>> ---
>> [...]
>> : NEEDED_PREAUTH: admin(a)EXAMPLE.COM for krbtgt/EXAMPLE.COM(a)EXAMPLE.COM,
>> Additional pre-authentication required
>> Jan 08 10:39:22 ipa2.example.com krb5kdc[21691](info): closing down
>> fd 11
>> Jan 08 10:39:39 ipa2.example.com krb5kdc[21692](info): AS_REQ (8 etypes
>> {18 17 20 19 16 23 25 26}) 141.51.x.y: ISSUE: authtime 1546940379,
>> etypes
>> {rep=18 tkt=18 ses=18}, admin(a)EXAMPLE.COM for
>> krbtgt/EXAMPLE.COM(a)EXAMPLE.COM
>> Jan 08 10:39:39 ipa2.example.com krb5kdc[21692](info): closing down
>> fd 11
>> Jan 08 10:40:32 ipa2.example.com krb5kdc[21692](info): TGS_REQ (8
>> etypes
>> {18 17 20 19 16 23 25 26}) 141.51.x.y: ISSUE: authtime 1546940379,
>> etypes
>> {rep=18 tkt=18 ses=18}, admin(a)EXAMPLE.COM for
>> HTTP/ipa2.example.com(a)EXAMPLE.COM
>> Jan 08 10:40:32 ipa2.example.com krb5kdc[21692](info): closing down
>> fd 11
>> ---
>>
>>>> httpd Service: RUNNING
>>>> ipa-custodia Service: STOPPED
>>>> ntpd Service: RUNNING
>>>> pki-tomcatd Service: RUNNING
>>>> ipa-otpd Service: STOPPED
>>>> ipa: INFO: The ipactl command was successful
>>>> ---
>>>>
>>>> --- ipa1
>>>> $ openssl x509 -in /var/lib/ipa/ra-agent.pem -text -noout | grep
>>>> Serial
>>>> Serial Number: 268304391 (0xffe0007)
>>>>
>>>> $ ldapsearch -x -b uid=ipara,ou=People,o=ipaca
>>>> [...]
>>>> description: 2;268304391;CN=Certificate
>>>> Authority,O=EXAMPLE.COM;CN=IPA
>>>> RA,O=EXAMPLE.COM
>>>> ---
>>>>
>>>> Looks good.
>>>>
>>>> --- ipa2
>>>> $ openssl x509 -in /var/lib/ipa/ra-agent.pem -text -noout | grep
>>>> Serial
>>>> Serial Number: 268304391 (0xffe0007)
>>>>
>>>> $ ldapsearch -x -b uid=ipara,ou=People,o=ipaca
>>>> [...]
>>>> description: 2;44;CN=Certificate Authority,O=EXAMPLE.COM;CN=IPA
>>>> RA,O=EXAMPLE.COM
>>>> ---
>>>>
>>>> Looks bad.
>>>>
>>> So yes, the replication of the suffix o=ipaca seems broken. In a
>>> deployment with CA, the ldap server contains 2 different suffixes, one
>>> for the IdM data (your base DN as defined in /etc/ipa/default.conf)
>>> and
>>> another one for the CA, below o=ipaca.
>>>
>>> You can have a look at
>>> https://www.freeipa.org/page/Troubleshooting/Directory_Server for
>>> replication troubleshooting, but the repl issue could be a
>>> consequence of
>>> kerberos server not starting.
>>
>> I checked the troubleshooting pages and carried out the steps
>> described in
>> there, but my system (ipa2) appears to be ok as far as the steps go.
>>
>>>> --- /var/log/ipaupgrade.log
>>>> [...]
>>>> 2018-12-28T10:55:53Z DEBUG The ipa-server-upgrade command failed,
>>>> exception: RemoteRetrieveError: Failed to authenticate to CA REST
>>>> API
>>>> ---
>>>>
>>>> --- /var/log/dirsrv/slapd-EXAMPLE-COM/errors
>>>> [...]
>>>> [28/Dec/2018:11:55:55.559648469 +0100] - ERR - set_krb5_creds -
>>>> Could
>>>> not
>>>> get initial credentials for principal
>>>> [ldap/ipa2.example.com(a)EXAMPLE.COM]
>>>> in keytab [FILE:/etc/dirsrv/ds.keytab]: -1765328228 (Cannot
>>>> contact any
>>>> KDC for requested realm)
>>>> ---
>>>>
>>>> --- ipa-checkcerts.py
>>>> [...]
>>>> ipa: ERROR: ra.get_certificate(): EXCEPTION (Invalid Credential.)
>>>> ra.get_certificate(): EXCEPTION (Invalid Credential.)
>>>> ipa: INFO: Checking permissions and ownership
>>>> Checking permissions and ownership
>>>> ipa: INFO: Failures:
>>>> Failures:
>>>> ipa: INFO: Unable to find request for serial 268304391
>>>> Unable to find request for serial 268304391
>>>> ipa: INFO: Unable to find request for serial 268304394
>>>> Unable to find request for serial 268304394
>>>> ipa: INFO: Unable to find request for serial 268304393
>>>> Unable to find request for serial 268304393
>>>> ipa: INFO: Unable to find request for serial 268304392
>>>> Unable to find request for serial 268304392
>>>> ipa: INFO: Unable to find request for serial 268304390
>>>> Unable to find request for serial 268304390
>>>> ipa: INFO: RA agent description does not match 2;44;CN=Certificate
>>>> Authority,O=EXAMPLE.COM;CN=IPA RA,O=EXAMPLE.COM in LDAP and
>>>> 2;268304391;CN=Certificate Authority,O=EXAMPLE.COM;CN=IPA
>>>> RA,O=EXAMPLE.COM
>>>> expected
>>>> ipa: INFO: RA agent certificate not found in LDAP RA agent
>>>> certificate
>>>> not
>>>> found in LDAP
>>>> [...]
>>>> ---
>>>>
>>>> The root cause appears to be the wrong IPA RA certificate in
ipa2's
>>>> LDAP, right? I guess this has to fixed by manually importing the
>>>> proper certificate using ldapmodify, similar to the procedure
>>>> described in [1]?
>>>>
>>> It is possible to manually modify the IPA RA cert in ipa2's ldap
>>> server
>>> but this won't solve the replication issue. It will however allow
>>> you to
>>> run ipa cert-show.
>>
>> Unfortunately, it doesn't (ipa2):
>>
>> ---
>> ipa: ERROR: Certificate operation cannot be completed: EXCEPTION
>> (Invalid
>> Credential.)
>> ---
>>
>> Kerberos log:
>>
>> ---
>> Jan 08 15:42:45 ipa2.example.com krb5kdc[23520](info): AS_REQ (8 etypes
>> {18 17 16 23 20 19 25 26}) 141.51.124.20: NEEDED_PREAUTH:
>> host/ipa2.example.com(a)EXAMPLE.COM for krbtgt/EXAMPLE.COM(a)EXAMPLE.COM,
>> Additional pre-authentication required
>> ---
>>
>> Works ok on ipa1.
>>
>> I'm unsure how to proceed here. My obvious problem is ipa2 not getting
>> upgraded properly:
>>
>> ---
>> [...]
>> 2019-01-08T09:46:07Z INFO [Updating HTTPD service IPA configuration]
>> 2019-01-08T09:46:07Z DEBUG Starting external process
>> 2019-01-08T09:46:07Z DEBUG args=/usr/sbin/selinuxenabled
>> 2019-01-08T09:46:07Z DEBUG Process finished, return code=1
>> 2019-01-08T09:46:07Z DEBUG stdout=
>> 2019-01-08T09:46:07Z DEBUG stderr=
>> 2019-01-08T09:46:07Z DEBUG Starting external process
>> 2019-01-08T09:46:07Z DEBUG args=/bin/systemctl --system daemon-reload
>> 2019-01-08T09:46:07Z DEBUG Process finished, return code=0
>> 2019-01-08T09:46:07Z DEBUG stdout=
>> 2019-01-08T09:46:07Z DEBUG stderr=
>> 2019-01-08T09:46:07Z INFO [Updating HTTPD service IPA WSGI
>> configuration]
>> 2019-01-08T09:46:07Z INFO Nothing to do for configure_httpd_wsgi_conf
>> 2019-01-08T09:46:07Z INFO [Updating mod_nss protocol versions]
>> 2019-01-08T09:46:07Z DEBUG Loading StateFile from
>> '/var/lib/ipa/sysupgrade/sysupgrade.state'
>> 2019-01-08T09:46:07Z INFO Protocol versions already updated
>> 2019-01-08T09:46:07Z INFO [Updating mod_nss cipher suite]
>> 2019-01-08T09:46:07Z DEBUG Loading StateFile from
>> '/var/lib/ipa/sysupgrade/sysupgrade.state'
>> 2019-01-08T09:46:07Z DEBUG Cipher suite already updated
>> 2019-01-08T09:46:07Z INFO [Updating mod_nss enabling OCSP]
>> 2019-01-08T09:46:07Z DEBUG Loading StateFile from
>> '/var/lib/ipa/sysupgrade/sysupgrade.state'
>> 2019-01-08T09:46:07Z INFO [Fixing trust flags in /etc/httpd/alias]
>> 2019-01-08T09:46:07Z DEBUG Loading StateFile from
>> '/var/lib/ipa/sysupgrade/sysupgrade.state'
>> 2019-01-08T09:46:07Z INFO Trust flags already processed
>> 2019-01-08T09:46:07Z INFO [Moving HTTPD service keytab to gssproxy]
>> 2019-01-08T09:46:07Z DEBUG Starting external process
>> 2019-01-08T09:46:07Z DEBUG args=/usr/sbin/selinuxenabled
>> 2019-01-08T09:46:07Z DEBUG Process finished, return code=1
>> 2019-01-08T09:46:07Z DEBUG stdout=
>> 2019-01-08T09:46:07Z DEBUG stderr=
>> 2019-01-08T09:46:07Z DEBUG Starting external process
>> 2019-01-08T09:46:07Z DEBUG args=/bin/systemctl restart gssproxy.service
>> 2019-01-08T09:46:07Z DEBUG Process finished, return code=0
>> 2019-01-08T09:46:07Z DEBUG stdout=
>> 2019-01-08T09:46:07Z DEBUG stderr=
>> 2019-01-08T09:46:07Z DEBUG Starting external process
>> 2019-01-08T09:46:07Z DEBUG args=/bin/systemctl is-active
>> gssproxy.service
>> 2019-01-08T09:46:07Z DEBUG Process finished, return code=0
>> 2019-01-08T09:46:07Z DEBUG stdout=active
>>
>> 2019-01-08T09:46:07Z DEBUG stderr=
>> 2019-01-08T09:46:07Z DEBUG Restart of gssproxy.service complete
>> 2019-01-08T09:46:07Z DEBUG Starting external process
>> 2019-01-08T09:46:07Z DEBUG args=/bin/systemctl start httpd.service
>> 2019-01-08T09:46:09Z DEBUG Process finished, return code=0
>> 2019-01-08T09:46:09Z DEBUG stdout=
>> 2019-01-08T09:46:09Z DEBUG stderr=
>> 2019-01-08T09:46:09Z DEBUG Starting external process
>> 2019-01-08T09:46:09Z DEBUG args=/bin/systemctl is-active httpd.service
>> 2019-01-08T09:46:09Z DEBUG Process finished, return code=0
>> 2019-01-08T09:46:09Z DEBUG stdout=active
>>
>> 2019-01-08T09:46:09Z DEBUG stderr=
>> 2019-01-08T09:46:09Z DEBUG Start of httpd.service complete
>> 2019-01-08T09:46:09Z INFO [Removing self-signed CA]
>> 2019-01-08T09:46:09Z DEBUG Self-signed CA is not installed
>> 2019-01-08T09:46:09Z INFO [Removing Dogtag 9 CA]
>> 2019-01-08T09:46:09Z DEBUG Dogtag is version 10 or above
>> 2019-01-08T09:46:09Z DEBUG Loading StateFile from
>> '/var/lib/ipa/sysrestore/sysrestore.state'
>> 2019-01-08T09:46:09Z DEBUG Loading Index file from
>> '/var/lib/ipa/sysrestore/sysrestore.index'
>> 2019-01-08T09:46:09Z INFO [Checking for deprecated KDC configuration
>> files]
>> 2019-01-08T09:46:09Z INFO [Checking for deprecated backups of Samba
>> configuration files]
>> 2019-01-08T09:46:09Z DEBUG raw: ca_is_enabled(version=u'2.229')
>> 2019-01-08T09:46:09Z DEBUG ca_is_enabled(version=u'2.229')
>> 2019-01-08T09:46:09Z DEBUG flushing
>> ldapi://%2fvar%2frun%2fslapd-EXAMPLE-COM.socket from SchemaCache
>> 2019-01-08T09:46:09Z DEBUG retrieving schema for SchemaCache
>> url=ldapi://%2fvar%2frun%2fslapd-EXAMPLE-COM.socket
>> conn=<ldap.ldapobject.SimpleLDAPObject instance at 0x7f8d6ccff5a8>
>> 2019-01-08T09:46:09Z DEBUG raw: kra_is_enabled(version=u'2.229')
>> 2019-01-08T09:46:09Z DEBUG kra_is_enabled(version=u'2.229')
>> 2019-01-08T09:46:09Z DEBUG Cleaning up after pkispawn for the CA
>> subsystem
>> 2019-01-08T09:46:09Z DEBUG Loading StateFile from
>> '/var/lib/ipa/sysrestore/sysrestore.state'
>> 2019-01-08T09:46:09Z DEBUG Loading Index file from
>> '/var/lib/ipa/sysrestore/sysrestore.index'
>> 2019-01-08T09:46:09Z DEBUG Loading StateFile from
>> '/var/lib/ipa/sysrestore/sysrestore.state'
>> 2019-01-08T09:46:09Z INFO [Add missing CA DNS records]
>> 2019-01-08T09:46:09Z DEBUG Loading StateFile from
>> '/var/lib/ipa/sysupgrade/sysupgrade.state'
>> 2019-01-08T09:46:09Z INFO IPA CA DNS records already processed
>> 2019-01-08T09:46:09Z INFO [Removing deprecated DNS configuration
>> options]
>> 2019-01-08T09:46:09Z INFO DNS is not configured
>> 2019-01-08T09:46:09Z INFO [Ensuring minimal number of connections]
>> 2019-01-08T09:46:09Z INFO DNS is not configured
>> 2019-01-08T09:46:09Z INFO [Updating GSSAPI configuration in DNS]
>> 2019-01-08T09:46:09Z INFO DNS is not configured
>> 2019-01-08T09:46:09Z INFO [Updating pid-file configuration in DNS]
>> 2019-01-08T09:46:09Z INFO DNS is not configured
>> 2019-01-08T09:46:09Z INFO DNS is not configured
>> 2019-01-08T09:46:09Z INFO DNS is not configured
>> 2019-01-08T09:46:09Z INFO DNS is not configured
>> 2019-01-08T09:46:09Z INFO DNS is not configured
>> 2019-01-08T09:46:09Z INFO DNS is not configured
>> 2019-01-08T09:46:09Z DEBUG Loading StateFile from
>> '/var/lib/ipa/sysrestore/sysrestore.state'
>> 2019-01-08T09:46:09Z DEBUG Loading Index file from
>> '/var/lib/ipa/sysrestore/sysrestore.index'
>> 2019-01-08T09:46:09Z INFO DNS is not configured
>> 2019-01-08T09:46:09Z DEBUG Loading StateFile from
>> '/var/lib/ipa/sysupgrade/sysupgrade.state'
>> 2019-01-08T09:46:09Z DEBUG Loading StateFile from
>> '/var/lib/ipa/sysupgrade/sysupgrade.state'
>> 2019-01-08T09:46:09Z DEBUG Loading StateFile from
>> '/var/lib/ipa/sysrestore/sysrestore.state'
>> 2019-01-08T09:46:09Z DEBUG Loading Index file from
>> '/var/lib/ipa/sysrestore/sysrestore.index'
>> 2019-01-08T09:46:09Z INFO DNS is not configured
>> 2019-01-08T09:46:09Z DEBUG Loading StateFile from
>> '/var/lib/ipa/sysrestore/sysrestore.state'
>> 2019-01-08T09:46:09Z DEBUG Loading Index file from
>> '/var/lib/ipa/sysrestore/sysrestore.index'
>> 2019-01-08T09:46:09Z DEBUG Loading StateFile from
>> '/var/lib/ipa/sysupgrade/sysupgrade.state'
>> 2019-01-08T09:46:09Z INFO [Upgrading CA schema]
>> 2019-01-08T09:46:10Z DEBUG Processing schema LDIF file
>> /usr/share/pki/server/conf/schema-certProfile.ldif
>> 2019-01-08T09:46:10Z DEBUG Processing schema LDIF file
>> /usr/share/pki/server/conf/schema-authority.ldif
>> 2019-01-08T09:46:10Z DEBUG Not updating schema
>> 2019-01-08T09:46:10Z INFO CA schema update complete (no changes)
>> 2019-01-08T09:46:10Z INFO [Verifying that CA audit signing cert has
>> 2 year
>> validity]
>> 2019-01-08T09:46:10Z DEBUG caSignedLogCert.cfg profile validity
>> range is
>> 720
>> 2019-01-08T09:46:10Z INFO [Update certmonger certificate renewal
>> configuration]
>> 2019-01-08T09:46:10Z DEBUG Loading Index file from
>> '/var/lib/ipa/sysrestore/sysrestore.index'
>> 2019-01-08T09:46:10Z DEBUG Starting external process
>> 2019-01-08T09:46:10Z DEBUG args=/usr/bin/certutil -d
>> dbm:/etc/httpd/alias
>> -L -n Server-Cert -a -f /etc/httpd/alias/pwdfile.txt
>> 2019-01-08T09:46:10Z DEBUG Process finished, return code=0
>> 2019-01-08T09:46:10Z DEBUG stdout=-----BEGIN CERTIFICATE-----
>> MIIFYjCCBEqgAwIBAgIED/4ABTANBgkqhkiG9w0BAQsFADBHMSUwIwYDVQQKExxE
>> [...]
>> Serial Number: 268304389 (0xffe0005)
>> [...]
>> n7aRLXFxWB8va5AxibMHYUJb08nWJRqHHwufF/YQnJdskFuDyAM=
>> -----END CERTIFICATE-----
>>
>> 2019-01-08T09:46:10Z DEBUG stderr=
>> 2019-01-08T09:46:10Z DEBUG Loading Index file from
>> '/var/lib/ipa/sysrestore/sysrestore.index'
>> 2019-01-08T09:46:10Z DEBUG Starting external process
>> 2019-01-08T09:46:10Z DEBUG args=/usr/bin/certutil -d
>> dbm:/etc/dirsrv/slapd-EXAMPLE-COM/ -L -n Server-Cert -a -f
>> /etc/dirsrv/slapd-EXAMPLE-COM/pwdfile.txt
>> 2019-01-08T09:46:10Z DEBUG Process finished, return code=0
>> 2019-01-08T09:46:10Z DEBUG stdout=-----BEGIN CERTIFICATE-----
>> MIIFYjCCBEqgAwIBAgIED/4ABDANBgkqhkiG9w0BAQsFADBHMSUwIwYDVQQKExxE
>> [...]
>> Serial Number: 268304388 (0xffe0004)
>> [...]
>> 7D6pk95Rgq6g73cZmogJBfJicVF0odkf2VkvV/aJCxTtZ4xkVds=
>> -----END CERTIFICATE-----
>>
>> 2019-01-08T09:46:10Z DEBUG stderr=
>> 2019-01-08T09:46:10Z DEBUG Loading Index file from
>> '/var/lib/ipa/sysrestore/sysrestore.index'
>> 2019-01-08T09:46:10Z DEBUG Starting external process
>> 2019-01-08T09:46:10Z DEBUG args=/usr/bin/certutil -d
>> dbm:/etc/pki/pki-tomcat/alias -L -f
>> /etc/pki/pki-tomcat/alias/pwdfile.txt
>> 2019-01-08T09:46:10Z DEBUG Process finished, return code=0
>> 2019-01-08T09:46:10Z DEBUG stdout=
>> Certificate Nickname Trust
>> Attributes
>>
>> SSL,S/MIME,JAR/XPI
>>
>> ocspSigningCert cert-pki-ca u,u,u
>> caSigningCert cert-pki-ca CTu,Cu,Cu
>> Server-Cert cert-pki-ca u,u,u
>> auditSigningCert cert-pki-ca u,u,Pu
>> subsystemCert cert-pki-ca u,u,u
>>
>> 2019-01-08T09:46:10Z DEBUG stderr=
>> 2019-01-08T09:46:11Z INFO Certmonger certificate renewal configuration
>> already up-to-date
>> 2019-01-08T09:46:11Z INFO [Enable PKIX certificate path discovery and
>> validation]
>> 2019-01-08T09:46:11Z DEBUG Loading StateFile from
>> '/var/lib/ipa/sysupgrade/sysupgrade.state'
>> 2019-01-08T09:46:11Z INFO PKIX already enabled
>> 2019-01-08T09:46:11Z INFO [Authorizing RA Agent to modify profiles]
>> 2019-01-08T09:46:11Z INFO [Authorizing RA Agent to manage
>> lightweight CAs]
>> 2019-01-08T09:46:11Z INFO [Ensuring Lightweight CAs container exists in
>> Dogtag database]
>> 2019-01-08T09:46:11Z DEBUG Created connection
>> context.ldap2_140245392902928
>> 2019-01-08T09:46:11Z DEBUG flushing
>> ldapi://%2fvar%2frun%2fslapd-EXAMPLE-COM.socket from SchemaCache
>> 2019-01-08T09:46:11Z DEBUG retrieving schema for SchemaCache
>> url=ldapi://%2fvar%2frun%2fslapd-EXAMPLE-COM.socket
>> conn=<ldap.ldapobject.SimpleLDAPObject instance at 0x7f8d6cea4fc8>
>> 2019-01-08T09:46:12Z DEBUG Destroyed connection
>> context.ldap2_140245392902928
>> 2019-01-08T09:46:12Z INFO [Adding default OCSP URI configuration]
>> 2019-01-08T09:46:12Z INFO [Ensuring CA is using LDAPProfileSubsystem]
>> 2019-01-08T09:46:12Z INFO [Migrating certificate profiles to LDAP]
>> 2019-01-08T09:46:12Z DEBUG Created connection
>> context.ldap2_140245366754832
>> 2019-01-08T09:46:12Z DEBUG flushing
>> ldapi://%2fvar%2frun%2fslapd-EXAMPLE-COM.socket from SchemaCache
>> 2019-01-08T09:46:12Z DEBUG retrieving schema for SchemaCache
>> url=ldapi://%2fvar%2frun%2fslapd-EXAMPLE-COM.socket
>> conn=<ldap.ldapobject.SimpleLDAPObject instance at 0x7f8d6b5db128>
>> 2019-01-08T09:46:12Z DEBUG Destroyed connection
>> context.ldap2_140245366754832
>> 2019-01-08T09:46:12Z DEBUG request GET
>> https://ipa2.example.com:8443/ca/rest/account/login
>> 2019-01-08T09:46:12Z DEBUG request body ''
>> 2019-01-08T09:46:12Z DEBUG response status 401
>> 2019-01-08T09:46:12Z DEBUG response headers Server: Apache-Coyote/1.1
>> Cache-Control: private
>> Expires: Thu, 01 Jan 1970 01:00:00 CET
>> WWW-Authenticate: Basic realm="Certificate Authority"
>> Content-Type: text/html;charset=utf-8
>> Content-Language: en
>> Content-Length: 951
>> Date: Tue, 08 Jan 2019 09:46:12 GMT
>>
>> 2019-01-08T09:46:12Z DEBUG response body '<html> [...] HTTP Status
401
>> [...] message [...] This request requires HTTP authentication. [...]
>> </html>'
>> 2019-01-08T09:46:12Z ERROR IPA server upgrade failed: Inspect
>> /var/log/ipaupgrade.log and run command ipa-server-upgrade manually.
>> 2019-01-08T09:46:12Z DEBUG File
>> "/usr/lib/python2.7/site-packages/ipapython/admintool.py", line 178,
in
>> execute
>> return_value = self.run()
>> File
>>
"/usr/lib/python2.7/site-packages/ipaserver/install/ipa_server_upgrade.py",
>>
>> line 54, in run
>> server.upgrade()
>> File
>>
"/usr/lib/python2.7/site-packages/ipaserver/install/server/upgrade.py",
>> line 2085, in upgrade
>> upgrade_configuration()
>> File
>>
"/usr/lib/python2.7/site-packages/ipaserver/install/server/upgrade.py",
>> line 1952, in upgrade_configuration
>> ca_enable_ldap_profile_subsystem(ca)
>> File
>>
"/usr/lib/python2.7/site-packages/ipaserver/install/server/upgrade.py",
>> line 396, in ca_enable_ldap_profile_subsystem
>> cainstance.migrate_profiles_to_ldap()
>> File
>> "/usr/lib/python2.7/site-packages/ipaserver/install/cainstance.py",
>> line
>> 1814, in migrate_profiles_to_ldap
>> _create_dogtag_profile(profile_id, profile_data, overwrite=False)
>> File
>> "/usr/lib/python2.7/site-packages/ipaserver/install/cainstance.py",
>> line
>> 1820, in _create_dogtag_profile
>> with api.Backend.ra_certprofile as profile_api:
>> File
"/usr/lib/python2.7/site-packages/ipaserver/plugins/dogtag.py",
>> line 1302, in __enter__
>> raise errors.RemoteRetrieveError(reason=_('Failed to
>> authenticate to
>> CA REST API'))
>>
>> 2019-01-08T09:46:12Z DEBUG The ipa-server-upgrade command failed,
>> exception: RemoteRetrieveError: Failed to authenticate to CA REST API
>> 2019-01-08T09:46:12Z ERROR Unexpected error - see
>> /var/log/ipaupgrade.log
>> for details: RemoteRetrieveError: Failed to authenticate to CA REST API
>> 2019-01-08T09:46:12Z ERROR The ipa-server-upgrade command failed. See
>> /var/log/ipaupgrade.log for more information
>> ---
>>
> on ipa2, what is the content of /var/lib/ipa/ra-agent.pem? Is it the
> same as on ipa1? Can you check the file permissions?
>
> Can you also compare with the ldap entry uid=ipara,ou=People,o=ipaca
> (description and usercertificate fields are important).
looked deeper this time, as you requested:
--- ipa1
$ ls -la /var/lib/ipa/ra-agent.*
-r--r----- 1 root ipaapi 1854 Jul 11 2018 /var/lib/ipa/ra-agent.key
-r--r----- 1 root ipaapi 1318 Jul 11 2018 /var/lib/ipa/ra-agent.pem
--
$ openssl x509 -in /var/lib/ipa/ra-agent.pem -text -noout | grep Serial
Serial Number: 268304391 (0xffe0007)
--
$ ldapsearch -x -b uid=ipara,ou=People,o=ipaca
# extended LDIF
#
# LDAPv3
# base <uid=ipara,ou=People,o=ipaca> with scope subtree
# filter: (objectclass=*)
# requesting: ALL
#
# ipara, people, ipaca
dn: uid=ipara,ou=people,o=ipaca
userCertificate::
MIIDozCCAougAwIBAgIBBzANBgkqhkiG9w0BAQsFADBHMSUwIwYDVQQKExxE
[...]
DoNVNCVX+P2rYhhKizm5W/+Q77YYMxs4=
userCertificate::
MIIDozCCAougAwIBAgIBLDANBgkqhkiG9w0BAQsFADBHMSUwIwYDVQQKExxE
[...]
DFVH/Om2gwOobX7Rt3L582BOjz97gRSQ=
userCertificate::
MIIDoTCCAomgAwIBAgIED/4ABzANBgkqhkiG9w0BAQsFADBHMSUwIwYDVQQK
[...]
zon51BrP2rvAW9tpXAgPE1XqQuRzn
userstate: 1
usertype: agentType
cn: ipara
sn: ipara
uid: ipara
objectClass: top
objectClass: person
objectClass: organizationalPerson
objectClass: inetOrgPerson
objectClass: cmsuser
description: 2;268304391;CN=Certificate Authority,O=EXAMPLE.COM;CN=IPA
RA,O=EXAMPLE.COM
# search result
search: 2
result: 0 Success
# numResponses: 2
# numEntries: 1
--
$ head -n 5 ra-agent.key
Bag Attributes
friendlyName: ipaCert
localKeyID: 5E 90 B6 ED CA 3E 5B FF 2E 76 F6 B8 73 FD AC 1A A7 84
D7 11
Key Attributes: <No Attributes>
-----BEGIN PRIVATE KEY-----
--- ipa2
$ ls -la /var/lib/ipa/ra-agent.*
-r--r----- 1 root ipaapi 1828 Jul 11 2018 /var/lib/ipa/ra-agent.key
-r--r----- 1 root ipaapi 1318 Jul 11 2018 /var/lib/ipa/ra-agent.pem
--
$ openssl x509 -in /var/lib/ipa/ra-agent.pem -text -noout | grep Serial
Serial Number: 268304391 (0xffe0007)
--
$ ldapsearch -x -b uid=ipara,ou=People,o=ipaca
# extended LDIF
#
# LDAPv3
# base <uid=ipara,ou=People,o=ipaca> with scope subtree
# filter: (objectclass=*)
# requesting: ALL
#
# ipara, people, ipaca
dn: uid=ipara,ou=people,o=ipaca
description: 2;44;CN=Certificate Authority,O=EXAMPLE.COM;CN=IPA
RA,O=EXAMPLE.COM
objectClass: top
objectClass: person
objectClass: organizationalPerson
objectClass: inetOrgPerson
objectClass: cmsuser
uid: ipara
sn: ipara
cn: ipara
usertype: agentType
userstate: 1
userCertificate::
MIIDozCCAougAwIBAgIBBzANBgkqhkiG9w0BAQsFADBHMSUwIwYDVQQKExxE
[...]
DoNVNCVX+P2rYhhKizm5W/+Q77YYMxs4=
userCertificate::
MIIDozCCAougAwIBAgIBLDANBgkqhkiG9w0BAQsFADBHMSUwIwYDVQQKExxE
[...]
DFVH/Om2gwOobX7Rt3L582BOjz97gRSQ=
# search result
search: 2
result: 0 Success
# numResponses: 2
# numEntries: 1
--
$ head -n 5 ra-agent.key
Bag Attributes
localKeyID: 5E 90 B6 ED CA 3E 5B FF 2E 76 F6 B8 73 FD AC 1A A7 84
D7 11
Key Attributes: <No Attributes>
-----BEGIN PRIVATE KEY-----
[...]
---
Two differences:
1. The key file on ipa1 contains a "Bag Attribute" "friendlyName",
that
the key key file on ipa2 misses - don't know if this is relevant, but
probably not.
2. The LDAP entry on ipa1 contains 3 "userCertificate"s, but only the
first 2 of them are in ipa2's LDAP. This is probably the interesting part.
Am i right to assume, that adding the missing userCertificate to ipa2's
LDAP will solve my problem? If yes, how would I achieve that?
Hi,
you can use ldapmodify to manually add the missing certificate:
1. transform the RA agent cert into der format
$ openssl x509 -outform der -in /var/lib/ipa/ra-agent.pem -out
/tmp/ra-agent.der
2. upload the cert in LDAP
$ ldapmodify -h ipa2 -p 389 -D "cn=directory manager" -W
Enter LDAP Password:
dn: uid=ipara,ou=people,o=ipaca
changetype: modify
add: usercertificate
usercertificate:< file:///tmp/ra-agent.der
modifying entry "uid=ipara,ou=people,o=ipaca"
<Ctrl-D> to exit
After that, you should be able to re-run ipa-server-upgrade. At this
point, please make sure that replication could be re-established between
the two nodes.
HTH,
flo
Mit freundlichen Gruessen/With best regards,
--Daniel.
_______________________________________________
FreeIPA-users mailing list -- freeipa-users(a)lists.fedorahosted.org
To unsubscribe send an email to freeipa-users-leave(a)lists.fedorahosted.org
Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives:
https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedoraho...
8:24 a.m.
Hi Florence,
On Thu, 10 Jan 2019, Florence Blanc-Renaud wrote:
On 1/10/19 1:46 PM, dbischof--- via FreeIPA-users wrote:
[...]
you can use ldapmodify to manually add the missing certificate:
1. transform the RA agent cert into der format $ openssl x509 -outform
der -in /var/lib/ipa/ra-agent.pem -out /tmp/ra-agent.der
2. upload the cert in LDAP
$ ldapmodify -h ipa2 -p 389 -D "cn=directory manager" -W
Enter LDAP Password:
dn: uid=ipara,ou=people,o=ipaca
changetype: modify
add: usercertificate
usercertificate:< file:///tmp/ra-agent.der
modifying entry "uid=ipara,ou=people,o=ipaca"
<Ctrl-D> to exit
After that, you should be able to re-run ipa-server-upgrade. At this
point, please make sure that replication could be re-established between
the two nodes.
your help is greatly appreciated.
I had to change the cert serial in "description" additionally the same way
via ldapmodify, but now ipa-server-upgrade goes smooth and IPA on ipa2
comes up properly after a reboot. Fine.
Regarding replication: Checking, whether replication works properly is
achieved with "ipa-replica-manage -v list <host>", right? Has to work on
both IPA servers and "last update ended" must be a reasonable recent
timestamp?
Mit freundlichen Gruessen/With best regards,
--Daniel.
9:32 a.m.
On 1/11/19 3:24 PM, dbischof--- via FreeIPA-users wrote:
Hi Florence,
On Thu, 10 Jan 2019, Florence Blanc-Renaud wrote:
> On 1/10/19 1:46 PM, dbischof--- via FreeIPA-users wrote:
> [...]
> you can use ldapmodify to manually add the missing certificate:
>
> 1. transform the RA agent cert into der format $ openssl x509 -outform
> der -in /var/lib/ipa/ra-agent.pem -out /tmp/ra-agent.der
>
> 2. upload the cert in LDAP
> $ ldapmodify -h ipa2 -p 389 -D "cn=directory manager" -W
> Enter LDAP Password:
> dn: uid=ipara,ou=people,o=ipaca
> changetype: modify
> add: usercertificate
> usercertificate:< file:///tmp/ra-agent.der
>
> modifying entry "uid=ipara,ou=people,o=ipaca"
>
> <Ctrl-D> to exit
>
> After that, you should be able to re-run ipa-server-upgrade. At this
> point, please make sure that replication could be re-established
> between the two nodes.
your help is greatly appreciated.
I had to change the cert serial in "description" additionally the same
way via ldapmodify, but now ipa-server-upgrade goes smooth and IPA on
ipa2 comes up properly after a reboot. Fine.
Regarding replication: Checking, whether replication works properly is
achieved with "ipa-replica-manage -v list <host>", right? Has to work on
both IPA servers and "last update ended" must be a reasonable recent
timestamp?
Yes, ipa-replica-manage -v list <host> will display the status of the
replication for the domain (user, hosts, ...). The value of "last update
status" must be "Replica acquired successfully: Incremental update
succeeded".
If the topology includes multiple CA instances, replication is also
configured for the CA data, and the status can be found using
ipa-csreplica-manage -v list <host>.
HTH,
flo
Mit freundlichen Gruessen/With best regards,
--Daniel.
_______________________________________________
FreeIPA-users mailing list -- freeipa-users(a)lists.fedorahosted.org
To unsubscribe send an email to freeipa-users-leave(a)lists.fedorahosted.org
Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives:
https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedoraho...
7:32 a.m.
Hi Florence,
On Fri, 11 Jan 2019, Florence Blanc-Renaud via FreeIPA-users wrote:
On 1/11/19 3:24 PM, dbischof--- via FreeIPA-users wrote:
> On Thu, 10 Jan 2019, Florence Blanc-Renaud wrote:
>
>> On 1/10/19 1:46 PM, dbischof--- via FreeIPA-users wrote:
>> [...]
>> you can use ldapmodify to manually add the missing certificate:
>>
>> 1. transform the RA agent cert into der format $ openssl x509 -outform
>> der -in /var/lib/ipa/ra-agent.pem -out /tmp/ra-agent.der
>>
>> 2. upload the cert in LDAP
>> $ ldapmodify -h ipa2 -p 389 -D "cn=directory manager" -W
>> Enter LDAP Password:
>> dn: uid=ipara,ou=people,o=ipaca
>> changetype: modify
>> add: usercertificate
>> usercertificate:< file:///tmp/ra-agent.der
>>
>> modifying entry "uid=ipara,ou=people,o=ipaca"
>>
>> <Ctrl-D> to exit
>>
>> After that, you should be able to re-run ipa-server-upgrade. At this
>> point, please make sure that replication could be re-established between
>> the two nodes.
>
> your help is greatly appreciated.
>
> I had to change the cert serial in "description" additionally the same
way
> via ldapmodify, but now ipa-server-upgrade goes smooth and IPA on ipa2
> comes up properly after a reboot. Fine.
>
> Regarding replication: Checking, whether replication works properly is
> achieved with "ipa-replica-manage -v list <host>", right? Has to
work on
> both IPA servers and "last update ended" must be a reasonable recent
> timestamp?
>
Yes, ipa-replica-manage -v list <host> will display the status of the
replication for the domain (user, hosts, ...). The value of "last update
status" must be "Replica acquired successfully: Incremental update
succeeded".
this is working now, thanks again.
If the topology includes multiple CA instances, replication is also
configured for the CA data, and the status can be found using
ipa-csreplica-manage -v list <host>.
I do have 2 CA instances and it appears that i'm not yet out of the woods
here:
---ipa2
$ ipa-csreplica-manage -v list ipa1.example.com
ipa2.example.com
last init status: None
last init ended: 1970-01-01 00:00:00+00:00
last update status: Error (-1) Problem connecting to replica - LDAP error: Can't
contact LDAP server (connection error)
last update ended: 1970-01-01 00:00:00+00:00
---
But i will start a new thread for this, if i can't get fixed myself.
Mit freundlichen Gruessen/With best regards,
--Daniel.
1924
days inactive
1947
days old
freeipa-users@lists.fedorahosted.org
14 comments
2 participants
participants (2)
-
dbischof＠hrz.uni-kassel.de -
Florence Blanc-Renaud