ipa-healthcheck: a replica says "RA agent description does not match", ""Request for certificate failed, Certificate operation cannot be completed: EXCEPTION (Invalid Credential.)"

[Alex Corcoles]

Hi,

I'm monitoring using ipa-healthcheck and I just started getting:

$ sudo ipa-healthcheck --severity CRITICAL --severity ERROR --failures-only
ra.get_certificate(): EXCEPTION (Invalid Credential.)
ra.get_certificate(): EXCEPTION (Invalid Credential.)
ra.get_certificate(): EXCEPTION (Invalid Credential.)
ra.get_certificate(): EXCEPTION (Invalid Credential.)
ra.get_certificate(): EXCEPTION (Invalid Credential.)
ra.get_certificate(): EXCEPTION (Invalid Credential.)
ra.get_certificate(): EXCEPTION (Invalid Credential.)
ra.get_certificate(): EXCEPTION (Invalid Credential.)
ra.get_certificate(): EXCEPTION (Invalid Credential.)
ra.get_certificate(): EXCEPTION (Invalid Credential.)
[
  {
    "source": "ipahealthcheck.ipa.certs",
    "kw": {
      "msg": "RA agent description does not match 2;44;CN=Certificate
Authority,O=IPA.PDP7.NET;CN=IPA RA,O=IPA.PDP7.NET in LDAP and
2;7;CN=Certificate Authority,O=IPA.PDP7.NET;CN=IPA RA,O=IPA.PDP7.NET
expected",
      "got": "2;44;CN=Certificate Authority,O=IPA.PDP7.NET;CN=IPA RA,O=
IPA.PDP7.NET",
      "expected": "2;7;CN=Certificate Authority,O=IPA.PDP7.NET;CN=IPA
RA,O=
IPA.PDP7.NET"
    },
    "uuid": "0bfa6af6-5dd9-4505-89dc-a733060042a4",
    "duration": "0.037322",
    "when": "20191221123847Z",
    "check": "IPARAAgent",
    "result": "ERROR"
  },
  {
    "source": "ipahealthcheck.ipa.certs",
    "kw": {
      "msg": "Request for certificate failed, Certificate operation cannot
be completed: EXCEPTION (Invalid Credential.)",
      "key": "20181108202133"
    },
    "uuid": "bd04fd67-7b3e-4d2f-a87e-ff15563808e0",
    "duration": "0.491949",
    "when": "20191221123848Z",
    "check": "IPACertRevocation",
    "result": "ERROR"
  },

... the second one is repeated a bunch of times. If I go into the replica
web UI to check cert 7, I get very much the same error:

An error has occurred (IPA Error 4301: CertificateOperationError)
Certificate operation cannot be completed: EXCEPTION (Invalid Credential.)

However, if I go to the first IPA server I created, I can view the cert
normally. How should I proceed?

Cheers,

Álex

-- 
   ___
 {~._.~}
  ( Y )
 ()~*~()  mail: alex at corcoles dot net
 (_)-(_)  http://alex.corcoles.net/

[Florence Blanc-Renaud]

On 12/21/19 1:49 PM, Alex Corcoles via FreeIPA-users wrote:
>  Hi,
>  
>  I'm monitoring using ipa-healthcheck and I just started getting:
>  
>  $ sudo ipa-healthcheck --severity CRITICAL --severity ERROR --failures-only
>  ra.get_certificate(): EXCEPTION (Invalid Credential.)
>  ra.get_certificate(): EXCEPTION (Invalid Credential.)
>  ra.get_certificate(): EXCEPTION (Invalid Credential.)
>  ra.get_certificate(): EXCEPTION (Invalid Credential.)
>  ra.get_certificate(): EXCEPTION (Invalid Credential.)
>  ra.get_certificate(): EXCEPTION (Invalid Credential.)
>  ra.get_certificate(): EXCEPTION (Invalid Credential.)
>  ra.get_certificate(): EXCEPTION (Invalid Credential.)
>  ra.get_certificate(): EXCEPTION (Invalid Credential.)
>  ra.get_certificate(): EXCEPTION (Invalid Credential.)
>  [
>     {
>       "source": "ipahealthcheck.ipa.certs",
>       "kw": {
>         "msg": "RA agent description does not match 2;44;CN=Certificate 
>  Authority,O=IPA.PDP7.NET <http://IPA.PDP7.NET>;CN=IPA RA,O=IPA.PDP7.NET 
>  <http://IPA.PDP7.NET> in LDAP and 2;7;CN=Certificate 
>  Authority,O=IPA.PDP7.NET <http://IPA.PDP7.NET>;CN=IPA RA,O=IPA.PDP7.NET 
>  <http://IPA.PDP7.NET> expected",
>         "got": "2;44;CN=Certificate Authority,O=IPA.PDP7.NET 
>  <http://IPA.PDP7.NET>;CN=IPA RA,O=IPA.PDP7.NET <http://IPA.PDP7.NET>";,
>         "expected": "2;7;CN=Certificate Authority,O=IPA.PDP7.NET 
>  <http://IPA.PDP7.NET>;CN=IPA RA,O=IPA.PDP7.NET <http://IPA.PDP7.NET>"
>       },
>       "uuid": "0bfa6af6-5dd9-4505-89dc-a733060042a4",
>       "duration": "0.037322",
>       "when": "20191221123847Z",
>       "check": "IPARAAgent",
>       "result": "ERROR"
>     },
>     {
>       "source": "ipahealthcheck.ipa.certs",
>       "kw": {
>         "msg": "Request for certificate failed, Certificate operation 
>  cannot be completed: EXCEPTION (Invalid Credential.)",
>         "key": "20181108202133"
>       },
>       "uuid": "bd04fd67-7b3e-4d2f-a87e-ff15563808e0",
>       "duration": "0.491949",
>       "when": "20191221123848Z",
>       "check": "IPACertRevocation",
>       "result": "ERROR"
>     },
>  
>  ... the second one is repeated a bunch of times. If I go into the 
>  replica web UI to check cert 7, I get very much the same error:
>  
>  An error has occurred (IPA Error 4301: CertificateOperationError)
>  Certificate operation cannot be completed: EXCEPTION (Invalid Credential.)
>  
>  However, if I go to the first IPA server I created, I can view the cert 
>  normally. How should I proceed?
>   
Hi,
this means that the RA agent cert was renewed on the master (which is 
probably your CA renewal master), but the new cert was not properly 
downloaded/updated on the replicas.

1. Check which master is the CA renewal master with
$ ipa config-show | grep renewal
   IPA CA renewal master: <hostname>

2. On the renewal master found above, check the RA agent cert serial 
number. Depending on your version of IPA, it is either in 
/var/lib/ipa/ra-agent.pem (IPA 4.5+) or in the NSS database 
/etc/httpd/alias/
$ certutil -L -d /etc/httpd/alias/ -n ipaCert | grep Serial
         Serial Number: 7 (0x7)
or
$ openssl x509 -noout -serial -in /var/lib/ipa/ra-agent.pem
serial=07

Note the serial number and compare with the content of the LDAP entry 
uid=ipara,ou=people,o=ipaca:

$ ldapsearch -D cn=directory\ manager -W -b uid=ipara,ou=people,o=ipaca 
-LLL -o ldif-wrap=no dn description usercertificate
Enter LDAP Password:
dn: uid=ipara,ou=people,o=ipaca
description: 2;7;CN=Certificate Authority,O=DOMAIN.COM;CN=IPA 
RA,O=DOMAIN.COM
usercertificate:: MIIDyD...

The description attribute must have the format 
"2;<serial>;<issuer>;<subject>" and the userCertificate value
must be 
the same as the RA agent cert. If it's not the case, you need to perform 
a ldapmodify in order to fix this issue.

3. On the other replicas, check that the replication is OK and that the 
entry uid=ipara,ou=people,o=ipaca has been properly updated. If it's not 
the case, you will need to fix replication first.

4. On the other replicas, check that the certificate has been properly 
installed in the NSS database /etc/httpd/alias/ or in 
/var/lib/ipa/ra-agent.pem.
If it's not the case, you can manually install the cert or call getcert 
resubmit -i <ID of the tracking for RA agent>
Make sure that the request completed successfully with
$ getcert list -i <ID>
(the status must be: MONITORING)

The ID can be found with:
getcert list -f /var/lib/ipa/ra-agent.pem
or
getcert list -n ipaCert

HTH,
flo
>  Cheers,
>  
>  Álex
>  
>  -- 
>      ___
>    {~._.~}
>     ( Y )
>    ()~*~() mail: alex at corcoles dot net
>    (_)-(_) http://alex.corcoles.net/
>  
>  
>  _______________________________________________
>  FreeIPA-users mailing list -- freeipa-users(a)lists.fedorahosted.org
>  To unsubscribe send an email to freeipa-users-leave(a)lists.fedorahosted.org
>  Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
>  List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
>  List Archives:
> https://lists.fedorahosted.org/archives/list/[e-mail]...
>

[Alex Corcoles]

Thanks!

On Sun, Dec 22, 2019 at 11:13 AM Florence Blanc-Renaud <flo(a)redhat.com&gt;
wrote:

>  4. On the other replicas, check that the certificate has been
> properly
>  installed in the NSS database /etc/httpd/alias/ or in
>  /var/lib/ipa/ra-agent.pem.
>  If it's not the case, you can manually install the cert or call getcert
>  resubmit -i <ID of the tracking for RA agent>
>  Make sure that the request completed successfully with
>  $ getcert list -i <ID>
>  (the status must be: MONITORING)
>
>  The ID can be found with:
>  getcert list -f /var/lib/ipa/ra-agent.pem
>  or
>  getcert list -n ipaCert
>  
So on my renewal master, this was the cert:

$ sudo  getcert list -i 20180929065626
Number of certificates and requests being tracked: 9.
Request ID '20180929065626':

but on the broken replica:

$ sudo getcert resubmit -i 20180929065626
No request found with specified nickname.

However, copying the file over worked. Thanks!

Hopefully, this now will be googleable, although I'd humbly suggest that
this could be documented somewhere? (and it would be brilliant if the
ipa-healthcheck output pointed to it).

Cheers,

Álex
-- 
   ___
 {~._.~}
  ( Y )
 ()~*~()  mail: alex at corcoles dot net
 (_)-(_)  http://alex.corcoles.net/

[Florence Blanc-Renaud]

On 12/22/19 6:28 PM, Alex Corcoles via FreeIPA-users wrote:
>  Thanks!
>  
>  On Sun, Dec 22, 2019 at 11:13 AM Florence Blanc-Renaud <flo(a)redhat.com 
>  <mailto:[e-mail]>> wrote:
>  
>      4. On the other replicas, check that the certificate has been properly
>      installed in the NSS database /etc/httpd/alias/ or in
>      /var/lib/ipa/ra-agent.pem.
>      If it's not the case, you can manually install the cert or call getcert
>      resubmit -i <ID of the tracking for RA agent>
>      Make sure that the request completed successfully with
>      $ getcert list -i <ID>
>      (the status must be: MONITORING)
>  
>      The ID can be found with:
>      getcert list -f /var/lib/ipa/ra-agent.pem
>      or
>      getcert list -n ipaCert
>  
>  
>  So on my renewal master, this was the cert:
>  
>  $ sudo  getcert list -i 20180929065626
>  Number of certificates and requests being tracked: 9.
>  Request ID '20180929065626':
>  
>  but on the broken replica:
>  
>  $ sudo getcert resubmit -i 20180929065626
>  No request found with specified nickname.
>   
That's expected, as the request ID is different on each host.

>  However, copying the file over worked. Thanks! 
Glad to know
that the solution worked for you. So it means that the 
renewal did not proceed on the replicas. Could you check if there were 
any logs in the replica journal, that could help us understand why 
renewal failed?
$ journalctl -t certmonger
(look for lines mentioning "Certificate in file 
"/var/lib/ipa/ra-agent.pem" issued by CA and saved." or errors for this 
cert renewal.

flo

>  
>  Hopefully, this now will be googleable, although I'd humbly suggest that 
>  this could be documented somewhere? (and it would be brilliant if the 
>  ipa-healthcheck output pointed to it).
>  
>  Cheers,
>  
>  Álex
>  -- 
>      ___
>    {~._.~}
>     ( Y )
>    ()~*~() mail: alex at corcoles dot net
>    (_)-(_) http://alex.corcoles.net/
>  
>  
>  _______________________________________________
>  FreeIPA-users mailing list -- freeipa-users(a)lists.fedorahosted.org
>  To unsubscribe send an email to freeipa-users-leave(a)lists.fedorahosted.org
>  Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
>  List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
>  List Archives:
> https://lists.fedorahosted.org/archives/list/[e-mail]...
>

[Alex Corcoles]

journal seems empty for certmonger. I just see some stuff in messages:

# grep certmonger /var/log/messages*
/var/log/messages-20191215:Dec  8 08:21:11 ipa certmonger: Certificate
named "subsystemCert cert-pki-ca" in token "NSS Certificate DB" in
database
"/etc/pki/pki-tomcat/alias" will not be valid after 20200104123512.
/var/log/messages-20191215:Dec  8 08:21:11 ipa certmonger: Certificate
named "auditSigningCert cert-pki-ca" in token "NSS Certificate DB" in
database "/etc/pki/pki-tomcat/alias" will not be valid after 20200104123512.
/var/log/messages-20191215:Dec  8 08:21:11 ipa certmonger: Certificate
named "ocspSigningCert cert-pki-ca" in token "NSS Certificate DB" in
database "/etc/pki/pki-tomcat/alias" will not be valid after 20200104123511.
/var/log/messages-20191215:Dec  8 08:21:11 ipa certmonger: Certificate in
file "/var/lib/ipa/ra-agent.pem" will not be valid after 20200104123538.
/var/log/messages-20191215:Dec  8 08:21:35 ipa certmonger: Certificate
named "auditSigningCert cert-pki-ca" in token "NSS Certificate DB" in
database "/etc/pki/pki-tomcat/alias" issued by CA and saved.
/var/log/messages-20191215:Dec  8 16:21:39 ipa certmonger: 2019-12-08
16:21:39 [15599] Invalid cookie: u''
/var/log/messages-20191215:Dec  8 16:21:49 ipa certmonger: 2019-12-08
16:21:49 [15599] Invalid cookie: u''
/var/log/messages-20191215:Dec  8 16:21:59 ipa certmonger: 2019-12-08
16:21:59 [15599] Invalid cookie: u''

, which is weird; 20200104123511 is in the future...

On Sun, Dec 22, 2019 at 9:04 PM Florence Blanc-Renaud <flo(a)redhat.com&gt;
wrote:

>  On 12/22/19 6:28 PM, Alex Corcoles via FreeIPA-users wrote:
>  > Thanks!
>  >
>  > On Sun, Dec 22, 2019 at 11:13 AM Florence Blanc-Renaud <flo(a)redhat.com
>  > <mailto:[e-mail]>> wrote:
>  >
>  >     4. On the other replicas, check that the certificate has been
>  properly
>  >     installed in the NSS database /etc/httpd/alias/ or in
>  >     /var/lib/ipa/ra-agent.pem.
>  >     If it's not the case, you can manually install the cert or call
>  getcert
>  >     resubmit -i <ID of the tracking for RA agent>
>  >     Make sure that the request completed successfully with
>  >     $ getcert list -i <ID>
>  >     (the status must be: MONITORING)
>  >
>  >     The ID can be found with:
>  >     getcert list -f /var/lib/ipa/ra-agent.pem
>  >     or
>  >     getcert list -n ipaCert
>  >
>  >
>  > So on my renewal master, this was the cert:
>  >
>  > $ sudo  getcert list -i 20180929065626
>  > Number of certificates and requests being tracked: 9.
>  > Request ID '20180929065626':
>  >
>  > but on the broken replica:
>  >
>  > $ sudo getcert resubmit -i 20180929065626
>  > No request found with specified nickname.
>  >
>  That's expected, as the request ID is different on each host.
>
>  > However, copying the file over worked. Thanks!
>  Glad to know that the solution worked for you. So it means that the
>  renewal did not proceed on the replicas. Could you check if there were
>  any logs in the replica journal, that could help us understand why
>  renewal failed?
>  $ journalctl -t certmonger
>  (look for lines mentioning "Certificate in file
>  "/var/lib/ipa/ra-agent.pem" issued by CA and saved." or errors for this
>  cert renewal.
>
>  flo
>
>  >
>  > Hopefully, this now will be googleable, although I'd humbly suggest that
>  > this could be documented somewhere? (and it would be brilliant if the
>  > ipa-healthcheck output pointed to it).
>  >
>  > Cheers,
>  >
>  > Álex
>  > --
>  >     ___
>  >   {~._.~}
>  >    ( Y )
>  >   ()~*~() mail: alex at corcoles dot net
>  >   (_)-(_) http://alex.corcoles.net/
>  >
>  >
>  > _______________________________________________
>  > FreeIPA-users mailing list -- freeipa-users(a)lists.fedorahosted.org
>  > To unsubscribe send an email to
>  freeipa-users-leave(a)lists.fedorahosted.org
>  > Fedora Code of Conduct:
>  https://docs.fedoraproject.org/en-US/project/code-of-conduct/
>  > List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
>  > List Archives:
>  https://lists.fedorahosted.org/archives/list/[e-mail]...
>  >
>
>  
-- 
   ___
 {~._.~}
  ( Y )
 ()~*~()  mail: alex at corcoles dot net
 (_)-(_)  http://alex.corcoles.net/