certlist shows ca-error after upgrade

[Cody Ashe-McNalley]

Hi All,

My primary CA's httpd and slapd certs show a 'ca-error' warning "4027
(RPC failed at server.  The search criteria was not specific enough.  Expected 1 and found
2." 

RHEL 7.9
ipa-server-4.6.8-5.el7.x86_64
CA and DNS enabled

Request ID '20180927235641':
        status: CA_UNREACHABLE
        ca-error: Server at https://<ipaserver>/ipa/xml failed request, will retry:
4027 (RPC failed at server.  The search criteria was not specific enough. Expected 1 and
found 2.).
        stuck: no
        key pair storage:
type=NSSDB,location='/etc/dirsrv/slapd-<DOMAIN>',nickname='Server-Cert',token='NSS
Certificate DB',pinfile='/etc/dirsrv/slapd-<DOMAIN>/pwdfile.txt'
        certificate:
type=NSSDB,location='/etc/dirsrv/slapd-<DOMAIN>',nickname='Server-Cert',token='NSS
Certificate DB'
        CA: IPA
        issuer: CN=Certificate Authority,O=<DOMAIN>
        subject: CN=<ipaserver>,O=<DOMAIN>
        expires: 2022-05-05 23:59:26 UTC
        principal name: ldap/<ipaserver>@<DOMAIN>
        key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment
        eku: id-kp-serverAuth,id-kp-clientAuth
        pre-save command:
        post-save command: /usr/libexec/ipa/certmonger/restart_dirsrv <DOMAIN>
        track: yes
        auto-renew: yes
Request ID '20180927235642':
        status: CA_UNREACHABLE
        ca-error: Server at https://<ipaserver>/ipa/xml failed request, will retry:
4027 (RPC failed at server.  The search criteria was not specific enough. Expected 1 and
found 2.).
        stuck: no
        key pair storage:
type=NSSDB,location='/etc/httpd/alias',nickname='Server-Cert',token='NSS
Certificate DB',pinfile='/etc/httpd/alias/pwdfile.txt'
        certificate:
type=NSSDB,location='/etc/httpd/alias',nickname='Server-Cert',token='NSS
Certificate DB'
        CA: IPA
        issuer: CN=Certificate Authority,O=<DOMAIN>
        subject: CN=<ipaserver>,O=<DOMAIN>
        expires: 2022-05-05 23:59:25 UTC
        principal name: HTTP/<ipaserver>@<DOMAIN>
        key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment
        eku: id-kp-serverAuth,id-kp-clientAuth
        pre-save command:
        post-save command: /usr/libexec/ipa/certmonger/restart_httpd
        track: yes
        auto-renew: yes

Advice and experience would be greatly appreciated.

Best regards,
Cody

[Rob Crittenden]

Cody Ashe-McNalley via FreeIPA-users wrote:
>  Hi All,
>  
>  My primary CA's httpd and slapd certs show a 'ca-error' warning "4027
> (RPC failed at server.  The search criteria was not specific enough.  Expected 1 and found
> 2." 
>  
>  RHEL 7.9
>  ipa-server-4.6.8-5.el7.x86_64
>  CA and DNS enabled
>  
>  Request ID '20180927235641':
>          status: CA_UNREACHABLE
>          ca-error: Server at https://<ipaserver>/ipa/xml failed request, will retry:
> 4027 (RPC failed at server.  The search criteria was not specific enough. Expected 1 and
> found 2.).
>          stuck: no
>          key pair storage:
> type=NSSDB,location='/etc/dirsrv/slapd-<DOMAIN>',nickname='Server-Cert',token='NSS
> Certificate DB',pinfile='/etc/dirsrv/slapd-<DOMAIN>/pwdfile.txt'
>          certificate:
> type=NSSDB,location='/etc/dirsrv/slapd-<DOMAIN>',nickname='Server-Cert',token='NSS
> Certificate DB'
>          CA: IPA
>          issuer: CN=Certificate Authority,O=<DOMAIN>
>          subject: CN=<ipaserver>,O=<DOMAIN>
>          expires: 2022-05-05 23:59:26 UTC
>          principal name: ldap/<ipaserver>@<DOMAIN>
>          key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment
>          eku: id-kp-serverAuth,id-kp-clientAuth
>          pre-save command:
>          post-save command: /usr/libexec/ipa/certmonger/restart_dirsrv <DOMAIN>
>          track: yes
>          auto-renew: yes
>  Request ID '20180927235642':
>          status: CA_UNREACHABLE
>          ca-error: Server at https://<ipaserver>/ipa/xml failed request, will retry:
> 4027 (RPC failed at server.  The search criteria was not specific enough. Expected 1 and
> found 2.).
>          stuck: no
>          key pair storage:
> type=NSSDB,location='/etc/httpd/alias',nickname='Server-Cert',token='NSS
> Certificate DB',pinfile='/etc/httpd/alias/pwdfile.txt'
>          certificate:
> type=NSSDB,location='/etc/httpd/alias',nickname='Server-Cert',token='NSS
> Certificate DB'
>          CA: IPA
>          issuer: CN=Certificate Authority,O=<DOMAIN>
>          subject: CN=<ipaserver>,O=<DOMAIN>
>          expires: 2022-05-05 23:59:25 UTC
>          principal name: HTTP/<ipaserver>@<DOMAIN>
>          key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment
>          eku: id-kp-serverAuth,id-kp-clientAuth
>          pre-save command:
>          post-save command: /usr/libexec/ipa/certmonger/restart_httpd
>          track: yes
>          auto-renew: yes
>  
>  Advice and experience would be greatly appreciated. 
I suspect replication conflict entries. I'd suggest starting with:

$ kinit admin
$ ldapseach -LLL -Y GSSAPI -b cn=services,cn=accounts,$BASEDN
'(krbprincipalname=ldap/<ipaserver>@DOMAIN)'

Similar for the HTTP principal.

rob

[Cody Ashe-McNalley]

Hi Rob,

Thanks for such a fast reply!

That ldapsearch results in just the following 4 lines of output:

SASL/GSSAPI authentication started
SASL username: admin@DOMAIN
SASL SSF: 256
SASL data security layer installed.

Is that good?

Best regards,
Cody

[Rob Crittenden]

Cody Ashe-McNalley via FreeIPA-users wrote:
>  Hi Rob,
>  
>  Thanks for such a fast reply!
>  
>  That ldapsearch results in just the following 4 lines of output:
>  
>  SASL/GSSAPI authentication started
>  SASL username: admin@DOMAIN
>  SASL SSF: 256
>  SASL data security layer installed.
>  
>  Is that good? 
It's unexpected. Any chance you can provide the ldapsearch you used?
Maybe I goofed in my example. If you need to obfuscate your domain it
would be best to limit it as much as possible and replace values (e.g.
use "example" in place of "real domain") rather than completely remove
them.

rob

[Cody Ashe-McNalley]

Sure, here it is with just the ipaserver hostname substituted.

ldapsearch -LLL -Y GSSAPI -b cn=services,cn=accounts,dc=atmos,dc=ucla,dc=edu
'(krbprincipalname=HTTP/<ipaserver>@ATMOS-UCLA-EDU)'

Thanks,
Cody

[Rob Crittenden]

Cody Ashe-McNalley via FreeIPA-users wrote:
>  Sure, here it is with just the ipaserver hostname substituted.
>  
>  ldapsearch -LLL -Y GSSAPI -b cn=services,cn=accounts,dc=atmos,dc=ucla,dc=edu
> '(krbprincipalname=HTTP/<ipaserver>@ATMOS-UCLA-EDU)' 
Assuming it's not obfuscation, you need to replace the value for
<ipaserver> with the name of the server that's failing.

rob

[Cody Ashe-McNalley]

Hi Rob,

I used the actual ipa servers.  All 3 replicas (all with CA & DNS) show that same
output (with their hostname substituted).

Cody

[Cody Ashe-McNalley]

One of the replicas does NOT show the ca-error in `getcert list`.  Should I resync the
other 2 from that replica?

[Rob Crittenden]

Cody Ashe-McNalley via FreeIPA-users wrote:
>  One of the replicas does NOT show the ca-error in `getcert list`. 
> Should I resync the other 2 from that replica? 
It's curious that no conflict entries were found. I'd suggest looking
explicitly before doing a force re-init.

ldapsearch -x -D 'cn=directory manager' -W -b dc=example,dc=test
"(&(!(objectclass=nstombstone))(nsds5ReplConflict=*))"

At least rule them out. If it isn't a conflict then I'm not sure what is
causing the too many entries error.

rob

[Rob Crittenden]

Cody Ashe-McNalley via FreeIPA-users wrote:
>  Hi Rob,
>  
>  I used the actual ipa servers.  All 3 replicas (all with CA & DNS) show that same
> output (with their hostname substituted). 
Very strange. Ok, let's attack it another way:

$ ipa service-find --all --raw ldap/<server>

(Or HTTP). This should be more or less doing the equivalent ldapsearch.

rob

[Cody Ashe-McNalley]

Thanks, that worked.  The initial server has 2 usercertificate attributes, while the other
two replicas only have one.  Also the initial server doesn't have a krbcanonicalname.

-----------------
1 service matched
-----------------
  dn:
krbprincipalname=ldap/ipaserver.atmos.ucla.edu(a)ATMOS.UCLA.EDU,cn=services,cn=accounts,dc=atmos,dc=ucla,dc=edu
  krbprincipalname: ldap/ipaserver.atmos.ucla.edu(a)ATMOS.UCLA.EDU
  usercertificate: MII ... xU=
  usercertificate: MII ... w==
  subject: CN=ipaserver.atmos.ucla.edu,O=ATMOS.UCLA.EDU
  serial_number: 8
  serial_number_hex: 0x8
  issuer: CN=Certificate Authority,O=ATMOS.UCLA.EDU
  valid_not_before: Fri Jun 27 17:38:28 2014 UTC
  valid_not_after: Mon Jun 27 17:38:28 2016 UTC
  sha1_fingerprint: ...
  sha256_fingerprint: ...
  has_keytab: TRUE
  managedby:
fqdn=ipaserver.atmos.ucla.edu,cn=computers,cn=accounts,dc=atmos,dc=ucla,dc=edu
  ipaKrbPrincipalAlias: ldap/ipaserver.atmos.ucla.edu(a)ATMOS.UCLA.EDU
  ipaUniqueID: UUID
  krbExtraData: ...=
  krbLastPwdChange: 20140627174009Z
  krbLastSuccessfulAuth: 20201115230924Z
  krbPwdPolicyReference: cn=Default Service Password
Policy,cn=services,cn=accounts,dc=atmos,dc=ucla,dc=edu
  memberof: cn=replication managers,cn=sysaccounts,cn=etc,dc=atmos,dc=ucla,dc=edu
  objectClass: ipaobject
  objectClass: top
  objectClass: ipaservice
  objectClass: pkiuser
  objectClass: ipakrbprincipal
  objectClass: krbprincipal
  objectClass: krbprincipalaux
  objectClass: krbTicketPolicyAux
----------------------------
Number of entries returned 1
----------------------------