ldap_sasl_interactive_bind_s: Inappropriate authentication (48) - help debugging

[Radoslaw Kujawa]

Hello list.

I am trying to understand a reason for certificate-based authentication 
failure to one of my directory servers.

A have 3 IPA replicas running on CentOS 7. After running yum update on 
one of the nodes, PKI Tomcat failed to start. That system was not 
updated for last year or so, so the problem might have existed earlier 
and now was merely triggered by the update.

At first I suspected contents of /etc/pki/tomcat being wrong, however 
that does not seem to be the case.

Trying to understand the issue, I decided to try to replicate the 
authentication process "by hand":

I've set:
LDAPTLS_CACERTDIR="/etc/pki/pki-tomcat/alias"
LDAPTLS_CERT="NSS Certificate DB:subsystemCert cert-pki-ca"

However:
${NODE1}# ldapsearch -H ldaps://${NODE1}:636 -b "" -s base -Y EXTERNAL 
-Q -LLL dn namingcontexts
ldap_sasl_interactive_bind_s: Inappropriate authentication (48)
	additional info: missing client certificate

Interestingly, I _can_ authenticate this way to the other two nodes 
(from the same node where authentication to a local dir server does not 
work):

${NODE1}# ldapsearch -H ldaps://${NODE2}:636 -b "" -s base -Y EXTERNAL 
-Q -LLL dn namingcontexts
dn:
namingcontexts: cn=changelog
namingcontexts: dc=infra,dc=linker,dc=shop
namingcontexts: o=ipaca

I don't understand what does "missing client certificate" mean in this 
case, after all client configuration is identical, I am merely changing 
the server to which I connect.

I've investigated the contents of /tmp/openldap-tlsmc*/*/*pem and it 
seems to be correct (and same on all nodes), if I use ldapsearch -Y 
EXTERNAL.

${NODE1}# openssl x509 -in /tmp/openldap-tlsmc-alias--*/cert.pem -text
Certificate:
     Data:
         Version: 3 (0x2)
         Serial Number: 228 (0xe4)
     Signature Algorithm: sha256WithRSAEncryption
         Issuer: O=xxx CN=Certificate Authority
         Validity
             Not Before: Jul 12 08:49:04 2022 GMT
             Not After : Jul  1 08:49:04 2024 GMT
         Subject: O=xxx, CN=CA Subsystem
         Subject Public Key Info:
             Public Key Algorithm: rsaEncryption
                 Public-Key: (2048 bit)
                 Modulus:
...

I suspected that maybe replication is somehow messed up, but it seems to 
be working:
${NODE1}# ipa-replica-manage list `hostname` -v
${NODE2}: replica
   last update status: Error (0) Replica acquired successfully: 
Incremental update succeeded
   last update ended: 2023-05-30 14:21:10+00:00
${NODE3}: replica
   last update status: Error (0) Replica acquired successfully: 
Incremental update succeeded
   last update ended: 2023-05-30 14:21:10+00:00

As I understand, the cert-pki-ca certificate and private key is shared 
between the nodes and can be used to authenticate to any of directory 
servers?

What can possibly be different in directory servers between these nodes, 
which results in certificate authentication failing to one server, and 
succeeding to another?

Any hints would be appreciated.

Best regards,
Radoslaw