I found that 'kinit -n' was prompting me for the password for WELLKNOWN/ANONYMOUS@IPA.EXAMPLE.COM. This happened on everal, but not all clients.
After setting the environment variable KRB5_TRACE=/dev/stderr, the useful parts of the output of 'kinit -n' were:
[826240] 1693432177.150062: PKINIT loading CA certs and CRLs from FILE /var/lib/ipa-client/pki/kdc-ca-bundle.pem [826240] 1693432177.150063: PKINIT loading CA certs and CRLs from FILE /var/lib/ipa-client/pki/ca-bundle.pem [826240] 1693432177.150064: PKINIT client computed kdc-req-body checksum 14/265A6783F1767B3DB744B8ED1FE333185EFFEB7B [826240] 1693432177.150066: PKINIT client making DH request [826240] 1693432177.150067: Preauth module pkinit (16) (real) returned: 0/Success [826240] 1693432177.150068: Produced preauth for next request: PA-FX-COOKIE (133), PA-PK-AS-REQ (16) [826240] 1693432177.150069: Sending request (1565 bytes) to IPA.EXAMPLE.COM [826240] 1693432177.150070: Initiating TCP connection to stream 192.0.2.5:88 [826240] 1693432177.150071: Sending TCP request to stream 192.0.2.5:88 [826240] 1693432177.150072: Received answer (1609 bytes) from stream 192.0.2.5:88 [826240] 1693432177.150073: Terminating TCP connection to stream 192.0.2.5:88 [826240] 1693432177.150074: Response was from primary KDC [826240] 1693432177.150075: Processing preauth types: PA-PK-AS-REP (17), PA-PKINIT-KX (147) [826240] 1693432177.150076: Preauth module pkinit (147) (info) returned: 0/Success [826240] 1693432177.150077: PKINIT client could not verify DH reply [826240] 1693432177.150078: Preauth module pkinit (17) (real) returned: -1765328360/Preauthentication failed
Searching online for "PKINIT client could not verify DH reply" didn't come up with anything useful so I am writing this message to help anyone else who might experience a similar problem.
I noticed that all the failing clients were talking to the same IPA server. And on that server, 'kinit -n' failed with the same message.
As for the underlying cause: it turns out that there was a problem with the KDC's PKI certificate on the server.
[root@ipa5 ~]# getcert list -f /var/kerberos/krb5kdc/kdc.crt Number of certificates and requests being tracked: 12. Request ID '20221103090547': status: MONITORING stuck: no key pair storage: type=FILE,location='/var/kerberos/krb5kdc/kdc.key' certificate: type=FILE,location='/var/kerberos/krb5kdc/kdc.crt' CA: SelfSign issuer: CN=ipa5.ipa.example.com,O=IPA.EXAMPLE.COM subject: CN=ipa5.ipa.example.com,O=IPA.EXAMPLE.COM issued: 2023-07-08 18:12:32 UTC expires: 2024-07-08 18:12:32 UTC dns: ipa5.ipa.example.com principal name: krbtgt/IPA.EXAMPLE.COM@IPA.EXAMPLE.COM key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment eku: id-kp-serverAuth,id-pkinit-KPKdc certificate template/profile: KDCs_PKINIT_Certs profile: KDCs_PKINIT_Certs pre-save command: post-save command: /usr/libexec/ipa/certmonger/renew_kdc_cert track: yes auto-renew: yes
Note the CA is "SelfSign", which caused certmonger to generate a self-signed certificate on 2023-07-08 when it renewed, instead of contacting the IPA CA.
The fix was to run:
[root@ipa5 ~]# getcert start-tracking -f /var/kerberos/krb5kdc/kdc.crt -c IPA Request "20221103090547" modified.
[root@ipa5 ~]# getcert resubmit -w -v -f /var/kerberos/krb5kdc/kdc.crt Resubmitting "20221103090547" to "IPA". State GENERATING_CSR, stuck: no. State SUBMITTING, stuck: no. State READING_CERT, stuck: no. State POST_SAVED_CERT, stuck: no. State MONITORING, stuck: no.
[root@ipa5 ~]# getcert list -f /var/kerberos/krb5kdc/kdc.crt Number of certificates and requests being tracked: 12. Request ID '20221103090547': status: MONITORING stuck: no key pair storage: type=FILE,location='/var/kerberos/krb5kdc/kdc.key' certificate: type=FILE,location='/var/kerberos/krb5kdc/kdc.crt' CA: IPA issuer: CN=Certificate Authority,O=IPA.EXAMPLE.COM subject: CN=ipa5.ipa.example.com,O=IPA.EXAMPLE.COM issued: 2023-08-30 21:51:48 UTC expires: 2023-11-28 21:51:48 UTC dns: ipa5.ipa.example.com principal name: krbtgt/IPA.EXAMPLE.COM@IPA.EXAMPLE.COM key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment eku: id-kp-serverAuth,id-pkinit-KPKdc certificate template/profile: KDCs_PKINIT_Certs profile: KDCs_PKINIT_Certs pre-save command: post-save command: /usr/libexec/ipa/certmonger/renew_kdc_cert track: yes auto-renew: yes
After this, 'kinit -n' on the server and clients was successful again.
I've no earthly idea how the tracking request was set up with the wrong CA, but at least this appears to have fixed it.
There are some other differences between the tracking request on this server and another server (both of which are running fully-updated RHEL 8):
[root@ipa3 ~]# getcert list -f /var/kerberos/krb5kdc/kdc.crt Number of certificates and requests being tracked: 12. Request ID '20210423163239': status: MONITORING stuck: no key pair storage: type=FILE,location='/var/kerberos/krb5kdc/kdc.key' certificate: type=FILE,location='/var/kerberos/krb5kdc/kdc.crt' CA: IPA issuer: CN=Certificate Authority,O=IPA.EXAMPLE.COM subject: CN=ipa3.ipa.example.com,O=IPA.EXAMPLE.COM issued: 2023-07-29 16:33:03 UTC expires: 2023-10-27 16:33:03 UTC principal name: krbtgt/IPA.EXAMPLE.COM@IPA.EXAMPLE.COM key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment eku: id-kp-serverAuth,id-pkinit-KPKdc profile: KDCs_PKINIT_Certs pre-save command: post-save command: /usr/libexec/ipa/certmonger/renew_kdc_cert track: yes auto-renew: yes
On ipa3 the tracking request has no 'dns' line (subsequently, in the issued certificate there is no dnsName in the subjectAltName extension). There's also no 'certificate template/profile' line. Neither of these seem to affect any functionality, but I do wonder where the difference came from. Perhaps this is a chance in behaviour in freeipa between when ipa3 and ipa5 were installed?
The tracking request with the wrong CA wasn't picked up by ipa-healthcheck, I can file an issue about that if it's helpful.
Hopefully someone else could find this info useful.