Hi,
I've encountered some authentication problems with my FreeIPA installation, which I've traced to RA Agent certification auth problems. I've done typical steps to verify certs in LDAP and hit a wall. Please suggest further steps.
My setup is 2 masters on Fedora 34: freeipa-server-4.9.6-2.fc34.x86_64 pki-base-10.10.6-1.fc34.noarch
Steps I've done: First I noticed problems when enabling ACME: $ ipa-acme-manage enable Failed to authenticate to CA REST API The ipa-acme-manage command failed.
This is caused by authentication failure (401 return code), which I confirmed using curl:
$ curl -X POST https://kaitain.pipebreaker.pl:8443/acme/enable --cert /var/lib/ipa/ra-agent.pem --key /var/lib/ipa/ra-agent.key
<!doctype html><html lang="en"><head><title>HTTP Status 401 – Unauthorized</title><style type="text/css">body {font-family:Tahoma,Arial,sans-serif;} h1, h2, h3, b {color:white;background-color:#525D76;} h1 {font-size:22px;} h2 {font-size:16px;} h3 {font-size:14px;} p {font-size:12px;} a {color:black;} .line {height:1px;background-color:#525D76;border:none;}</style></head><body><h1>HTTP Status 401 – Unauthorized</h1><hr class="line" /><p><b>Type</b> Status Report</p><p><b>Description</b> The request has not been applied because it lacks valid authentication credentials for the target resource.</p><hr class="line" /><h3>Apache Tomcat/9.0.52</h3></body></html>
Errors from catalina.out:
02-Oct-2021 16:29:39.618 INFO [https-jsse-nio-8443-exec-6] com.netscape.cms.tomcat.ExternalAuthenticationValve.invoke ExternalAuthenticationValve: authType: null 02-Oct-2021 16:29:39.618 INFO [https-jsse-nio-8443-exec-6] com.netscape.cms.tomcat.ExternalAuthenticationValve.invoke ExternalAuthenticationValve: principal: null 02-Oct-2021 16:29:39.620 INFO [https-jsse-nio-8443-exec-6] com.netscape.cms.tomcat.AbstractPKIAuthenticator.doAuthenticate PKIAuthenticator: Authenticate with client certificate authentication 02-Oct-2021 16:29:39.620 INFO [https-jsse-nio-8443-exec-6] com.netscape.cms.tomcat.ProxyRealm.authenticate Authenticating certificate chain: 02-Oct-2021 16:29:39.621 INFO [https-jsse-nio-8443-exec-6] com.netscape.cms.tomcat.ProxyRealm.authenticate - CN=IPA RA,O=PIPEBREAKER.PL 02-Oct-2021 16:29:39.621 INFO [https-jsse-nio-8443-exec-6] com.netscape.cms.tomcat.ProxyRealm.authenticate - CN=Certificate Authority,O=PIPEBREAKER.PL 02-Oct-2021 16:29:39.624 INFO [https-jsse-nio-8443-exec-6] com.netscape.cms.tomcat.AbstractPKIAuthenticator.doAuthenticate PKIAuthenticator: Result: false
I've made sure that /var/lib/ipa/ra-agent.pem is the same as in LDAP. $ openssl x509 -in /var/lib/ipa/ra-agent.pem -noout -text | grep Serial Serial Number: 105 (0x69)
$ ldapsearch -o ldif_wrap=no -D "cn=directory manager" -W -b o=ipaca "(uid=ipara)" Enter LDAP Password: # extended LDIF # # LDAPv3 # base <o=ipaca> with scope subtree # filter: (uid=ipara) # requesting: ALL #
# ipara, people, ipaca dn: uid=ipara,ou=people,o=ipaca description: 2;105;CN=Certificate Authority,O=PIPEBREAKER.PL;CN=IPA RA,O=PIPEBREAKER.PL userCertificate;binary:: <SNIPPED> cn: ipara objectClass: top objectClass: person objectClass: organizationalPerson objectClass: inetOrgPerson objectClass: cmsuser usertype: agentType sn: ipara uid: ipara userstate: 1
# search result search: 2 result: 0 Success
# numResponses: 2 # numEntries: 1
Then SNIPPED portion is the same data as in /var/lib/ipa/ra-agent.pem. This is the same certificate; serial number matches, too.
Certificate is NOT expired: $ openssl x509 -in /var/lib/ipa/ra-agent.pem -noout -dates notBefore=Jun 16 04:34:42 2021 GMT notAfter=Jun 6 04:34:42 2023 GMT
What should I do next to resolve this authentication issue?
On Sat, Oct 02, 2021 at 04:38:34PM +0200, Tomasz Torcz via FreeIPA-users wrote:
$ ipa-acme-manage enable Failed to authenticate to CA REST API The ipa-acme-manage command failed.
Then SNIPPED portion is the same data as in /var/lib/ipa/ra-agent.pem. This is the same certificate; serial number matches, too.
What should I do next to resolve this authentication issue?
No ideas how to proceed? Most troubleshooting guides end at comparing certs on the filesystem and in LDAP. What's the next step?
Tomasz Torcz via FreeIPA-users wrote:
On Sat, Oct 02, 2021 at 04:38:34PM +0200, Tomasz Torcz via FreeIPA-users wrote:
$ ipa-acme-manage enable Failed to authenticate to CA REST API The ipa-acme-manage command failed.
Then SNIPPED portion is the same data as in /var/lib/ipa/ra-agent.pem. This is the same certificate; serial number matches, too.
What should I do next to resolve this authentication issue?
No ideas how to proceed? Most troubleshooting guides end at comparing certs on the filesystem and in LDAP. What's the next step?
I'd suggest trying ipa-healthcheck. It does these comparisons and more.
Does the RA cert work in other contexts? Does ipa cert-find work? Can you request a test certificate?
rob
On Tue, Oct 12, 2021 at 02:33:01PM -0400, Rob Crittenden via FreeIPA-users wrote:
Tomasz Torcz via FreeIPA-users wrote:
On Sat, Oct 02, 2021 at 04:38:34PM +0200, Tomasz Torcz via FreeIPA-users wrote:
$ ipa-acme-manage enable Failed to authenticate to CA REST API The ipa-acme-manage command failed.
Then SNIPPED portion is the same data as in /var/lib/ipa/ra-agent.pem. This is the same certificate; serial number matches, too.
What should I do next to resolve this authentication issue?
No ideas how to proceed? Most troubleshooting guides end at comparing certs on the filesystem and in LDAP. What's the next step?
I'd suggest trying ipa-healthcheck. It does these comparisons and more.
Run that, some minor warnings, but nothing about RA cert.
"source": "ipahealthcheck.ds.replication", "check": "ReplicationCheck", "result": "WARNING", "uuid": "10a0ad23-dc7a-4f43-a5f5-fac08c55a7b9", "when": "20211014120305Z", "duration": "0.392689", "kw": { "key": "DSREPLLE0002", "items": [ "Replication", "Conflict Entries" ], "msg": "There were 1 conflict entries found under the replication suffix "dc=pipebreaker,dc=pl"." }
Not much actionable info here.
{ "source": "ipahealthcheck.ipa.certs", "check": "IPACertTracking", "result": "WARNING", "uuid": "e4a545a3-ad22-4b8e-b4f0-70287eae98a9", "when": "20211014120309Z", "duration": "2.828753", "kw": { "key": "20141107202922", "msg": "certmonger tracking request {key} found and is not expected on an IPA master." } },
$ getcert list -i 20141107202922 Number of certificates and requests being tracked: 10. Request ID '20141107202922': status: MONITORING stuck: no key pair storage: type=FILE,location='/etc/pki/tls/private/kaitain.pipebreaker.pl.key' certificate: type=FILE,location='/etc/pki/tls/certs/kaitain.pipebreaker.pl.crt' CA: IPA issuer: CN=Certificate Authority,O=PIPEBREAKER.PL subject: CN=kaitain.pipebreaker.pl,O=PIPEBREAKER.PL issued: 2020-08-24 06:23:58 CEST expires: 2022-08-25 06:23:58 CEST dns: kaitain.pipebreaker.pl principal name: host/kaitain.pipebreaker.pl@PIPEBREAKER.PL key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment eku: id-kp-serverAuth,id-kp-clientAuth pre-save command: post-save command: track: yes auto-renew: yes
Looks fine, I have this cert/key configured in systemd-journal-upload service, this is not a part of FreeIPA.
{ "source": "ipahealthcheck.ipa.certs", "check": "IPACertDNSSAN", "result": "ERROR", "uuid": "87699232-f56d-47e4-802b-afab4f1d1b9b", "when": "20211014120312Z", "duration": "2.300274", "kw": { "key": "20200624045303", "hostname": "kaitain.pipebreaker.pl", "san": [], "ca": "IPA", "profile": "caIPAserviceCert", "msg": "Certificate request id {key} with profile {profile} for CA {ca} does not have a DNS SAN {san} matching name {hostname}" } } ]
$ getcert list -i 20200624045303 Number of certificates and requests being tracked: 10. Request ID '20200624045303': status: MONITORING stuck: no key pair storage: type=NSSDB,location='/etc/dirsrv/slapd-PIPEBREAKER-PL',nickname='Server-Cert',token='NSS Certificate DB',pinfile='/etc/dirsrv/slapd-PIPEBREAKER-PL/pwdfile.txt' certificate: type=NSSDB,location='/etc/dirsrv/slapd-PIPEBREAKER-PL',nickname='Server-Cert',token='NSS Certificate DB' CA: IPA issuer: CN=Certificate Authority,O=PIPEBREAKER.PL subject: CN=kaitain.pipebreaker.pl,O=PIPEBREAKER.PL issued: 2021-08-18 14:27:32 CEST expires: 2023-08-19 14:27:32 CEST principal name: ldap/kaitain.pipebreaker.pl@PIPEBREAKER.PL key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment eku: id-kp-serverAuth,id-kp-clientAuth profile: caIPAserviceCert pre-save command: post-save command: /usr/libexec/ipa/certmonger/restart_dirsrv PIPEBREAKER-PL track: yes auto-renew: y
Also looks fine, SAN requirement in certificates only appeared few years ago, after this particular server was installed. I doubt it is even used in context of LDAP connection.
Does the RA cert work in other contexts? Does ipa cert-find work? Can you request a test certificate?
It looks so:
root@kaitain ~$ ipa cert-find ipa: ERROR: did not receive Kerberos credentials
root@kaitain ~$ kinit admin Password for admin@PIPEBREAKER.PL:
root@kaitain ~$ ipa cert-find ipa: WARNING: Search result has been truncated: Configured size limit exceeded ------------------------ 100 certificates matched ------------------------ [ … hundred certificates listed … ]
When I check in WebUI I see that latest certificate was Issued On Tue Oct 05 20:27:05 2021 UTC
So it worked last week.
What would be next step?
Tomasz Torcz via FreeIPA-users wrote:
On Tue, Oct 12, 2021 at 02:33:01PM -0400, Rob Crittenden via FreeIPA-users wrote:
Tomasz Torcz via FreeIPA-users wrote:
On Sat, Oct 02, 2021 at 04:38:34PM +0200, Tomasz Torcz via FreeIPA-users wrote:
$ ipa-acme-manage enable Failed to authenticate to CA REST API The ipa-acme-manage command failed.
Then SNIPPED portion is the same data as in /var/lib/ipa/ra-agent.pem. This is the same certificate; serial number matches, too.
What should I do next to resolve this authentication issue?
No ideas how to proceed? Most troubleshooting guides end at comparing certs on the filesystem and in LDAP. What's the next step?
I'd suggest trying ipa-healthcheck. It does these comparisons and more.
Run that, some minor warnings, but nothing about RA cert.
"source": "ipahealthcheck.ds.replication", "check": "ReplicationCheck", "result": "WARNING", "uuid": "10a0ad23-dc7a-4f43-a5f5-fac08c55a7b9", "when": "20211014120305Z", "duration": "0.392689", "kw": { "key": "DSREPLLE0002", "items": [ "Replication", "Conflict Entries" ], "msg": "There were 1 conflict entries found under the replication suffix "dc=pipebreaker,dc=pl"." }
Not much actionable info here.
{ "source": "ipahealthcheck.ipa.certs", "check": "IPACertTracking", "result": "WARNING", "uuid": "e4a545a3-ad22-4b8e-b4f0-70287eae98a9", "when": "20211014120309Z", "duration": "2.828753", "kw": { "key": "20141107202922", "msg": "certmonger tracking request {key} found and is not expected on an IPA master." } },
$ getcert list -i 20141107202922 Number of certificates and requests being tracked: 10. Request ID '20141107202922': status: MONITORING stuck: no key pair storage: type=FILE,location='/etc/pki/tls/private/kaitain.pipebreaker.pl.key' certificate: type=FILE,location='/etc/pki/tls/certs/kaitain.pipebreaker.pl.crt' CA: IPA issuer: CN=Certificate Authority,O=PIPEBREAKER.PL subject: CN=kaitain.pipebreaker.pl,O=PIPEBREAKER.PL issued: 2020-08-24 06:23:58 CEST expires: 2022-08-25 06:23:58 CEST dns: kaitain.pipebreaker.pl principal name: host/kaitain.pipebreaker.pl@PIPEBREAKER.PL key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment eku: id-kp-serverAuth,id-kp-clientAuth pre-save command: post-save command: track: yes auto-renew: yes
Looks fine, I have this cert/key configured in systemd-journal-upload service, this is not a part of FreeIPA.
{ "source": "ipahealthcheck.ipa.certs", "check": "IPACertDNSSAN", "result": "ERROR", "uuid": "87699232-f56d-47e4-802b-afab4f1d1b9b", "when": "20211014120312Z", "duration": "2.300274", "kw": { "key": "20200624045303", "hostname": "kaitain.pipebreaker.pl", "san": [], "ca": "IPA", "profile": "caIPAserviceCert", "msg": "Certificate request id {key} with profile {profile} for CA {ca} does not have a DNS SAN {san} matching name {hostname}" } } ]
$ getcert list -i 20200624045303 Number of certificates and requests being tracked: 10. Request ID '20200624045303': status: MONITORING stuck: no key pair storage: type=NSSDB,location='/etc/dirsrv/slapd-PIPEBREAKER-PL',nickname='Server-Cert',token='NSS Certificate DB',pinfile='/etc/dirsrv/slapd-PIPEBREAKER-PL/pwdfile.txt' certificate: type=NSSDB,location='/etc/dirsrv/slapd-PIPEBREAKER-PL',nickname='Server-Cert',token='NSS Certificate DB' CA: IPA issuer: CN=Certificate Authority,O=PIPEBREAKER.PL subject: CN=kaitain.pipebreaker.pl,O=PIPEBREAKER.PL issued: 2021-08-18 14:27:32 CEST expires: 2023-08-19 14:27:32 CEST principal name: ldap/kaitain.pipebreaker.pl@PIPEBREAKER.PL key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment eku: id-kp-serverAuth,id-kp-clientAuth profile: caIPAserviceCert pre-save command: post-save command: /usr/libexec/ipa/certmonger/restart_dirsrv PIPEBREAKER-PL track: yes auto-renew: y
Also looks fine, SAN requirement in certificates only appeared few years ago, after this particular server was installed. I doubt it is even used in context of LDAP connection.
Does the RA cert work in other contexts? Does ipa cert-find work? Can you request a test certificate?
It looks so:
root@kaitain ~$ ipa cert-find ipa: ERROR: did not receive Kerberos credentials
root@kaitain ~$ kinit admin Password for admin@PIPEBREAKER.PL:
root@kaitain ~$ ipa cert-find ipa: WARNING: Search result has been truncated: Configured size limit exceeded
100 certificates matched
[ … hundred certificates listed … ]
When I check in WebUI I see that latest certificate was Issued On Tue Oct 05 20:27:05 2021 UTC
So it worked last week.
What would be next step?
So this shows that the RA certificate is fine. It looks like a group permission issue within the CA that the RA is not allowed to perform ACME actions.
Some things to check:
- uid=acme-<IPA SERVER HOSTNAME>,ou=people,o=ipaca and uid=ipara,ou=People,o=ipaca are both uniqueMember attributes of cn=Enterprise ACME Administrators,ou=groups,o=ipaca - the entry id=acme-<IPA SERVER HOSTNAME>,ou=people,o=ipaca exists - In cn=aclResources,o=ipaca there is the value: resourceACLS: certServer.ca.certs:execute:allow (execute) group="Enterprise ACME Administrators":ACME Agents may execute cert operations
rob
On Fri, Oct 15, 2021 at 02:04:42PM -0400, Rob Crittenden via FreeIPA-users wrote:
Tomasz Torcz via FreeIPA-users wrote:
On Tue, Oct 12, 2021 at 02:33:01PM -0400, Rob Crittenden via FreeIPA-users wrote:
Tomasz Torcz via FreeIPA-users wrote:
On Sat, Oct 02, 2021 at 04:38:34PM +0200, Tomasz Torcz via FreeIPA-users wrote:
$ ipa-acme-manage enable Failed to authenticate to CA REST API The ipa-acme-manage command failed.
No ideas how to proceed? Most troubleshooting guides end at comparing certs on the filesystem and in LDAP. What's the next step?
So this shows that the RA certificate is fine. It looks like a group permission issue within the CA that the RA is not allowed to perform ACME actions.
Some things to check:
All below seem to be correct:
- uid=acme-<IPA SERVER HOSTNAME>,ou=people,o=ipaca and
uid=ipara,ou=People,o=ipaca are both uniqueMember attributes of cn=Enterprise ACME Administrators,ou=groups,o=ipaca
# base <cn=Enterprise ACME Administrators,ou=groups,o=ipaca> with scope # subtree # filter: (objectclass=*) # requesting: uniqueMember #
# Enterprise ACME Administrators, groups, ipaca dn: cn=Enterprise ACME Administrators,ou=groups,o=ipaca uniqueMember: uid=acme-kaitain.pipebreaker.pl,ou=people,o=ipaca uniqueMember: uid=ipara,ou=people,o=ipaca uniqueMember: uid=acme-okda.pipebreaker.pl,ou=people,o=ipaca
- the entry id=acme-<IPA SERVER HOSTNAME>,ou=people,o=ipaca exists
There is no entry with id=, but there is one with uid= (I assume you made a typo):
# acme-kaitain.pipebreaker.pl, people, ipaca dn: uid=acme-kaitain.pipebreaker.pl,ou=people,o=ipaca objectClass: top objectClass: person objectClass: organizationalPerson objectClass: inetOrgPerson objectClass: cmsuser uid: acme-kaitain.pipebreaker.pl cn: acme-kaitain.pipebreaker.pl sn: acme-kaitain.pipebreaker.pl usertype: agentType userstate: 1 userPassword:: …
- In cn=aclResources,o=ipaca there is the value:
resourceACLS: certServer.ca.certs:execute:allow (execute) group="Enterprise ACME Administrators":ACME Agents may execute cert operations
$ ldapsrch -b cn=aclResources,o=ipaca resourceACLs | grep ACME Enter LDAP Password: resourceACLs: certServer.ca.certs:execute:allow (execute) group="Enterprise ACME Administrators":ACME Agents may execute cert operations
So everything looks to be in order. Maybe there is a way to increase logging in com.netscape.cms.tomcat.AbstractPKIAuthenticator.doAuthenticate PKIAuthenticator ?
Tomasz Torcz via FreeIPA-users wrote:
On Fri, Oct 15, 2021 at 02:04:42PM -0400, Rob Crittenden via FreeIPA-users wrote:
Tomasz Torcz via FreeIPA-users wrote:
On Tue, Oct 12, 2021 at 02:33:01PM -0400, Rob Crittenden via FreeIPA-users wrote:
Tomasz Torcz via FreeIPA-users wrote:
On Sat, Oct 02, 2021 at 04:38:34PM +0200, Tomasz Torcz via FreeIPA-users wrote:
$ ipa-acme-manage enable Failed to authenticate to CA REST API The ipa-acme-manage command failed.
No ideas how to proceed? Most troubleshooting guides end at comparing certs on the filesystem and in LDAP. What's the next step?
So this shows that the RA certificate is fine. It looks like a group permission issue within the CA that the RA is not allowed to perform ACME actions.
Some things to check:
All below seem to be correct:
- uid=acme-<IPA SERVER HOSTNAME>,ou=people,o=ipaca and
uid=ipara,ou=People,o=ipaca are both uniqueMember attributes of cn=Enterprise ACME Administrators,ou=groups,o=ipaca
# base <cn=Enterprise ACME Administrators,ou=groups,o=ipaca> with scope # subtree # filter: (objectclass=*) # requesting: uniqueMember #
# Enterprise ACME Administrators, groups, ipaca dn: cn=Enterprise ACME Administrators,ou=groups,o=ipaca uniqueMember: uid=acme-kaitain.pipebreaker.pl,ou=people,o=ipaca uniqueMember: uid=ipara,ou=people,o=ipaca uniqueMember: uid=acme-okda.pipebreaker.pl,ou=people,o=ipaca
- the entry id=acme-<IPA SERVER HOSTNAME>,ou=people,o=ipaca exists
There is no entry with id=, but there is one with uid= (I assume you made a typo):
# acme-kaitain.pipebreaker.pl, people, ipaca dn: uid=acme-kaitain.pipebreaker.pl,ou=people,o=ipaca objectClass: top objectClass: person objectClass: organizationalPerson objectClass: inetOrgPerson objectClass: cmsuser uid: acme-kaitain.pipebreaker.pl cn: acme-kaitain.pipebreaker.pl sn: acme-kaitain.pipebreaker.pl usertype: agentType userstate: 1 userPassword:: …
- In cn=aclResources,o=ipaca there is the value:
resourceACLS: certServer.ca.certs:execute:allow (execute) group="Enterprise ACME Administrators":ACME Agents may execute cert operations
$ ldapsrch -b cn=aclResources,o=ipaca resourceACLs | grep ACME Enter LDAP Password: resourceACLs: certServer.ca.certs:execute:allow (execute) group="Enterprise ACME Administrators":ACME Agents may execute cert operations
So everything looks to be in order. Maybe there is a way to increase logging in com.netscape.cms.tomcat.AbstractPKIAuthenticator.doAuthenticate PKIAuthenticator ?
I don't know. Endi, what would you suggest here?
thanks
rob
Hi,
I think error 401 means that the client cert could not be mapped to the user in DS.
Could you check the uid=ipara,ou=people,o=ipaca to make sure that the userCertificate and the description attributes contain the right certificate?
You can also try setting the log level to INFO or FINE to see the authentication process on the server side: https://github.com/dogtagpki/pki/wiki/Configuring-Server-Logging
On Wed, Oct 20, 2021 at 08:40:30PM -0500, Endi Dewata via FreeIPA-users wrote:
Hi,
I think error 401 means that the client cert could not be mapped to the user in DS.
Could you check the uid=ipara,ou=people,o=ipaca to make sure that the userCertificate and the description attributes contain the right certificate?
That was the first thing I've checked. userCertificate:: (after base64 decoding) is the same as /var/lib/ipa/ra-agent.pem - the same description, fingerprint, etc. openssl x509 -serial return "69" for both, and LDAP contains: description: 2;105;CN=Certificate Authority,O=PIPEBREAKER.PL;CN=IPA RA,O=PIPEBREAKER.PL
105 (dec) == 69 (hex) so this is correct, too.
You can also try setting the log level to INFO or FINE to see the authentication process on the server side: https://github.com/dogtagpki/pki/wiki/Configuring-Server-Logging
This is something! There are new lines between starting certificate authentication and returning failure. First I thought there are libraries missing, but in the end all finish with "Loading class from parent":
FINE: Calling authenticate() INFO: PKIAuthenticator: Authenticate with client certificate authentication INFO: Authenticating certificate chain: INFO: - CN=IPA RA,O=PIPEBREAKER.PL INFO: - CN=Certificate Authority,O=PIPEBREAKER.PL FINE: loadClass(org.mozilla.jss.netscape.security.util.Cert, false) FINE: Searching local repositories FINE: findClass(org.mozilla.jss.netscape.security.util.Cert) FINE: --> Returning ClassNotFoundException FINE: Delegating to parent classloader at end: java.net.URLClassLoader@5fcfe4b2 FINE: Loading class from parent FINE: loadClass(netscape.ldap.LDAPSearchResults, false) FINE: Searching local repositories FINE: findClass(netscape.ldap.LDAPSearchResults) FINE: --> Returning ClassNotFoundException FINE: Delegating to parent classloader at end: java.net.URLClassLoader@5fcfe4b2 FINE: Loading class from parent FINE: loadClass(netscape.ldap.LDAPEntry, false) FINE: Searching local repositories FINE: findClass(netscape.ldap.LDAPEntry) FINE: --> Returning ClassNotFoundException FINE: Delegating to parent classloader at end: java.net.URLClassLoader@5fcfe4b2 FINE: Loading class from parent FINE: loadClass(com.netscape.cmscore.usrgrp.User, false) FINE: Searching local repositories FINE: findClass(com.netscape.cmscore.usrgrp.User) FINE: Loading class from local repository FINE: loadClass(netscape.ldap.LDAPAttribute, false) FINE: Searching local repositories FINE: findClass(netscape.ldap.LDAPAttribute) FINE: --> Returning ClassNotFoundException FINE: Delegating to parent classloader at end: java.net.URLClassLoader@5fcfe4b2 FINE: Loading class from parent INFO: PKIAuthenticator: Result: false FINE: Failed authenticate() test
Second invocation of "pki-acme-manage status" do not generate those class messages:
FINE: Calling hasUserDataPermission() FINE: User data constraint already satisfied FINE: Calling authenticate() INFO: PKIAuthenticator: Authenticate with client certificate authentication INFO: Authenticating certificate chain: INFO: - CN=IPA RA,O=PIPEBREAKER.PL INFO: - CN=Certificate Authority,O=PIPEBREAKER.PL INFO: PKIAuthenticator: Result: false FINE: Failed authenticate() test FINE: JSSEngine: wrap(ssl_fd=org.mozilla.jss.nss.SSLFDProxy[1522605810@00079ea974550000])
On Thu, Oct 21, 2021 at 5:36 AM Tomasz Torcz via FreeIPA-users < freeipa-users@lists.fedorahosted.org> wrote:
On Wed, Oct 20, 2021 at 08:40:30PM -0500, Endi Dewata via FreeIPA-users wrote:
Hi,
I think error 401 means that the client cert could not be mapped to the user in DS.
Could you check the uid=ipara,ou=people,o=ipaca to make sure that the userCertificate and the description attributes contain the right certificate?
That was the first thing I've checked. userCertificate:: (after base64 decoding) is the same as /var/lib/ipa/ra-agent.pem - the same description, fingerprint, etc. openssl x509 -serial return "69" for both, and LDAP contains: description: 2;105;CN=Certificate Authority,O=PIPEBREAKER.PL;CN=IPA RA,O= PIPEBREAKER.PL
105 (dec) == 69 (hex) so this is correct, too.
You can also try setting the log level to INFO or FINE to see the authentication process on the server side: https://github.com/dogtagpki/pki/wiki/Configuring-Server-Logging
This is something! There are new lines between starting certificate authentication and returning failure. First I thought there are libraries missing, but in the end all finish with "Loading class from parent":
FINE: Calling authenticate() INFO: PKIAuthenticator: Authenticate with client certificate authentication INFO: Authenticating certificate chain: INFO: - CN=IPA RA,O=PIPEBREAKER.PL INFO: - CN=Certificate Authority,O=PIPEBREAKER.PL FINE: loadClass(org.mozilla.jss.netscape.security.util.Cert, false) FINE: Searching local repositories FINE: findClass(org.mozilla.jss.netscape.security.util.Cert) FINE: --> Returning ClassNotFoundException FINE: Delegating to parent classloader at end: java.net.URLClassLoader@5fcfe4b2 FINE: Loading class from parent FINE: loadClass(netscape.ldap.LDAPSearchResults, false) FINE: Searching local repositories FINE: findClass(netscape.ldap.LDAPSearchResults) FINE: --> Returning ClassNotFoundException FINE: Delegating to parent classloader at end: java.net.URLClassLoader@5fcfe4b2 FINE: Loading class from parent FINE: loadClass(netscape.ldap.LDAPEntry, false) FINE: Searching local repositories FINE: findClass(netscape.ldap.LDAPEntry) FINE: --> Returning ClassNotFoundException FINE: Delegating to parent classloader at end: java.net.URLClassLoader@5fcfe4b2 FINE: Loading class from parent FINE: loadClass(com.netscape.cmscore.usrgrp.User, false) FINE: Searching local repositories FINE: findClass(com.netscape.cmscore.usrgrp.User) FINE: Loading class from local repository FINE: loadClass(netscape.ldap.LDAPAttribute, false) FINE: Searching local repositories FINE: findClass(netscape.ldap.LDAPAttribute) FINE: --> Returning ClassNotFoundException FINE: Delegating to parent classloader at end: java.net.URLClassLoader@5fcfe4b2 FINE: Loading class from parent INFO: PKIAuthenticator: Result: false FINE: Failed authenticate() test
Second invocation of "pki-acme-manage status" do not generate those class messages:
FINE: Calling hasUserDataPermission() FINE: User data constraint already satisfied FINE: Calling authenticate() INFO: PKIAuthenticator: Authenticate with client certificate authentication INFO: Authenticating certificate chain: INFO: - CN=IPA RA,O=PIPEBREAKER.PL INFO: - CN=Certificate Authority,O=PIPEBREAKER.PL INFO: PKIAuthenticator: Result: false FINE: Failed authenticate() test FINE: JSSEngine: wrap(ssl_fd=org.mozilla.jss.nss.SSLFDProxy[1522605810@00079ea974550000])
I think the class loading messages above were generated by Tomcat. That's probably how it resolves the classes, so I don't think that's an issue.
Could you raise the debug level in the CA subsystem too? https://github.com/dogtagpki/pki/wiki/Configuring-Subsystem-Debug-Log The authenticator uses the LDAP connection in the CA to find the user in DS, so there might be an issue there.
On Thu, Oct 21, 2021 at 12:44 PM Endi Dewata edewata@redhat.com wrote:
I think the class loading messages above were generated by Tomcat. That's probably how it resolves the classes, so I don't think that's an issue.
Could you raise the debug level in the CA subsystem too? https://github.com/dogtagpki/pki/wiki/Configuring-Subsystem-Debug-Log The authenticator uses the LDAP connection in the CA to find the user in DS, so there might be an issue there.
Actually it's a bit different for ACME. I just updated the above page. Could you raise the logging level in ACME too?
ACME also has a realm configuration: https://github.com/dogtagpki/pki/blob/master/docs/installation/acme/Configur... https://github.com/dogtagpki/pki/blob/master/docs/installation/acme/Configur... so there could be an issue there.
On Thu, Oct 21, 2021 at 01:25:09PM -0500, Endi Dewata via FreeIPA-users wrote:
On Thu, Oct 21, 2021 at 12:44 PM Endi Dewata edewata@redhat.com wrote:
I think the class loading messages above were generated by Tomcat. That's probably how it resolves the classes, so I don't think that's an issue.
Could you raise the debug level in the CA subsystem too? https://github.com/dogtagpki/pki/wiki/Configuring-Subsystem-Debug-Log The authenticator uses the LDAP connection in the CA to find the user in DS, so there might be an issue there.
Actually it's a bit different for ACME. I just updated the above page. Could you raise the logging level in ACME too?
I've increased all the loglevels to FINE, there are no additional logs: INFO: PKIAuthenticator: Authenticate with client certificate authentication INFO: Authenticating certificate chain: INFO: - CN=IPA RA,O=PIPEBREAKER.PL INFO: - CN=Certificate Authority,O=PIPEBREAKER.PL INFO: PKIAuthenticator: Result: false
and in ca/debug-<date>.log:
2021-10-22 19:55:32 [Timer-0] FINE: LdapBoundConnFactory: number of connections: 3 2021-10-22 19:59:16 [https-jsse-nio-8443-exec-3] FINE: Certificates: 2021-10-22 19:59:16 [https-jsse-nio-8443-exec-3] FINE: - CN=IPA RA,O=PIPEBREAKER.PL 2021-10-22 19:59:16 [https-jsse-nio-8443-exec-3] FINE: parent: CN=Certificate Authority,O=PIPEBREAKER.PL 2021-10-22 19:59:16 [https-jsse-nio-8443-exec-3] FINE: - CN=Certificate Authority,O=PIPEBREAKER.PL 2021-10-22 19:59:16 [https-jsse-nio-8443-exec-3] FINE: child: CN=IPA RA,O=PIPEBREAKER.PL
dirsrv log shows 1 certificate is being found: [22/Oct/2021:20:05:35.119328765 +0200] conn=4469 op=9 SRCH base="ou=people,o=ipaca" scope=1 filter="(description=2;105;CN=Certificate Authority,O=PIPEBREAKER.PL;CN=IP A RA,O=PIPEBREAKER.PL)" attrs=ALL [22/Oct/2021:20:05:35.119963856 +0200] conn=4469 op=9 RESULT err=0 tag=101 nentries=1 wtime=0.000231485 optime=0.000638726 etime=0.000867770 [22/Oct/2021:20:05:35.277708077 +0200] conn=4508 op=3 UNBIND
BUT in acme/debug.log:
2021-10-22 20:01:12 [https-jsse-nio-8443-exec-4] FINE: Looking up certificates 2021-10-22 20:01:12 [https-jsse-nio-8443-exec-4] INFO: Authenticating user with client certificate 2021-10-22 20:01:12 [https-jsse-nio-8443-exec-4] INFO: Finding user by cert: 2021-10-22 20:01:12 [https-jsse-nio-8443-exec-4] INFO: - base DN: ou=people,o=ipaca 2021-10-22 20:01:12 [https-jsse-nio-8443-exec-4] INFO: - filter: description=2;105;CN=Certificate Authority,O=PIPEBREAKER.PL;CN=IPA RA,O=PIPEBREAKER.PL 2021-10-22 20:01:12 [https-jsse-nio-8443-exec-4] INFO: User: uid=ipara,ou=people,o=ipaca 2021-10-22 20:01:12 [https-jsse-nio-8443-exec-4] FINE: Realm.authenticate() returned false
So a problem with realm?
ACME also has a realm configuration: https://github.com/dogtagpki/pki/blob/master/docs/installation/acme/Configur... https://github.com/dogtagpki/pki/blob/master/docs/installation/acme/Configur... so there could be an issue there.
This look to be configured, but I found a possible discrepancy in "password":
$ cat /etc/pki/pki-tomcat/acme/realm.conf # VERSION 2 - DO NOT REMOVE THIS LINE authType=BasicAuth class=org.dogtagpki.acme.realm.DSRealm groupsDN=ou=groups,o=ipaca usersDN=ou=people,o=ipaca url=ldaps://kaitain.pipebreaker.pl:636 configFile=/etc/pki/pki-tomcat/ca/CS.cfg username=acme-kaitain.pipebreaker.pl password=<40-character long text string>
While userPassword:: field of uid=acme-kaitain.pipebreaker.pl,ou=people,o=ipaca contains very long base64 string, which decodes to 447 string starting with {PBKDF2_SHA256}. How to make sure it's corresponds to the same value?
Tomasz Torcz via FreeIPA-users wrote:
On Thu, Oct 21, 2021 at 01:25:09PM -0500, Endi Dewata via FreeIPA-users wrote:
On Thu, Oct 21, 2021 at 12:44 PM Endi Dewata edewata@redhat.com wrote:
I think the class loading messages above were generated by Tomcat. That's probably how it resolves the classes, so I don't think that's an issue.
Could you raise the debug level in the CA subsystem too? https://github.com/dogtagpki/pki/wiki/Configuring-Subsystem-Debug-Log The authenticator uses the LDAP connection in the CA to find the user in DS, so there might be an issue there.
Actually it's a bit different for ACME. I just updated the above page. Could you raise the logging level in ACME too?
I've increased all the loglevels to FINE, there are no additional logs: INFO: PKIAuthenticator: Authenticate with client certificate authentication INFO: Authenticating certificate chain: INFO: - CN=IPA RA,O=PIPEBREAKER.PL INFO: - CN=Certificate Authority,O=PIPEBREAKER.PL INFO: PKIAuthenticator: Result: false
and in ca/debug-<date>.log:
2021-10-22 19:55:32 [Timer-0] FINE: LdapBoundConnFactory: number of connections: 3 2021-10-22 19:59:16 [https-jsse-nio-8443-exec-3] FINE: Certificates: 2021-10-22 19:59:16 [https-jsse-nio-8443-exec-3] FINE: - CN=IPA RA,O=PIPEBREAKER.PL 2021-10-22 19:59:16 [https-jsse-nio-8443-exec-3] FINE: parent: CN=Certificate Authority,O=PIPEBREAKER.PL 2021-10-22 19:59:16 [https-jsse-nio-8443-exec-3] FINE: - CN=Certificate Authority,O=PIPEBREAKER.PL 2021-10-22 19:59:16 [https-jsse-nio-8443-exec-3] FINE: child: CN=IPA RA,O=PIPEBREAKER.PL
dirsrv log shows 1 certificate is being found: [22/Oct/2021:20:05:35.119328765 +0200] conn=4469 op=9 SRCH base="ou=people,o=ipaca" scope=1 filter="(description=2;105;CN=Certificate Authority,O=PIPEBREAKER.PL;CN=IP A RA,O=PIPEBREAKER.PL)" attrs=ALL [22/Oct/2021:20:05:35.119963856 +0200] conn=4469 op=9 RESULT err=0 tag=101 nentries=1 wtime=0.000231485 optime=0.000638726 etime=0.000867770 [22/Oct/2021:20:05:35.277708077 +0200] conn=4508 op=3 UNBIND
BUT in acme/debug.log:
2021-10-22 20:01:12 [https-jsse-nio-8443-exec-4] FINE: Looking up certificates 2021-10-22 20:01:12 [https-jsse-nio-8443-exec-4] INFO: Authenticating user with client certificate 2021-10-22 20:01:12 [https-jsse-nio-8443-exec-4] INFO: Finding user by cert: 2021-10-22 20:01:12 [https-jsse-nio-8443-exec-4] INFO: - base DN: ou=people,o=ipaca 2021-10-22 20:01:12 [https-jsse-nio-8443-exec-4] INFO: - filter: description=2;105;CN=Certificate Authority,O=PIPEBREAKER.PL;CN=IPA RA,O=PIPEBREAKER.PL 2021-10-22 20:01:12 [https-jsse-nio-8443-exec-4] INFO: User: uid=ipara,ou=people,o=ipaca 2021-10-22 20:01:12 [https-jsse-nio-8443-exec-4] FINE: Realm.authenticate() returned false
So a problem with realm?
ACME also has a realm configuration: https://github.com/dogtagpki/pki/blob/master/docs/installation/acme/Configur... https://github.com/dogtagpki/pki/blob/master/docs/installation/acme/Configur... so there could be an issue there.
This look to be configured, but I found a possible discrepancy in "password":
$ cat /etc/pki/pki-tomcat/acme/realm.conf # VERSION 2 - DO NOT REMOVE THIS LINE authType=BasicAuth class=org.dogtagpki.acme.realm.DSRealm groupsDN=ou=groups,o=ipaca usersDN=ou=people,o=ipaca url=ldaps://kaitain.pipebreaker.pl:636 configFile=/etc/pki/pki-tomcat/ca/CS.cfg username=acme-kaitain.pipebreaker.pl password=<40-character long text string>
While userPassword:: field of uid=acme-kaitain.pipebreaker.pl,ou=people,o=ipaca contains very long base64 string, which decodes to 447 string starting with {PBKDF2_SHA256}. How to make sure it's corresponds to the same value?
This is the password for the username in the file. It is basically unused by IPA as IPA uses client auth with the RA agent certificate.
rob
On Mon, Oct 25, 2021 at 7:42 AM Rob Crittenden via FreeIPA-users < freeipa-users@lists.fedorahosted.org> wrote:
Tomasz Torcz via FreeIPA-users wrote:
ACME also has a realm configuration:
https://github.com/dogtagpki/pki/blob/master/docs/installation/acme/Configur...
https://github.com/dogtagpki/pki/blob/master/docs/installation/acme/Configur...
so there could be an issue there.
This look to be configured, but I found a possible discrepancy in
"password":
$ cat /etc/pki/pki-tomcat/acme/realm.conf # VERSION 2 - DO NOT REMOVE THIS LINE authType=BasicAuth class=org.dogtagpki.acme.realm.DSRealm groupsDN=ou=groups,o=ipaca usersDN=ou=people,o=ipaca url=ldaps://kaitain.pipebreaker.pl:636 configFile=/etc/pki/pki-tomcat/ca/CS.cfg username=acme-kaitain.pipebreaker.pl password=<40-character long text string>
While userPassword:: field of uid=acme-kaitain.pipebreaker.pl
,ou=people,o=ipaca
contains very long base64 string, which decodes to 447 string starting with {PBKDF2_SHA256}. How to make sure it's corresponds to the same value?
This is the password for the username in the file. It is basically unused by IPA as IPA uses client auth with the RA agent certificate.
rob
Looks like the realm is configured with BasicAuth, so it should be using bindDN and bindPassword params as described here: https://github.com/dogtagpki/pki/blob/master/docs/installation/acme/Configur...
If you want to use SslClientAuth, I think you would need to specify the nickname param: https://github.com/dogtagpki/pki/blob/master/base/acme/src/main/java/org/dog... https://github.com/dogtagpki/pki/blob/master/base/server/src/main/java/com/n... https://github.com/dogtagpki/pki/wiki/Configuring-Client-Certificate-Authent...
But IIRC in IPA case it's configured to reuse the internaldb connection defined in CS.cfg so these params don't need to be specified again. Is there a working IPA instance with ACME that can be compared against?
On Mon, Oct 25, 2021 at 10:09 AM Endi Dewata edewata@redhat.com wrote:
On Mon, Oct 25, 2021 at 7:42 AM Rob Crittenden via FreeIPA-users < freeipa-users@lists.fedorahosted.org> wrote:
Tomasz Torcz via FreeIPA-users wrote:
ACME also has a realm configuration:
https://github.com/dogtagpki/pki/blob/master/docs/installation/acme/Configur...
https://github.com/dogtagpki/pki/blob/master/docs/installation/acme/Configur...
so there could be an issue there.
This look to be configured, but I found a possible discrepancy in
"password":
$ cat /etc/pki/pki-tomcat/acme/realm.conf # VERSION 2 - DO NOT REMOVE THIS LINE authType=BasicAuth class=org.dogtagpki.acme.realm.DSRealm groupsDN=ou=groups,o=ipaca usersDN=ou=people,o=ipaca url=ldaps://kaitain.pipebreaker.pl:636 configFile=/etc/pki/pki-tomcat/ca/CS.cfg username=acme-kaitain.pipebreaker.pl password=<40-character long text string>
While userPassword:: field of uid=acme-kaitain.pipebreaker.pl
,ou=people,o=ipaca
contains very long base64 string, which decodes to 447 string starting with {PBKDF2_SHA256}. How to make sure it's corresponds to the same value?
This is the password for the username in the file. It is basically unused by IPA as IPA uses client auth with the RA agent certificate.
rob
Looks like the realm is configured with BasicAuth, so it should be using bindDN and bindPassword params as described here:
https://github.com/dogtagpki/pki/blob/master/docs/installation/acme/Configur...
If you want to use SslClientAuth, I think you would need to specify the nickname param:
https://github.com/dogtagpki/pki/blob/master/base/acme/src/main/java/org/dog...
https://github.com/dogtagpki/pki/blob/master/base/server/src/main/java/com/n...
https://github.com/dogtagpki/pki/wiki/Configuring-Client-Certificate-Authent...
But IIRC in IPA case it's configured to reuse the internaldb connection defined in CS.cfg so these params don't need to be specified again. Is there a working IPA instance with ACME that can be compared against?
Yeah, the realm config has a configFile param, so it will ignore the other params above, and use the params from CS.cfg instead: https://github.com/dogtagpki/pki/blob/master/base/acme/src/main/java/org/dog... https://github.com/dogtagpki/pki/blob/master/base/acme/src/main/java/org/dog...
On Mon, Oct 25, 2021 at 10:09:56AM -0500, Endi Dewata via FreeIPA-users wrote:
On Mon, Oct 25, 2021 at 7:42 AM Rob Crittenden via FreeIPA-users < freeipa-users@lists.fedorahosted.org> wrote:
Tomasz Torcz via FreeIPA-users wrote:
ACME also has a realm configuration:
https://github.com/dogtagpki/pki/blob/master/docs/installation/acme/Configur...
https://github.com/dogtagpki/pki/blob/master/docs/installation/acme/Configur...
so there could be an issue there.
But IIRC in IPA case it's configured to reuse the internaldb connection defined in CS.cfg so these params don't need to be specified again. Is there a working IPA instance with ACME that can be compared against?
So I did a clean install of Fedora 34 and FreeIPA. Clean install works as expected. I did comparison between fresh and mine install, there were discrepancies I mostly fixed, but it didn't change my problem. Failure looks like that in logs (pki-tomcat/acme/debug-<data>.log):
2021-11-03 18:43:07 [https-jsse-nio-8443-exec-12] INFO: Finding user by cert: 2021-11-03 18:43:07 [https-jsse-nio-8443-exec-12] INFO: - base DN: ou=people,o=ipaca 2021-11-03 18:43:07 [https-jsse-nio-8443-exec-12] INFO: - filter: description=2;105;CN=Certificate Authority,O=PIPEBREAKER.PL;CN=IPA RA,O=PIPEBREAKER.PL 2021-11-03 18:43:07 [https-jsse-nio-8443-exec-12] INFO: User: uid=ipara,ou=people,o=ipaca 2021-11-03 18:43:08 [https-jsse-nio-8443-exec-12] FINE: Realm.authenticate() returned false
While on _fresh install_ correct log looks like:
2021-10-31 13:51:47 [https-jsse-nio-8443-exec-13] INFO: Authenticating user with client certificate 2021-10-31 13:51:48 [https-jsse-nio-8443-exec-13] INFO: Finding user by cert: 2021-10-31 13:51:48 [https-jsse-nio-8443-exec-13] INFO: - base DN: ou=people,o=ipaca 2021-10-31 13:51:48 [https-jsse-nio-8443-exec-13] INFO: - filter: description=2;7;CN=Certificate Authority,O=IPADEV.PIPEBREAKER.PL;CN=IPA RA,O=IPADEV.PIPEBREAKER.PL 2021-10-31 13:51:48 [https-jsse-nio-8443-exec-13] INFO: User: uid=ipara,ou=people,o=ipaca 2021-10-31 13:51:48 [https-jsse-nio-8443-exec-13] INFO: Getting user roles: 2021-10-31 13:51:48 [https-jsse-nio-8443-exec-13] INFO: - base DN: ou=groups,o=ipaca 2021-10-31 13:51:48 [https-jsse-nio-8443-exec-13] INFO: - filter: uniqueMember=uid=ipara,ou=people,o=ipaca 2021-10-31 13:51:48 [https-jsse-nio-8443-exec-13] INFO: Roles: 2021-10-31 13:51:48 [https-jsse-nio-8443-exec-13] INFO: - cn=Certificate Manager Agents,ou=groups,o=ipaca 2021-10-31 13:51:48 [https-jsse-nio-8443-exec-13] INFO: - cn=Registration Manager Agents,ou=groups,o=ipaca 2021-10-31 13:51:48 [https-jsse-nio-8443-exec-13] INFO: - cn=Enterprise ACME Administrators,ou=groups,o=ipaca 2021-10-31 13:51:48 [https-jsse-nio-8443-exec-13] INFO: Initializing ACMEApplication 2021-10-31 13:51:48 [https-jsse-nio-8443-exec-13] INFO: ACMELoginService: Session: 3DBCD2FB21ADFDD04ADC518C97AA07B4 2021-10-31 13:51:48 [https-jsse-nio-8443-exec-13] INFO: ACMELoginService: Principal: GenericPrincipal[ipara(Certificate Manager Agents,Enterprise ACME Administrators,Registration Manager Agents,)] 2021-10-31 13:51:48 [https-jsse-nio-8443-exec-13] INFO: ACMELoginService: Principal: ipara 2021-10-31 13:51:48 [https-jsse-nio-8443-exec-13] INFO: ACMELoginService: Roles: 2021-10-31 13:51:48 [https-jsse-nio-8443-exec-13] INFO: ACMELoginService: - Certificate Manager Agents 2021-10-31 13:51:48 [https-jsse-nio-8443-exec-13] INFO: ACMELoginService: - Enterprise ACME Administrators 2021-10-31 13:51:48 [https-jsse-nio-8443-exec-13] INFO: ACMELoginService: - Registration Manager Agents 2021-10-31 13:51:48 [ajp-nio-0:0:0:0:0:0:0:1-8009-exec-1] INFO: LDAP: search ou=config,ou=acme,o=ipaca 2021-10-31 13:51:49 [ajp-nio-0:0:0:0:0:0:0:1-8009-exec-1] INFO: ACMERequestFilter: ACME service is disabled
Things I've observed on fresh install, which I've implemented on my production (it changed nothing, provided here for documentation only):
# in /etc/pki/pki-tomcat/ca/CS.cfg: - added lines: features.authority.description=Lightweight CAs features.authority.enabled=true features.authority.version=1.0
- 36 profile.* lines were missing; carefully added them, for example: profile.AdminCert.class_id=caEnrollImpl profile.AdminCert.config=/var/lib/pki/pki-tomcat/ca/profiles/ca/AdminCert.cfg
- also copied a long line starting with profile.listprofile.list=
- /var/lib/pki/pki-tomcat/ca/profiles/ca on prod server contained 74 files, while fresh install had over 90. I've copied missing ones from /usr/share/pki/ca/profiles/ca/
# in LDAP - ipaca / groups / Certificate Manager Agents had entry for pkidbuser; added on prod uniqueMember: uid=pkidbuser,ou=People,o=ipaca - pkidbuser had 3 userCertificate: entries, two of them were expired; removed those
Tomasz Torcz via FreeIPA-users wrote:
On Mon, Oct 25, 2021 at 10:09:56AM -0500, Endi Dewata via FreeIPA-users wrote:
On Mon, Oct 25, 2021 at 7:42 AM Rob Crittenden via FreeIPA-users < freeipa-users@lists.fedorahosted.org> wrote:
Tomasz Torcz via FreeIPA-users wrote:
ACME also has a realm configuration:
https://github.com/dogtagpki/pki/blob/master/docs/installation/acme/Configur...
https://github.com/dogtagpki/pki/blob/master/docs/installation/acme/Configur...
so there could be an issue there.
But IIRC in IPA case it's configured to reuse the internaldb connection defined in CS.cfg so these params don't need to be specified again. Is there a working IPA instance with ACME that can be compared against?
So I did a clean install of Fedora 34 and FreeIPA. Clean install works as expected. I did comparison between fresh and mine install, there were discrepancies I mostly fixed, but it didn't change my problem. Failure looks like that in logs (pki-tomcat/acme/debug-<data>.log):
2021-11-03 18:43:07 [https-jsse-nio-8443-exec-12] INFO: Finding user by cert: 2021-11-03 18:43:07 [https-jsse-nio-8443-exec-12] INFO: - base DN: ou=people,o=ipaca 2021-11-03 18:43:07 [https-jsse-nio-8443-exec-12] INFO: - filter: description=2;105;CN=Certificate Authority,O=PIPEBREAKER.PL;CN=IPA RA,O=PIPEBREAKER.PL 2021-11-03 18:43:07 [https-jsse-nio-8443-exec-12] INFO: User: uid=ipara,ou=people,o=ipaca 2021-11-03 18:43:08 [https-jsse-nio-8443-exec-12] FINE: Realm.authenticate() returned false
Yeah, I noticed this in your logs as well. I have no insight into what PKI does to authenticate beyond the things you've already checked. We know that this cert is ok because you can authenticate to the CA using it in other ways. It would be nice if they logged some reason for the failure to authenticate but I'm not sure how to get that.
rob
While on _fresh install_ correct log looks like:
2021-10-31 13:51:47 [https-jsse-nio-8443-exec-13] INFO: Authenticating user with client certificate 2021-10-31 13:51:48 [https-jsse-nio-8443-exec-13] INFO: Finding user by cert: 2021-10-31 13:51:48 [https-jsse-nio-8443-exec-13] INFO: - base DN: ou=people,o=ipaca 2021-10-31 13:51:48 [https-jsse-nio-8443-exec-13] INFO: - filter: description=2;7;CN=Certificate Authority,O=IPADEV.PIPEBREAKER.PL;CN=IPA RA,O=IPADEV.PIPEBREAKER.PL 2021-10-31 13:51:48 [https-jsse-nio-8443-exec-13] INFO: User: uid=ipara,ou=people,o=ipaca 2021-10-31 13:51:48 [https-jsse-nio-8443-exec-13] INFO: Getting user roles: 2021-10-31 13:51:48 [https-jsse-nio-8443-exec-13] INFO: - base DN: ou=groups,o=ipaca 2021-10-31 13:51:48 [https-jsse-nio-8443-exec-13] INFO: - filter: uniqueMember=uid=ipara,ou=people,o=ipaca 2021-10-31 13:51:48 [https-jsse-nio-8443-exec-13] INFO: Roles: 2021-10-31 13:51:48 [https-jsse-nio-8443-exec-13] INFO: - cn=Certificate Manager Agents,ou=groups,o=ipaca 2021-10-31 13:51:48 [https-jsse-nio-8443-exec-13] INFO: - cn=Registration Manager Agents,ou=groups,o=ipaca 2021-10-31 13:51:48 [https-jsse-nio-8443-exec-13] INFO: - cn=Enterprise ACME Administrators,ou=groups,o=ipaca 2021-10-31 13:51:48 [https-jsse-nio-8443-exec-13] INFO: Initializing ACMEApplication 2021-10-31 13:51:48 [https-jsse-nio-8443-exec-13] INFO: ACMELoginService: Session: 3DBCD2FB21ADFDD04ADC518C97AA07B4 2021-10-31 13:51:48 [https-jsse-nio-8443-exec-13] INFO: ACMELoginService: Principal: GenericPrincipal[ipara(Certificate Manager Agents,Enterprise ACME Administrators,Registration Manager Agents,)] 2021-10-31 13:51:48 [https-jsse-nio-8443-exec-13] INFO: ACMELoginService: Principal: ipara 2021-10-31 13:51:48 [https-jsse-nio-8443-exec-13] INFO: ACMELoginService: Roles: 2021-10-31 13:51:48 [https-jsse-nio-8443-exec-13] INFO: ACMELoginService: - Certificate Manager Agents 2021-10-31 13:51:48 [https-jsse-nio-8443-exec-13] INFO: ACMELoginService: - Enterprise ACME Administrators 2021-10-31 13:51:48 [https-jsse-nio-8443-exec-13] INFO: ACMELoginService: - Registration Manager Agents 2021-10-31 13:51:48 [ajp-nio-0:0:0:0:0:0:0:1-8009-exec-1] INFO: LDAP: search ou=config,ou=acme,o=ipaca 2021-10-31 13:51:49 [ajp-nio-0:0:0:0:0:0:0:1-8009-exec-1] INFO: ACMERequestFilter: ACME service is disabled
Things I've observed on fresh install, which I've implemented on my production (it changed nothing, provided here for documentation only):
# in /etc/pki/pki-tomcat/ca/CS.cfg:
- added lines:
features.authority.description=Lightweight CAs features.authority.enabled=true features.authority.version=1.0
- 36 profile.* lines were missing; carefully added them, for example:
profile.AdminCert.class_id=caEnrollImpl profile.AdminCert.config=/var/lib/pki/pki-tomcat/ca/profiles/ca/AdminCert.cfg
also copied a long line starting with profile.listprofile.list=
/var/lib/pki/pki-tomcat/ca/profiles/ca on prod server contained 74 files, while fresh install had over 90. I've copied missing ones from /usr/share/pki/ca/profiles/ca/
# in LDAP
- ipaca / groups / Certificate Manager Agents had entry for pkidbuser; added on prod uniqueMember: uid=pkidbuser,ou=People,o=ipaca
- pkidbuser had 3 userCertificate: entries, two of them were expired; removed those
On Thu, Nov 4, 2021 at 12:32 PM Rob Crittenden via FreeIPA-users < freeipa-users@lists.fedorahosted.org> wrote:
Tomasz Torcz via FreeIPA-users wrote:
On Mon, Oct 25, 2021 at 10:09:56AM -0500, Endi Dewata via FreeIPA-users
wrote:
On Mon, Oct 25, 2021 at 7:42 AM Rob Crittenden via FreeIPA-users < freeipa-users@lists.fedorahosted.org> wrote:
Tomasz Torcz via FreeIPA-users wrote:
ACME also has a realm configuration:
https://github.com/dogtagpki/pki/blob/master/docs/installation/acme/Configur...
https://github.com/dogtagpki/pki/blob/master/docs/installation/acme/Configur...
so there could be an issue there.
But IIRC in IPA case it's configured to reuse the internaldb connection defined in CS.cfg so these params don't need to be specified again. Is there a working IPA instance with ACME that can be compared against?
So I did a clean install of Fedora 34 and FreeIPA. Clean install works as expected. I did comparison between fresh and mine install, there were discrepancies I mostly fixed, but it didn't change my problem. Failure looks like that in logs (pki-tomcat/acme/debug-<data>.log):
2021-11-03 18:43:07 [https-jsse-nio-8443-exec-12] INFO: Finding user by
cert:
2021-11-03 18:43:07 [https-jsse-nio-8443-exec-12] INFO: - base DN:
ou=people,o=ipaca
2021-11-03 18:43:07 [https-jsse-nio-8443-exec-12] INFO: - filter:
description=2;105;CN=Certificate Authority,O=PIPEBREAKER.PL;CN=IPA RA,O= PIPEBREAKER.PL
2021-11-03 18:43:07 [https-jsse-nio-8443-exec-12] INFO: User:
uid=ipara,ou=people,o=ipaca
2021-11-03 18:43:08 [https-jsse-nio-8443-exec-12] FINE:
Realm.authenticate() returned false
Yeah, I noticed this in your logs as well. I have no insight into what PKI does to authenticate beyond the things you've already checked. We know that this cert is ok because you can authenticate to the CA using it in other ways. It would be nice if they logged some reason for the failure to authenticate but I'm not sure how to get that.
rob
While on _fresh install_ correct log looks like:
2021-10-31 13:51:47 [https-jsse-nio-8443-exec-13] INFO: Authenticating
user with client certificate
2021-10-31 13:51:48 [https-jsse-nio-8443-exec-13] INFO: Finding user by
cert:
2021-10-31 13:51:48 [https-jsse-nio-8443-exec-13] INFO: - base DN:
ou=people,o=ipaca
2021-10-31 13:51:48 [https-jsse-nio-8443-exec-13] INFO: - filter:
description=2;7;CN=Certificate Authority,O=IPADEV.PIPEBREAKER.PL;CN=IPA RA,O=IPADEV.PIPEBREAKER.PL
2021-10-31 13:51:48 [https-jsse-nio-8443-exec-13] INFO: User:
uid=ipara,ou=people,o=ipaca
2021-10-31 13:51:48 [https-jsse-nio-8443-exec-13] INFO: Getting user
roles:
2021-10-31 13:51:48 [https-jsse-nio-8443-exec-13] INFO: - base DN:
ou=groups,o=ipaca
2021-10-31 13:51:48 [https-jsse-nio-8443-exec-13] INFO: - filter:
uniqueMember=uid=ipara,ou=people,o=ipaca
2021-10-31 13:51:48 [https-jsse-nio-8443-exec-13] INFO: Roles: 2021-10-31 13:51:48 [https-jsse-nio-8443-exec-13] INFO: - cn=Certificate
Manager Agents,ou=groups,o=ipaca
2021-10-31 13:51:48 [https-jsse-nio-8443-exec-13] INFO: -
cn=Registration Manager Agents,ou=groups,o=ipaca
2021-10-31 13:51:48 [https-jsse-nio-8443-exec-13] INFO: - cn=Enterprise
ACME Administrators,ou=groups,o=ipaca
2021-10-31 13:51:48 [https-jsse-nio-8443-exec-13] INFO: Initializing
ACMEApplication
2021-10-31 13:51:48 [https-jsse-nio-8443-exec-13] INFO:
ACMELoginService: Session: 3DBCD2FB21ADFDD04ADC518C97AA07B4
2021-10-31 13:51:48 [https-jsse-nio-8443-exec-13] INFO:
ACMELoginService: Principal: GenericPrincipal[ipara(Certificate Manager Agents,Enterprise ACME Administrators,Registration Manager Agents,)]
2021-10-31 13:51:48 [https-jsse-nio-8443-exec-13] INFO:
ACMELoginService: Principal: ipara
2021-10-31 13:51:48 [https-jsse-nio-8443-exec-13] INFO:
ACMELoginService: Roles:
2021-10-31 13:51:48 [https-jsse-nio-8443-exec-13] INFO:
ACMELoginService: - Certificate Manager Agents
2021-10-31 13:51:48 [https-jsse-nio-8443-exec-13] INFO:
ACMELoginService: - Enterprise ACME Administrators
2021-10-31 13:51:48 [https-jsse-nio-8443-exec-13] INFO:
ACMELoginService: - Registration Manager Agents
2021-10-31 13:51:48 [ajp-nio-0:0:0:0:0:0:0:1-8009-exec-1] INFO: LDAP:
search ou=config,ou=acme,o=ipaca
2021-10-31 13:51:49 [ajp-nio-0:0:0:0:0:0:0:1-8009-exec-1] INFO:
ACMERequestFilter: ACME service is disabled
Things I've observed on fresh install, which I've implemented on my
production
(it changed nothing, provided here for documentation only):
# in /etc/pki/pki-tomcat/ca/CS.cfg:
- added lines:
features.authority.description=Lightweight CAs features.authority.enabled=true features.authority.version=1.0
- 36 profile.* lines were missing; carefully added them, for example:
profile.AdminCert.class_id=caEnrollImpl
profile.AdminCert.config=/var/lib/pki/pki-tomcat/ca/profiles/ca/AdminCert.cfg
also copied a long line starting with profile.listprofile.list=
/var/lib/pki/pki-tomcat/ca/profiles/ca on prod server contained 74
files, while
fresh install had over 90. I've copied missing ones from
/usr/share/pki/ca/profiles/ca/
# in LDAP
- ipaca / groups / Certificate Manager Agents had entry for pkidbuser;
added on prod
uniqueMember: uid=pkidbuser,ou=People,o=ipaca
- pkidbuser had 3 userCertificate: entries, two of them were expired;
removed those
I added some log messages into this file if you want to try again: https://github.com/edewata/pki/blob/debug-v10.10/base/acme/src/main/java/org...
The build is available from this repo: https://copr.fedorainfracloud.org/coprs/edewata/pki-10.10/builds/
On Thu, Nov 04, 2021 at 08:45:17PM -0500, Endi Dewata via FreeIPA-users wrote:
I added some log messages into this file if you want to try again: https://github.com/edewata/pki/blob/debug-v10.10/base/acme/src/main/java/org...
The build is available from this repo: https://copr.fedorainfracloud.org/coprs/edewata/pki-10.10/builds/
Thanks… For most problems, the root cause is almost always DNS. For IPA, it's almost always certificate ;)
Verbose builds produced following logs:
2021-11-05 11:15:14 [https-jsse-nio-8443-exec-26] INFO: LDAP search: 2021-11-05 11:15:14 [https-jsse-nio-8443-exec-26] INFO: - base DN: ou=people,o=ipaca 2021-11-05 11:15:14 [https-jsse-nio-8443-exec-26] INFO: - filter: (description=2;105;CN=Certificate Authority,O=PIPEBREAKER.PL;CN=IPA RA,O=PIPEBREAKER.PL) 2021-11-05 11:15:14 [https-jsse-nio-8443-exec-26] INFO: User: uid=ipara,ou=people,o=ipaca 2021-11-05 11:15:14 [https-jsse-nio-8443-exec-26] INFO: Validating cert data in uid=ipara,ou=people,o=ipaca 2021-11-05 11:15:14 [https-jsse-nio-8443-exec-26] WARNING: User uid=ipara,ou=people,o=ipaca has no certificates
Impossible! I triple checked that. Let check again and compare with fresh install:
% ldapsearch -h kaitain.pipebreaker.pl -D cn=directory\ manager -W -o ldif-wrap=no \ -b uid=ipara,ou=people,o=ipaca
[…] dn: uid=ipara,ou=people,o=ipaca description: 2;105;CN=Certificate Authority,O=PIPEBREAKER.PL;CN=IPA RA,O=PIPEBREAKER.PL userCertificate;binary:: MIIDajCCAlKgAw… […]
While fresh install gives:
dn: uid=ipara,ou=people,o=ipaca description: 2;7;CN=Certificate Authority,O=IPADEV.PIPEBREAKER.PL;CN=IPA RA,O=IPADEV.PIPEBREAKER.PL userCertificate:: MIID/zCCAmegAwIBA…
There's an additional ";binary" in certificate attribute on my prod server. And I was comparing only base64 encoded part. And that was it. After removing ';binary' from attribute name, `pki-acme-manage` can authenticate.
Thank you very much for patience and assistance!
It is always a certificate.
Tomasz Torcz via FreeIPA-users wrote:
On Thu, Nov 04, 2021 at 08:45:17PM -0500, Endi Dewata via FreeIPA-users wrote:
I added some log messages into this file if you want to try again: https://github.com/edewata/pki/blob/debug-v10.10/base/acme/src/main/java/org...
The build is available from this repo: https://copr.fedorainfracloud.org/coprs/edewata/pki-10.10/builds/
Thanks… For most problems, the root cause is almost always DNS. For IPA, it's almost always certificate ;)
Verbose builds produced following logs:
2021-11-05 11:15:14 [https-jsse-nio-8443-exec-26] INFO: LDAP search: 2021-11-05 11:15:14 [https-jsse-nio-8443-exec-26] INFO: - base DN: ou=people,o=ipaca 2021-11-05 11:15:14 [https-jsse-nio-8443-exec-26] INFO: - filter: (description=2;105;CN=Certificate Authority,O=PIPEBREAKER.PL;CN=IPA RA,O=PIPEBREAKER.PL) 2021-11-05 11:15:14 [https-jsse-nio-8443-exec-26] INFO: User: uid=ipara,ou=people,o=ipaca 2021-11-05 11:15:14 [https-jsse-nio-8443-exec-26] INFO: Validating cert data in uid=ipara,ou=people,o=ipaca 2021-11-05 11:15:14 [https-jsse-nio-8443-exec-26] WARNING: User uid=ipara,ou=people,o=ipaca has no certificates
Impossible! I triple checked that. Let check again and compare with fresh install:
% ldapsearch -h kaitain.pipebreaker.pl -D cn=directory\ manager -W -o ldif-wrap=no \ -b uid=ipara,ou=people,o=ipaca
[…] dn: uid=ipara,ou=people,o=ipaca description: 2;105;CN=Certificate Authority,O=PIPEBREAKER.PL;CN=IPA RA,O=PIPEBREAKER.PL userCertificate;binary:: MIIDajCCAlKgAw… […]
While fresh install gives:
dn: uid=ipara,ou=people,o=ipaca description: 2;7;CN=Certificate Authority,O=IPADEV.PIPEBREAKER.PL;CN=IPA RA,O=IPADEV.PIPEBREAKER.PL userCertificate:: MIID/zCCAmegAwIBA…
There's an additional ";binary" in certificate attribute on my prod server. And I was comparing only base64 encoded part. And that was it. After removing ';binary' from attribute name, `pki-acme-manage` can authenticate.
Thank you very much for patience and assistance!
It is always a certificate.
That is quite unexpected as the same entry worked in other areas of dogtag. I'm not sure if I can detect this in ipa-healthcheck but I'll take a look. According to https://www.rfc-editor.org/rfc/rfc4522 they should be treated equivalently within the LDAP server since it isn't technically a subtype.
Endi, thanks for improving the logging! I hope that can be incorporated into a future build.
rob
freeipa-users@lists.fedorahosted.org
-
Endi Dewata -
Rob Crittenden -
Tomasz Torcz