If the subject isn't vague enough, perhaps I can explain in some better detail. I have IPA setup with a couple of replicas and it's been running fine for a few months; perriodic runs of ipa-healthcheck didn't show any issues. During an update of the system packages, something went wrong and the CA is now unavailable from the webui and ipa tools.
When accessing from the webui (Authentication->Certificates) I get th following error: "Certificate operation cannot be completed: Unable to communicate with CMS (403)"
From some cli tools (ipa cert-show) I get this error: "ipa: ERROR: Certificate operation cannot be completed: Request failed with status 403: Non-2xx response from CA REST API: 403. (403)"
It appears that dogtag is working, as I can use it's toolset to query "pki ca-cert-show 0x1 --pretty" for example returns the certificate as expected so I suspect the issue lies somewhere in the API. I compared the cert which I believe to be used for pki with the following: "grep internal /var/lib/pki/pki-tomcat/conf/password.conf | cut -d= -f2 > /tmp/pwdfile.txt" "certutil -K -d /etc/pki/pki-tomcat/alias -f /tmp/pwdfile.txt -n 'subsystemCert cert-pki-ca'" So from what I've read it seems pki-tomcat should be able to access it's private key and certificate.
I also compared the cert with the one in LDAP and the two are the same. "ldapsearch -LLL -D 'cn=directory manager' -W -b uid=pkidbuser,ou=people,o=ipaca userCertificate"
In digging through some old backups, I found that entries in /etc/pki/pki-tomcat/server.xml had changed, so I reset the "requiredSecret" back to an earlier value, which seemed to allow the webui to work again, at least partially as I can now get a listing from the CA in the webui though I cannot see details of any particular certificate.
Assuming I'm on the right path, is there a documented process to regenerate the server.xml file or am I chasing the wrong problem?
D Trom via FreeIPA-users wrote:
If the subject isn't vague enough, perhaps I can explain in some better detail. I have IPA setup with a couple of replicas and it's been running fine for a few months; perriodic runs of ipa-healthcheck didn't show any issues. During an update of the system packages, something went wrong and the CA is now unavailable from the webui and ipa tools.
When accessing from the webui (Authentication->Certificates) I get th following error: "Certificate operation cannot be completed: Unable to communicate with CMS (403)"
From some cli tools (ipa cert-show) I get this error: "ipa: ERROR: Certificate operation cannot be completed: Request failed with status 403: Non-2xx response from CA REST API: 403. (403)"
It appears that dogtag is working, as I can use it's toolset to query "pki ca-cert-show 0x1 --pretty" for example returns the certificate as expected so I suspect the issue lies somewhere in the API. I compared the cert which I believe to be used for pki with the following: "grep internal /var/lib/pki/pki-tomcat/conf/password.conf | cut -d= -f2 > /tmp/pwdfile.txt" "certutil -K -d /etc/pki/pki-tomcat/alias -f /tmp/pwdfile.txt -n 'subsystemCert cert-pki-ca'" So from what I've read it seems pki-tomcat should be able to access it's private key and certificate.
I also compared the cert with the one in LDAP and the two are the same. "ldapsearch -LLL -D 'cn=directory manager' -W -b uid=pkidbuser,ou=people,o=ipaca userCertificate"
In digging through some old backups, I found that entries in /etc/pki/pki-tomcat/server.xml had changed, so I reset the "requiredSecret" back to an earlier value, which seemed to allow the webui to work again, at least partially as I can now get a listing from the CA in the webui though I cannot see details of any particular certificate.
Assuming I'm on the right path, is there a documented process to regenerate the server.xml file or am I chasing the wrong problem?
First, it was a most excellent idea to use the pki command to verify that the CA was alive and well!
The value of requiredSecret needs to match the value in /etc/httpd/conf.d/ipa-pki-proxy.conf in the ProxyPassMatch statements (the last argument)
The argument name in server.xml is determined by the version of tomcat. pre-9.0.31.0 uses 'requiredSecret' and afterward uses 'secret'.
There is no documented process on regenerating server.xml that I know of. But since you already did the detective work to figure out it is requiredSecret related look at your version of tomcat vs what was added in the IPA upgrader (run in rpm %post).
Note that any manual changes you make may be overwritten on the next package update.
rob
Thank you for the hint, it's gotten me farther. I can now see cert details in the webui; however, cli tools still fail with "ipa: ERROR: Certificate operation cannot be completed: Request failed with status 403: Non-2xx response from CA REST API: 403. (403)" Specifically, "ipa cert show 4" (where 4 is a valid certificate serial number)
Here's the output of "ipa-healthcheck". Of note, valid.tld is sanitized, it really is valid and not literally "valid.tld". The replica server4.valid.tld is a failed server which has been removed and does not show in the output of "ipa-replica-manage list" "ipa topologysuffix-verify [domain|ca]" "ipa topologysegment-find [domain|ca]"
# ipa-healthcheck Internal server error HTTPSConnectionPool(host='server4.valid.tld', port=443): Max retries exceeded with url: /ca/rest/certs/search?size=3 (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x7f8ac490a8d0>: Failed to establish a new connection: [Errno -2] Name or service not known',)) ra.get_certificate(): Request failed with status 403: Non-2xx response from CA REST API: 403. (403) ra.get_certificate(): Request failed with status 403: Non-2xx response from CA REST API: 403. (403) ra.get_certificate(): Request failed with status 403: Non-2xx response from CA REST API: 403. (403) ra.get_certificate(): Request failed with status 403: Non-2xx response from CA REST API: 403. (403) ra.get_certificate(): Request failed with status 403: Non-2xx response from CA REST API: 403. (403) ra.get_certificate(): Request failed with status 403: Non-2xx response from CA REST API: 403. (403) ra.get_certificate(): Request failed with status 403: Non-2xx response from CA REST API: 403. (403) ra.get_certificate(): Request failed with status 403: Non-2xx response from CA REST API: 403. (403) ra.get_certificate(): Request failed with status 403: Non-2xx response from CA REST API: 403. (403) ra.get_certificate(): Request failed with status 403: Non-2xx response from CA REST API: 403. (403) [ { "source": "pki.server.healthcheck.clones.connectivity_and_data", "check": "ClonesConnectivyAndDataCheck", "result": "ERROR", "uuid": "d6d3a36d-f2fd-4793-971f-9bacadfe5881", "when": "20210910184505Z", "duration": "1.538118", "kw": { "status": "ERROR: pki-tomcat : Internal error testing CA clone. Host: server4.valid.tld Port: 443" } }, { "source": "ipahealthcheck.dogtag.ca", "check": "DogtagCertsConnectivityCheck", "result": "ERROR", "uuid": "fa1ac443-9ce2-457a-a814-2b127eff8541", "when": "20210910184507Z", "duration": "0.246410", "kw": { "msg": "Request for certificate failed, Certificate operation cannot be completed: Request failed with status 403: Non-2xx response from CA REST API: 403. (403)" } }, { "source": "ipahealthcheck.ds.replication", "check": "ReplicationCheck", "result": "ERROR", "uuid": "2ecf8b7f-78c7-4527-9d0b-716b1ba8061b", "when": "20210910184508Z", "duration": "0.742027", "kw": { "key": "DSREPLLE0003", "items": [ "Replication", "Agreement" ], "msg": "The replication agreement (catoserver2.valid.tld) under "o=ipaca" is not in synchronization.\nStatus message: error (18) can't acquire replica (incremental update transient warning. backing off, will retry update later.)" } }, { "source": "ipahealthcheck.ds.replication", "check": "ReplicationCheck", "result": "ERROR", "uuid": "498d7a58-68d4-44ad-966a-0d8e918df33c", "when": "20210910184508Z", "duration": "0.742055", "kw": { "key": "DSREPLLE0003", "items": [ "Replication", "Agreement" ], "msg": "The replication agreement (catoserver3.valid.tld) under "o=ipaca" is not in synchronization.\nStatus message: error (18) can't acquire replica (incremental update transient warning. backing off, will retry update later.)" } }, { "source": "ipahealthcheck.ipa.certs", "check": "IPACertRevocation", "result": "ERROR", "uuid": "519e1eb9-8229-4695-9f86-2c3d834543d1", "when": "20210910184514Z", "duration": "0.424361", "kw": { "key": "20210303190407", "serial": 7, "error": "Certificate operation cannot be completed: Request failed with status 403: Non-2xx response from CA REST API: 403. (403)", "msg": "Request for certificate serial number {serial} in request {key} failed: {error}" } }, { "source": "ipahealthcheck.ipa.certs", "check": "IPACertRevocation", "result": "ERROR", "uuid": "7f3dd497-2125-4f64-bff3-52cd65291d9c", "when": "20210910184514Z", "duration": "0.528265", "kw": { "key": "20210303190402", "serial": 5, "error": "Certificate operation cannot be completed: Request failed with status 403: Non-2xx response from CA REST API: 403. (403)", "msg": "Request for certificate serial number {serial} in request {key} failed: {error}" } }, { "source": "ipahealthcheck.ipa.certs", "check": "IPACertRevocation", "result": "ERROR", "uuid": "b242bb04-7a86-446b-b2c6-3c1c65994a21", "when": "20210910184514Z", "duration": "0.630944", "kw": { "key": "20210303190403", "serial": 2, "error": "Certificate operation cannot be completed: Request failed with status 403: Non-2xx response from CA REST API: 403. (403)", "msg": "Request for certificate serial number {serial} in request {key} failed: {error}" } }, { "source": "ipahealthcheck.ipa.certs", "check": "IPACertRevocation", "result": "ERROR", "uuid": "5b6aad97-4a48-477c-bf45-503b6a2df426", "when": "20210910184515Z", "duration": "0.735810", "kw": { "key": "20210303190404", "serial": 4, "error": "Certificate operation cannot be completed: Request failed with status 403: Non-2xx response from CA REST API: 403. (403)", "msg": "Request for certificate serial number {serial} in request {key} failed: {error}" } }, { "source": "ipahealthcheck.ipa.certs", "check": "IPACertRevocation", "result": "ERROR", "uuid": "4c68d780-aaab-4d28-8920-e0396433b969", "when": "20210910184515Z", "duration": "0.838743", "kw": { "key": "20210303190405", "serial": 1, "error": "Certificate operation cannot be completed: Request failed with status 403: Non-2xx response from CA REST API: 403. (403)", "msg": "Request for certificate serial number {serial} in request {key} failed: {error}" } }, { "source": "ipahealthcheck.ipa.certs", "check": "IPACertRevocation", "result": "ERROR", "uuid": "8e8e7e65-3081-47b1-b3fd-d35ee444b7a6", "when": "20210910184515Z", "duration": "0.939950", "kw": { "key": "20210303190406", "serial": 3, "error": "Certificate operation cannot be completed: Request failed with status 403: Non-2xx response from CA REST API: 403. (403)", "msg": "Request for certificate serial number {serial} in request {key} failed: {error}" } }, { "source": "ipahealthcheck.ipa.certs", "check": "IPACertRevocation", "result": "ERROR", "uuid": "e22c4c88-92dd-4326-ae54-9ce626348e5f", "when": "20210910184515Z", "duration": "0.992323", "kw": { "key": "20210303190409", "serial": 58, "error": "Certificate operation cannot be completed: Request failed with status 403: Non-2xx response from CA REST API: 403. (403)", "msg": "Request for certificate serial number {serial} in request {key} failed: {error}" } }, { "source": "ipahealthcheck.ipa.certs", "check": "IPACertRevocation", "result": "ERROR", "uuid": "c885ae6c-4365-47ea-905c-e09429aa6f21", "when": "20210910184515Z", "duration": "1.091397", "kw": { "key": "20210303190408", "serial": 8, "error": "Certificate operation cannot be completed: Request failed with status 403: Non-2xx response from CA REST API: 403. (403)", "msg": "Request for certificate serial number {serial} in request {key} failed: {error}" } }, { "source": "ipahealthcheck.ipa.certs", "check": "IPACertRevocation", "result": "ERROR", "uuid": "3c788561-f1a5-4d3e-8ad6-312fc4b335f3", "when": "20210910184515Z", "duration": "1.144757", "kw": { "key": "20201102193636", "serial": 10, "error": "Certificate operation cannot be completed: Request failed with status 403: Non-2xx response from CA REST API: 403. (403)", "msg": "Request for certificate serial number {serial} in request {key} failed: {error}" } } ]
I ran into similar issues after upgrading from FreeIPA 4.9.3 to 4.9.6 on Centos Stream 8 last week.
You could check /var/log/httpd/error_log - I had trouble with TLS 1.3 (leading to error "Request failed with status 403: Non-2xx response from CA REST API: 403.") which could be solved by disabling SSLProtocol in /etc/httpd/conf.d/ssl.conf. There were other issues with disabling TLS 1.3 (see https://bugzilla.redhat.com/show_bug.cgi?id=1775158) and due to lack of time, finally I recovered from backup.
Thank you. Setting requiredSecret to the same value as secret in /etc/pki/pki-tomcat/server.xml fixed it for me on CentOS Stream 8. It stopped working after upgrading FreeIPA from 4.9.3 to 4.9.6. Seems I barely missed the version that uses "secret": java -cp catalina.jar org.apache.catalina.util.ServerInfo Server version: Apache Tomcat/9.0.30 Server built: Jun 29 2021 22:00:15 UTC Server number: 9.0.30.0 OS Name: Linux OS Version: 4.18.0-338.el8.x86_64 Architecture: amd64 JVM Version: 1.8.0_302-b08 JVM Vendor: Red Hat, Inc.
Now to figure out why sudo 1.9.8 segfaults when FreeIPA sudo rules are enabled...
freeipa-users@lists.fedorahosted.org
-
D Trom -
Dirk Silkenbaeumer -
pp -
Rob Crittenden