Thank you for the hint, it's gotten me farther. I can now see cert details in the webui; however, cli tools still fail with "ipa: ERROR: Certificate operation cannot be completed: Request failed with status 403: Non-2xx response from CA REST API: 403. (403)" Specifically, "ipa cert show 4" (where 4 is a valid certificate serial number)
Here's the output of "ipa-healthcheck". Of note, valid.tld is sanitized, it really is valid and not literally "valid.tld". The replica server4.valid.tld is a failed server which has been removed and does not show in the output of "ipa-replica-manage list" "ipa topologysuffix-verify [domain|ca]" "ipa topologysegment-find [domain|ca]"
# ipa-healthcheck Internal server error HTTPSConnectionPool(host='server4.valid.tld', port=443): Max retries exceeded with url: /ca/rest/certs/search?size=3 (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x7f8ac490a8d0>: Failed to establish a new connection: [Errno -2] Name or service not known',)) ra.get_certificate(): Request failed with status 403: Non-2xx response from CA REST API: 403. (403) ra.get_certificate(): Request failed with status 403: Non-2xx response from CA REST API: 403. (403) ra.get_certificate(): Request failed with status 403: Non-2xx response from CA REST API: 403. (403) ra.get_certificate(): Request failed with status 403: Non-2xx response from CA REST API: 403. (403) ra.get_certificate(): Request failed with status 403: Non-2xx response from CA REST API: 403. (403) ra.get_certificate(): Request failed with status 403: Non-2xx response from CA REST API: 403. (403) ra.get_certificate(): Request failed with status 403: Non-2xx response from CA REST API: 403. (403) ra.get_certificate(): Request failed with status 403: Non-2xx response from CA REST API: 403. (403) ra.get_certificate(): Request failed with status 403: Non-2xx response from CA REST API: 403. (403) ra.get_certificate(): Request failed with status 403: Non-2xx response from CA REST API: 403. (403) [ { "source": "pki.server.healthcheck.clones.connectivity_and_data", "check": "ClonesConnectivyAndDataCheck", "result": "ERROR", "uuid": "d6d3a36d-f2fd-4793-971f-9bacadfe5881", "when": "20210910184505Z", "duration": "1.538118", "kw": { "status": "ERROR: pki-tomcat : Internal error testing CA clone. Host: server4.valid.tld Port: 443" } }, { "source": "ipahealthcheck.dogtag.ca", "check": "DogtagCertsConnectivityCheck", "result": "ERROR", "uuid": "fa1ac443-9ce2-457a-a814-2b127eff8541", "when": "20210910184507Z", "duration": "0.246410", "kw": { "msg": "Request for certificate failed, Certificate operation cannot be completed: Request failed with status 403: Non-2xx response from CA REST API: 403. (403)" } }, { "source": "ipahealthcheck.ds.replication", "check": "ReplicationCheck", "result": "ERROR", "uuid": "2ecf8b7f-78c7-4527-9d0b-716b1ba8061b", "when": "20210910184508Z", "duration": "0.742027", "kw": { "key": "DSREPLLE0003", "items": [ "Replication", "Agreement" ], "msg": "The replication agreement (catoserver2.valid.tld) under "o=ipaca" is not in synchronization.\nStatus message: error (18) can't acquire replica (incremental update transient warning. backing off, will retry update later.)" } }, { "source": "ipahealthcheck.ds.replication", "check": "ReplicationCheck", "result": "ERROR", "uuid": "498d7a58-68d4-44ad-966a-0d8e918df33c", "when": "20210910184508Z", "duration": "0.742055", "kw": { "key": "DSREPLLE0003", "items": [ "Replication", "Agreement" ], "msg": "The replication agreement (catoserver3.valid.tld) under "o=ipaca" is not in synchronization.\nStatus message: error (18) can't acquire replica (incremental update transient warning. backing off, will retry update later.)" } }, { "source": "ipahealthcheck.ipa.certs", "check": "IPACertRevocation", "result": "ERROR", "uuid": "519e1eb9-8229-4695-9f86-2c3d834543d1", "when": "20210910184514Z", "duration": "0.424361", "kw": { "key": "20210303190407", "serial": 7, "error": "Certificate operation cannot be completed: Request failed with status 403: Non-2xx response from CA REST API: 403. (403)", "msg": "Request for certificate serial number {serial} in request {key} failed: {error}" } }, { "source": "ipahealthcheck.ipa.certs", "check": "IPACertRevocation", "result": "ERROR", "uuid": "7f3dd497-2125-4f64-bff3-52cd65291d9c", "when": "20210910184514Z", "duration": "0.528265", "kw": { "key": "20210303190402", "serial": 5, "error": "Certificate operation cannot be completed: Request failed with status 403: Non-2xx response from CA REST API: 403. (403)", "msg": "Request for certificate serial number {serial} in request {key} failed: {error}" } }, { "source": "ipahealthcheck.ipa.certs", "check": "IPACertRevocation", "result": "ERROR", "uuid": "b242bb04-7a86-446b-b2c6-3c1c65994a21", "when": "20210910184514Z", "duration": "0.630944", "kw": { "key": "20210303190403", "serial": 2, "error": "Certificate operation cannot be completed: Request failed with status 403: Non-2xx response from CA REST API: 403. (403)", "msg": "Request for certificate serial number {serial} in request {key} failed: {error}" } }, { "source": "ipahealthcheck.ipa.certs", "check": "IPACertRevocation", "result": "ERROR", "uuid": "5b6aad97-4a48-477c-bf45-503b6a2df426", "when": "20210910184515Z", "duration": "0.735810", "kw": { "key": "20210303190404", "serial": 4, "error": "Certificate operation cannot be completed: Request failed with status 403: Non-2xx response from CA REST API: 403. (403)", "msg": "Request for certificate serial number {serial} in request {key} failed: {error}" } }, { "source": "ipahealthcheck.ipa.certs", "check": "IPACertRevocation", "result": "ERROR", "uuid": "4c68d780-aaab-4d28-8920-e0396433b969", "when": "20210910184515Z", "duration": "0.838743", "kw": { "key": "20210303190405", "serial": 1, "error": "Certificate operation cannot be completed: Request failed with status 403: Non-2xx response from CA REST API: 403. (403)", "msg": "Request for certificate serial number {serial} in request {key} failed: {error}" } }, { "source": "ipahealthcheck.ipa.certs", "check": "IPACertRevocation", "result": "ERROR", "uuid": "8e8e7e65-3081-47b1-b3fd-d35ee444b7a6", "when": "20210910184515Z", "duration": "0.939950", "kw": { "key": "20210303190406", "serial": 3, "error": "Certificate operation cannot be completed: Request failed with status 403: Non-2xx response from CA REST API: 403. (403)", "msg": "Request for certificate serial number {serial} in request {key} failed: {error}" } }, { "source": "ipahealthcheck.ipa.certs", "check": "IPACertRevocation", "result": "ERROR", "uuid": "e22c4c88-92dd-4326-ae54-9ce626348e5f", "when": "20210910184515Z", "duration": "0.992323", "kw": { "key": "20210303190409", "serial": 58, "error": "Certificate operation cannot be completed: Request failed with status 403: Non-2xx response from CA REST API: 403. (403)", "msg": "Request for certificate serial number {serial} in request {key} failed: {error}" } }, { "source": "ipahealthcheck.ipa.certs", "check": "IPACertRevocation", "result": "ERROR", "uuid": "c885ae6c-4365-47ea-905c-e09429aa6f21", "when": "20210910184515Z", "duration": "1.091397", "kw": { "key": "20210303190408", "serial": 8, "error": "Certificate operation cannot be completed: Request failed with status 403: Non-2xx response from CA REST API: 403. (403)", "msg": "Request for certificate serial number {serial} in request {key} failed: {error}" } }, { "source": "ipahealthcheck.ipa.certs", "check": "IPACertRevocation", "result": "ERROR", "uuid": "3c788561-f1a5-4d3e-8ad6-312fc4b335f3", "when": "20210910184515Z", "duration": "1.144757", "kw": { "key": "20201102193636", "serial": 10, "error": "Certificate operation cannot be completed: Request failed with status 403: Non-2xx response from CA REST API: 403. (403)", "msg": "Request for certificate serial number {serial} in request {key} failed: {error}" } } ]