"Internal server error 'Link'" from ClonesConnectivyAndDataCheck health check on RHEL 8 when talking to RHEL 9 server

[Sam Morris]

I tracked down the source of the myserious "Internal server error
'Link'" message when running this health check. It's caused by having a
mixture of both RHEL 8 and RHEL 9 servers.

The error message in context:

# ipa-healthcheck --source=pki.server.healthcheck.clones.connectivity_and_data --check=ClonesConnectivyAndDataCheck --output-type=json --debug
    [...]
    stderr=
    Calling check <pki.server.healthcheck.clones.connectivity_and_data.ClonesConnectivyAndDataCheck object at 0x7f8c87e8cf60>
    [...]
    About to check the subsystem clones
    Starting new HTTPS connection (1): ipa3.example.com:443
    https://ipa3.example.com:443 "POST /ca/rest/certs/search?size=3 HTTP/1.1" 200 431
    Cert data successfully obtained from clone.
    Starting new HTTPS connection (1): ipa5.example.com:443
    https://ipa5.example.com:443 "POST /ca/rest/certs/search?size=3 HTTP/1.1" 200 431
    Cert data successfully obtained from clone.
    Starting new HTTPS connection (1): ipa6.example.com:443
    https://ipa6.example.com:443 "POST /ca/rest/certs/search?size=3 HTTP/1.1" 200 317
    Internal server error 'Link'
    [...]
    [
      {
    "source": "pki.server.healthcheck.clones.connectivity_and_data",
    "check": "ClonesConnectivyAndDataCheck",
    "result": "ERROR",
    "uuid": "f672f185-6251-47e9-a772-8f9796a34986",
    "when": "20240312021736Z",
    "duration": "0.521790",
    "kw": {
      "status": "ERROR:  pki-tomcat : Internal error testing CA clone. Host: ipa6.example.com Port: 443"
    }
      }
    ]

I edited ClonesConnectivyAndDataCheck.check_ca_clones to call
logger.exception in its BaseException handler instead of just
logger.error. This logs the traceback of the original exception:

Internal server error 'Link'
    Traceback (most recent call last):
      File "/usr/lib/python3.6/site-packages/pki/server/healthcheck/clones/connectivity_and_data.py", line 35, in check_ca_clones
    certs = cert_client.list_certs(size=3)
      File "/usr/lib/python3.6/site-packages/pki/__init__.py", line 431, in handler
    return fn_call(inst, *args, **kwargs)
      File "/usr/lib/python3.6/site-packages/pki/cert.py", line 674, in list_certs
    return CertDataInfoCollection.from_json(response.json())
      File "/usr/lib/python3.6/site-packages/pki/cert.py", line 179, in from_json
    links = json_value['Link']
    KeyError: 'Link'

I guess the newer version of Dogtag in RHEL 9 doesn't include this
"Link" attribute, but pki.cert:CertDataInfoCollection.from_json in RHEL
8 expects it to be present.

--
Sam Morris https://robots.org.uk/
CAAA AA1A CA69 A83A 892B  1855 D20B 4202 5CDA 27B9

[Rob Crittenden]

Sam Morris via FreeIPA-users wrote:
> I tracked down the source of the myserious "Internal server error
> 'Link'" message when running this health check. It's caused by having a
> mixture of both RHEL 8 and RHEL 9 servers.
>
> The error message in context:
> # ipa-healthcheck --source=pki.server.healthcheck.clones.connectivity_and_data --check=ClonesConnectivyAndDataCheck --output-type=json --debug
> [...]
> stderr=
> Calling check <pki.server.healthcheck.clones.connectivity_and_data.ClonesConnectivyAndDataCheck object at 0x7f8c87e8cf60>
> [...]
> About to check the subsystem clones
> Starting new HTTPS connection (1): ipa3.example.com:443
> https://ipa3.example.com:443 "POST /ca/rest/certs/search?size=3 HTTP/1.1" 200 431
> Cert data successfully obtained from clone.
> Starting new HTTPS connection (1): ipa5.example.com:443
> https://ipa5.example.com:443 "POST /ca/rest/certs/search?size=3 HTTP/1.1" 200 431
> Cert data successfully obtained from clone.
> Starting new HTTPS connection (1): ipa6.example.com:443
> https://ipa6.example.com:443 "POST /ca/rest/certs/search?size=3 HTTP/1.1" 200 317
> Internal server error 'Link'
> [...]
> [
>   {
>
> "source": "pki.server.healthcheck.clones.connectivity_and_data",
>    "check": "ClonesConnectivyAndDataCheck",
>    "result": "ERROR",
>    "uuid": "f672f185-6251-47e9-a772-8f9796a34986",
>    "when": "20240312021736Z",
>    "duration": "0.521790",
>    "kw": {
>      "status": "ERROR:  pki-tomcat : Internal error testing CA clone. Host: ipa6.example.com Port: 443"
>    }
>       }
>     ]
>
> I edited ClonesConnectivyAndDataCheck.check_ca_clones to call
> logger.exception in its BaseException handler instead of just
> logger.error. This logs the traceback of the original exception:
> Internal server error 'Link'
> Traceback (most recent call last):
>   File "/usr/lib/python3.6/site-packages/pki/server/healthcheck/clones/connectivity_and_data.py", line 35, in check_ca_clones
>
> certs = cert_client.list_certs(size=3)
>       File "/usr/lib/python3.6/site-packages/pki/__init__.py", line 431, in handler
>    return fn_call(inst, *args, **kwargs)
>       File "/usr/lib/python3.6/site-packages/pki/cert.py", line 674, in list_certs
>    return CertDataInfoCollection.from_json(response.json())
>       File "/usr/lib/python3.6/site-packages/pki/cert.py", line 179, in from_json
>    links = json_value['Link']
>     KeyError: 'Link'
>
> I guess the newer version of Dogtag in RHEL 9 doesn't include this
> "Link" attribute, but pki.cert:CertDataInfoCollection.from_json in RHEL
> 8 expects it to be present.

Thanks for doing the research, this is great! Any chance you can file a
ticket against the "Red Hat Certificate System" project at
https://issues.redhat.com/ ? I don't own this particular check.

I'd encourage you to run your entire topology on the same IPA release.
We recognize that this isn't always possible, or desirable immediately,
but the transition is hopefully kept short. I usually recommend weeks
not months.

rob

[Sam Morris]

On 12/03/2024 12:27, Rob Crittenden via FreeIPA-users wrote:
> > I guess the newer version of Dogtag in RHEL 9 doesn't include this
> > "Link" attribute, but pki.cert:CertDataInfoCollection.from_json in RHEL
> > 8 expects it to be present.
>
> Thanks for doing the research, this is great! Any chance you can file a
> ticket against the "Red Hat Certificate System" project at
> https://issues.redhat.com/ ? I don't own this particular check.

Done at https://issues.redhat.com/browse/RHEL-29137

Please check the component, it's definitely wrong but I don't know what 
the right value is.
> I'd encourage you to run your entire topology on the same IPA release.
> We recognize that this isn't always possible, or desirable immediately,
> but the transition is hopefully kept short. I usually recommend weeks
> not months.

Agreed, I'm way behind regarding getting my whole homelab running RHEL 
9... ;)

[Rob Crittenden]

Sam Morris via FreeIPA-users wrote:
> On 12/03/2024 12:27, Rob Crittenden via FreeIPA-users wrote:
> > > I guess the newer version of Dogtag in RHEL 9 doesn't include this
> > > "Link" attribute, but pki.cert:CertDataInfoCollection.from_json in RHEL
> > > 8 expects it to be present.
> >
> > Thanks for doing the research, this is great! Any chance you can file a
> > ticket against the "Red Hat Certificate System" project at
> > https://issues.redhat.com/ ? I don't own this particular check.
>
> Done at https://issues.redhat.com/browse/RHEL-29137
>
> Please check the component, it's definitely wrong but I don't know what
> the right value is.

This looks great, pki-core is correct.

thanks

rob
> > I'd encourage you to run your entire topology on the same IPA release.
> > We recognize that this isn't always possible, or desirable immediately,
> > but the transition is hopefully kept short. I usually recommend weeks
> > not months.
>
> Agreed, I'm way behind regarding getting my whole homelab running RHEL
> 9... ;)