Healthckeck help

[Bob Strachan]

I wonder I if have gotten myself in a bind.

I have a small realm of a dozen Rhel7 and Rhel8 servers, a few dozen users and three IDM
servers.  We moved from yellow pages to IDM for 2FA a couple of years ago.  The original
IDM servers were all Rhel7.  In November of 2021, we moved to IDM 4.9 on Rhel8.4 by
standing up new IDM servers and replicating.  

When we finished the migration there were no significant healtcheck (hc) errors.  Over the
next year we upgraded to 8.5, 8.6 and then 8.7. 

At some point and I believe it was when we got to Rhel8.6 we started getting hc errors
with this type of message:  
"msg": "Certificate 'subsystemCert cert-pki-ca' does not match the
value of kra.subsystem.cert in /var/lib/pki/pki-tomcat/kra/conf/CS.cfg" 

On two of the IDM servers the messages were for: 
	kra_subsystem 
	kra_transport
	kra_storage
	kra_audit_signing  
all showing a missmatch in /var/lib/pki/pki-tomcat/kra/conf/CS.cfg
There was one other similar message:
	"transportCert cert-pki-kra  had a mismatch in
/var/lib/pki/pki-tomcat/conf/ca/CS.cfg

At the time we googled the issue and came up with:
https://github.com/freeipa/freeipa-healthcheck/issues/154  
where Rob Crittenden says:
"We seem to have lost traction on this. It is my understanding that the certificates
not matching the values in CS.cfg is not a critical problem. It is checked to ensure
consistency."

So we ignored the errors
After upgrading to 8.7 in November 2022, we started getting more hc errors regarding iDNS.
 Google led us to a script to fix the problem which we ran and the problem went away.  

I was still concerned with the CS.cfg mismatches, so I decided to fix the messages.  An
inventory of the messages showed two IDM servers with the messages listed above and one
IDM server with an sslserverCert in ...kra/CS.cfg and transportCert in ...ca/CS.cfg
mismatch errors.

Since this was only a sanity check, I modified the cert hashes in each CS.cfg to match the
cert generated by:
certutil -d /etc/pki/pki-tomcat/alias -L -n '<cert nickname>' -a by pasting
the trimmed hash into the appropriate directive in the corresponding CS.cfg.

Now when I run ipa-healthcheck, there are no errors.  My concern is that perhaps I have
broken Dogtag and when some of these certs go to be renewed in February, there will be
problems.

I also note that for all 14 of these cert directives I modified in CS.cfg there was also a
 "<directive.certreq=>" pointing to a hash in CS.cfg that I didn't
modify.  

Is my system in trouble?  How do I resolve this?  Since all the errors are only in CS.cfg,
is there a way to generate a fresh CS.cfg?  Most of the articles I've found regarding
fixing corrupt CS.cfg files, are old and incredibly complex.

[Jochen Kellner]

Bob Strachan via FreeIPA-users <freeipa-users(a)lists.fedorahosted.org&gt;
writes:

>  At some point and I believe it was when we got to Rhel8.6 we started
>  getting hc errors with this type of message:
>  "msg": "Certificate 'subsystemCert cert-pki-ca' does not match
> the
>  value of kra.subsystem.cert in
>  /var/lib/pki/pki-tomcat/kra/conf/CS.cfg" 
My analysis is documented here, more or less what you found:
https://pagure.io/freeipa/issue/9277

I've posted my take on an ansible script to fix my servers:
https://lists.fedorahosted.org/archives/list/[e-mail]...

HTH
Jochen

-- 
This space is intentionally left blank.

[Rob Crittenden]

Bob Strachan via FreeIPA-users wrote:
>  I wonder I if have gotten myself in a bind.
>  
>  I have a small realm of a dozen Rhel7 and Rhel8 servers, a few dozen users and three IDM
> servers.  We moved from yellow pages to IDM for 2FA a couple of years ago.  The original
> IDM servers were all Rhel7.  In November of 2021, we moved to IDM 4.9 on Rhel8.4 by
> standing up new IDM servers and replicating.  
>  
>  When we finished the migration there were no significant healtcheck (hc) errors.  Over
> the next year we upgraded to 8.5, 8.6 and then 8.7. 
>  
>  At some point and I believe it was when we got to Rhel8.6 we started getting hc errors
> with this type of message:  
>  "msg": "Certificate 'subsystemCert cert-pki-ca' does not match the
> value of kra.subsystem.cert in /var/lib/pki/pki-tomcat/kra/conf/CS.cfg" 
>  
>  On two of the IDM servers the messages were for: 
>  	kra_subsystem 
>  	kra_transport
>  	kra_storage
>  	kra_audit_signing  
>  all showing a missmatch in /var/lib/pki/pki-tomcat/kra/conf/CS.cfg
>  There was one other similar message:
>  	"transportCert cert-pki-kra  had a mismatch in
> /var/lib/pki/pki-tomcat/conf/ca/CS.cfg
>  
>  At the time we googled the issue and came up with:
>  https://github.com/freeipa/freeipa-healthcheck/issues/154  
>  where Rob Crittenden says:
>  "We seem to have lost traction on this. It is my understanding that the certificates
> not matching the values in CS.cfg is not a critical problem. It is checked to ensure
> consistency."
>  
>  So we ignored the errors
>  After upgrading to 8.7 in November 2022, we started getting more hc errors regarding
> iDNS.  Google led us to a script to fix the problem which we ran and the problem went
> away.  
>  
>  I was still concerned with the CS.cfg mismatches, so I decided to fix the messages.  An
> inventory of the messages showed two IDM servers with the messages listed above and one
> IDM server with an sslserverCert in ...kra/CS.cfg and transportCert in ...ca/CS.cfg
> mismatch errors.
>  
>  Since this was only a sanity check, I modified the cert hashes in each CS.cfg to match
> the cert generated by:
>  certutil -d /etc/pki/pki-tomcat/alias -L -n '<cert nickname>' -a by pasting
> the trimmed hash into the appropriate directive in the corresponding CS.cfg.
>  
>  Now when I run ipa-healthcheck, there are no errors.  My concern is that perhaps I have
> broken Dogtag and when some of these certs go to be renewed in February, there will be
> problems.
>  
>  I also note that for all 14 of these cert directives I modified in CS.cfg there was also
> a  "<directive.certreq=>" pointing to a hash in CS.cfg that I didn't
> modify.  
>  
>  Is my system in trouble?  How do I resolve this?  Since all the errors are only in
> CS.cfg, is there a way to generate a fresh CS.cfg?  Most of the articles I've found
> regarding fixing corrupt CS.cfg files, are old and incredibly complex.  
As I mentioned in healthcheck upstream, the CS does not consider the
certificate blobs to be a critical issue. You fixed them, that's fine.
I'd probably make sure to keep a copy of CS.cfg prior to tweaking it for
general safety. There is no way that I know of to regenerate it from
scratch for a given system.

I'm not sure why you think you've broken dogtag by doing this. Does the
CA start up? Can it issue certificates? Can you create a vault?

You don't have to wait on pins and needles. If you want extra breathing
room for when renewal happens you can set enroll_ttls in
/etc/certmonger/certmonger.conf and restart it (see man page for details).

You can, for example, add a 60-day renewal value, then restart certmonger:

enroll_ttls = 5184000, 2419200, 604800, 259200, 172800, 86400, 43200,
21600, 7200, 3600

That would cover past the end of February so it should start renewing
immediately.

Note that renewal may not complete in one pass and that's usually ok.
Renewal involves restarting the CS for each updated cert and there are
minimum 5 to renew, plus the LDAP and HTTP certs, so there is a lot of
work to do.

rob

[Bob Strachan]

Rob and Jochen,

Thank you both for your speedy reply.  

My IDM system seems to be working fine.  I can issue certs.  My concern is with the two
CS.cfg files, as  I have no idea what they are for.  I don't know if the csr blobs in
CS.cfg are necessary or if they need to be in sync with the cert blobs I manually updated.
 

After reading Jochen's notes, and my experience, I am guessing that the renewal master
updates the .../kra/conf/CS.cfg but not the kra CS.cfg files on the other replicas.    I
am also guessing that my renewal server was a fresh install and it hit the bug:
https://bugzilla.redhat.com/show_bug.cgi?id=1871188

So I am still wondering, what are the CS.cfg files for????  I    

I would guess that they might be called when using ipa-cert-fix, but I am not skilled
enough to unpack what ipa-cert-fix does.  If the CS.cfg files are full deprecated on a
Rhel 8.6 replicated IDM system, then I would like to know, so I can relax.

As for advancing the certmonger configuration, It appears that my certs should get renewed
in 7 days.  As such, I will just wait for the 7 days and see if the renewal works.  I have
no expectation that the kra CS.cfg file will get updated. 

Thanks again for your replies.

Bob

[Rob Crittenden]

Bob Strachan via FreeIPA-users wrote:
>  Rob and Jochen,
>  
>  Thank you both for your speedy reply.  
>  
>  My IDM system seems to be working fine.  I can issue certs.  My concern is with the two
> CS.cfg files, as  I have no idea what they are for.  I don't know if the csr blobs in
> CS.cfg are necessary or if they need to be in sync with the cert blobs I manually updated.
>   
A CSR is a Certificate Signing Request. You don't want or need to touch
these.

>  
>  After reading Jochen's notes, and my experience, I am guessing that the renewal
> master updates the .../kra/conf/CS.cfg but not the kra CS.cfg files on the other replicas.
>    I am also guessing that my renewal server was a fresh install and it hit the bug:
>  https://bugzilla.redhat.com/show_bug.cgi?id=1871188 
Each subsystem (kra, ca) has only one CS.cfg. This is the subsystem
configuration file which defines how it works.

>  
>  So I am still wondering, what are the CS.cfg files for????  I    
>  
>  I would guess that they might be called when using ipa-cert-fix, but I am not skilled
> enough to unpack what ipa-cert-fix does.  If the CS.cfg files are full deprecated on a
> Rhel 8.6 replicated IDM system, then I would like to know, so I can relax. 
CS.cfg is a configuration file for each subsystem. It is definitely not
deprecated.

Whether certain values within the file are important is another matter.
It is unclear how important the cert blobs are but we try to keep them
up to date for neatness. Except for KRA which I totally missed doing. It
does not appear to result in any problems though, beyond healthcheck
mentioning it.

Healthcheck is not an end-all-be-all grade of IPA health. It is a set of
common things that cause problems that we can easily check on and
report. There may be things missing and there may be false positives.
The point is to keep admins watching for issues before they become major
problems.

>  
>  As for advancing the certmonger configuration, It appears that my certs should get
> renewed in 7 days.  As such, I will just wait for the 7 days and see if the renewal works.
>  I have no expectation that the kra CS.cfg file will get updated. 
>   
I can almost guarantee it won't because the issue Jochen filed hasn't
been addressed yet.

rob