Roderick Johnstone via FreeIPA-users wrote:
On 15/01/2018 20:07, Rob Crittenden via FreeIPA-users wrote:
> Roderick Johnstone via FreeIPA-users wrote:
>> On 15/01/2018 16:06, Rob Crittenden via FreeIPA-users wrote:
>>> Roderick Johnstone via FreeIPA-users wrote:
>>>> Hi
>>>>
>>>> Our freeipa certificates need to be renewed due to passing their
>>>> expiry
>>>> dates.
>>>>
>>>> While some certificates have renewed ok, the ipaCert and
>>>> auditSigningCert are renewing but the new certificates have the wrong
>>>> Subject.
>>>>
>>>> Environment is:
>>>> serverA (CRL, first, master) RHEL 7.3, ipa 4.4
>>>> serverB (replica) RHEL 7.3, ipa 4.4
>>>> serverC (replica) RHEL 7.4, ipa 4.5
>>>>
>>>> Once there are renewed certificates with the wrong Subject present,
>>>> there are various problems with renewing the remaining certificates,
>>>> which I think might be related to the bad Subject:
>>>>
>>>> 1) When just ipaCert has the wrong subject no further renewals happen
>>>>
>>>> 2) When auditSigningCert has the wrong subject the ipa pki-tomcatd
>>>> service will not start and no further renewals happen.
>>>>
>>>> I've been round the following loop many times on ServerA, our first
>>>> master:
>>>>
>>>> 1) Restore good certificates from backup
>>>> 2) Put the clock back to a time when certificates are all valid
>>>> 3) Resubmit certificates for renewal
>>>>
>>>> Each time the ipaCert renews it has the same wrong Subject. The wrong
>>>> Subject includes the host name of one of our ipa client systems.
>>>>
>>>> Each time the auditSigningCert renews it has the same wrong Subject
>>>> but
>>>> a different subject to the ipaCert. The wrong Subject in this case
>>>> includes the host name of a system which has never been an ipa client,
>>>> but might have been added and removed with ipa host-add and ipa
>>>> host-del
>>>> for testing something, a while ago.
>>>>
>>>> As far as I can see, the "cert_subject" is set correctly in the
file
>>>> /var/lib/certmonger/<request id> until the point at which the
>>>> certificate is actually renewed.
>>>>
>>>> I'd be very grateful for some pointers as to which configuration
>>>> options
>>>> and logs to check through to resolve this problem on our production
>>>> system.
>>>>
>>>> If its of any relevance we did change which server is the first master
>>>> some time ago.
>>>
>>> I'd pull the CSR out of dogtag (CS.cfg) and/or certmonger to see what
>>> the subject is.
>>
>> I'm not seeing any obvious CSR fields in the
>> /etc/pki/pki-tomcat/ca/CS.cfg file.
>
> foo.bar.certreq=
>
>> The CSR in the certmonger requests file for the auditSigningCert seems
>> to be showing with the correct Subject. This is different from the bad
>> subject showing in the requests file field:
>> cert_subject=
>
> The value of cert_subject comes from the issued certificate.
>
>> and the Subject which is showing in the 'getcert list' output (which is
>> the same as that in the cert_subject= field.>
>> I'm not quite sure what this all means.
>
> It is displayed from the data within the tracked certmonger request.
>
> certmonger logs to syslog so you can check there or you can stop the
> process and run it manually with: certmonger -n -d 9 2>&1 | tee
> certmonger.log
>
> That will provide a lot of debugging output that may show what is
> going on.
I've restored certificate databases from backup and put the clock back
to a time when certificates are valid and renewed the ocspSigining
certificate with:
getcert resubmit -N "CN=OCSP Subsystem,O=<REALM>" -i 20161124081302
(I've previously tried without the -N with similar results)
What I am seeing in the certmonger logs is:
2017-10-23 00:05:28 [438] Located the key 'ocspSigningCert cert-pki-ca'.
2017-10-23 00:05:28 [438] Converted private key 'ocspSigningCert
cert-pki-ca' to public key.
2017-10-23 00:05:28 [439] Located the certificate "ocspSigningCert
cert-pki-ca".
2017-10-23 00:05:28 [440] 0x1d Certificate named "ocspSigningCert
cert-pki-ca" in token "NSS Certificate DB" in database
"/etc/pki/pki-tomcat/alias" will not be valid after 20171025122401.
2017-10-23 00:05:28 [442] Located the key 'ocspSigningCert cert-pki-ca'.
2017-10-23 00:05:28 [442] Converted private key 'ocspSigningCert
cert-pki-ca' to public key.
2017-10-23 00:05:28 [443] Located the certificate "ocspSigningCert
cert-pki-ca".
2017-10-23 00:05:28 [444] Located the key 'ocspSigningCert cert-pki-ca'.
2017-10-23 00:05:28 [444] Converted private key 'ocspSigningCert
cert-pki-ca' to public key.
2017-10-23 00:05:39 [581] Found a certificate with the same nickname but
different subject, removing certificate "ocspSigningCert cert-pki-ca"
with subject "CN=OCSP Subsystem,O=<REALM>".
2017-10-23 00:05:39 [581] Imported certificate "ocspSigningCert
cert-pki-ca", got nickname "ocspSigningCert cert-pki-ca".
2017-10-23 00:05:39 [583] Located the certificate "ocspSigningCert
cert-pki-ca".
2017-10-23 00:05:39 [48576] Adding hook
"/usr/libexec/ipa/certmonger/renew_ca_cert "ocspSigningCert
cert-pki-ca"" (0).
2017-10-23 00:10:43 [942] 0x1d Certificate named "ocspSigningCert
cert-pki-ca" in token "NSS Certificate DB" in database
"/etc/pki/pki-tomcat/alias" issued by CA and saved.
I now have a date valid ocspSigningCertificate, but with the wrong
subject, and a broken certificate system which will no longer start.
ipactl status
...
pki-tomcatd Service: STOPPED
I can't renew other expired certificates.
I also note that there is now no key for ocspSigningCertificate as shown
by:
certutil -K -d /etc/pki/pki-tomcat/alias
I wonder if this is because the certificate subject changed? There was a
key before the certificate renewed.
The ca debug logs are showing:
[23/Oct/2017:00:55:47][localhost-startStop-1]: Found cert by nickname:
'ocspSigningCert cert-pki-ca' with serial number: 268370108
[23/Oct/2017:00:55:47][localhost-startStop-1]: converted to x509CertImpl
[23/Oct/2017:00:55:47][localhost-startStop-1]: SigningUnit: Certificate
object not found
Certificate object not found
at com.netscape.ca.SigningUnit.init(SigningUnit.java:184)
Any help in repairing my broken ipa system will be much appreciated.
Sorry for the delay. I think my previous reading of the certmonger
csrgen code was wrong.
IIRC in your certmonger entry the cert_subject has the hostname value
right? Do you also have cert_subject_der?
You can decode that by:
1. Running a hex-to-bin, something hacky like this in python:
import binascii
hex_string = "<hex value>"
binary_string = binascii.unhexlify(hex_string)
fd = open("out", "wb")
fd.write(binary_string)
fd.close()
2. Run: openssl asn1parse -in out -inform der
I'm going to assume this also has the hostname encoded in it.
Can you try this:
1. Make a backup of the request file (just in case)
2. Remove cert_subject_der
3. Modify cert_subject to be CN=OCSP Subsystem,O=<YOUR_REALM>
3. Try the renewal again
rob