Clarification on CA Cert renewal requested

[Khurrum Maqb]

I am running ipa-server 4.6.6 with the same version clients. This IPA server has been
around since pre-v1 and has been upgraded till the current version that is shipped with
Centos7.
The IPA CA Cert was/is set to expire on Aug 10, 2020.

On the server that is the  IPA CA renewal master, I checked the output of `getcert list`
and the CA with the certificate "subject: CN=Certificate Authority,O=DOMAIN.COM"
and nickname nickname='caSigningCert cert-pki-ca'  is shown as renewed till 2040.
All other certs that appear in that list are updated without intervention to 2022.
It's located at `location='/etc/pki/pki-tomcat/alias'`. So far so good;  

BUT I noticed that /etc/ipa/ca.crt on the same server shows as still expiring on August
10:

#  openssl x509 -inform pem -enddate -noout -in /etc/ipa/ca.crt
notAfter=Aug 10 21:29:31 2020 GMT

So that means that the caSigningCert cert-pki-ca is set to automatically renew for 20
years But the IPA CA Cert is not.  

Next, I saw that there are certs located in /etc/pki/pki-tomcat/alias, /etc/ipa/nssdb/,
/etc/httpd/alias/, and /etc/pki/nssdb/. 

My questions:

* Is my self-signed IPA CA Cert supposed to be automatically renewed? 
* Or is it required that I run `ipa-cacert-manage renew` on the IPA CA renewal master, and
then `ipa-certupdate` on all the other server replicas and clients?
* Why do I appear to have duplicate DOMAIN IPA CA certs listed in /etc/ipa/nssdb/,
/etc/httpd/alias/? Is one location deprecated? 

Thank you for your help!

[Rob Crittenden]

Khurrum Maqb via FreeIPA-users wrote:
>  I am running ipa-server 4.6.6 with the same version clients. This IPA
> server has been around since pre-v1 and has been upgraded till the current version that is
> shipped with Centos7.
>  The IPA CA Cert was/is set to expire on Aug 10, 2020.
>  
>  On the server that is the  IPA CA renewal master, I checked the output of `getcert list`
> and the CA with the certificate "subject: CN=Certificate Authority,O=DOMAIN.COM"
> and nickname nickname='caSigningCert cert-pki-ca'  is shown as renewed till 2040.
> All other certs that appear in that list are updated without intervention to 2022.
> It's located at `location='/etc/pki/pki-tomcat/alias'`. So far so good;  
>  
>  BUT I noticed that /etc/ipa/ca.crt on the same server shows as still expiring on August
> 10:
>  
>  #  openssl x509 -inform pem -enddate -noout -in /etc/ipa/ca.crt
>  notAfter=Aug 10 21:29:31 2020 GMT
>  
>  So that means that the caSigningCert cert-pki-ca is set to automatically renew for 20
> years But the IPA CA Cert is not.  
>  
>  Next, I saw that there are certs located in /etc/pki/pki-tomcat/alias, /etc/ipa/nssdb/,
> /etc/httpd/alias/, and /etc/pki/nssdb/. 
>  
>  My questions:
>  
>  * Is my self-signed IPA CA Cert supposed to be automatically renewed?  
It can automatically renew, I don't know if it did or not in your case.
The updated cert just isn't automatically pushed out to all the file
locations it needs to be.

Run ipa-certupdate on all IPA-enrolled machines, including servers, to
update local files.

>  * Or is it required that I run `ipa-cacert-manage renew` on the IPA
> CA renewal master, and then `ipa-certupdate` on all the other server replicas and clients?
Just the ipa-certupdate part since the renewal is already done. Run on
all machines.

>  * Why do I appear to have duplicate DOMAIN IPA CA certs listed in
> /etc/ipa/nssdb/, /etc/httpd/alias/? Is one location deprecated?  
They are separate NSS databases for separate purposes. dbm-based NSS
databases should not be shared between multiple services hence each one
has its own db (389 has one as well).

rob

[Khurrum Maqb]

>  Run ipa-certupdate on all IPA-enrolled machines, including servers,
> to
>  update local files. 
Thanks. I ran ipa-certupdate on a client and I see that it completed successfully. 

The output of `certutil -L -d /etc/ipa/nssdb/` shows a second `DOMAIN IPA CA` now with the
new certificate with the new expiration date. It still has the old cert as well which is
expected.

However, `/etc/ipa/ca.crt` changed in file size AND the filesystem modified data changed
to the time where ipa-certupdate was ran today. but the output of `openssl x509 -inform
pem -enddate -noout -in /etc/ipa/ca.crt` is still "notAfter=Aug 10 21:29:31 2020
GMT" 

-rw-r--r--.   1 root root 12351 Aug  6 12:20 ca.crt
-rw-r--r--.   1 root root  4145 Aug  6 12:20 ca.crt.original

When I ran `ipa-certupdate -v` it showed a "File not found" for `IPA CA` but
then it found `DOMAIN IPA CA` so that's probably not relevant. 

So far it looks like everything updates but ca.crt does not show the updated cert. 

Is this something that can be changed so that ca.crt is also showing the correct
certificate? 

K

[Florence Blanc-Renaud]

On 8/6/20 6:35 PM, Khurrum Maqb via FreeIPA-users wrote:
> > Run ipa-certupdate on all IPA-enrolled machines, including
> servers, to
> > update local files.
>  
>  Thanks. I ran ipa-certupdate on a client and I see that it completed successfully.
>  
>  The output of `certutil -L -d /etc/ipa/nssdb/` shows a second `DOMAIN IPA CA` now with
> the new certificate with the new expiration date. It still has the old cert as well which
> is expected.
>  
>  However, `/etc/ipa/ca.crt` changed in file size AND the filesystem modified data changed
> to the time where ipa-certupdate was ran today. but the output of `openssl x509 -inform
> pem -enddate -noout -in /etc/ipa/ca.crt` is still "notAfter=Aug 10 21:29:31 2020
> GMT" 
`openssl x509` has a limitation when used with a PEM file that contains 
multiple certificates: it displays the information for the first cert only.
If you want to see all the certs, you can use instead:
openssl crl2pkcs7 -nocrl -certfile /etc/ipa/ca.crt | openssl pkcs7 
-print_certs -text -noout

HTH,
flo
>  
>  -rw-r--r--.   1 root root 12351 Aug  6 12:20 ca.crt
>  -rw-r--r--.   1 root root  4145 Aug  6 12:20 ca.crt.original
>  
>  When I ran `ipa-certupdate -v` it showed a "File not found" for `IPA CA` but
> then it found `DOMAIN IPA CA` so that's probably not relevant.
>  
>  So far it looks like everything updates but ca.crt does not show the updated cert.
>  
>  Is this something that can be changed so that ca.crt is also showing the correct
> certificate?
>  
>  K
>  _______________________________________________
>  FreeIPA-users mailing list -- freeipa-users(a)lists.fedorahosted.org
>  To unsubscribe send an email to freeipa-users-leave(a)lists.fedorahosted.org
>  Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
>  List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
>  List Archives:
> https://lists.fedorahosted.org/archives/list/[e-mail]...
>

[Khurrum Maqb]

That was it! Thank you all!

[Stefan Ubbink]

On Thu, 6 Aug 2020 19:16:58 +0200
Florence Blanc-Renaud via FreeIPA-users
<freeipa-users(a)lists.fedorahosted.org&gt; wrote:


>  On 8/6/20 6:35 PM, Khurrum Maqb via FreeIPA-users wrote:
>  > However, `/etc/ipa/ca.crt` changed in file size AND the filesystem
>  > modified data changed to the time where ipa-certupdate was ran
>  > today. but the output of `openssl x509 -inform pem -enddate -noout
>  > -in /etc/ipa/ca.crt` is still "notAfter=Aug 10 21:29:31 2020 GMT"  
>  `openssl x509` has a limitation when used with a PEM file that
>  contains multiple certificates: it displays the information for the
>  first cert only. If you want to see all the certs, you can use
>  instead: openssl crl2pkcs7 -nocrl -certfile /etc/ipa/ca.crt | openssl
>  pkcs7 -print_certs -text -noout 
It is also possible to use certtool [1] to get all the information
`certtool -i --infile /etc/ipa/ca.crt`

[1] certtool is provided by the gnutls-utils package.
-- 
Stefan Ubbink
DNS & Systems Engineer
Present: Mon, Tue, Wed, Fri
SIDN | Meander 501 | 6825 MD | ARNHEM | The Netherlands
T [phone]
https://www.sidn.nl

[Khurrum Maqb]

Unfortunately, despite `getcert list` returning the updated cert, and /etc/ipa/ca.crt
showing the correct certificate, IPA is showing a certificate error after the old CA Cert
expired at midnight. 

One example is:

# ipa-certupdate
trying https:/URL/ipa/json
Connection to https:/URL/ipa/json failed with [SSL: CERTIFICATE_VERIFY_FAILED] certificate
verify failed (_ssl.c:618)

I also can't log in using LDAP from some services that depend on IPA. I had run
ipa-certupdate on all these clients when it wasn't expired.

Another oddity is that if I go to https://IPA-URL and view the cert, the CA cert looks
fine ie expiration is 2040. 

Any idea how I can resolve this? There is an outdated cert somewhere that ipa-certupdate
did not fix. 
Note I had run ipa-certupdate everywhere including the FreeIPA servers before the cert
expired.

Thanks

[Khurrum Maqb]

One other thing that I noticed, when I manually edit /etc/ipa/ca.crt and remove the top
most cert block, then ipa-certupdate starts working without any errors ie there is no
more" [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:618)"

And then once ipa-certupate runs the update the errors come back. Do I also need to remove
the old ca cert from the IPA server manually?

[Khurrum Maqb]

I do see the old CA Certificate listed here:

# ipa-cacert-manage list
DOMAIN.COM IPA CA
DOMAIN.COM IPA CA
.
.
.
The ipa-cacert-manage command was successful

and running  `certutil -L -d /etc/httpd/alias -n "DOMAIN.COM IPA CA"` returns
details of both the expired CA Cert and the current renewed CA Cert. Is there any other
place where IPA is giving the older CA Cert a higher priority than the renewed cert? And
would deleting the old expired cert be the 'fix' for this issue? 

It appears like I wouldn't be able to run ipa-cacert-manage delete DOMAIN.COM IPA CA
because there's a duplicate name in that list. Is there a way to tell it to delete by
Certificate ID or another unique identifier? 

I also am currently unable to log into the Web UI for this same reason. For external
websites, I have disabled Cert verification temporarily until this is resolved so users
are able to log in. 

Thanks!

[Rob Crittenden]

Khurrum Maqb via FreeIPA-users wrote:
>  I do see the old CA Certificate listed here:
>  
>  # ipa-cacert-manage list
>  DOMAIN.COM IPA CA
>  DOMAIN.COM IPA CA
>  .
>  .
>  .
>  The ipa-cacert-manage command was successful
>  
>  and running  `certutil -L -d /etc/httpd/alias -n "DOMAIN.COM IPA CA"` returns
> details of both the expired CA Cert and the current renewed CA Cert. Is there any other
> place where IPA is giving the older CA Cert a higher priority than the renewed cert? And
> would deleting the old expired cert be the 'fix' for this issue? 
>  
>  It appears like I wouldn't be able to run ipa-cacert-manage delete DOMAIN.COM IPA CA
> because there's a duplicate name in that list. Is there a way to tell it to delete by
> Certificate ID or another unique identifier? 
>  
>  I also am currently unable to log into the Web UI for this same reason. For external
> websites, I have disabled Cert verification temporarily until this is resolved so users
> are able to log in.  
You can't delete the IPA CA using the delete command so it wouldn't help
you in this case.

It is OpenSSL that is not handling the multiple certs, I'm not sure why
in this case.

But you can clean up the expired one pretty easily:

Use your favorite LDAP tool (Apache studio, ldapmodify, etc) to look in:
cn=certificates,cn=ipa,cn=etc,dc=example,dc=test

Identify which caCertificate is the expired one and remove the value.

On output it will likely appear (depending on tool) as a base64-encoded
blob. You'll need to either convert it to DER (easier) or PEM format to
see the details of it.

I'd base64-decode it and write it to a file and then use openssl x509
-text -inform der -in /path/to/file to examine it.

rob

[Khurrum Maqb]

Thank you! That did it. I had to delete the old cert from cn=certificates,cn=ipa,cn=etc
Then I had to edit /etc/ipa/ca.crt on the IPA CA renewal master and remote the expired
cert. Then I ran ipa-certupdate. Then I was able to confirm that running openssl x509
-inform pem -enddate -noout -in /etc/ipa/ca.crt gave me the expected endDate of 2020. 

Then I did the same on all other IPA servers. And then finally had to run ipactl restart
and all the WebUIs became accessible again. Now I'll run ipa-certupdate on all
remaining systems. 

Thanks again for your help!