handling certificate expirations

[Grant Janssen]

When I upgraded the servers to EL8 (I rebuilt from scratch using the old hostnames), I had
neglected to assign an IPA CA renewal master after the old “boss” was retired.
This crime is of course it’s own punishment.

I found the documentation for handling this to actually be pretty good.
https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/8/...

fraser’s blog was also helpful (in confirming I executed this correctly)
https://frasertweedale.github.io/blog-redhat/posts/2019-05-24-ipa-cert-fi...

I progressed through the other three IPA servers, but the last one still has a bad
expiration on the CA cert.

[root@ef-idm01 ~]# date
Wed Feb 14 07:08:38 PST 2024
[root@ef-idm01 ~]# getcert list | egrep '^Request|status:|subject:|expir'
Request ID '20230530175932':
        status: MONITORING
        subject:
CN=ef-idm01.production.efilm.com<http://ef-idm01.production.efilm.com>,O=PRODUCTION.EFILM.COM<http://PRODUCTION.EFILM.COM>
        expires: 2025-05-30 10:59:53 PDT
Request ID '20230530180022':
        status: MONITORING
        subject:
CN=ef-idm01.production.efilm.com<http://ef-idm01.production.efilm.com>,O=PRODUCTION.EFILM.COM<http://PRODUCTION.EFILM.COM>
        expires: 2025-05-30 11:00:30 PDT
Request ID '20230530180438':
        status: NEED_CA
        subject: CN=IPA RA,O=PRODUCTION.EFILM.COM<http://PRODUCTION.EFILM.COM>
        expires: 2024-01-02 07:58:28 PST
[root@ef-idm01 ~]# ipa-cert-fix

                          WARNING

ipa-cert-fix is intended for recovery when expired certificates
prevent the normal operation of IPA.  It should ONLY be used
in such scenarios, and backup of the system, especially certificates
and keys, is STRONGLY RECOMMENDED.


The following certificates will be renewed:

IPA IPA RA certificate:
  Subject: CN=IPA RA,O=PRODUCTION.EFILM.COM<http://PRODUCTION.EFILM.COM>
  Serial:  162
  Expires: 2024-01-02 15:58:28

Enter "yes" to proceed: yes
Proceeding.
Renewed IPA IPA RA certificate:
  Subject: CN=IPA RA,O=PRODUCTION.EFILM.COM<http://PRODUCTION.EFILM.COM>
  Serial:  1341915142
  Expires: 2026-02-03 20:18:20

Becoming renewal master.
Restarting IPA

Note: Monitor the certmonger-initiated renewal of
certificates after ipa-cert-fix and wait for its completion before
any other administrative task.

The ipa-cert-fix command was successful
[root@ef-idm01 ~]#

I checked the cert expiration several times yesterday, but it never updated on this
server.
I waited a full day to let certmonger do its thing, below is my result this morning.

[root@ef-idm01 ~]# ipactl restart
Restarting Directory Service
Restarting krb5kdc Service
Restarting kadmin Service
Restarting httpd Service
Restarting ipa-custodia Service
Restarting pki-tomcatd Service
Restarting ipa-otpd Service
ipa: INFO: The ipactl command was successful
[root@ef-idm01 ~]# ipactl status
Directory Service: RUNNING
krb5kdc Service: RUNNING
kadmin Service: RUNNING
httpd Service: RUNNING
ipa-custodia Service: RUNNING
pki-tomcatd Service: RUNNING
ipa-otpd Service: RUNNING
ipa: INFO: The ipactl command was successful
[root@ef-idm01 ~]# getcert list | egrep '^Request|status:|subject:|expir'
Request ID '20230530175932':
        status: MONITORING
        subject:
CN=ef-idm01.production.efilm.com<http://ef-idm01.production.efilm.com>,O=PRODUCTION.EFILM.COM<http://PRODUCTION.EFILM.COM>
        expires: 2025-05-30 10:59:53 PDT
Request ID '20230530180022':
        status: MONITORING
        subject:
CN=ef-idm01.production.efilm.com<http://ef-idm01.production.efilm.com>,O=PRODUCTION.EFILM.COM<http://PRODUCTION.EFILM.COM>
        expires: 2025-05-30 11:00:30 PDT
Request ID '20230530180438':
        status: NEED_CA
        subject: CN=IPA RA,O=PRODUCTION.EFILM.COM<http://PRODUCTION.EFILM.COM>
        expires: 2024-01-02 07:58:28 PST
[root@ef-idm01 ~]# ipa-cert-fix
Nothing to do.
The ipa-cert-fix command was successful
[root@ef-idm01 ~]# getcert list | egrep '^Request|status:|subject:|expir'
Request ID '20230530175932':
        status: MONITORING
        subject:
CN=ef-idm01.production.efilm.com<http://ef-idm01.production.efilm.com>,O=PRODUCTION.EFILM.COM<http://PRODUCTION.EFILM.COM>
        expires: 2025-05-30 10:59:53 PDT
Request ID '20230530180022':
        status: MONITORING
        subject:
CN=ef-idm01.production.efilm.com<http://ef-idm01.production.efilm.com>,O=PRODUCTION.EFILM.COM<http://PRODUCTION.EFILM.COM>
        expires: 2025-05-30 11:00:30 PDT
Request ID '20230530180438':
        status: NEED_CA
        subject: CN=IPA RA,O=PRODUCTION.EFILM.COM<http://PRODUCTION.EFILM.COM>
        expires: 2024-01-02 07:58:28 PST
[root@ef-idm01 ~]#

How can I sort out this one remaining issue?
Do I just make assign another server as the renewal master?

thanx

- grant

[Rob Crittenden]

Grant Janssen via FreeIPA-users wrote:
>  When I upgraded the servers to EL8 (I rebuilt from scratch using the
> old
>  hostnames), I had neglected to assign an IPA CA renewal master after the
>  old “boss” was retired.
>  This crime is of course it’s own punishment.
>  
>  I found the documentation for handling this to actually be pretty good.
>
> https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/8/...
>  
>  fraser’s blog was also helpful (in confirming I executed this correctly)
>  https://frasertweedale.github.io/blog-redhat/posts/2019-05-24-ipa-cert-fi...
>  
>  I progressed through the other three IPA servers, but the last one still
>  has a bad expiration on the CA cert.
>  
>      [root@ef-idm01 ~]# date
>      Wed Feb 14 07:08:38 PST 2024
>      [root@ef-idm01 ~]# getcert list | egrep
>      '^Request|status:|subject:|expir'
>      Request ID '20230530175932':
>              status: MONITORING
>              subject: CN=ef-idm01.production.efilm.com
>      <http://ef-idm01.production.efilm.com>,O=PRODUCTION.EFILM.COM
>      <http://PRODUCTION.EFILM.COM>
>              expires: 2025-05-30 10:59:53 PDT
>      Request ID '20230530180022':
>              status: MONITORING
>              subject: CN=ef-idm01.production.efilm.com
>      <http://ef-idm01.production.efilm.com>,O=PRODUCTION.EFILM.COM
>      <http://PRODUCTION.EFILM.COM>
>              expires: 2025-05-30 11:00:30 PDT
>      Request ID '20230530180438':
>              status: NEED_CA
>              subject: CN=IPA RA,O=PRODUCTION.EFILM.COM
>      <http://PRODUCTION.EFILM.COM>
>              expires: 2024-01-02 07:58:28 PST
>      [root@ef-idm01 ~]# ipa-cert-fix
>  
>                                WARNING
>  
>      ipa-cert-fix is intended for recovery when expired certificates
>      prevent the normal operation of IPA.  It should ONLY be used
>      in such scenarios, and backup of the system, especially certificates
>      and keys, is STRONGLY RECOMMENDED.
>  
>  
>      The following certificates will be renewed:
>  
>      IPA IPA RA certificate:
>        Subject: CN=IPA RA,O=PRODUCTION.EFILM.COM
>      <http://PRODUCTION.EFILM.COM>
>        Serial:  162
>        Expires: 2024-01-02 15:58:28
>  
>      Enter "yes" to proceed: yes
>      Proceeding.
>      Renewed IPA IPA RA certificate:
>        Subject: CN=IPA RA,O=PRODUCTION.EFILM.COM
>      <http://PRODUCTION.EFILM.COM>
>        Serial:  1341915142
>        Expires: 2026-02-03 20:18:20
>  
>      Becoming renewal master.
>      Restarting IPA
>  
>      Note: Monitor the certmonger-initiated renewal of
>      certificates after ipa-cert-fix and wait for its completion before
>      any other administrative task.
>  
>      The ipa-cert-fix command was successful
>      [root@ef-idm01 ~]#
>  
>  
>  I checked the cert expiration several times yesterday, but it never
>  updated on this server.
>  I waited a full day to let certmonger do its thing, below is my result
>  this morning.
>  
>      [root@ef-idm01 ~]#ipactl restart
>      Restarting Directory Service
>      Restarting krb5kdc Service
>      Restarting kadmin Service
>      Restarting httpd Service
>      Restarting ipa-custodia Service
>      Restarting pki-tomcatd Service
>      Restarting ipa-otpd Service
>      ipa: INFO: The ipactl command was successful
>      [root@ef-idm01 ~]# ipactl status
>      Directory Service: RUNNING
>      krb5kdc Service: RUNNING
>      kadmin Service: RUNNING
>      httpd Service: RUNNING
>      ipa-custodia Service: RUNNING
>      pki-tomcatd Service: RUNNING
>      ipa-otpd Service: RUNNING
>      ipa: INFO: The ipactl command was successful
>      [root@ef-idm01 ~]# getcert list | egrep
>      '^Request|status:|subject:|expir'
>      Request ID '20230530175932':
>              status: MONITORING
>              subject: CN=ef-idm01.production.efilm.com
>      <http://ef-idm01.production.efilm.com>,O=PRODUCTION.EFILM.COM
>      <http://PRODUCTION.EFILM.COM>
>              expires: 2025-05-30 10:59:53 PDT
>      Request ID '20230530180022':
>              status: MONITORING
>              subject: CN=ef-idm01.production.efilm.com
>      <http://ef-idm01.production.efilm.com>,O=PRODUCTION.EFILM.COM
>      <http://PRODUCTION.EFILM.COM>
>              expires: 2025-05-30 11:00:30 PDT
>      Request ID '20230530180438':
>              status: NEED_CA
>              subject: CN=IPA RA,O=PRODUCTION.EFILM.COM
>      <http://PRODUCTION.EFILM.COM>
>              expires: 2024-01-02 07:58:28 PST
>      [root@ef-idm01 ~]#ipa-cert-fix
>      Nothing to do.
>      The ipa-cert-fix command was successful
>      [root@ef-idm01 ~]# getcert list | egrep
>      '^Request|status:|subject:|expir'
>      Request ID '20230530175932':
>              status: MONITORING
>              subject: CN=ef-idm01.production.efilm.com
>      <http://ef-idm01.production.efilm.com>,O=PRODUCTION.EFILM.COM
>      <http://PRODUCTION.EFILM.COM>
>              expires: 2025-05-30 10:59:53 PDT
>      Request ID '20230530180022':
>              status: MONITORING
>              subject: CN=ef-idm01.production.efilm.com
>      <http://ef-idm01.production.efilm.com>,O=PRODUCTION.EFILM.COM
>      <http://PRODUCTION.EFILM.COM>
>              expires: 2025-05-30 11:00:30 PDT
>      Request ID '20230530180438':
>              status: NEED_CA
>              subject: CN=IPA RA,O=PRODUCTION.EFILM.COM
>      <http://PRODUCTION.EFILM.COM>
>              expires: 2024-01-02 07:58:28 PST
>      [root@ef-idm01 ~]#
>  
>  
>  How can I sort out this one remaining issue?
>  Do I just make assign another server as the renewal master? 
You only need to run ipa-cert-fix on the CA renewal master. Once that is
done the RA Agent can be renewed on the machine and made available to
the replicas.

Updated CA/RA certificates are stored in LDAP. Once replication is
working then all replicas will pick up the necessary certificates from
there. In fact you don't want to renew the CA/RA certificates
per-machine. They all must be the same.

I'd need to see the full RA cert tracking to be able to tell what is
going on but I'd suggest running ipa-server-upgrade first. That will fix
any bad tracking. Then manually resubmit it with getcert if it is still
in the NEED_CA state and it should pull the updated cert out of LDAP.

rob

[Grant Janssen]

this was definitely the hot tip.
executing a server upgrade fixed everything for me.

thanx rob

[Grant Janssen]

well, I thought I was out of the woods, but I still have some issues.
the services are running, but kinit gets me a ticket to nowhere.

"ipa: ERROR: No valid Negotiate header in server response"

grant@ef-idm01:~[20240220-14:36][#785]$ klist
Ticket cache: KCM:555
Default principal: grant(a)PRODUCTION.EFILM.COM

Valid starting       Expires              Service principal
02/20/2024 14:36:12  02/21/2024 13:51:10 
krbtgt/PRODUCTION.EFILM.COM(a)PRODUCTION.EFILM.COM
grant@ef-idm01:~[20240220-14:36][#786]$ ipa server-find
ipa: ERROR: No valid Negotiate header in server response
grant@ef-idm01:~[20240220-14:36][#787]$ sudo systemctl status gssproxy.service
● gssproxy.service - GSSAPI Proxy Daemon
   Loaded: loaded (/usr/lib/systemd/system/gssproxy.service; disabled; vendor preset:
disabled)
   Active: active (running) since Tue 2024-02-20 13:57:40 PST; 39min ago
  Process: 2158008 ExecStart=/usr/sbin/gssproxy -D (code=exited, status=0/SUCCESS)
 Main PID: 2158009 (gssproxy)
    Tasks: 6 (limit: 74714)
   Memory: 4.2M
   CGroup: /system.slice/gssproxy.service
           └─2158009 /usr/sbin/gssproxy -D

Feb 20 13:57:40 ef-idm01.production.efilm.com systemd[1]: gssproxy.service: Succeeded.
Feb 20 13:57:40 ef-idm01.production.efilm.com systemd[1]: Stopped GSSAPI Proxy Daemon.
Feb 20 13:57:40 ef-idm01.production.efilm.com systemd[1]: Starting GSSAPI Proxy Daemon...
Feb 20 13:57:40 ef-idm01.production.efilm.com systemd[1]: Started GSSAPI Proxy Daemon.
grant@ef-idm01:~[20240220-14:37][#788]$ sudo ipactl status
Directory Service: RUNNING
krb5kdc Service: RUNNING
kadmin Service: RUNNING
httpd Service: RUNNING
ipa-custodia Service: RUNNING
pki-tomcatd Service: RUNNING
ipa-otpd Service: RUNNING
ipa: INFO: The ipactl command was successful
grant@ef-idm01:~[20240220-14:37][#789]$

I looked online for some references and it was suggested I replace the
/var/lib/ipa/gssproxy/http.keytab
The file looks OKAY to me though.