Health check issues

[Alex Corcoles]

Hi all,

Sorry I didn't keep track of this more accurately. Some time ago, the ipa-healthcheck service started failing (September 23rd, I think). I took a look, and IIRC, it said something like some certs were about to expire. I ignored that (because they renew automatically?). But then I checked some time after that, and ipa-healthcheck started reporting:

[
  {
    "source": "pki.server.healthcheck.meta.csconfig",
    "check": "CADogtagCertsConfigCheck",
    "result": "ERROR",
    "uuid": "af584c7d-6288-4848-acf8-9e59946e298b",
    "when": "20231004180708Z",
    "duration": "0.093486",
    "kw": {
      "key": "ca_audit_signing",
      "nickname": "auditSigningCert cert-pki-ca",
      "directive": "ca.audit_signing.cert",
      "configfile": "/etc/pki/pki-tomcat/ca/CS.cfg",
      "msg": "Certificate 'auditSigningCert cert-pki-ca' does not match the value of ca.audit_signing.cert in /etc/pki/pki-tomcat/ca/CS.cfg"
    }
  },
  {
    "source": "ipahealthcheck.dogtag.ca",
    "check": "DogtagCertsConfigCheck",
    "result": "ERROR",
    "uuid": "94d21af1-63d1-4bc8-80ff-dc974b3bafc2",
    "when": "20231004180708Z",
    "duration": "0.401906",
    "kw": {
      "key": "auditSigningCert cert-pki-ca",
      "directive": "ca.audit_signing.cert",
      "configfile": "/var/lib/pki/pki-tomcat/conf/ca/CS.cfg",
      "msg": "Certificate 'auditSigningCert cert-pki-ca' does not match the value of ca.audit_signing.cert in /var/lib/pki/pki-tomcat/conf/ca/CS.cfg"
    }
  }
]

I suppose the automatic renewal process went awry? I have seen messages on this list with similar errors, but the path forward does not seem clear to me.

I'm running:

ipa-healthcheck-0.12-1.el9.noarch
ipa-healthcheck-core-0.12-1.el9.noarch
ipa-server-4.10.1-9.el9_2.x86_64

Coincidentally, some updates went out around those dates:

2023-08-26T06:56:04+0000 SUBDEBUG Upgraded: ipa-server-dns-4.10.1-7.el9_2.noarch
2023-08-26T06:56:05+0000 SUBDEBUG Upgraded: ipa-server-4.10.1-7.el9_2.x86_64
2023-08-26T06:56:05+0000 SUBDEBUG Upgraded: python3-ipaserver-4.10.1-7.el9_2.noarch
2023-08-26T06:56:05+0000 SUBDEBUG Upgraded: ipa-client-4.10.1-7.el9_2.x86_64
2023-08-26T06:56:05+0000 SUBDEBUG Upgraded: python3-ipaclient-4.10.1-7.el9_2.noarch
2023-08-26T06:56:05+0000 SUBDEBUG Upgraded: python3-ipalib-4.10.1-7.el9_2.noarch
2023-08-26T06:56:05+0000 SUBDEBUG Upgraded: ipa-common-4.10.1-7.el9_2.noarch
2023-08-26T06:56:05+0000 SUBDEBUG Upgraded: ipa-server-common-4.10.1-7.el9_2.noarch
2023-08-26T06:56:05+0000 SUBDEBUG Upgraded: ipa-client-common-4.10.1-7.el9_2.noarch
2023-08-26T06:56:05+0000 SUBDEBUG Upgraded: ipa-selinux-4.10.1-7.el9_2.noarch
2023-09-24T06:56:28+0000 SUBDEBUG Upgraded: ipa-server-dns-4.10.1-8.el9_2.noarch
2023-09-24T06:56:28+0000 SUBDEBUG Upgraded: ipa-server-4.10.1-8.el9_2.x86_64
2023-09-24T06:56:29+0000 SUBDEBUG Upgraded: python3-ipaserver-4.10.1-8.el9_2.noarch
2023-09-24T06:56:29+0000 SUBDEBUG Upgraded: ipa-client-4.10.1-8.el9_2.x86_64
2023-09-24T06:56:29+0000 SUBDEBUG Upgraded: python3-ipaclient-4.10.1-8.el9_2.noarch
2023-09-24T06:56:29+0000 SUBDEBUG Upgraded: python3-ipalib-4.10.1-8.el9_2.noarch
2023-09-24T06:56:29+0000 SUBDEBUG Upgraded: ipa-common-4.10.1-8.el9_2.noarch
2023-09-24T06:56:30+0000 SUBDEBUG Upgraded: ipa-server-common-4.10.1-8.el9_2.noarch
2023-09-24T06:56:30+0000 SUBDEBUG Upgraded: ipa-client-common-4.10.1-8.el9_2.noarch
2023-09-24T06:56:30+0000 SUBDEBUG Upgraded: ipa-selinux-4.10.1-8.el9_2.noarch

Any thoughts?

Thanks,

Álex

[Alex Corcoles]

I forgot to add; I'm running two replicas, both are CAs and provisioned identically, and only one of them shows this issue.

[Jochen Kellner]

Alex Corcoles via FreeIPA-users 
writes:
> Hi all,
>
> Sorry I didn't keep track of this more accurately. Some time ago, the
> ipa-healthcheck service started failing (September 23rd, I think). I
> took a look, and IIRC, it said something like some certs were about to
> expire. I ignored that (because they renew automatically?). But then I
> checked some time after that, and ipa-healthcheck started reporting:

...
>   "msg": "Certificate 'auditSigningCert cert-pki-ca' does not match the value of ca.audit_signing.cert in /etc/pki/pki-tomcat/ca/CS.cfg"

...
> Any thoughts?

This looks similar to
https://pagure.io/freeipa/issue/9277
https://github.com/dogtagpki/pki/issues/2157

I've used this play to fix my system:
---
# file: freeipa-fixes.yml
- name: Fix problems in IPA installations or configurations after install / postinstall or later
  hosts:
  - ipaservers
  become: true

tasks:
# ...
  # Another healthcheck fix: when the PKI server certificate is renewed
  # the new certificate is written to /var/lib/pki/pki-tomcat/ca/conf/CS.cfg.
  # It needs to be in /var/lib/pki/pki-tomcat/kra/conf/CS.cfg too.
  # {
  #   "source": "pki.server.healthcheck.meta.csconfig",
  #   "check": "KRADogtagCertsConfigCheck",
  #   "result": "ERROR",
  #   "uuid": "892ad5b7-8612-4476-8120-2a5fe6c6b005",
  #   "when": "20221116030029Z",
  #   "duration": "0.024925",
  #   "kw": {
  #     "key": "kra_sslserver",
  #     "nickname": "Server-Cert cert-pki-ca",
  #     "directive": "kra.sslserver.cert",
  #     "configfile": "/var/lib/pki/pki-tomcat/kra/conf/CS.cfg",
  #     "msg": "Certificate 'Server-Cert cert-pki-ca' does not match the value
  #  of kra.sslserver.cert in /var/lib/pki/pki-tomcat/kra/conf/CS.cfg"
  #   }
  # },
  # This is likely a bug in /usr/libexec/ipa/certmonger/renew_ca_cert
  - name: Fetch ca.sslserver.cert from /var/lib/pki/pki-tomcat/ca/conf/CS.cfg
    ansible.builtin.command:
      cmd: awk -F '=' '/^ca.sslserver.cert=/ { print $2 }' /var/lib/pki/pki-tomcat/ca/conf/CS.cfg
    register: ca_sslserver_cert
    check_mode: false
    changed_when: false

- name: Fetch kra.sslserver.cert= from /var/lib/pki/pki-tomcat/kra/conf/CS.cfg
    ansible.builtin.command:
      cmd: awk -F '=' '/^kra.sslserver.cert=/ { print $2 }' /var/lib/pki/pki-tomcat/kra/conf/CS.cfg
    register: kra_sslserver_cert
    check_mode: false
    changed_when: false

#  - name: debug display the possibly different certs
#    ansible.builtin.debug:
#      var: "{{ item }}"
#    loop:
#    - ca_sslserver_cert.stdout
#    - kra_sslserver_cert.stdout

- name: Fix ipa-healthcheck, KRADogtagCertsConfigCheck
    ansible.builtin.lineinfile:
      dest: /var/lib/pki/pki-tomcat/kra/conf/CS.cfg
      regexp: '^kra.sslserver.cert='
      line: 'kra.sslserver.cert={{ ca_sslserver_cert.stdout }}'
      owner: pkiuser
      group: pkiuser
      mode: '0660'
      backup: true
    when: ca_sslserver_cert.stdout != kra_sslserver_cert.stdout
    notify: Restart pki-tomcat

#      "key": "transportCert cert-pki-kra",
#     "directive": "ca.connector.KRA.transportCert",
#     "configfile": "/var/lib/pki/pki-tomcat/conf/ca/CS.cfg",
#     "msg": "Certificate 'transportCert cert-pki-kra' does not match the value of
#      ca.connector.KRA.transportCert in /var/lib/pki/pki-tomcat/c onf/ca/CS.cfg"
  - name: Fetch Certificate 'transportCert cert-pki-kra'
    ansible.builtin.shell:
      cmd: certutil -d /etc/pki/pki-tomcat/alias/ -L -n 'transportCert cert-pki-kra' -a | awk '/^[^-]/ { sub(/\r/, ""); printf("%s", $0) }'
    register: transportcert
    check_mode: false
    changed_when: false

- name: Fetch Certificate ca.connector.KRA.transportCert
    ansible.builtin.shell:
      cmd: awk -F '=' '/^ca.connector.KRA.transportCert=/ { print $2 }' /var/lib/pki/pki-tomcat/ca/conf/CS.cfg
    register: ca_connector_transportcert
    check_mode: false
    changed_when: false

- name: Fix ipa-healthcheck, ca.connector.KRA.transportCert
    ansible.builtin.lineinfile:
      dest: /var/lib/pki/pki-tomcat/ca/conf/CS.cfg
      regexp: '^ca.connector.KRA.transportCert='
      line: 'ca.connector.KRA.transportCert={{ transportcert.stdout }}'
      owner: pkiuser
      group: pkiuser
      mode: '0660'
      backup: true
    when: ca_connector_transportcert.stdout != transportcert.stdout
    notify: Restart pki-tomcat

- name: Fetch Certificate kra.transport.cert
    ansible.builtin.shell:
      cmd: awk -F '=' '/^kra.transport.cert=/ { print $2 }' /var/lib/pki/pki-tomcat/kra/conf/CS.cfg
    register: kra_transport_cert
    check_mode: false
    changed_when: false

- name: Fix ipa-healthcheck, kra.transport.cert
    ansible.builtin.lineinfile:
      dest: /var/lib/pki/pki-tomcat/kra/conf/CS.cfg
      regexp: '^kra.transport.cert='
      line: 'kra.transport.cert={{ transportcert.stdout }}'
      owner: pkiuser
      group: pkiuser
      mode: '0660'
      backup: true
    when: kra_transport_cert.stdout != transportcert.stdout
    notify: Restart pki-tomcat

- name: Fetch Certificate ca.connector.KRA.transportCert
    ansible.builtin.shell:
      cmd: awk -F '=' '/^ca.connector.KRA.transportCert=/ { print $2 }' /var/lib/pki/pki-tomcat/ca/conf/CS.cfg
    register: ca_connector_transportcert
    check_mode: false
    changed_when: false

- name: Fix ipa-healthcheck, ca.connector.KRA.transportCert
    ansible.builtin.lineinfile:
      dest: /var/lib/pki/pki-tomcat/ca/conf/CS.cfg
      regexp: '^ca.connector.KRA.transportCert='
      line: 'ca.connector.KRA.transportCert={{ transportcert.stdout }}'
      owner: pkiuser
      group: pkiuser
      mode: '0660'
      backup: true
    when: ca_connector_transportcert.stdout != transportcert.stdout
    notify: Restart pki-tomcat

#      "nickname": "subsystemCert cert-pki-ca",
#      "directive": "kra.subsystem.cert",
#      "configfile": "/var/lib/pki/pki-tomcat/kra/conf/CS.cfg",
#      "msg": "Certificate 'subsystemCert cert-pki-ca' does not match the value
#      of kra.subsystem.cert in /var/lib/pki/pki-tomcat/kra/conf/CS.cfg"

- name: Fetch Certificate 'subsystemCert cert-pki-ca'
    ansible.builtin.shell:
      cmd: certutil -d /etc/pki/pki-tomcat/alias/ -L -n 'subsystemCert cert-pki-ca' -a | awk '/^[^-]/ { sub(/\r/, ""); printf("%s", $0) }'
    register: subsystemcert
    check_mode: false
    changed_when: false

- name: Fetch Certificate kra.subsystem.cert
    ansible.builtin.shell:
      cmd: awk -F '=' '/^kra.subsystem.cert=/ { print $2 }' /var/lib/pki/pki-tomcat/kra/conf/CS.cfg
    register: kra_subsystem_cert
    check_mode: false
    changed_when: false

- name: Fix ipa-healthcheck, kra.subsystem.cert
    ansible.builtin.lineinfile:
      dest: /var/lib/pki/pki-tomcat/kra/conf/CS.cfg
      regexp: '^kra.subsystem.cert='
      line: 'kra.subsystem.cert={{ subsystemcert.stdout }}'
      owner: pkiuser
      group: pkiuser
      mode: '0660'
      backup: true
    when: kra_subsystem_cert.stdout != subsystemcert.stdout
    notify: Restart pki-tomcat

#      "nickname": "storageCert cert-pki-kra",
#      "directive": "kra.storage.cert",
#      "configfile": "/var/lib/pki/pki-tomcat/kra/conf/CS.cfg",
#      "msg": "Certificate 'storageCert cert-pki-kra' does not match the value
#      of kra.storage.cert in /var/lib/pki/pki-tomcat/kra/conf/CS.cfg"

- name: Fetch Certificate 'storageCert cert-pki-kra'
    ansible.builtin.shell:
      cmd: certutil -d /etc/pki/pki-tomcat/alias/ -L -n 'storageCert cert-pki-kra' -a | awk '/^[^-]/ { sub(/\r/, ""); printf("%s", $0) }'
    register: storagecert
    check_mode: false
    changed_when: false

- name: Fetch Certificate kra.storage.cert
    ansible.builtin.shell:
      cmd: awk -F '=' '/^kra.storage.cert=/ { print $2 }' /var/lib/pki/pki-tomcat/kra/conf/CS.cfg
    register: kra_storage_cert
    check_mode: false
    changed_when: false

- name: Fix ipa-healthcheck, kra.storage.cert
    ansible.builtin.lineinfile:
      dest: /var/lib/pki/pki-tomcat/kra/conf/CS.cfg
      regexp: '^kra.storage.cert='
      line: 'kra.storage.cert={{ storagecert.stdout }}'
      owner: pkiuser
      group: pkiuser
      mode: '0660'
      backup: true
    when: storagecert.stdout != kra_storage_cert.stdout
    notify: Restart pki-tomcat

#      "nickname": "auditSigningCert cert-pki-kra",
#      "directive": "kra.audit_signing.cert",
#      "configfile": "/var/lib/pki/pki-tomcat/kra/conf/CS.cfg",
#      "msg": "Certificate 'auditSigningCert cert-pki-kra' does not match the
# value of kra.audit_signing.cert in /var/lib/pki/pki-tomcat/kra/conf/CS.cfg"

- name: Fetch Certificate 'auditSigningCert cert-pki-kra'
    ansible.builtin.shell:
      cmd: certutil -d /etc/pki/pki-tomcat/alias/ -L -n 'auditSigningCert cert-pki-kra' -a | awk '/^[^-]/ { sub(/\r/, ""); printf("%s", $0) }'
    register: auditsigningcert
    check_mode: false
    changed_when: false

- name: Fetch Certificate kra.audit_signing.cert
    ansible.builtin.shell:
      cmd: awk -F '=' '/^kra.audit_signing.cert=/ { print $2 }' /var/lib/pki/pki-tomcat/kra/conf/CS.cfg
    register: kra_audit_signing_cert
    check_mode: false
    changed_when: false

- name: Fix ipa-healthcheck, kra.audit_signing.cert
    ansible.builtin.lineinfile:
      dest: /var/lib/pki/pki-tomcat/kra/conf/CS.cfg
      regexp: '^kra.audit_signing.cert='
      line: 'kra.audit_signing.cert={{ auditsigningcert.stdout }}'
      owner: pkiuser
      group: pkiuser
      mode: '0660'
      backup: true
    when: kra_audit_signing_cert.stdout != auditsigningcert.stdout
    notify: Restart pki-tomcat

handlers:
# ...
  - name: Restart pki-tomcat
    ansible.builtin.service:
      name: [e-mail]
      state: restarted

[Alex Corcoles]

Oh, thanks for the playbook- I appreciate it.

It's surprising that some of the bugs you posted mention SELinux- the replica that doesn't have issues is running SELinux, while the replica that has issues doesn't (it's an LXC container).

[Rob Crittenden]

Jochen Kellner via FreeIPA-users wrote:
> Alex Corcoles via FreeIPA-users 
> writes:
> > Hi all,
> >
> > Sorry I didn't keep track of this more accurately. Some time ago, the
> > ipa-healthcheck service started failing (September 23rd, I think). I
> > took a look, and IIRC, it said something like some certs were about to
> > expire. I ignored that (because they renew automatically?). But then I
> > checked some time after that, and ipa-healthcheck started reporting:
>
> ...
> >   "msg": "Certificate 'auditSigningCert cert-pki-ca' does not match the value of ca.audit_signing.cert in /etc/pki/pki-tomcat/ca/CS.cfg"
>
> ...
> > Any thoughts?
>
> This looks similar to
> https://pagure.io/freeipa/issue/9277
> https://github.com/dogtagpki/pki/issues/2157

The KRA values are definitely not being updated. That shouldn't be the
case for the CA values.

rob
> I've used this play to fix my system:
>
> # file: freeipa-fixes.yml
>
> name: Fix problems in IPA installations or configurations after install / postinstall or later
> hosts:
> ipaservers
>
> become: true
>
> tasks:
>
> # ...
>   # Another healthcheck fix: when the PKI server certificate is renewed
>   # the new certificate is written to /var/lib/pki/pki-tomcat/ca/conf/CS.cfg.
>   # It needs to be in /var/lib/pki/pki-tomcat/kra/conf/CS.cfg too.
>   # {
>   #   "source": "pki.server.healthcheck.meta.csconfig",
>   #   "check": "KRADogtagCertsConfigCheck",
>   #   "result": "ERROR",
>   #   "uuid": "892ad5b7-8612-4476-8120-2a5fe6c6b005",
>   #   "when": "20221116030029Z",
>   #   "duration": "0.024925",
>   #   "kw": {
>   #     "key": "kra_sslserver",
>   #     "nickname": "Server-Cert cert-pki-ca",
>   #     "directive": "kra.sslserver.cert",
>   #     "configfile": "/var/lib/pki/pki-tomcat/kra/conf/CS.cfg",
>   #     "msg": "Certificate 'Server-Cert cert-pki-ca' does not match the value
>   #  of kra.sslserver.cert in /var/lib/pki/pki-tomcat/kra/conf/CS.cfg"
>   #   }
>   # },
>   # This is likely a bug in /usr/libexec/ipa/certmonger/renew_ca_cert
>
> name: Fetch ca.sslserver.cert from /var/lib/pki/pki-tomcat/ca/conf/CS.cfg
> ansible.builtin.command:
>   cmd: awk -F '=' '/^ca.sslserver.cert=/ { print $2 }' /var/lib/pki/pki-tomcat/ca/conf/CS.cfg
> register: ca_sslserver_cert
> check_mode: false
> changed_when: false
>
> name: Fetch kra.sslserver.cert= from /var/lib/pki/pki-tomcat/kra/conf/CS.cfg
> ansible.builtin.command:
>   cmd: awk -F '=' '/^kra.sslserver.cert=/ { print $2 }' /var/lib/pki/pki-tomcat/kra/conf/CS.cfg
> register: kra_sslserver_cert
> check_mode: false
> changed_when: false
>
> #  - name: debug display the possibly different certs
> #    ansible.builtin.debug:
> #      var: "{{ item }}"
> #    loop:
> #    - ca_sslserver_cert.stdout
> #    - kra_sslserver_cert.stdout
> name: Fix ipa-healthcheck, KRADogtagCertsConfigCheck
> ansible.builtin.lineinfile:
>   dest: /var/lib/pki/pki-tomcat/kra/conf/CS.cfg
>   regexp: '^kra.sslserver.cert='
>   line: 'kra.sslserver.cert={{ ca_sslserver_cert.stdout }}'
>   owner: pkiuser
>   group: pkiuser
>   mode: '0660'
>   backup: true
> when: ca_sslserver_cert.stdout != kra_sslserver_cert.stdout
> notify: Restart pki-tomcat
>
> #      "key": "transportCert cert-pki-kra",
> #     "directive": "ca.connector.KRA.transportCert",
> #     "configfile": "/var/lib/pki/pki-tomcat/conf/ca/CS.cfg",
> #     "msg": "Certificate 'transportCert cert-pki-kra' does not match the value of
> #      ca.connector.KRA.transportCert in /var/lib/pki/pki-tomcat/c onf/ca/CS.cfg"
>
> name: Fetch Certificate 'transportCert cert-pki-kra'
> ansible.builtin.shell:
>   cmd: certutil -d /etc/pki/pki-tomcat/alias/ -L -n 'transportCert cert-pki-kra' -a | awk '/^[^-]/ { sub(/\r/, ""); printf("%s", $0) }'
> register: transportcert
> check_mode: false
> changed_when: false
>
> name: Fetch Certificate ca.connector.KRA.transportCert
> ansible.builtin.shell:
>   cmd: awk -F '=' '/^ca.connector.KRA.transportCert=/ { print $2 }' /var/lib/pki/pki-tomcat/ca/conf/CS.cfg
> register: ca_connector_transportcert
> check_mode: false
> changed_when: false
>
> name: Fix ipa-healthcheck, ca.connector.KRA.transportCert
> ansible.builtin.lineinfile:
>   dest: /var/lib/pki/pki-tomcat/ca/conf/CS.cfg
>   regexp: '^ca.connector.KRA.transportCert='
>   line: 'ca.connector.KRA.transportCert={{ transportcert.stdout }}'
>   owner: pkiuser
>   group: pkiuser
>   mode: '0660'
>   backup: true
> when: ca_connector_transportcert.stdout != transportcert.stdout
> notify: Restart pki-tomcat
>
> name: Fetch Certificate kra.transport.cert
> ansible.builtin.shell:
>   cmd: awk -F '=' '/^kra.transport.cert=/ { print $2 }' /var/lib/pki/pki-tomcat/kra/conf/CS.cfg
> register: kra_transport_cert
> check_mode: false
> changed_when: false
>
> name: Fix ipa-healthcheck, kra.transport.cert
> ansible.builtin.lineinfile:
>   dest: /var/lib/pki/pki-tomcat/kra/conf/CS.cfg
>   regexp: '^kra.transport.cert='
>   line: 'kra.transport.cert={{ transportcert.stdout }}'
>   owner: pkiuser
>   group: pkiuser
>   mode: '0660'
>   backup: true
> when: kra_transport_cert.stdout != transportcert.stdout
> notify: Restart pki-tomcat
>
> name: Fetch Certificate ca.connector.KRA.transportCert
> ansible.builtin.shell:
>   cmd: awk -F '=' '/^ca.connector.KRA.transportCert=/ { print $2 }' /var/lib/pki/pki-tomcat/ca/conf/CS.cfg
> register: ca_connector_transportcert
> check_mode: false
> changed_when: false
>
> name: Fix ipa-healthcheck, ca.connector.KRA.transportCert
> ansible.builtin.lineinfile:
>   dest: /var/lib/pki/pki-tomcat/ca/conf/CS.cfg
>   regexp: '^ca.connector.KRA.transportCert='
>   line: 'ca.connector.KRA.transportCert={{ transportcert.stdout }}'
>   owner: pkiuser
>   group: pkiuser
>   mode: '0660'
>   backup: true
> when: ca_connector_transportcert.stdout != transportcert.stdout
> notify: Restart pki-tomcat
>
> #      "nickname": "subsystemCert cert-pki-ca",
> #      "directive": "kra.subsystem.cert",
> #      "configfile": "/var/lib/pki/pki-tomcat/kra/conf/CS.cfg",
> #      "msg": "Certificate 'subsystemCert cert-pki-ca' does not match the value
> #      of kra.subsystem.cert in /var/lib/pki/pki-tomcat/kra/conf/CS.cfg"
>
> name: Fetch Certificate 'subsystemCert cert-pki-ca'
> ansible.builtin.shell:
>   cmd: certutil -d /etc/pki/pki-tomcat/alias/ -L -n 'subsystemCert cert-pki-ca' -a | awk '/^[^-]/ { sub(/\r/, ""); printf("%s", $0) }'
> register: subsystemcert
> check_mode: false
> changed_when: false
>
> name: Fetch Certificate kra.subsystem.cert
> ansible.builtin.shell:
>   cmd: awk -F '=' '/^kra.subsystem.cert=/ { print $2 }' /var/lib/pki/pki-tomcat/kra/conf/CS.cfg
> register: kra_subsystem_cert
> check_mode: false
> changed_when: false
>
> name: Fix ipa-healthcheck, kra.subsystem.cert
> ansible.builtin.lineinfile:
>   dest: /var/lib/pki/pki-tomcat/kra/conf/CS.cfg
>   regexp: '^kra.subsystem.cert='
>   line: 'kra.subsystem.cert={{ subsystemcert.stdout }}'
>   owner: pkiuser
>   group: pkiuser
>   mode: '0660'
>   backup: true
> when: kra_subsystem_cert.stdout != subsystemcert.stdout
> notify: Restart pki-tomcat
>
> #      "nickname": "storageCert cert-pki-kra",
> #      "directive": "kra.storage.cert",
> #      "configfile": "/var/lib/pki/pki-tomcat/kra/conf/CS.cfg",
> #      "msg": "Certificate 'storageCert cert-pki-kra' does not match the value
> #      of kra.storage.cert in /var/lib/pki/pki-tomcat/kra/conf/CS.cfg"
>
> name: Fetch Certificate 'storageCert cert-pki-kra'
> ansible.builtin.shell:
>   cmd: certutil -d /etc/pki/pki-tomcat/alias/ -L -n 'storageCert cert-pki-kra' -a | awk '/^[^-]/ { sub(/\r/, ""); printf("%s", $0) }'
> register: storagecert
> check_mode: false
> changed_when: false
>
> name: Fetch Certificate kra.storage.cert
> ansible.builtin.shell:
>   cmd: awk -F '=' '/^kra.storage.cert=/ { print $2 }' /var/lib/pki/pki-tomcat/kra/conf/CS.cfg
> register: kra_storage_cert
> check_mode: false
> changed_when: false
>
> name: Fix ipa-healthcheck, kra.storage.cert
> ansible.builtin.lineinfile:
>   dest: /var/lib/pki/pki-tomcat/kra/conf/CS.cfg
>   regexp: '^kra.storage.cert='
>   line: 'kra.storage.cert={{ storagecert.stdout }}'
>   owner: pkiuser
>   group: pkiuser
>   mode: '0660'
>   backup: true
> when: storagecert.stdout != kra_storage_cert.stdout
> notify: Restart pki-tomcat
>
> #      "nickname": "auditSigningCert cert-pki-kra",
> #      "directive": "kra.audit_signing.cert",
> #      "configfile": "/var/lib/pki/pki-tomcat/kra/conf/CS.cfg",
> #      "msg": "Certificate 'auditSigningCert cert-pki-kra' does not match the
> # value of kra.audit_signing.cert in /var/lib/pki/pki-tomcat/kra/conf/CS.cfg"
>
> name: Fetch Certificate 'auditSigningCert cert-pki-kra'
> ansible.builtin.shell:
>   cmd: certutil -d /etc/pki/pki-tomcat/alias/ -L -n 'auditSigningCert cert-pki-kra' -a | awk '/^[^-]/ { sub(/\r/, ""); printf("%s", $0) }'
> register: auditsigningcert
> check_mode: false
> changed_when: false
>
> name: Fetch Certificate kra.audit_signing.cert
> ansible.builtin.shell:
>   cmd: awk -F '=' '/^kra.audit_signing.cert=/ { print $2 }' /var/lib/pki/pki-tomcat/kra/conf/CS.cfg
> register: kra_audit_signing_cert
> check_mode: false
> changed_when: false
>
> name: Fix ipa-healthcheck, kra.audit_signing.cert
> ansible.builtin.lineinfile:
>   dest: /var/lib/pki/pki-tomcat/kra/conf/CS.cfg
>   regexp: '^kra.audit_signing.cert='
>   line: 'kra.audit_signing.cert={{ auditsigningcert.stdout }}'
>   owner: pkiuser
>   group: pkiuser
>   mode: '0660'
>   backup: true
> when: kra_audit_signing_cert.stdout != auditsigningcert.stdout
> notify: Restart pki-tomcat
>
> handlers:
> # ...
> name: Restart pki-tomcat
> ansible.builtin.service:
>   name: [e-mail]
>   state: restarted

[Rob Crittenden]

Alex Corcoles via FreeIPA-users wrote:
> Hi all,
>
> Sorry I didn't keep track of this more accurately. Some time ago, the ipa-healthcheck service started failing (September 23rd, I think). I took a look, and IIRC, it said something like some certs were about to expire. I ignored that (because they renew automatically?). But then I checked some time after that, and ipa-healthcheck started reporting:

I'd start by verifying that the certificates indeed did renew.
> [
>   {
>     "source": "pki.server.healthcheck.meta.csconfig",
>     "check": "CADogtagCertsConfigCheck",
>     "result": "ERROR",
>     "uuid": "af584c7d-6288-4848-acf8-9e59946e298b",
>     "when": "20231004180708Z",
>     "duration": "0.093486",
>     "kw": {
>       "key": "ca_audit_signing",
>       "nickname": "auditSigningCert cert-pki-ca",
>       "directive": "ca.audit_signing.cert",
>       "configfile": "/etc/pki/pki-tomcat/ca/CS.cfg",
>       "msg": "Certificate 'auditSigningCert cert-pki-ca' does not match the value of ca.audit_signing.cert in /etc/pki/pki-tomcat/ca/CS.cfg"
>     }
>   },
>   {
>     "source": "ipahealthcheck.dogtag.ca",
>     "check": "DogtagCertsConfigCheck",
>     "result": "ERROR",
>     "uuid": "94d21af1-63d1-4bc8-80ff-dc974b3bafc2",
>     "when": "20231004180708Z",
>     "duration": "0.401906",
>     "kw": {
>       "key": "auditSigningCert cert-pki-ca",
>       "directive": "ca.audit_signing.cert",
>       "configfile": "/var/lib/pki/pki-tomcat/conf/ca/CS.cfg",
>       "msg": "Certificate 'auditSigningCert cert-pki-ca' does not match the value of ca.audit_signing.cert in /var/lib/pki/pki-tomcat/conf/ca/CS.cfg"
>     }
>   }
> ]
>
> I suppose the automatic renewal process went awry? I have seen messages on this list with similar errors, but the path forward does not seem clear to me.

There is some disagreement whether CS.cfg being updated is important or
not. The PKI team is looking into this now. If you really want to update
it you can get the base64 blob:

# certutil -L -d /etc/pki/pki-tomcat/alias -n 'auditSigningCert
cert-pki-ca' -a

Then stop pki-tomcat@pki-tomcatd, update the mentioned blob in CS.cfg,
and restart tomcat.

rob
> I'm running:
>
> ipa-healthcheck-0.12-1.el9.noarch
> ipa-healthcheck-core-0.12-1.el9.noarch
> ipa-server-4.10.1-9.el9_2.x86_64
>
> Coincidentally, some updates went out around those dates:
>
> 2023-08-26T06:56:04+0000 SUBDEBUG Upgraded: ipa-server-dns-4.10.1-7.el9_2.noarch
> 2023-08-26T06:56:05+0000 SUBDEBUG Upgraded: ipa-server-4.10.1-7.el9_2.x86_64
> 2023-08-26T06:56:05+0000 SUBDEBUG Upgraded: python3-ipaserver-4.10.1-7.el9_2.noarch
> 2023-08-26T06:56:05+0000 SUBDEBUG Upgraded: ipa-client-4.10.1-7.el9_2.x86_64
> 2023-08-26T06:56:05+0000 SUBDEBUG Upgraded: python3-ipaclient-4.10.1-7.el9_2.noarch
> 2023-08-26T06:56:05+0000 SUBDEBUG Upgraded: python3-ipalib-4.10.1-7.el9_2.noarch
> 2023-08-26T06:56:05+0000 SUBDEBUG Upgraded: ipa-common-4.10.1-7.el9_2.noarch
> 2023-08-26T06:56:05+0000 SUBDEBUG Upgraded: ipa-server-common-4.10.1-7.el9_2.noarch
> 2023-08-26T06:56:05+0000 SUBDEBUG Upgraded: ipa-client-common-4.10.1-7.el9_2.noarch
> 2023-08-26T06:56:05+0000 SUBDEBUG Upgraded: ipa-selinux-4.10.1-7.el9_2.noarch
> 2023-09-24T06:56:28+0000 SUBDEBUG Upgraded: ipa-server-dns-4.10.1-8.el9_2.noarch
> 2023-09-24T06:56:28+0000 SUBDEBUG Upgraded: ipa-server-4.10.1-8.el9_2.x86_64
> 2023-09-24T06:56:29+0000 SUBDEBUG Upgraded: python3-ipaserver-4.10.1-8.el9_2.noarch
> 2023-09-24T06:56:29+0000 SUBDEBUG Upgraded: ipa-client-4.10.1-8.el9_2.x86_64
> 2023-09-24T06:56:29+0000 SUBDEBUG Upgraded: python3-ipaclient-4.10.1-8.el9_2.noarch
> 2023-09-24T06:56:29+0000 SUBDEBUG Upgraded: python3-ipalib-4.10.1-8.el9_2.noarch
> 2023-09-24T06:56:29+0000 SUBDEBUG Upgraded: ipa-common-4.10.1-8.el9_2.noarch
> 2023-09-24T06:56:30+0000 SUBDEBUG Upgraded: ipa-server-common-4.10.1-8.el9_2.noarch
> 2023-09-24T06:56:30+0000 SUBDEBUG Upgraded: ipa-client-common-4.10.1-8.el9_2.noarch
> 2023-09-24T06:56:30+0000 SUBDEBUG Upgraded: ipa-selinux-4.10.1-8.el9_2.noarch
>
> Any thoughts?
>
> Thanks,
>
> Álex
> _______________________________________________
> FreeIPA-users mailing list -- [e-mail]
> To unsubscribe send an email to [e-mail]
> Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives: https://lists.fedorahosted.org/archives/list/[e-mail]...
> Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue