Wow ok, that was easy. `getcert list` now reports correct expiration dates for those certificates and they're all in MONITORING. It still has that ca-error field although it's no longer trying to renew. Is that going to be an issue or is it just going to try again when it's time to renew and succeed?

On Wed, Jul 8, 2020, 9:43 AM Florence Blanc-Renaud <flo@redhat.com> wrote:
On 7/6/20 7:59 PM, Ilya Kogan via FreeIPA-users wrote:
> Hi,
>
> Thanks for the help so far! I've actually run `ipa-cert-fix` on both
> nodes, it says everything is ok on both nodes. When I run it with
> verbose mode, it spits out the command it's running and the certificate
> it got, for example:
>
>     ```
>     ipapython.ipautil: DEBUG: args=['/usr/bin/certutil', '-d',
>     'sql:/etc/pki/pki-tomcat/alias', '-L', '-n', 'storageCert
>     cert-pki-kra', '-a', '-f', '/etc/pki/pki-tomcat/alias/pwdfile.txt']
>     ```
>
>
> If I then take that cert and ask what `openssl x509 -text -noout` thinks
> about it, it tells me that it's valid from 2020-06-29 to 2022-06-29.
> Strangely, though, when I ask `getcert list`, it shows that the certificate:
>
>     ```
>     certificate:
>     type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='storageCert cert-pki-kra',token='NSS
>     Certificate DB'
>     ```
>
>
> expires on 2020-06-27. It's almost as if this node's certificate has
> _already_ been renewed but certmonger (I think) doesn't know about it,
> which might be why it's having trouble renewing it.
>
Hi,

you may want to restart certmonger to force it re-reading the
certificate information:
# sudo systemctl restart certmonger

flo

> Here's what the two nodes say about replication:
>
>  From node one:
>
>     ```
>     ipa-two.mydomain.org <http://ipa-two.mydomain.org>
>        last update status: Error (0) Replica acquired successfully:
>     Incremental update succeeded
>        last update ended: 2020-07-06 17:46:17+00:00
>     ```
>
>
>  From node two:
>
>     ```
>     ipa-one.gaea.mythicnet.org <http://ipa-one.gaea.mythicnet.org>
>        last update status: Error (0) Replica acquired successfully:
>     Incremental update succeeded
>        last update ended: 2020-07-06 17:46:17+00:00
>     ```
>
>
> I suppose this might be a good time to mention that this is a simple two
> node multi-master setup. Finally, I'm not sure if I'm doing this
> correctly, but to make absolutely sure about which node is the renewal
> master, I ran this on both nodes:
>
>     ```
>     ldapsearch -H ldap://ipa-one.gaea.mythicnet.org
>     <http://ipa-one.gaea.mythicnet.org> -D 'cn=Directory Manager' -W -b
>     'cn=masters,cn=ipa,cn=etc,dc=mydomain,dc=org'
>     '(&(cn=CA)(ipaConfigString=caRenewalMaster))' dn
>     ldapsearch -H ldap://ipa-two.gaea.mythicnet.org
>     <http://ipa-two.gaea.mythicnet.org> -D 'cn=Directory Manager' -W -b
>     'cn=masters,cn=ipa,cn=etc,dc=mydomain,dc=org'
>     '(&(cn=CA)(ipaConfigString=caRenewalMaster))' dn
>     ```
>
>
> The result for both is:
>
>     ```
>     dn: cn=CA,cn=ipa-one.gaea.mythicnet.org
>     <http://ipa-one.gaea.mythicnet.org>,cn=masters,cn=ipa,cn=etc,dc=mydomain,dc=org
>     ```
>
>
> So it looks like the renewal master is the one having this problem.
>
>       
> Ilya Kogan
> w:    github.com/ikogan <http://github.com/ikogan> e: ikogan@mythicnet.org
> <mailto:ikogan@mythicnet.org>
> <http://twitter.com/ilkogan> <https://www.linkedin.com/in/ilyakogan/>
>
>
>
> On Mon, Jul 6, 2020 at 1:25 PM Rob Crittenden <rcritten@redhat.com
> <mailto:rcritten@redhat.com>> wrote:
>
>     Florence Blanc-Renaud via FreeIPA-users wrote:
>      > On 7/5/20 2:23 AM, Ilya Kogan via FreeIPA-users wrote:
>      >> Hi,
>      >>
>      >> I seem to be facing a similar issue with one of my KRAs. My KRA
>      >> certificates were, for some reason, not automatically renewed when
>      >> they expired last month. Using `ipa-cert-fix` correctly fixed
>     them on
>      >> _one_ host. On the other, they seem to be stuck in the renewal state
>      >> and `ipa-cert-fix` claims there's nothing to do:
>      >>
>      >> ```
>      >> Request ID '20191031183458':
>      >>          status: MONITORING
>      >>          ca-error: Server at
>      >> "http://ipa-one.mydomain.org:8080/ca/ee/ca/profileSubmit" replied:
>      >> Missing credential: sessionID
>      >>          stuck: no
>      >>          key pair storage:
>      >>
>     type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='auditSigningCert
>      >> cert-pki-kra',token='NSS Certificate DB',pin set
>      >>          certificate:
>      >>
>     type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='auditSigningCert
>      >> cert-pki-kra',token='NSS Certificate DB'
>      >>          CA: dogtag-ipa-ca-renew-agent
>      >>          issuer: CN=Certificate Authority,O=MYDOMAIN.ORG
>     <http://MYDOMAIN.ORG>
>      >> <http://MYDOMAIN.ORG>
>      >>          subject: CN=KRA Audit,O=MYDOMAIN.ORG
>     <http://MYDOMAIN.ORG> <http://MYDOMAIN.ORG>
>      >>          expires: 2020-06-27 01:54:34 EDT
>      >>          key usage: digitalSignature,nonRepudiation
>      >>          pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad
>      >>          post-save command:
>     /usr/libexec/ipa/certmonger/renew_ca_cert
>      >> "auditSigningCert cert-pki-kra"
>      >>          track: yes
>      >>          auto-renew: yes
>      >> Request ID '20191031183459':
>      >>          status: MONITORING
>      >>          ca-error: Server at
>      >> "http://ipa-one.mydomain.org:8080/ca/ee/ca/profileSubmit" replied:
>      >> Missing credential: sessionID
>      >>          stuck: no
>      >>          key pair storage:
>      >>
>     type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='transportCert
>     cert-pki-kra',token='NSS
>      >> Certificate DB',pin set
>      >>          certificate:
>      >>
>     type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='transportCert
>     cert-pki-kra',token='NSS
>      >> Certificate DB'
>      >>          CA: dogtag-ipa-ca-renew-agent
>      >>          issuer: CN=Certificate Authority,O=MYDOMAIN.ORG
>     <http://MYDOMAIN.ORG>
>      >> <http://MYDOMAIN.ORG>
>      >>          subject: CN=KRA Transport Certificate,O=MYDOMAIN.ORG
>     <http://MYDOMAIN.ORG>
>      >> <http://MYDOMAIN.ORG>
>      >>          expires: 2020-06-27 01:54:30 EDT
>      >>          key usage:
>      >> digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment
>      >>          eku: id-kp-clientAuth
>      >>          pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad
>      >>          post-save command:
>     /usr/libexec/ipa/certmonger/renew_ca_cert
>      >> "transportCert cert-pki-kra"
>      >>          track: yes
>      >>          auto-renew: yes
>      >> Request ID '20191031183500':
>      >>          status: MONITORING
>      >>          ca-error: Server at
>      >> "http://ipa-one.mydomain.org:8080/ca/ee/ca/profileSubmit" replied:
>      >> Missing credential: sessionID
>      >>          stuck: no
>      >>          key pair storage:
>      >>
>     type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='storageCert
>      >> cert-pki-kra',token='NSS Certificate DB',pin set
>      >>          certificate:
>      >>
>     type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='storageCert
>      >> cert-pki-kra',token='NSS Certificate DB'
>      >>          CA: dogtag-ipa-ca-renew-agent
>      >>          issuer: CN=Certificate Authority,O=MYDOMAIN.ORG
>     <http://MYDOMAIN.ORG>
>      >> <http://MYDOMAIN.ORG>
>      >>          subject: CN=KRA Storage Certificate,O=MYDOMAIN.ORG
>     <http://MYDOMAIN.ORG>
>      >> <http://MYDOMAIN.ORG>
>      >>          expires: 2020-06-27 01:54:32 EDT
>      >>          key usage:
>      >> digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment
>      >>          eku: id-kp-clientAuth
>      >>          pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad
>      >>          post-save command:
>     /usr/libexec/ipa/certmonger/renew_ca_cert
>      >> "storageCert cert-pki-kra"
>      >>          track: yes
>      >>          auto-renew: yes
>      >> ```
>      >>
>      >> Here are the sequence of events that seem to have led to this:
>      >>
>      >> 1. Install FreeIPA Master many years ago and continue to upgrade it
>      >> from time to time.
>      >> 2. Install FreeIPA Replica a few years after and continue to upgrade
>      >> it from time to time.
>      >> 3. Allow the certificates to expire on both nodes.
>      >> 4. Attempt to patch the replica via `yum upgrade` on the second
>     node.
>      >> 5. Notice after reboot that `pki-tomcatd` is having trouble and
>      >> discover certificate issues.
>      >> 5. Issue `ipa-cert-fix`, reboot again, and notice that things are
>      >> working. Try and create a key in the vault.
>      >> 6. Attempt to patch the master via `yum upgrade` on the first node.
>      >> 7. Notice after reboot that everything seems to be ok. Try and
>     create
>      >> a key in the vault.
>      >> 8. Notice a few days later that renewal seems to be broken on the
>      >> first node.
>      >>
>      >> At this point `ipa-cert-fix` just shows that everything is fine.
>     If I
>      >> run it with -v, and then check the "storageCert cert-pki-kra"
>      >> certificate with `openssl x509 -text -in`, I'm shown:
>      >
>      > Hi,
>      > just double-checking, but did you run ipa-cert-fix on the replica
>     that
>      > was repaired in step 5? If that's the case, it's normal that
>      > ipa-cert-fix does not see any issue as it's running only locally and
>      > does not attempt to repair remote nodes.
>      >
>      > You will need to login to the node with expired certs and run
>      > ipa-cert-fix there.
>
>     I'd also look to see which one is the renewal master. That is the one
>     that should renew the cert. I'm too curious why the renewal raised an
>     error (as if it actually tried to renew) rather than either go into
>     CA_WORKING or pick up the updated cert.
>
>     I'd also make sure that replication is working. On each master:
>
>     # ipa-csreplica-manage list -v `hostname`
>
>     rob
>
>      >
>      > HTH,
>      > flo
>      >
>      >>
>      >>          Validity
>      >>              Not Before: Jun 29 00:52:33 2020 GMT
>      >>              Not After : Jun 19 00:52:33 2022 GMT
>      >>
>      >> On the second known, `getcert list` shows correct expirations for
>      >> those certificates:
>      >>
>      >> Request ID '20191206005909':
>      >>          status: MONITORING
>      >>          stuck: no
>      >>          key pair storage:
>      >>
>     type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='storageCert
>      >> cert-pki-kra',token='NSS Certificate DB',pin set
>      >>          certificate:
>      >>
>     type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='storageCert
>      >> cert-pki-kra',token='NSS Certificate DB'
>      >>          CA: dogtag-ipa-ca-renew-agent
>      >>          issuer: CN=Certificate Authority,O=MYDOMAIN.ORG
>     <http://MYDOMAIN.ORG>
>      >> <http://MYDOMAIN.ORG>
>      >>          subject: CN=KRA Storage Certificate,O=MYDOMAIN.ORG
>     <http://MYDOMAIN.ORG>
>      >> <http://MYDOMAIN.ORG>
>      >>          expires: 2022-06-18 20:52:33 EDT
>      >>          key usage:
>      >> digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment
>      >>          eku: id-kp-clientAuth
>      >>          pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad
>      >>          post-save command:
>     /usr/libexec/ipa/certmonger/renew_ca_cert
>      >> "storageCert cert-pki-kra"
>      >>          track: yes
>      >>          auto-renew: yes
>      >>
>      >> It seems like _something_, perhaps `ipa-cert-fix` somehow renewed
>      >> these certificates but...outside of certmonger? Is this some other
>      >> version of https://bugzilla.redhat.com/show_bug.cgi?id=1788907? The
>      >> certificates are not in CA_WORKING though, they're in MONITORING.
>      >>
>      >> What can I do to get myself out of this state as it seems like
>     I'm in
>      >> a "this could explode at any moment" situation?
>      >>
>      >> This is on Fedora 30 with IP version:
>      >>
>      >> Last metadata expiration check: 0:23:05 ago on Sat 04 Jul 2020
>      >> 07:59:16 PM EDT.
>      >> Installed Packages
>      >> Name         : certmonger
>      >> Version      : 0.79.9
>      >> Release      : 1.fc30
>      >> Architecture : x86_64
>      >> Size         : 3.4 M
>      >> Source       : certmonger-0.79.9-1.fc30.src.rpm
>      >> Repository   : @System
>      >>  From repo    : updates
>      >>
>      >> .. snip ..
>      >>
>      >> Name         : freeipa-server
>      >> Version      : 4.8.3
>      >> Release      : 1.fc30
>      >> Architecture : x86_64
>      >> Size         : 1.3 M
>      >> Source       : freeipa-4.8.3-1.fc30.src.rpm
>      >> Repository   : @System
>      >>  From repo    : updates
>      >>
>      >> .. snip ..
>      >>
>      >> Thanks!
>      >>
>      >>
>      >> Ilya Kogan
>      >> w: github.com/ikogan <http://github.com/ikogan>
>     <http://github.com/ikogan> e:
>      >> ikogan@mythicnet.org <mailto:ikogan@mythicnet.org>
>     <mailto:ikogan@mythicnet.org <mailto:ikogan@mythicnet.org>>
>      >> <http://twitter.com/ilkogan>
>     <https://www.linkedin.com/in/ilyakogan/>
>      >>
>      >>
>      >> _______________________________________________
>      >> FreeIPA-users mailing list --
>     freeipa-users@lists.fedorahosted.org
>     <mailto:freeipa-users@lists.fedorahosted.org>
>      >> To unsubscribe send an email to
>      >> freeipa-users-leave@lists.fedorahosted.org
>     <mailto:freeipa-users-leave@lists.fedorahosted.org>
>      >> Fedora Code of Conduct:
>      >> https://docs.fedoraproject.org/en-US/project/code-of-conduct/
>      >> List Guidelines:
>     https://fedoraproject.org/wiki/Mailing_list_guidelines
>      >> List Archives:
>      >>
>     https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedorahosted.org
>      >>
>      >>
>      > _______________________________________________
>      > FreeIPA-users mailing list --
>     freeipa-users@lists.fedorahosted.org
>     <mailto:freeipa-users@lists.fedorahosted.org>
>      > To unsubscribe send an email to
>     freeipa-users-leave@lists.fedorahosted.org
>     <mailto:freeipa-users-leave@lists.fedorahosted.org>
>      > Fedora Code of Conduct:
>      > https://docs.fedoraproject.org/en-US/project/code-of-conduct/
>      > List Guidelines:
>     https://fedoraproject.org/wiki/Mailing_list_guidelines
>      > List Archives:
>      >
>     https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedorahosted.org
>      >
>
>
> _______________________________________________
> FreeIPA-users mailing list -- freeipa-users@lists.fedorahosted.org
> To unsubscribe send an email to freeipa-users-leave@lists.fedorahosted.org
> Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives: https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedorahosted.org
>