Ilya Kogan wrote:
> Wow ok, that was easy. `getcert list` now reports correct expiration
> dates for those certificates and they're all in MONITORING. It still has
> that ca-error field although it's no longer trying to renew. Is that
> going to be an issue or is it just going to try again when it's time to
> renew and succeed?
I don't know. I'd check the journal to see if it logged any errors
post-restart. I don't believe that the ca-error is stored between
restarts. You could grep in /var/lib/certmonger/requests to see I suppose.
rob
>
> On Wed, Jul 8, 2020, 9:43 AM Florence Blanc-Renaud <flo@redhat.com
> <mailto:flo@redhat.com>> wrote:
>
> On 7/6/20 7:59 PM, Ilya Kogan via FreeIPA-users wrote:
> > Hi,
> >
> > Thanks for the help so far! I've actually run `ipa-cert-fix` on both
> > nodes, it says everything is ok on both nodes. When I run it with
> > verbose mode, it spits out the command it's running and the
> certificate
> > it got, for example:
> >
> > ```
> > ipapython.ipautil: DEBUG: args=['/usr/bin/certutil', '-d',
> > 'sql:/etc/pki/pki-tomcat/alias', '-L', '-n', 'storageCert
> > cert-pki-kra', '-a', '-f',
> '/etc/pki/pki-tomcat/alias/pwdfile.txt']
> > ```
> >
> >
> > If I then take that cert and ask what `openssl x509 -text -noout`
> thinks
> > about it, it tells me that it's valid from 2020-06-29 to 2022-06-29.
> > Strangely, though, when I ask `getcert list`, it shows that the
> certificate:
> >
> > ```
> > certificate:
> >
> type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='storageCert
> cert-pki-kra',token='NSS
> > Certificate DB'
> > ```
> >
> >
> > expires on 2020-06-27. It's almost as if this node's certificate has
> > _already_ been renewed but certmonger (I think) doesn't know about
> it,
> > which might be why it's having trouble renewing it.
> >
> Hi,
>
> you may want to restart certmonger to force it re-reading the
> certificate information:
> # sudo systemctl restart certmonger
>
> flo
>
> > Here's what the two nodes say about replication:
> >
> > From node one:
> >
> > ```
> > ipa-two.mydomain.org <http://ipa-two.mydomain.org>
> <http://ipa-two.mydomain.org>
> > last update status: Error (0) Replica acquired successfully:
> > Incremental update succeeded
> > last update ended: 2020-07-06 17:46:17+00:00
> > ```
> >
> >
> > From node two:
> >
> > ```
> > ipa-one.gaea.mythicnet.org <http://ipa-one.gaea.mythicnet.org>
> <http://ipa-one.gaea.mythicnet.org>
> > last update status: Error (0) Replica acquired successfully:
> > Incremental update succeeded
> > last update ended: 2020-07-06 17:46:17+00:00
> > ```
> >
> >
> > I suppose this might be a good time to mention that this is a
> simple two
> > node multi-master setup. Finally, I'm not sure if I'm doing this
> > correctly, but to make absolutely sure about which node is the
> renewal
> > master, I ran this on both nodes:
> >
> > ```
> > ldapsearch -H ldap://ipa-one.gaea.mythicnet.org
> <http://ipa-one.gaea.mythicnet.org>
> > <http://ipa-one.gaea.mythicnet.org> -D 'cn=Directory Manager'
> -W -b
> > 'cn=masters,cn=ipa,cn=etc,dc=mydomain,dc=org'
> > '(&(cn=CA)(ipaConfigString=caRenewalMaster))' dn
> > ldapsearch -H ldap://ipa-two.gaea.mythicnet.org
> <http://ipa-two.gaea.mythicnet.org>
> > <http://ipa-two.gaea.mythicnet.org> -D 'cn=Directory Manager'
> -W -b
> > 'cn=masters,cn=ipa,cn=etc,dc=mydomain,dc=org'
> > '(&(cn=CA)(ipaConfigString=caRenewalMaster))' dn
> > ```
> >
> >
> > The result for both is:
> >
> > ```
> > dn: cn=CA,cn=ipa-one.gaea.mythicnet.org
> <http://ipa-one.gaea.mythicnet.org>
> >
> <http://ipa-one.gaea.mythicnet.org>,cn=masters,cn=ipa,cn=etc,dc=mydomain,dc=org
> > ```
> >
> >
> > So it looks like the renewal master is the one having this problem.
> >
> >
> > Ilya Kogan
> > w: github.com/ikogan <http://github.com/ikogan>
> <http://github.com/ikogan> e: ikogan@mythicnet.org
> <mailto:ikogan@mythicnet.org>
> > <mailto:ikogan@mythicnet.org <mailto:ikogan@mythicnet.org>>
> > <http://twitter.com/ilkogan> <https://www.linkedin.com/in/ilyakogan/>
> >
> >
> >
> > On Mon, Jul 6, 2020 at 1:25 PM Rob Crittenden <rcritten@redhat.com
> <mailto:rcritten@redhat.com>
> > <mailto:rcritten@redhat.com <mailto:rcritten@redhat.com>>> wrote:
> >
> > Florence Blanc-Renaud via FreeIPA-users wrote:
> > > On 7/5/20 2:23 AM, Ilya Kogan via FreeIPA-users wrote:
> > >> Hi,
> > >>
> > >> I seem to be facing a similar issue with one of my KRAs.
> My KRA
> > >> certificates were, for some reason, not automatically
> renewed when
> > >> they expired last month. Using `ipa-cert-fix` correctly fixed
> > them on
> > >> _one_ host. On the other, they seem to be stuck in the
> renewal state
> > >> and `ipa-cert-fix` claims there's nothing to do:
> > >>
> > >> ```
> > >> Request ID '20191031183458':
> > >> status: MONITORING
> > >> ca-error: Server at
> > >> "http://ipa-one.mydomain.org:8080/ca/ee/ca/profileSubmit"
> replied:
> > >> Missing credential: sessionID
> > >> stuck: no
> > >> key pair storage:
> > >>
> >
> type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='auditSigningCert
> > >> cert-pki-kra',token='NSS Certificate DB',pin set
> > >> certificate:
> > >>
> >
> type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='auditSigningCert
> > >> cert-pki-kra',token='NSS Certificate DB'
> > >> CA: dogtag-ipa-ca-renew-agent
> > >> issuer: CN=Certificate Authority,O=MYDOMAIN.ORG
> <http://MYDOMAIN.ORG>
> > <http://MYDOMAIN.ORG>
> > >> <http://MYDOMAIN.ORG>
> > >> subject: CN=KRA Audit,O=MYDOMAIN.ORG
> <http://MYDOMAIN.ORG>
> > <http://MYDOMAIN.ORG> <http://MYDOMAIN.ORG>
> > >> expires: 2020-06-27 01:54:34 EDT
> > >> key usage: digitalSignature,nonRepudiation
> > >> pre-save command:
> /usr/libexec/ipa/certmonger/stop_pkicad
> > >> post-save command:
> > /usr/libexec/ipa/certmonger/renew_ca_cert
> > >> "auditSigningCert cert-pki-kra"
> > >> track: yes
> > >> auto-renew: yes
> > >> Request ID '20191031183459':
> > >> status: MONITORING
> > >> ca-error: Server at
> > >> "http://ipa-one.mydomain.org:8080/ca/ee/ca/profileSubmit"
> replied:
> > >> Missing credential: sessionID
> > >> stuck: no
> > >> key pair storage:
> > >>
> >
> type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='transportCert
> > cert-pki-kra',token='NSS
> > >> Certificate DB',pin set
> > >> certificate:
> > >>
> >
> type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='transportCert
> > cert-pki-kra',token='NSS
> > >> Certificate DB'
> > >> CA: dogtag-ipa-ca-renew-agent
> > >> issuer: CN=Certificate Authority,O=MYDOMAIN.ORG
> <http://MYDOMAIN.ORG>
> > <http://MYDOMAIN.ORG>
> > >> <http://MYDOMAIN.ORG>
> > >> subject: CN=KRA Transport
> Certificate,O=MYDOMAIN.ORG <http://MYDOMAIN.ORG>
> > <http://MYDOMAIN.ORG>
> > >> <http://MYDOMAIN.ORG>
> > >> expires: 2020-06-27 01:54:30 EDT
> > >> key usage:
> > >>
> digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment
> > >> eku: id-kp-clientAuth
> > >> pre-save command:
> /usr/libexec/ipa/certmonger/stop_pkicad
> > >> post-save command:
> > /usr/libexec/ipa/certmonger/renew_ca_cert
> > >> "transportCert cert-pki-kra"
> > >> track: yes
> > >> auto-renew: yes
> > >> Request ID '20191031183500':
> > >> status: MONITORING
> > >> ca-error: Server at
> > >> "http://ipa-one.mydomain.org:8080/ca/ee/ca/profileSubmit"
> replied:
> > >> Missing credential: sessionID
> > >> stuck: no
> > >> key pair storage:
> > >>
> >
> type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='storageCert
> > >> cert-pki-kra',token='NSS Certificate DB',pin set
> > >> certificate:
> > >>
> >
> type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='storageCert
> > >> cert-pki-kra',token='NSS Certificate DB'
> > >> CA: dogtag-ipa-ca-renew-agent
> > >> issuer: CN=Certificate Authority,O=MYDOMAIN.ORG
> <http://MYDOMAIN.ORG>
> > <http://MYDOMAIN.ORG>
> > >> <http://MYDOMAIN.ORG>
> > >> subject: CN=KRA Storage
> Certificate,O=MYDOMAIN.ORG <http://MYDOMAIN.ORG>
> > <http://MYDOMAIN.ORG>
> > >> <http://MYDOMAIN.ORG>
> > >> expires: 2020-06-27 01:54:32 EDT
> > >> key usage:
> > >>
> digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment
> > >> eku: id-kp-clientAuth
> > >> pre-save command:
> /usr/libexec/ipa/certmonger/stop_pkicad
> > >> post-save command:
> > /usr/libexec/ipa/certmonger/renew_ca_cert
> > >> "storageCert cert-pki-kra"
> > >> track: yes
> > >> auto-renew: yes
> > >> ```
> > >>
> > >> Here are the sequence of events that seem to have led to this:
> > >>
> > >> 1. Install FreeIPA Master many years ago and continue to
> upgrade it
> > >> from time to time.
> > >> 2. Install FreeIPA Replica a few years after and continue
> to upgrade
> > >> it from time to time.
> > >> 3. Allow the certificates to expire on both nodes.
> > >> 4. Attempt to patch the replica via `yum upgrade` on the
> second
> > node.
> > >> 5. Notice after reboot that `pki-tomcatd` is having
> trouble and
> > >> discover certificate issues.
> > >> 5. Issue `ipa-cert-fix`, reboot again, and notice that
> things are
> > >> working. Try and create a key in the vault.
> > >> 6. Attempt to patch the master via `yum upgrade` on the
> first node.
> > >> 7. Notice after reboot that everything seems to be ok. Try and
> > create
> > >> a key in the vault.
> > >> 8. Notice a few days later that renewal seems to be broken
> on the
> > >> first node.
> > >>
> > >> At this point `ipa-cert-fix` just shows that everything is
> fine.
> > If I
> > >> run it with -v, and then check the "storageCert cert-pki-kra"
> > >> certificate with `openssl x509 -text -in`, I'm shown:
> > >
> > > Hi,
> > > just double-checking, but did you run ipa-cert-fix on the
> replica
> > that
> > > was repaired in step 5? If that's the case, it's normal that
> > > ipa-cert-fix does not see any issue as it's running only
> locally and
> > > does not attempt to repair remote nodes.
> > >
> > > You will need to login to the node with expired certs and run
> > > ipa-cert-fix there.
> >
> > I'd also look to see which one is the renewal master. That is
> the one
> > that should renew the cert. I'm too curious why the renewal
> raised an
> > error (as if it actually tried to renew) rather than either go
> into
> > CA_WORKING or pick up the updated cert.
> >
> > I'd also make sure that replication is working. On each master:
> >
> > # ipa-csreplica-manage list -v `hostname`
> >
> > rob
> >
> > >
> > > HTH,
> > > flo
> > >
> > >>
> > >> Validity
> > >> Not Before: Jun 29 00:52:33 2020 GMT
> > >> Not After : Jun 19 00:52:33 2022 GMT
> > >>
> > >> On the second known, `getcert list` shows correct
> expirations for
> > >> those certificates:
> > >>
> > >> Request ID '20191206005909':
> > >> status: MONITORING
> > >> stuck: no
> > >> key pair storage:
> > >>
> >
> type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='storageCert
> > >> cert-pki-kra',token='NSS Certificate DB',pin set
> > >> certificate:
> > >>
> >
> type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='storageCert
> > >> cert-pki-kra',token='NSS Certificate DB'
> > >> CA: dogtag-ipa-ca-renew-agent
> > >> issuer: CN=Certificate Authority,O=MYDOMAIN.ORG
> <http://MYDOMAIN.ORG>
> > <http://MYDOMAIN.ORG>
> > >> <http://MYDOMAIN.ORG>
> > >> subject: CN=KRA Storage
> Certificate,O=MYDOMAIN.ORG <http://MYDOMAIN.ORG>
> > <http://MYDOMAIN.ORG>
> > >> <http://MYDOMAIN.ORG>
> > >> expires: 2022-06-18 20:52:33 EDT
> > >> key usage:
> > >>
> digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment
> > >> eku: id-kp-clientAuth
> > >> pre-save command:
> /usr/libexec/ipa/certmonger/stop_pkicad
> > >> post-save command:
> > /usr/libexec/ipa/certmonger/renew_ca_cert
> > >> "storageCert cert-pki-kra"
> > >> track: yes
> > >> auto-renew: yes
> > >>
> > >> It seems like _something_, perhaps `ipa-cert-fix` somehow
> renewed
> > >> these certificates but...outside of certmonger? Is this
> some other
> > >> version of
> https://bugzilla.redhat.com/show_bug.cgi?id=1788907? The
> > >> certificates are not in CA_WORKING though, they're in
> MONITORING.
> > >>
> > >> What can I do to get myself out of this state as it seems like
> > I'm in
> > >> a "this could explode at any moment" situation?
> > >>
> > >> This is on Fedora 30 with IP version:
> > >>
> > >> Last metadata expiration check: 0:23:05 ago on Sat 04 Jul 2020
> > >> 07:59:16 PM EDT.
> > >> Installed Packages
> > >> Name : certmonger
> > >> Version : 0.79.9
> > >> Release : 1.fc30
> > >> Architecture : x86_64
> > >> Size : 3.4 M
> > >> Source : certmonger-0.79.9-1.fc30.src.rpm
> > >> Repository : @System
> > >> From repo : updates
> > >>
> > >> .. snip ..
> > >>
> > >> Name : freeipa-server
> > >> Version : 4.8.3
> > >> Release : 1.fc30
> > >> Architecture : x86_64
> > >> Size : 1.3 M
> > >> Source : freeipa-4.8.3-1.fc30.src.rpm
> > >> Repository : @System
> > >> From repo : updates
> > >>
> > >> .. snip ..
> > >>
> > >> Thanks!
> > >>
> > >>
> > >> Ilya Kogan
> > >> w: github.com/ikogan <http://github.com/ikogan>
> <http://github.com/ikogan>
> > <http://github.com/ikogan> e:
> > >> ikogan@mythicnet.org <mailto:ikogan@mythicnet.org>
> <mailto:ikogan@mythicnet.org <mailto:ikogan@mythicnet.org>>
> > <mailto:ikogan@mythicnet.org <mailto:ikogan@mythicnet.org>
> <mailto:ikogan@mythicnet.org <mailto:ikogan@mythicnet.org>>>
> > >> <http://twitter.com/ilkogan>
> > <https://www.linkedin.com/in/ilyakogan/>
> > >>
> > >>
> > >> _______________________________________________
> > >> FreeIPA-users mailing list --
> > freeipa-users@lists.fedorahosted.org
> <mailto:freeipa-users@lists.fedorahosted.org>
> > <mailto:freeipa-users@lists.fedorahosted.org
> <mailto:freeipa-users@lists.fedorahosted.org>>
> > >> To unsubscribe send an email to
> > >> freeipa-users-leave@lists.fedorahosted.org
> <mailto:freeipa-users-leave@lists.fedorahosted.org>
> > <mailto:freeipa-users-leave@lists.fedorahosted.org
> <mailto:freeipa-users-leave@lists.fedorahosted.org>>
> > >> Fedora Code of Conduct:
> > >> https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> > >> List Guidelines:
> > https://fedoraproject.org/wiki/Mailing_list_guidelines
> > >> List Archives:
> > >>
> >
> https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedorahosted.org
> > >>
> > >>
> > > _______________________________________________
> > > FreeIPA-users mailing list --
> > freeipa-users@lists.fedorahosted.org
> <mailto:freeipa-users@lists.fedorahosted.org>
> > <mailto:freeipa-users@lists.fedorahosted.org
> <mailto:freeipa-users@lists.fedorahosted.org>>
> > > To unsubscribe send an email to
> > freeipa-users-leave@lists.fedorahosted.org
> <mailto:freeipa-users-leave@lists.fedorahosted.org>
> > <mailto:freeipa-users-leave@lists.fedorahosted.org
> <mailto:freeipa-users-leave@lists.fedorahosted.org>>
> > > Fedora Code of Conduct:
> > > https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> > > List Guidelines:
> > https://fedoraproject.org/wiki/Mailing_list_guidelines
> > > List Archives:
> > >
> >
> https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedorahosted.org
> > >
> >
> >
> > _______________________________________________
> > FreeIPA-users mailing list -- freeipa-users@lists.fedorahosted.org
> <mailto:freeipa-users@lists.fedorahosted.org>
> > To unsubscribe send an email to
> freeipa-users-leave@lists.fedorahosted.org
> <mailto:freeipa-users-leave@lists.fedorahosted.org>
> > Fedora Code of Conduct:
> https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> > List Guidelines:
> https://fedoraproject.org/wiki/Mailing_list_guidelines
> > List Archives:
> https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedorahosted.org
> >
>