On 7/6/20 7:59 PM, Ilya Kogan via FreeIPA-users wrote:
> Hi,
>
> Thanks for the help so far! I've actually run `ipa-cert-fix` on both
> nodes, it says everything is ok on both nodes. When I run it with
> verbose mode, it spits out the command it's running and the certificate
> it got, for example:
>
> ```
> ipapython.ipautil: DEBUG: args=['/usr/bin/certutil', '-d',
> 'sql:/etc/pki/pki-tomcat/alias', '-L', '-n', 'storageCert
> cert-pki-kra', '-a', '-f', '/etc/pki/pki-tomcat/alias/pwdfile.txt']
> ```
>
>
> If I then take that cert and ask what `openssl x509 -text -noout` thinks
> about it, it tells me that it's valid from 2020-06-29 to 2022-06-29.
> Strangely, though, when I ask `getcert list`, it shows that the certificate:
>
> ```
> certificate:
> type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='storageCert cert-pki-kra',token='NSS
> Certificate DB'
> ```
>
>
> expires on 2020-06-27. It's almost as if this node's certificate has
> _already_ been renewed but certmonger (I think) doesn't know about it,
> which might be why it's having trouble renewing it.
>
Hi,
you may want to restart certmonger to force it re-reading the
certificate information:
# sudo systemctl restart certmonger
flo
> Here's what the two nodes say about replication:
>
> From node one:
>
> ```
> ipa-two.mydomain.org <http://ipa-two.mydomain.org>
> last update status: Error (0) Replica acquired successfully:
> Incremental update succeeded
> last update ended: 2020-07-06 17:46:17+00:00
> ```
>
>
> From node two:
>
> ```
> ipa-one.gaea.mythicnet.org <http://ipa-one.gaea.mythicnet.org>
> last update status: Error (0) Replica acquired successfully:
> Incremental update succeeded
> last update ended: 2020-07-06 17:46:17+00:00
> ```
>
>
> I suppose this might be a good time to mention that this is a simple two
> node multi-master setup. Finally, I'm not sure if I'm doing this
> correctly, but to make absolutely sure about which node is the renewal
> master, I ran this on both nodes:
>
> ```
> ldapsearch -H ldap://ipa-one.gaea.mythicnet.org
> <http://ipa-one.gaea.mythicnet.org> -D 'cn=Directory Manager' -W -b
> 'cn=masters,cn=ipa,cn=etc,dc=mydomain,dc=org'
> '(&(cn=CA)(ipaConfigString=caRenewalMaster))' dn
> ldapsearch -H ldap://ipa-two.gaea.mythicnet.org
> <http://ipa-two.gaea.mythicnet.org> -D 'cn=Directory Manager' -W -b
> 'cn=masters,cn=ipa,cn=etc,dc=mydomain,dc=org'
> '(&(cn=CA)(ipaConfigString=caRenewalMaster))' dn
> ```
>
>
> The result for both is:
>
> ```
> dn: cn=CA,cn=ipa-one.gaea.mythicnet.org
> <http://ipa-one.gaea.mythicnet.org>,cn=masters,cn=ipa,cn=etc,dc=mydomain,dc=org
> ```
>
>
> So it looks like the renewal master is the one having this problem.
>
>
> Ilya Kogan
> w: github.com/ikogan <http://github.com/ikogan> e: ikogan@mythicnet.org
> <mailto:ikogan@mythicnet.org>
> <http://twitter.com/ilkogan> <https://www.linkedin.com/in/ilyakogan/>
>
>
>
> On Mon, Jul 6, 2020 at 1:25 PM Rob Crittenden <rcritten@redhat.com
> <mailto:rcritten@redhat.com>> wrote:
>
> Florence Blanc-Renaud via FreeIPA-users wrote:
> > On 7/5/20 2:23 AM, Ilya Kogan via FreeIPA-users wrote:
> >> Hi,
> >>
> >> I seem to be facing a similar issue with one of my KRAs. My KRA
> >> certificates were, for some reason, not automatically renewed when
> >> they expired last month. Using `ipa-cert-fix` correctly fixed
> them on
> >> _one_ host. On the other, they seem to be stuck in the renewal state
> >> and `ipa-cert-fix` claims there's nothing to do:
> >>
> >> ```
> >> Request ID '20191031183458':
> >> status: MONITORING
> >> ca-error: Server at
> >> "http://ipa-one.mydomain.org:8080/ca/ee/ca/profileSubmit" replied:
> >> Missing credential: sessionID
> >> stuck: no
> >> key pair storage:
> >>
> type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='auditSigningCert
> >> cert-pki-kra',token='NSS Certificate DB',pin set
> >> certificate:
> >>
> type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='auditSigningCert
> >> cert-pki-kra',token='NSS Certificate DB'
> >> CA: dogtag-ipa-ca-renew-agent
> >> issuer: CN=Certificate Authority,O=MYDOMAIN.ORG
> <http://MYDOMAIN.ORG>
> >> <http://MYDOMAIN.ORG>
> >> subject: CN=KRA Audit,O=MYDOMAIN.ORG
> <http://MYDOMAIN.ORG> <http://MYDOMAIN.ORG>
> >> expires: 2020-06-27 01:54:34 EDT
> >> key usage: digitalSignature,nonRepudiation
> >> pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad
> >> post-save command:
> /usr/libexec/ipa/certmonger/renew_ca_cert
> >> "auditSigningCert cert-pki-kra"
> >> track: yes
> >> auto-renew: yes
> >> Request ID '20191031183459':
> >> status: MONITORING
> >> ca-error: Server at
> >> "http://ipa-one.mydomain.org:8080/ca/ee/ca/profileSubmit" replied:
> >> Missing credential: sessionID
> >> stuck: no
> >> key pair storage:
> >>
> type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='transportCert
> cert-pki-kra',token='NSS
> >> Certificate DB',pin set
> >> certificate:
> >>
> type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='transportCert
> cert-pki-kra',token='NSS
> >> Certificate DB'
> >> CA: dogtag-ipa-ca-renew-agent
> >> issuer: CN=Certificate Authority,O=MYDOMAIN.ORG
> <http://MYDOMAIN.ORG>
> >> <http://MYDOMAIN.ORG>
> >> subject: CN=KRA Transport Certificate,O=MYDOMAIN.ORG
> <http://MYDOMAIN.ORG>
> >> <http://MYDOMAIN.ORG>
> >> expires: 2020-06-27 01:54:30 EDT
> >> key usage:
> >> digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment
> >> eku: id-kp-clientAuth
> >> pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad
> >> post-save command:
> /usr/libexec/ipa/certmonger/renew_ca_cert
> >> "transportCert cert-pki-kra"
> >> track: yes
> >> auto-renew: yes
> >> Request ID '20191031183500':
> >> status: MONITORING
> >> ca-error: Server at
> >> "http://ipa-one.mydomain.org:8080/ca/ee/ca/profileSubmit" replied:
> >> Missing credential: sessionID
> >> stuck: no
> >> key pair storage:
> >>
> type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='storageCert
> >> cert-pki-kra',token='NSS Certificate DB',pin set
> >> certificate:
> >>
> type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='storageCert
> >> cert-pki-kra',token='NSS Certificate DB'
> >> CA: dogtag-ipa-ca-renew-agent
> >> issuer: CN=Certificate Authority,O=MYDOMAIN.ORG
> <http://MYDOMAIN.ORG>
> >> <http://MYDOMAIN.ORG>
> >> subject: CN=KRA Storage Certificate,O=MYDOMAIN.ORG
> <http://MYDOMAIN.ORG>
> >> <http://MYDOMAIN.ORG>
> >> expires: 2020-06-27 01:54:32 EDT
> >> key usage:
> >> digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment
> >> eku: id-kp-clientAuth
> >> pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad
> >> post-save command:
> /usr/libexec/ipa/certmonger/renew_ca_cert
> >> "storageCert cert-pki-kra"
> >> track: yes
> >> auto-renew: yes
> >> ```
> >>
> >> Here are the sequence of events that seem to have led to this:
> >>
> >> 1. Install FreeIPA Master many years ago and continue to upgrade it
> >> from time to time.
> >> 2. Install FreeIPA Replica a few years after and continue to upgrade
> >> it from time to time.
> >> 3. Allow the certificates to expire on both nodes.
> >> 4. Attempt to patch the replica via `yum upgrade` on the second
> node.
> >> 5. Notice after reboot that `pki-tomcatd` is having trouble and
> >> discover certificate issues.
> >> 5. Issue `ipa-cert-fix`, reboot again, and notice that things are
> >> working. Try and create a key in the vault.
> >> 6. Attempt to patch the master via `yum upgrade` on the first node.
> >> 7. Notice after reboot that everything seems to be ok. Try and
> create
> >> a key in the vault.
> >> 8. Notice a few days later that renewal seems to be broken on the
> >> first node.
> >>
> >> At this point `ipa-cert-fix` just shows that everything is fine.
> If I
> >> run it with -v, and then check the "storageCert cert-pki-kra"
> >> certificate with `openssl x509 -text -in`, I'm shown:
> >
> > Hi,
> > just double-checking, but did you run ipa-cert-fix on the replica
> that
> > was repaired in step 5? If that's the case, it's normal that
> > ipa-cert-fix does not see any issue as it's running only locally and
> > does not attempt to repair remote nodes.
> >
> > You will need to login to the node with expired certs and run
> > ipa-cert-fix there.
>
> I'd also look to see which one is the renewal master. That is the one
> that should renew the cert. I'm too curious why the renewal raised an
> error (as if it actually tried to renew) rather than either go into
> CA_WORKING or pick up the updated cert.
>
> I'd also make sure that replication is working. On each master:
>
> # ipa-csreplica-manage list -v `hostname`
>
> rob
>
> >
> > HTH,
> > flo
> >
> >>
> >> Validity
> >> Not Before: Jun 29 00:52:33 2020 GMT
> >> Not After : Jun 19 00:52:33 2022 GMT
> >>
> >> On the second known, `getcert list` shows correct expirations for
> >> those certificates:
> >>
> >> Request ID '20191206005909':
> >> status: MONITORING
> >> stuck: no
> >> key pair storage:
> >>
> type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='storageCert
> >> cert-pki-kra',token='NSS Certificate DB',pin set
> >> certificate:
> >>
> type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='storageCert
> >> cert-pki-kra',token='NSS Certificate DB'
> >> CA: dogtag-ipa-ca-renew-agent
> >> issuer: CN=Certificate Authority,O=MYDOMAIN.ORG
> <http://MYDOMAIN.ORG>
> >> <http://MYDOMAIN.ORG>
> >> subject: CN=KRA Storage Certificate,O=MYDOMAIN.ORG
> <http://MYDOMAIN.ORG>
> >> <http://MYDOMAIN.ORG>
> >> expires: 2022-06-18 20:52:33 EDT
> >> key usage:
> >> digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment
> >> eku: id-kp-clientAuth
> >> pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad
> >> post-save command:
> /usr/libexec/ipa/certmonger/renew_ca_cert
> >> "storageCert cert-pki-kra"
> >> track: yes
> >> auto-renew: yes
> >>
> >> It seems like _something_, perhaps `ipa-cert-fix` somehow renewed
> >> these certificates but...outside of certmonger? Is this some other
> >> version of https://bugzilla.redhat.com/show_bug.cgi?id=1788907? The
> >> certificates are not in CA_WORKING though, they're in MONITORING.
> >>
> >> What can I do to get myself out of this state as it seems like
> I'm in
> >> a "this could explode at any moment" situation?
> >>
> >> This is on Fedora 30 with IP version:
> >>
> >> Last metadata expiration check: 0:23:05 ago on Sat 04 Jul 2020
> >> 07:59:16 PM EDT.
> >> Installed Packages
> >> Name : certmonger
> >> Version : 0.79.9
> >> Release : 1.fc30
> >> Architecture : x86_64
> >> Size : 3.4 M
> >> Source : certmonger-0.79.9-1.fc30.src.rpm
> >> Repository : @System
> >> From repo : updates
> >>
> >> .. snip ..
> >>
> >> Name : freeipa-server
> >> Version : 4.8.3
> >> Release : 1.fc30
> >> Architecture : x86_64
> >> Size : 1.3 M
> >> Source : freeipa-4.8.3-1.fc30.src.rpm
> >> Repository : @System
> >> From repo : updates
> >>
> >> .. snip ..
> >>
> >> Thanks!
> >>
> >>
> >> Ilya Kogan
> >> w: github.com/ikogan <http://github.com/ikogan>
> <http://github.com/ikogan> e:
> >> ikogan@mythicnet.org <mailto:ikogan@mythicnet.org>
> <mailto:ikogan@mythicnet.org <mailto:ikogan@mythicnet.org>>
> >> <http://twitter.com/ilkogan>
> <https://www.linkedin.com/in/ilyakogan/>
> >>
> >>
> >> _______________________________________________
> >> FreeIPA-users mailing list --
> freeipa-users@lists.fedorahosted.org
> <mailto:freeipa-users@lists.fedorahosted.org>
> >> To unsubscribe send an email to
> >> freeipa-users-leave@lists.fedorahosted.org
> <mailto:freeipa-users-leave@lists.fedorahosted.org>
> >> Fedora Code of Conduct:
> >> https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> >> List Guidelines:
> https://fedoraproject.org/wiki/Mailing_list_guidelines
> >> List Archives:
> >>
> https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedorahosted.org
> >>
> >>
> > _______________________________________________
> > FreeIPA-users mailing list --
> freeipa-users@lists.fedorahosted.org
> <mailto:freeipa-users@lists.fedorahosted.org>
> > To unsubscribe send an email to
> freeipa-users-leave@lists.fedorahosted.org
> <mailto:freeipa-users-leave@lists.fedorahosted.org>
> > Fedora Code of Conduct:
> > https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> > List Guidelines:
> https://fedoraproject.org/wiki/Mailing_list_guidelines
> > List Archives:
> >
> https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedorahosted.org
> >
>
>
> _______________________________________________
> FreeIPA-users mailing list -- freeipa-users@lists.fedorahosted.org
> To unsubscribe send an email to freeipa-users-leave@lists.fedorahosted.org
> Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives: https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedorahosted.org
>