Re: Why is ipa-ods-exporter broken after running ipa-dns-install? (Was - Unable to start directory server after updates)

[Rob Crittenden]

Jeremy Tourville via FreeIPA-users wrote:
> I was doing some reading and troubleshooting
>
> https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/htm...
>
> which basically says:
> #1 ipa-cacert-manage renew
> #2 ipa-certupdate
> #3 certutil -L -d /etc/pki/pki-tomcat/alias (to test the certs)
>
> See my output.  Step #1 and #3 work now but #2 still fails
>
> [root@utility certs]# ipa-certupdate
>
> cannot connect to 'https://utility.idm.nac-issa.org/ipa/json': [SSL:
> CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:897)
> The ipa-certupdate command failed.

So update-ca-trust had no affect or was this run beforehand?
> [root@utility certs]# certutil -L -d /etc/pki/pik-tomcat/alias
>
> certutil: function failed: SEC_ERROR_BAD_DATABASE: security library: bad
> database.

It failed because of a typo, pik -> pki.
> [root@utility certs]# ipa-cacert-manage renew
>
> Renewing CA certificate, please wait
> CA certificate successfully renewed
> The ipa-cacert-manage command was successful

This renews the CA certificate. The CA is good for 20 years, you didn't
need to do this.
> [root@utility certs]# ipa-certupdate
>
> cannot connect to 'https://utility.idm.nac-issa.org/ipa/json': [SSL:
> CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:897)
> The ipa-certupdate command failed.

We now have another CA certificate for IPA in the mix because of the
renewal.
> [root@utility certs]# certutil -L -d /etc/pki/pki-tomcat/alias
>
> Certificate Nickname                                         Trust
> Attributes
>                                                            
>  SSL,S/MIME,JAR/XPI
>
> ocspSigningCert cert-pki-ca                                  u,u,u
> subsystemCert cert-pki-ca                                    u,u,u
> auditSigningCert cert-pki-ca                                 u,u,Pu
> Server-Cert cert-pki-ca                                      u,u,u
> caSigningCert cert-pki-ca                                    CTu,Cu,Cu
> IDM.NAC-ISSA.ORG IPA CA                                      CTu,Cu,Cu
> [root@utility certs]# reboot

It isn't a problem with the CA. The system doesn't trust the CA for some
reason, though the openssl command verified that it is ok.
> [root@utility certs]# reboot
>
> [root@utility ~]# ipa-certupdate
>
> cannot connect to 'https://utility.idm.nac-issa.org/ipa/json': [SSL:
> CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:897)
> The ipa-certupdate command failed.

You didn't happen to touch /etc/httpd/conf.d/ssl.conf did you?

rob
> [root@utility ~]# ipactl status
>
> Directory Service: RUNNING
> krb5kdc Service: RUNNING
> kadmin Service: RUNNING
> named Service: RUNNING
> httpd Service: RUNNING
> ipa-custodia Service: RUNNING
> pki-tomcatd Service: RUNNING
> smb Service: RUNNING
> winbind Service: RUNNING
> ipa-otpd Service: RUNNING
> ipa-ods-exporter Service: STOPPED
> ods-enforcerd Service: RUNNING
> ipa-dnskeysyncd Service: RUNNING
> ipa: INFO: The ipactl command was successful
>
> *From:* Rob Crittenden 
> *Sent:* Friday, September 10, 2021 9:49 AM
> *To:* Jeremy Tourville ; FreeIPA users
> list 
> *Cc:* Florence Renaud 
> *Subject:* Re: [Freeipa-users] Re: Why is ipa-ods-exporter broken after
> running ipa-dns-install? (Was - Unable to start directory server after
> updates)
>  
> Jeremy Tourville wrote:
> > [root@utility certs]# curl https://utility.idm.nac-issa.org/
> > curl: (60) SSL certificate problem: self signed certificate in
> > certificate chain
> > More details here: https://curl.haxx.se/docs/sslcerts.html
> >
> > curl failed to verify the legitimacy of the server and therefore could not
> > establish a secure connection to it. To learn more about this situation and
> > how to fix it, please visit the web page mentioned above.
> >
> > [root@utility certs]# update-ca-trust
> >
> > [root@utility certs]# ausearch -m AVC -ts recent
> >
> > <no matches>
> >
> > [root@utility certs]# ipa-healthcheck
> > -bash: ipa-healthcheck: command not found
>
> I should have mentioned, try the curl after running update-ca-trust.
>
> ipa-healthcheck is not installed by default, you'd need to install the
> {free}ipa-healthcheck package.
>
> rob
> > *From:* Rob Crittenden 
> > *Sent:* Friday, September 10, 2021 9:33 AM
> > *To:* Jeremy Tourville ; FreeIPA users
> > list 
> > *Cc:* Florence Renaud 
> > *Subject:* Re: [Freeipa-users] Re: Why is ipa-ods-exporter broken after
> > running ipa-dns-install? (Was - Unable to start directory server after
> > updates)
> >  
> > Jeremy Tourville wrote:
> > > [root@utility certs]# ipa-certupdate
> > > cannot connect to 'https://utility.idm.nac-issa.org/ipa/json': [SSL:
> > > CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:897)
> > > The ipa-certupdate command failed.
> > >
> > > Sort of a bad catch 22 I guess?
> >
> > Yeah, I was afraid of that.
> >
> > Let's walk through it. Try a simple command for another data point. I'm
> > not sure what we'd do with this but it will exercise the system-wide
> > trust as well:
> >
> > $ curl https://%60hostname%60/
> >
> > Rebuilding the CA trust db may help
> >
> > # update-ca-trust
> >
> > I suppose also look for AVCs in case something is way out-of-whack:
> >
> > # ausearch -m AVC -ts recent
> >
> > ipa-healthcheck may be something to try as well but you're likely to get
> > a crapton of false positives since it can't talk to the web interface.
> >
> > rob
> > > *From:* Rob Crittenden 
> > > *Sent:* Friday, September 10, 2021 9:09 AM
> > > *To:* Jeremy Tourville ; FreeIPA users
> > > list 
> > > *Cc:* Florence Renaud 
> > > *Subject:* Re: [Freeipa-users] Re: Why is ipa-ods-exporter broken after
> > > running ipa-dns-install? (Was - Unable to start directory server after
> > > updates)
> > >  
> > > Jeremy Tourville wrote:
> > > > Now I understand how to test the cert(s) after re-reading your comments
> > > > Rob and Flo 🙂
> > > >
> > > > [root@utility certs]# openssl verify -verbose -show_chain -CAfile
> > > > /etc/ipa/ca.crt /var/lib/ipa/certs/httpd.crt
> > > > /var/lib/ipa/certs/httpd.crt: OK
> > > > Chain:
> > > > depth=0: O = IDM.NAC-ISSA.ORG, CN = utility.idm.nac-issa.org (untrusted)
> > > > depth=1: O = IDM.NAC-ISSA.ORG, CN = Certificate Authority
> > >
> > > I'd try running ipa-certupdate. I have the feeling some of the
> > > system-wide certificates are out-of-sync.
> > >
> > > rob
> > > > *From:* Jeremy Tourville 
> > > > *Sent:* Thursday, September 9, 2021 5:45 PM
> > > > *To:* FreeIPA users list 
> > > > *Cc:* Florence Renaud ; Rob Crittenden 
> > > > *Subject:* Re: [Freeipa-users] Re: Why is ipa-ods-exporter broken after
> > > > running ipa-dns-install? (Was - Unable to start directory server after
> > > > updates)
> > > >  
> > > > Oh wait!!! Which set of certs do I need to test against for my
> > > > certificate chain?
> > > > I realized I didn't include the proper path when testing.  It should be
> > > > something like-
> > > >
> > > > # openssl verify -verbose -show_chain -CAfile <path to root or
> > > > intermediate cert> /etc/ipa/ca.crt
> > > > # openssl verify -verbose -show_chain -CAfile <path to root or
> > > > intermediate cert> /var/lib/ipa/certs/httpd.crt
> > > >
> > > > This would give you output (presuming you are using the correct set of
> > > > certs)
> > > >  /etc/ipa/ca.crt: OK
> > > > /var/lib/ipa/certs/httpd.crt: OK
> > > >
> > > > Which path contains the intermediate or root CA certs I need to test
> > > > against?
> > > >
> > > > [root@utility ~]# ls -la | find / -name *.crt
> > > > /etc/pki/ca-trust/extracted/openssl/ca-bundle.trust.crt
> > > > /etc/pki/ca-trust/source/ca-bundle.legacy.crt
> > > > /etc/pki/tls/certs/ca-bundle.crt
> > > > /etc/pki/tls/certs/ca-bundle.trust.crt
> > > > /etc/pki/tls/certs/localhost.crt
> > > > /etc/pki/pki-tomcat/alias/ca.crt
> > > > /etc/ipa/ca.crt
> > > > /etc/dirsrv/ssca/ca.crt
> > > > /etc/dirsrv/slapd-IDM-NAC-ISSA-ORG/Server-Cert.crt
> > > > /etc/dirsrv/slapd-IDM-NAC-ISSA-ORG/ca.crt
> > > > /var/lib/ipa/certs/httpd.crt
> > > > /var/kerberos/krb5kdc/kdc.crt
> > > > /usr/share/pki/ca-trust-legacy/ca-bundle.legacy.default.crt
> > > > /usr/share/pki/ca-trust-legacy/ca-bundle.legacy.disable.crt
> > > > /usr/share/ipa/html/ca.crt
> > > >
> > > > *From:* Jeremy Tourville 
> > > > *Sent:* Thursday, September 9, 2021 3:13 PM
> > > > *To:* FreeIPA users list 
> > > > *Cc:* Florence Renaud ; Rob Crittenden 
> > > > *Subject:* Re: [Freeipa-users] Re: Why is ipa-ods-exporter broken after
> > > > running ipa-dns-install? (Was - Unable to start directory server after
> > > > updates)
> > > > > > >It isn't complaining that the certificate isn't valid, it's complaining
> > > >
> > > > that it isn't trusted.
> > > >  Thanksfor pointing out my mistake.  I'm wearing some egg on my face.  I
> > > > was thinking about it wrong at the time of my reply.
> > > >
> > > > I attempted to verify trust-
> > > > [root@utility ipa]# openssl verify -verbose -show_chain -CAfile
> > > > /etc/ipa/ca.crt
> > > > ^C
> > > > [root@utility ipa]# openssl verify -verbose -show_chain -CAfile
> > > > /var/lib/ipa/certs/httpd.crt
> > > > ^C
> > > >
> > > > As you can see, no output, so yeah, they are not trusted.
> > > > > > Where did httpd.crt come from/what issuer?
> > > >
> > > > I recall not using a 3rd party CA.  The certs were just self-signed when
> > > > the ipa server was initially built.  I never did replace the certs as it
> > > > wasn't required for our situation.
> > > >
> > > > Next steps I guess would be to generate some new certs?  Thoughts?
> > > >
> > > > *From:* Rob Crittenden 
> > > > *Sent:* Thursday, September 9, 2021 12:53 PM
> > > > *To:* FreeIPA users list 
> > > > *Cc:* Florence Renaud ; Jeremy Tourville
> > > >
> > > > *Subject:* Re: [Freeipa-users] Re: Why is ipa-ods-exporter broken after
> > > > running ipa-dns-install? (Was - Unable to start directory server after
> > > > updates)
> > > >  
> > > > Jeremy Tourville via FreeIPA-users wrote:
> > > > > /var/lib/ipa/certs/httpd.crt
> > > > > looks valid and has a 3 year validity date starting from Nov 23, 2020
> > > > >
> > > > > /etc/ipa/ca.crt
> > > > > looks valid and has a 20 year validity date starting from Nov 23, 2020
> > > >
> > > > It isn't complaining that the certificate isn't valid, it's complaining
> > > > that it isn't trusted. You also need to look at the signer and ensure
> > > > that the system trusts it globally. Where did httpd.crt come from/what
> > > > issuer?
> > > >
> > > > You might try running:
> > > >
> > > > openssl verify -verbose -show_chain -CAfile /etc/ipa/ca.crt
> > > > /var/lib/ipa/certs/httpd.crt
> > > >
> > > > See the default.conf(5) man page for a description of default.conf,
> > > > server.conf, etc. In this case server is a context so the configuration
> > > > only applies there.
> > > >
> > > > rob
> > > > > *From:* Florence Renaud 
> > > > > *Sent:* Tuesday, September 7, 2021 11:38 AM
> > > > > *To:* Jeremy Tourville 
> > > > > *Cc:* FreeIPA users list 
> > > > > *Subject:* Re: [Freeipa-users] Re: Why is ipa-ods-exporter broken after
> > > > > running ipa-dns-install? (Was - Unable to start directory server after
> > > > > updates)
> > > > >  
> > > > > Hi Jeremy,
> > > > >
> > > > > to enable debugging you can simply create /etc/ipa/server.conf if the
> > > > > file does not exist:
> > > > > # cat /etc/ipa/server.conf
> > > > > [global]
> > > > > debug=True
> > > > > # systemctl restart httpd
> > > > >
> > > > > The HTTPd certificate is stored in /var/lib/ipa/certs/httpd.crt, you can
> > > > > examine its content with
> > > > > # openssl x509 -noout -text -in /var/lib/ipa/certs/httpd.crt
> > > > > If the IPA deployment includes an embedded CA, the CA that issued the
> > > > > httpd cert is stored in /etc/ipa/ca.crt and can also be checked with
> > > > > openssl command.
> > > > >
> > > > > flo
> > > > >
> > > > > On Tue, Sep 7, 2021 at 6:09 PM Jeremy Tourville
> > > > > <[e-mail] > wrote:
> > > > >
> > > > > I think I see the issue but I am unsure what to do to fix it.  See
> > > > >      below.
> > > > >
> > > > > To answer your question, yes I did accept the security exception.
> > > > >
> > > > > Also, I don't see a server.conf file at /etc/ipa so that I may
> > > > >      enable debugging.  What can you suggest for this issue?
> > > > >
> > > > > [root@utility ~]# ipactl status
> > > > >      Directory Service: RUNNING
> > > > >      krb5kdc Service: RUNNING
> > > > >      kadmin Service: RUNNING
> > > > >      named Service: RUNNING
> > > > >      httpd Service: RUNNING
> > > > >      ipa-custodia Service: RUNNING
> > > > >      pki-tomcatd Service: RUNNING
> > > > >      smb Service: RUNNING
> > > > >      winbind Service: RUNNING
> > > > >      ipa-otpd Service: RUNNING
> > > > >      ipa-ods-exporter Service: STOPPED
> > > > >      ods-enforcerd Service: RUNNING
> > > > >      ipa-dnskeysyncd Service: RUNNING
> > > > >      ipa: INFO: The ipactl command was successful
> > > > >
> > > > > [root@utility ~]# kinit admin
> > > > >      Password for [e-mail] :
> > > > >
> > > > > [root@utility ~]# klist
> > > > >      Ticket cache: KCM:0:43616
> > > > >      Default principal: [e-mail]
> > > > >     
> > > > >
> > > > > Valid starting       Expires              Service principal
> > > > >      09/07/2021 10:59:23  09/08/2021 10:09:04
> > > > >       krbtgt/[e-mail]
> > > > >     
> > > > >
> > > > > [root@utility ~]# ipa config-show
> > > > >      ipa: ERROR: cannot connect to
> > > > >      'https://utility.idm.nac-issa.org/ipa/json': [SSL:
> > > > >      CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:897)
> > > > >
> > > > > ------------------------------------------------------------------------
> > > > >      *From:* Florence Renaud <[e-mail] >
> > > > >      *Sent:* Tuesday, September 7, 2021 10:47 AM
> > > > >      *To:* FreeIPA users list <[e-mail]
> > > > >      >
> > > > >      *Cc:* Jeremy Tourville <[e-mail]
> > > > >      >
> > > > >      *Subject:* Re: [Freeipa-users] Re: Why is ipa-ods-exporter broken
> > > > >      after running ipa-dns-install? (Was - Unable to start directory
> > > > >      server after updates)
> > > > >       
> > > > >      Hi Jeremy,
> > > > >      Did you accept the security exception displayed by the browser (I'm
> > > > >      trying to eliminate obvious issues)?
> > > > >      If nothing is displayed, can you check if ipa command-line is
> > > > >      working as expected (for instance do "kinit admin; ipa config-show")?
> > > > >      You may want to enable debug logs (add debug=True to the [global]
> > > > >      section of /etc/ipa/server.conf and restart httpd service), retry
> > > > >      WebUI authentication and check the generated logs in
> > > > >      /var/log/http/error_log
> > > > >
> > > > > flo
> > > > >
> > > > > On Tue, Sep 7, 2021 at 2:01 PM Jeremy Tourville via FreeIPA-users
> > > > >      <[e-mail]
> > > > >      > wrote:
> > > > >
> > > > > OK,
> > > > >          Why don't I see anything on the initial login page?
> > > > >          All I see is the URL and the fact that the certificate is not
> > > > >          trusted.  The certificate is not expired yet.  Not until Nov 2021.
> > > > >          The login in page is mostly solid white with no login or
> > > > >          password field.
> > > > >          _______________________________________________
> > > > >          FreeIPA-users mailing list --
> > > > >          [e-mail]
> > > > >          
> > > > >          To unsubscribe send an email to
> > > > >          [e-mail]
> > > > >          
> > > > >          Fedora Code of Conduct:
> > > > >          https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> > > > >          List Guidelines:
> > > > >          https://fedoraproject.org/wiki/Mailing_list_guidelines
> > > > >          List Archives:
> > > > >          https://lists.fedorahosted.org/archives/list/[e-mail]...
> > > > >          Do not reply to spam on the list, report it:
> > > > >          https://pagure.io/fedora-infrastructure
> > > > >
> > > > > FreeIPA-users mailing list -- [e-mail]
> > > > > To unsubscribe send an email to [e-mail]
> > > > > Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> > > > > List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> > > > > List Archives: https://lists.fedorahosted.org/archives/list/[e-mail]...
> > > > > Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
>
> FreeIPA-users mailing list -- [e-mail]
> To unsubscribe send an email to [e-mail]
> Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives: https://lists.fedorahosted.org/archives/list/[e-mail]...
> Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure

Why is ipa-ods-exporter broken after running ipa-dns-install? (Was - Unable to start directory server after updates)

[Jeremy Tourville]

ACK, the typo was why certutil failed initially.  It works now.
> > > So update-ca-trust had no affect or was this run beforehand?

Update-ca-trust had no affect.  it was run after doing the ipa-cacert-mange renew .
> > > You didn't happen to touch /etc/httpd/conf.d/ssl.conf did you?

No, I left it alone
> > > This renews the CA certificate. The CA is good for 20 years, you didn't

need to do this.
 ACK
> > > We now have another CA certificate for IPA in the mix because of the

renewal.

OK, I'll stand by so I don't really mess it up.

________________________________
From: Rob Crittenden 
Sent: Friday, September 10, 2021 3:26 PM
To: FreeIPA users list 
Cc: Florence Renaud ; Jeremy Tourville 
Subject: Re: [Freeipa-users] Re: Why is ipa-ods-exporter broken after running ipa-dns-install? (Was - Unable to start directory server after updates)

Jeremy Tourville via FreeIPA-users wrote:
> I was doing some reading and troubleshooting
>
> https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/htm...
>
> which basically says:
> #1 ipa-cacert-manage renew
> #2 ipa-certupdate
> #3 certutil -L -d /etc/pki/pki-tomcat/alias (to test the certs)
>
> See my output.  Step #1 and #3 work now but #2 still fails
>
> [root@utility certs]# ipa-certupdate
>
> cannot connect to 'https://utility.idm.nac-issa.org/ipa/json': [SSL:
> CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:897)
> The ipa-certupdate command failed.

So update-ca-trust had no affect or was this run beforehand?
> [root@utility certs]# certutil -L -d /etc/pki/pik-tomcat/alias
>
> certutil: function failed: SEC_ERROR_BAD_DATABASE: security library: bad
> database.

It failed because of a typo, pik -> pki.
> [root@utility certs]# ipa-cacert-manage renew
>
> Renewing CA certificate, please wait
> CA certificate successfully renewed
> The ipa-cacert-manage command was successful

This renews the CA certificate. The CA is good for 20 years, you didn't
need to do this.
> [root@utility certs]# ipa-certupdate
>
> cannot connect to 'https://utility.idm.nac-issa.org/ipa/json': [SSL:
> CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:897)
> The ipa-certupdate command failed.

We now have another CA certificate for IPA in the mix because of the
renewal.
> [root@utility certs]# certutil -L -d /etc/pki/pki-tomcat/alias
>
> Certificate Nickname                                         Trust
> Attributes
>
> SSL,S/MIME,JAR/XPI
>
> ocspSigningCert cert-pki-ca                                  u,u,u
> subsystemCert cert-pki-ca                                    u,u,u
> auditSigningCert cert-pki-ca                                 u,u,Pu
> Server-Cert cert-pki-ca                                      u,u,u
> caSigningCert cert-pki-ca                                    CTu,Cu,Cu
> IDM.NAC-ISSA.ORG IPA CA                                      CTu,Cu,Cu
> [root@utility certs]# reboot

It isn't a problem with the CA. The system doesn't trust the CA for some
reason, though the openssl command verified that it is ok.
> [root@utility certs]# reboot
>
> [root@utility ~]# ipa-certupdate
>
> cannot connect to 'https://utility.idm.nac-issa.org/ipa/json': [SSL:
> CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:897)
> The ipa-certupdate command failed.

You didn't happen to touch /etc/httpd/conf.d/ssl.conf did you?

rob
> [root@utility ~]# ipactl status
>
> Directory Service: RUNNING
> krb5kdc Service: RUNNING
> kadmin Service: RUNNING
> named Service: RUNNING
> httpd Service: RUNNING
> ipa-custodia Service: RUNNING
> pki-tomcatd Service: RUNNING
> smb Service: RUNNING
> winbind Service: RUNNING
> ipa-otpd Service: RUNNING
> ipa-ods-exporter Service: STOPPED
> ods-enforcerd Service: RUNNING
> ipa-dnskeysyncd Service: RUNNING
> ipa: INFO: The ipactl command was successful
>
> *From:* Rob Crittenden 
> *Sent:* Friday, September 10, 2021 9:49 AM
> *To:* Jeremy Tourville ; FreeIPA users
> list 
> *Cc:* Florence Renaud 
> *Subject:* Re: [Freeipa-users] Re: Why is ipa-ods-exporter broken after
> running ipa-dns-install? (Was - Unable to start directory server after
> updates)
>
> Jeremy Tourville wrote:
> > [root@utility certs]# curl https://utility.idm.nac-issa.org/
> > curl: (60) SSL certificate problem: self signed certificate in
> > certificate chain
> > More details here: https://curl.haxx.se/docs/sslcerts.html
> >
> > curl failed to verify the legitimacy of the server and therefore could not
> > establish a secure connection to it. To learn more about this situation and
> > how to fix it, please visit the web page mentioned above.
> >
> > [root@utility certs]# update-ca-trust
> >
> > [root@utility certs]# ausearch -m AVC -ts recent
> >
> > <no matches>
> >
> > [root@utility certs]# ipa-healthcheck
> > -bash: ipa-healthcheck: command not found
>
> I should have mentioned, try the curl after running update-ca-trust.
>
> ipa-healthcheck is not installed by default, you'd need to install the
> {free}ipa-healthcheck package.
>
> rob
> > *From:* Rob Crittenden 
> > *Sent:* Friday, September 10, 2021 9:33 AM
> > *To:* Jeremy Tourville ; FreeIPA users
> > list 
> > *Cc:* Florence Renaud 
> > *Subject:* Re: [Freeipa-users] Re: Why is ipa-ods-exporter broken after
> > running ipa-dns-install? (Was - Unable to start directory server after
> > updates)
> >
> > Jeremy Tourville wrote:
> > > [root@utility certs]# ipa-certupdate
> > > cannot connect to 'https://utility.idm.nac-issa.org/ipa/json': [SSL:
> > > CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:897)
> > > The ipa-certupdate command failed.
> > >
> > > Sort of a bad catch 22 I guess?
> >
> > Yeah, I was afraid of that.
> >
> > Let's walk through it. Try a simple command for another data point. I'm
> > not sure what we'd do with this but it will exercise the system-wide
> > trust as well:
> >
> > $ curl https://%60hostname%60/
> >
> > Rebuilding the CA trust db may help
> >
> > # update-ca-trust
> >
> > I suppose also look for AVCs in case something is way out-of-whack:
> >
> > # ausearch -m AVC -ts recent
> >
> > ipa-healthcheck may be something to try as well but you're likely to get
> > a crapton of false positives since it can't talk to the web interface.
> >
> > rob
> > > *From:* Rob Crittenden 
> > > *Sent:* Friday, September 10, 2021 9:09 AM
> > > *To:* Jeremy Tourville ; FreeIPA users
> > > list 
> > > *Cc:* Florence Renaud 
> > > *Subject:* Re: [Freeipa-users] Re: Why is ipa-ods-exporter broken after
> > > running ipa-dns-install? (Was - Unable to start directory server after
> > > updates)
> > >
> > > Jeremy Tourville wrote:
> > > > Now I understand how to test the cert(s) after re-reading your comments
> > > > Rob and Flo 🙂
> > > >
> > > > [root@utility certs]# openssl verify -verbose -show_chain -CAfile
> > > > /etc/ipa/ca.crt /var/lib/ipa/certs/httpd.crt
> > > > /var/lib/ipa/certs/httpd.crt: OK
> > > > Chain:
> > > > depth=0: O = IDM.NAC-ISSA.ORG, CN = utility.idm.nac-issa.org (untrusted)
> > > > depth=1: O = IDM.NAC-ISSA.ORG, CN = Certificate Authority
> > >
> > > I'd try running ipa-certupdate. I have the feeling some of the
> > > system-wide certificates are out-of-sync.
> > >
> > > rob
> > > > *From:* Jeremy Tourville 
> > > > *Sent:* Thursday, September 9, 2021 5:45 PM
> > > > *To:* FreeIPA users list 
> > > > *Cc:* Florence Renaud ; Rob Crittenden 
> > > > *Subject:* Re: [Freeipa-users] Re: Why is ipa-ods-exporter broken after
> > > > running ipa-dns-install? (Was - Unable to start directory server after
> > > > updates)
> > > >
> > > > Oh wait!!! Which set of certs do I need to test against for my
> > > > certificate chain?
> > > > I realized I didn't include the proper path when testing.  It should be
> > > > something like-
> > > >
> > > > # openssl verify -verbose -show_chain -CAfile <path to root or
> > > > intermediate cert> /etc/ipa/ca.crt
> > > > # openssl verify -verbose -show_chain -CAfile <path to root or
> > > > intermediate cert> /var/lib/ipa/certs/httpd.crt
> > > >
> > > > This would give you output (presuming you are using the correct set of
> > > > certs)
> > > >  /etc/ipa/ca.crt: OK
> > > > /var/lib/ipa/certs/httpd.crt: OK
> > > >
> > > > Which path contains the intermediate or root CA certs I need to test
> > > > against?
> > > >
> > > > [root@utility ~]# ls -la | find / -name *.crt
> > > > /etc/pki/ca-trust/extracted/openssl/ca-bundle.trust.crt
> > > > /etc/pki/ca-trust/source/ca-bundle.legacy.crt
> > > > /etc/pki/tls/certs/ca-bundle.crt
> > > > /etc/pki/tls/certs/ca-bundle.trust.crt
> > > > /etc/pki/tls/certs/localhost.crt
> > > > /etc/pki/pki-tomcat/alias/ca.crt
> > > > /etc/ipa/ca.crt
> > > > /etc/dirsrv/ssca/ca.crt
> > > > /etc/dirsrv/slapd-IDM-NAC-ISSA-ORG/Server-Cert.crt
> > > > /etc/dirsrv/slapd-IDM-NAC-ISSA-ORG/ca.crt
> > > > /var/lib/ipa/certs/httpd.crt
> > > > /var/kerberos/krb5kdc/kdc.crt
> > > > /usr/share/pki/ca-trust-legacy/ca-bundle.legacy.default.crt
> > > > /usr/share/pki/ca-trust-legacy/ca-bundle.legacy.disable.crt
> > > > /usr/share/ipa/html/ca.crt
> > > >
> > > > *From:* Jeremy Tourville 
> > > > *Sent:* Thursday, September 9, 2021 3:13 PM
> > > > *To:* FreeIPA users list 
> > > > *Cc:* Florence Renaud ; Rob Crittenden 
> > > > *Subject:* Re: [Freeipa-users] Re: Why is ipa-ods-exporter broken after
> > > > running ipa-dns-install? (Was - Unable to start directory server after
> > > > updates)
> > > > > > >It isn't complaining that the certificate isn't valid, it's complaining
> > > >
> > > > that it isn't trusted.
> > > >  Thanksfor pointing out my mistake.  I'm wearing some egg on my face.  I
> > > > was thinking about it wrong at the time of my reply.
> > > >
> > > > I attempted to verify trust-
> > > > [root@utility ipa]# openssl verify -verbose -show_chain -CAfile
> > > > /etc/ipa/ca.crt
> > > > ^C
> > > > [root@utility ipa]# openssl verify -verbose -show_chain -CAfile
> > > > /var/lib/ipa/certs/httpd.crt
> > > > ^C
> > > >
> > > > As you can see, no output, so yeah, they are not trusted.
> > > > > > Where did httpd.crt come from/what issuer?
> > > >
> > > > I recall not using a 3rd party CA.  The certs were just self-signed when
> > > > the ipa server was initially built.  I never did replace the certs as it
> > > > wasn't required for our situation.
> > > >
> > > > Next steps I guess would be to generate some new certs?  Thoughts?
> > > >
> > > > *From:* Rob Crittenden 
> > > > *Sent:* Thursday, September 9, 2021 12:53 PM
> > > > *To:* FreeIPA users list 
> > > > *Cc:* Florence Renaud ; Jeremy Tourville
> > > >
> > > > *Subject:* Re: [Freeipa-users] Re: Why is ipa-ods-exporter broken after
> > > > running ipa-dns-install? (Was - Unable to start directory server after
> > > > updates)
> > > >
> > > > Jeremy Tourville via FreeIPA-users wrote:
> > > > > /var/lib/ipa/certs/httpd.crt
> > > > > looks valid and has a 3 year validity date starting from Nov 23, 2020
> > > > >
> > > > > /etc/ipa/ca.crt
> > > > > looks valid and has a 20 year validity date starting from Nov 23, 2020
> > > >
> > > > It isn't complaining that the certificate isn't valid, it's complaining
> > > > that it isn't trusted. You also need to look at the signer and ensure
> > > > that the system trusts it globally. Where did httpd.crt come from/what
> > > > issuer?
> > > >
> > > > You might try running:
> > > >
> > > > openssl verify -verbose -show_chain -CAfile /etc/ipa/ca.crt
> > > > /var/lib/ipa/certs/httpd.crt
> > > >
> > > > See the default.conf(5) man page for a description of default.conf,
> > > > server.conf, etc. In this case server is a context so the configuration
> > > > only applies there.
> > > >
> > > > rob
> > > > > *From:* Florence Renaud 
> > > > > *Sent:* Tuesday, September 7, 2021 11:38 AM
> > > > > *To:* Jeremy Tourville 
> > > > > *Cc:* FreeIPA users list 
> > > > > *Subject:* Re: [Freeipa-users] Re: Why is ipa-ods-exporter broken after
> > > > > running ipa-dns-install? (Was - Unable to start directory server after
> > > > > updates)
> > > > >
> > > > > Hi Jeremy,
> > > > >
> > > > > to enable debugging you can simply create /etc/ipa/server.conf if the
> > > > > file does not exist:
> > > > > # cat /etc/ipa/server.conf
> > > > > [global]
> > > > > debug=True
> > > > > # systemctl restart httpd
> > > > >
> > > > > The HTTPd certificate is stored in /var/lib/ipa/certs/httpd.crt, you can
> > > > > examine its content with
> > > > > # openssl x509 -noout -text -in /var/lib/ipa/certs/httpd.crt
> > > > > If the IPA deployment includes an embedded CA, the CA that issued the
> > > > > httpd cert is stored in /etc/ipa/ca.crt and can also be checked with
> > > > > openssl command.
> > > > >
> > > > > flo
> > > > >
> > > > > On Tue, Sep 7, 2021 at 6:09 PM Jeremy Tourville
> > > > > <[e-mail] > wrote:
> > > > > I think I see the issue but I am unsure what to do to fix it.  See
> > > > > below.
> > > > >
> > > > > To answer your question, yes I did accept the security exception.
> > > > >
> > > > > Also, I don't see a server.conf file at /etc/ipa so that I may
> > > > > enable debugging.  What can you suggest for this issue?
> > > > >
> > > > >
> > > > > [root@utility ~]# ipactl status
> > > > > Directory Service: RUNNING
> > > > > krb5kdc Service: RUNNING
> > > > > kadmin Service: RUNNING
> > > > > named Service: RUNNING
> > > > > httpd Service: RUNNING
> > > > > ipa-custodia Service: RUNNING
> > > > > pki-tomcatd Service: RUNNING
> > > > > smb Service: RUNNING
> > > > > winbind Service: RUNNING
> > > > > ipa-otpd Service: RUNNING
> > > > > ipa-ods-exporter Service: STOPPED
> > > > > ods-enforcerd Service: RUNNING
> > > > > ipa-dnskeysyncd Service: RUNNING
> > > > > ipa: INFO: The ipactl command was successful
> > > > >
> > > > > [root@utility ~]# kinit admin
> > > > > Password for [e-mail] <mailto:[e-mail]>:
> > > > >
> > > > > [root@utility ~]# klist
> > > > > Ticket cache: KCM:0:43616
> > > > > Default principal: [e-mail]
> > > > > <mailto:[e-mail]>
> > > > >
> > > > > Valid starting       Expires              Service principal
> > > > > 09/07/2021 10:59:23  09/08/2021 10:09:04
> > > > >  krbtgt/[e-mail]
> > > > > <mailto:[e-mail]>
> > > > >
> > > > > [root@utility ~]# ipa config-show
> > > > > ipa: ERROR: cannot connect to
> > > > > 'https://utility.idm.nac-issa.org/ipa/json': [SSL:
> > > > > CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:897)
> > > > >
> > > > >
> > > > > ------------------------------------------------------------------------
> > > > > *From:* Florence Renaud <[e-mail] <mailto:[e-mail]>>
> > > > > *Sent:* Tuesday, September 7, 2021 10:47 AM
> > > > > *To:* FreeIPA users list <[e-mail]
> > > > > <mailto:[e-mail]>>
> > > > > *Cc:* Jeremy Tourville <[e-mail]
> > > > > <mailto:[e-mail]>>
> > > > > *Subject:* Re: [Freeipa-users] Re: Why is ipa-ods-exporter broken
> > > > > after running ipa-dns-install? (Was - Unable to start directory
> > > > > server after updates)
> > > > >
> > > > > Hi Jeremy,
> > > > > Did you accept the security exception displayed by the browser (I'm
> > > > > trying to eliminate obvious issues)?
> > > > > If nothing is displayed, can you check if ipa command-line is
> > > > > working as expected (for instance do "kinit admin; ipa config-show")?
> > > > > You may want to enable debug logs (add debug=True to the [global]
> > > > > section of /etc/ipa/server.conf and restart httpd service), retry
> > > > > WebUI authentication and check the generated logs in
> > > > > /var/log/http/error_log
> > > > >
> > > > > flo
> > > > >
> > > > > On Tue, Sep 7, 2021 at 2:01 PM Jeremy Tourville via FreeIPA-users
> > > > > <[e-mail]
> > > > > <mailto:[e-mail]>> wrote:
> > > > >
> > > > >     OK,
> > > > >     Why don't I see anything on the initial login page?
> > > > >     All I see is the URL and the fact that the certificate is not
> > > > >     trusted.  The certificate is not expired yet.  Not until Nov 2021.
> > > > >     The login in page is mostly solid white with no login or
> > > > >     password field.
> > > > >     _______________________________________________
> > > > >     FreeIPA-users mailing list --
> > > > >     [e-mail]
> > > > >     <mailto:[e-mail]>
> > > > >     To unsubscribe send an email to
> > > > >     [e-mail]
> > > > >     <mailto:[e-mail]>
> > > > >     Fedora Code of Conduct:
> > > > >     https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> > > > >     List Guidelines:
> > > > >     https://fedoraproject.org/wiki/Mailing_list_guidelines
> > > > >     List Archives:
> > > > >     https://lists.fedorahosted.org/archives/list/[e-mail]
> > > > >     Do not reply to spam on the list, report it:
> > > > >     https://pagure.io/fedora-infrastructure
> > > > >
> > > > >
> > > > > FreeIPA-users mailing list -- [e-mail]
> > > > > To unsubscribe send an email to [e-mail]
> > > > > Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> > > > > List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> > > > > List Archives: https://lists.fedorahosted.org/archives/list/[e-mail]...
> > > > > Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
>
> FreeIPA-users mailing list -- [e-mail]
> To unsubscribe send an email to [e-mail]
> Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives: https://lists.fedorahosted.org/archives/list/[e-mail]...
> Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure