Problems after replacing SSL certificates

[Andreas Bulling]

Dear all,

I have recently started using FreeIPA (4.8.1 on Ubuntu) and now wanted to replace the original SSL certificates for the web UI and the LDAP server with official ones issued by our university.

I've followed the procedure described here (no errors):
https://www.freeipa.org/page/Using_3rd_part_certificates_for_HTTP/LDAP

I could confirm in the browser that the certificate for the web UI has been replaced and I therefore assume so has the LDAP certificate. Authentication from other hosts/services using LDAP still works but in the server log file I see errors like these for all hosts in the domain:

Apr 20 19:57:11 auth krb5kdc[24895]: AS_REQ (8 etypes {18 17 20 19 16 23 25 26}) X: NEEDED_PREAUTH: host/X@X for krbtgt/X@X, Additional pre-authentication required
Apr 20 19:57:11 auth krb5kdc[24895]: closing down fd 12
Apr 20 19:57:11 auth krb5kdc[24895]: AS_REQ (8 etypes {18 17 20 19 16 23 25 26}) X: ISSUE: authtime 1587405431, etypes {rep=18 tkt=18 ses=18}, host/X@X for krbtgt/X@X
Apr 20 19:57:11 auth krb5kdc[24895]: closing down fd 12
Apr 20 19:57:11 auth krb5kdc[24895]: TGS_REQ (8 etypes {18 17 20 19 16 23 25 26}) X: ISSUE: authtime 1587405431, etypes {rep=18 tkt=18 ses=18}, host/X@X for ldap/X@X
Apr 20 19:57:11 auth krb5kdc[24895]: closing down fd 12

Also, ipa-certupdate on the respective clients shows

ipa-certupdate
trying https://X/ipa/json
[try 1]: Forwarding 'schema' to json server 'https://X/ipa/json'
cannot connect to 'https://X/ipa/json': [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:727)
The ipa-certupdate command failed.

Also, I can't login to the web UI anymore. I tried

ipa-getkeytab -s X -p HTTP/X@X -k /var/lib/ipa/gssproxy/http.keytab

on the freeipa server (followed by ipactl restart) but this didn't help.

Any idea/suggestions for how to get everything working again?

Thanks a lot!

[Rob Crittenden]

Andreas Bulling via FreeIPA-users wrote:
> Dear all,
>
> I have recently started using FreeIPA (4.8.1 on Ubuntu) and now wanted to replace the original SSL certificates for the web UI and the LDAP server with official ones issued by our university.
>
> I've followed the procedure described here (no errors):
> https://www.freeipa.org/page/Using_3rd_part_certificates_for_HTTP/LDAP
>
> I could confirm in the browser that the certificate for the web UI has been replaced and I therefore assume so has the LDAP certificate. Authentication from other hosts/services using LDAP still works but in the server log file I see errors like these for all hosts in the domain:
>
> Apr 20 19:57:11 auth krb5kdc[24895]: AS_REQ (8 etypes {18 17 20 19 16 23 25 26}) X: NEEDED_PREAUTH: host/X@X for krbtgt/X@X, Additional pre-authentication required
> Apr 20 19:57:11 auth krb5kdc[24895]: closing down fd 12
> Apr 20 19:57:11 auth krb5kdc[24895]: AS_REQ (8 etypes {18 17 20 19 16 23 25 26}) X: ISSUE: authtime 1587405431, etypes {rep=18 tkt=18 ses=18}, host/X@X for krbtgt/X@X
> Apr 20 19:57:11 auth krb5kdc[24895]: closing down fd 12
> Apr 20 19:57:11 auth krb5kdc[24895]: TGS_REQ (8 etypes {18 17 20 19 16 23 25 26}) X: ISSUE: authtime 1587405431, etypes {rep=18 tkt=18 ses=18}, host/X@X for ldap/X@X
> Apr 20 19:57:11 auth krb5kdc[24895]: closing down fd 12
>
> Also, ipa-certupdate on the respective clients shows
>
> ipa-certupdate
> trying https://X/ipa/json
> [try 1]: Forwarding 'schema' to json server 'https://X/ipa/json'
> cannot connect to 'https://X/ipa/json': [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:727)
> The ipa-certupdate command failed.
>
> Also, I can't login to the web UI anymore. I tried
>
> ipa-getkeytab -s X -p HTTP/X@X -k /var/lib/ipa/gssproxy/http.keytab
>
> on the freeipa server (followed by ipactl restart) but this didn't help.

Kerberos and TLS are separate crypto engines. Changing the certs
shouldn't affect it at all.

You have a chicken and egg problem. When replacing your certs on an
existing infrastructure you first have to add your new CA certs using
ipa-cacert-manage, then run ipa-certupdate on all enrolled machines,
including masters, then you can run ipa-servercert-install to replace them.

Otherwise your clients will not trust the CA that issued the new certs.

For the UI error I'd start with the apache error log for details.

rob

[Andreas Bulling]

> Andreas Bulling via FreeIPA-users wrote:
>
> You have a chicken and egg problem. When replacing your certs on an
> existing infrastructure you first have to add your new CA certs using
> ipa-cacert-manage, then run ipa-certupdate on all enrolled machines,
> including masters, then you can run ipa-servercert-install to replace them.

This seems to be the routine described on the freeipa page - which I followed except for running ipa-certupdate on all enrolled machines prior to ipa-servercert-install. The documentation doesn't mention this, should probably be fixed before more people end up in this situation.

Is there any way for me to fix this? client uninstall and reinstall?

[Florence Blanc-Renaud]

On 4/20/20 8:39 PM, Andreas Bulling via FreeIPA-users wrote:
> > Andreas Bulling via FreeIPA-users wrote:
> >
> > You have a chicken and egg problem. When replacing your certs on an
> > existing infrastructure you first have to add your new CA certs using
> > ipa-cacert-manage, then run ipa-certupdate on all enrolled machines,
> > including masters, then you can run ipa-servercert-install to replace them.
>
> This seems to be the routine described on the freeipa page - which I followed except for running ipa-certupdate on all enrolled machines prior to ipa-servercert-install. The documentation doesn't mention this, should probably be fixed before more people end up in this situation.

Hi,

I just updated the page 
https://www.freeipa.org/page/Using_3rd_part_certificates_for_HTTP/LDAP 
with a note mentioning that ipa-certupdate needs to be run on all the nodes.
> Is there any way for me to fix this? client uninstall and reinstall?

You just need to add the new CA to /etc/ipa/ca.crt (append the 
-----BEGIN CERTIFICATE---- .... -----END CERTIFICATE----- blob at the 
end of the file) and to /etc/ipa/nssdb with
$ certutil -A -d /etc/ipa/nssdb -n nickname -t CT,C,C -a -i 
/path/to/cacert.crt

Once it's done you can check if everything is working with 
ipa-certupdate or any ipa *-find command.

HTH,
flo
> FreeIPA-users mailing list -- [e-mail]
> To unsubscribe send an email to [e-mail]
> Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives: https://lists.fedorahosted.org/archives/list/[e-mail]...

[Andreas Bulling]

Thanks for your help. I ended up uninstalling and reinstalling all clients and saw the new CA certificates during the process. But the ISSUE/NEEDED_PREAUTH messages remain - is that normal?

Any idea how I can fix my other problem, that of not being able to login to the admin interface? In the apache server log I see

[Tue Apr 21 13:50:50.888429 2020] [wsgi:error] [pid 28066:tid 140524982961920] [remote X:51978] ipa: INFO: 401 Unauthorized: HTTPSConnectionPool(host='Xipa', port=443): Max retries exceeded with url: /session/cookie (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x7fce86b4c310>: Failed to establish a new connection: [Errno -2] Name or service not known'))

Note that, for some reason, "ipa" is added to the hostname which, of course, results in a host not found error.

[Andreas Bulling]

The admin login problem I just managed to fix - missing trailing slash in a permanent redirect from http to https in Apache.

But the ISSUE/NEEDED_PREAUTH messages I'd still like to figure out if these are not normal.

Thanks!

[Simo Sorce]

On Tue, 2020-04-21 at 12:25 +0000, Andreas Bulling via FreeIPA-users
wrote:
> The admin login problem I just managed to fix - missing trailing slash in a permanent redirect from http to https in Apache.
>
> But the ISSUE/NEEDED_PREAUTH messages I'd still like to figure out if these are not normal.

They are perfectly normal if immediately followed by a pre-
authenticated request.

Simo.

[Andreas Bulling]

This is what the Apache error log shows:

[Mon Apr 20 20:40:32.719986 2020] [wsgi:error] [pid 24934:tid 139866966574848] [remote 141.58.21.12:59320] ipa: INFO: 401 Unauthorized: HTTPSConnectionPool(host='Xipa', port=443): Max retries exceeded with url: /session/cookie (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x7f3551df4090>: Failed to establish a new connection: [Errno -2] Name or service not known'))

[Andreas Bulling]

I noticed a weird "ipa" directly appended to the host URL. Not sure where this is coming from...

[John Burns]

I have this exact same error on ipa-certupdate, after deleting certs that expired on May 30. Were you able to find any leads in the time since this post?

ipa-certupdate is needed after "ipa-cacert-manage install" commands, prior to ipa-server-certinstall.

[Florence Blanc-Renaud]

On 6/5/20 7:50 PM, John Burns via FreeIPA-users wrote:
> I have this exact same error on ipa-certupdate, after deleting certs that expired on May 30. Were you able to find any leads in the time since this post?
>
> ipa-certupdate is needed after "ipa-cacert-manage install" commands, prior to ipa-server-certinstall.
> _______________________________________________
> FreeIPA-users mailing list -- [e-mail]
> To unsubscribe send an email to [e-mail]
> Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives: https://lists.fedorahosted.org/archives/list/[e-mail]...

Hi,

I believe this question was already answered in the thread:
https://lists.fedorahosted.org/archives/list/[e-mail]...

If you forgot to run ipa-certupdate on a node, you need to add the new 
CA to /etc/ipa/ca.crt and /etc/ipa/nssdb. After that, ipa-certupdate 
should work.

HTH,
flo

[cicek adam]

Hi, I’ve been suffocating the same problem. I applied ipa-server-certinstall without adding ca first. 
I applied your steps and added my ca.crt to /etc/ipa/ca.crt and /etc/ipa/nssdb with certutil, after than I run ipa-certupdate and it still fails.

[root@xxx ~]# certutil -d sql:/etc/ipa/nssdb/ -L

Certificate Nickname                                         Trust Attributes
                                                            SSL,S/MIME,JAR/XPI

Xxx IPA CA                                   		   CT,C,C
globalsign                                                   CT,C,C

After this I ran ipa-certupdate and it says

cannot connect to 'any of the configured servers’: …. (List of my ipaservers goes here)
The ipa-certupdate command failed.

Should I do this process for all servers, or I am missing something? Related to this problem I am having login failure at the web ui. Would it work if I created a new db and added my GlobalSign ca there? Do I need the self signed ipa ca?

PS: I'm running freeipa on rhel8

Thanks.

[Florence Blanc-Renaud]

Hi,
you can manually add the new CA to the NSS databases:
- /etc/dirsrv/slapd-xxx
- /etc/ipa/nssdb
- /etc/pki/pki-tomcat/alias (if you have configured an embedded CA)
- /etc/httpd/alias (if IPA version < 4.7)

and to the PEM files /etc/ipa/ca.crt and /usr/share/ipa/html/ca.crt.

ipa-certupdate needs the services to be up and running, what is the output
of "ipactl status" on your server?

flo

On Sun, Oct 17, 2021 at 1:21 AM cicek adam via FreeIPA-users <
[e-mail]> wrote:
> Hi, I’ve been suffocating the same problem. I applied
> ipa-server-certinstall without adding ca first.
> I applied your steps and added my ca.crt to /etc/ipa/ca.crt and
> /etc/ipa/nssdb with certutil, after than I run ipa-certupdate and it still
> fails.
>
> [root@xxx ~]# certutil -d sql:/etc/ipa/nssdb/ -L
>
> Certificate Nickname                                         Trust
> Attributes
>
> SSL,S/MIME,JAR/XPI
>
> Xxx IPA CA                                                 CT,C,C
> globalsign                                                   CT,C,C
>
> After this I ran ipa-certupdate and it says
>
> cannot connect to 'any of the configured servers’: …. (List of my
> ipaservers goes here)
> The ipa-certupdate command failed.
>
> Should I do this process for all servers, or I am missing something?
> Related to this problem I am having login failure at the web ui. Would it
> work if I created a new db and added my GlobalSign ca there? Do I need the
> self signed ipa ca?
>
> PS: I'm running freeipa on rhel8
>
> Thanks.
> _______________________________________________
> FreeIPA-users mailing list -- [e-mail]
> To unsubscribe send an email to [e-mail]
> Fedora Code of Conduct:
> https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives:
> https://lists.fedorahosted.org/archives/list/[e-mail]...
> Do not reply to spam on the list, report it:
> https://pagure.io/fedora-infrastructure

[cicek adam]

Here is my ipactl status:

[root@xxx ~]# ipactl status
Directory Service: RUNNING
krb5kdc Service: RUNNING
kadmin Service: RUNNING
named Service: RUNNING
httpd Service: RUNNING
ipa-custodia Service: RUNNING
pki-tomcatd Service: RUNNING
ipa-otpd Service: RUNNING
ipa-dnskeysyncd Service: RUNNING
ipa: INFO: The ipactl command was successful

I think I am doing something wrong. I've made a fresh installation, then added ca.crt by "ipa-cacert-manage -n globalsign -t C,, install /root/ca.crt"

After this I ran ipa-certupdate and it was successful, I had no errors. So I tought it to be safe to run ipa-server-certinstall and ran it. 
As a result I get  login failure in the web ui again. When I check httpd error_log I see this:

[Wed Oct 20 14:02:17.214267 2021] [wsgi:error] [pid 20252:tid 140636607313664] [remote 10.212.238.92:52437] ipa: INFO: 401 Unauthorized: HTTPSConnectionPool(host='xxx', port=443): Max retries exceeded with url: /ipa/session/cookie (Caused by SSLError(SSLError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:897)'),))

After I saw this, I tried ipa-certupdate again and it gave the "cannot connect to 'any of the configured servers’:" error again.

What am I doing wrong? I did ipactl restart after ipa-server-certinstall.

I think I am missing some basics :/

[Florence Blanc-Renaud]

Hi,

I think you're hitting this issue: https://pagure.io/freeipa/issue/7759

What is the full certificate chain of your new server cert? If the chain
contains a root CA and one or multiple subCAs, each subCA also needs to be
added using ipa-cacert-manage install.
HTH,
flo

On Wed, Oct 20, 2021 at 1:29 PM cicek adam via FreeIPA-users <
[e-mail]> wrote:
> Here is my ipactl status:
>
> [root@xxx ~]# ipactl status
> Directory Service: RUNNING
> krb5kdc Service: RUNNING
> kadmin Service: RUNNING
> named Service: RUNNING
> httpd Service: RUNNING
> ipa-custodia Service: RUNNING
> pki-tomcatd Service: RUNNING
> ipa-otpd Service: RUNNING
> ipa-dnskeysyncd Service: RUNNING
> ipa: INFO: The ipactl command was successful
>
> I think I am doing something wrong. I've made a fresh installation, then
> added ca.crt by "ipa-cacert-manage -n globalsign -t C,, install
> /root/ca.crt"
>
> After this I ran ipa-certupdate and it was successful, I had no errors. So
> I tought it to be safe to run ipa-server-certinstall and ran it.
> As a result I get  login failure in the web ui again. When I check httpd
> error_log I see this:
>
> [Wed Oct 20 14:02:17.214267 2021] [wsgi:error] [pid 20252:tid
> 140636607313664] [remote 10.212.238.92:52437] ipa: INFO: 401
> Unauthorized: HTTPSConnectionPool(host='xxx', port=443): Max retries
> exceeded with url: /ipa/session/cookie (Caused by SSLError(SSLError(1,
> '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed
> (_ssl.c:897)'),))
>
> After I saw this, I tried ipa-certupdate again and it gave the "cannot
> connect to 'any of the configured servers’:" error again.
>
> What am I doing wrong? I did ipactl restart after ipa-server-certinstall.
>
> I think I am missing some basics :/
> _______________________________________________
> FreeIPA-users mailing list -- [e-mail]
> To unsubscribe send an email to [e-mail]
> Fedora Code of Conduct:
> https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives:
> https://lists.fedorahosted.org/archives/list/[e-mail]...
> Do not reply to spam on the list, report it:
> https://pagure.io/fedora-infrastructure

[cicek adam]

Yay! It worked. As you guessed I had an subca but it is from the same issuer as rootca, so i didn't consider it but when I add subca it really worked. Thank you very much, I appreciate it.