FreeIPA w. letsencrypt for HTTPS/LDAP failing to communicate with itself

[Chris Moody]

Hello folks.

Hopefully I'm just missing something face-palm level obvious, but I am 
running into some trouble when interfacing with my CA functionality on 
an IPA server cluster.  My attempts at scouring all my saved prior-comms 
from the mailing-list as well as several search-engines are not 
enchanting me with much clue.

It appears that my need for the LetsEncrypt certs for the user-facing 
Web-UI and LDAPs components are causing IPA to dis-trust itself.

===
4-node cluster - Ubuntu19.10
(all nodes currently fully updated/patched via the official Ubuntu repos)
===
ipa --version
VERSION: 4.8.1, API_VERSION: 2.233
===
running letsencrypt certificates successfully for HTTPs & LDAPs connectivity
===

These 4-nodes are all happily running and replicating betwixt each 
other.  LDAPs is functioning great and many linux systems are able to 
all join as freeipa-clients.  Users and groups are replicating and being 
used elegantly for many LDAP-based authentication/authorization needs.

Overall, for these nodes, life is good.


Where I'm running into trouble is in finally wanting to leverage 
certificate issuance on a per-user basis.  End goal is integrating 
things like yubikeys, user-cert auth, and so on.


In the UI, when I enter a user's account and select Actions->New 
Certificate, I am able to successfully issue the couple prompted 
'certutil' commands to generate the user's CSR.  I then paste in the 

contents of the CSR and hit 'Issue' and run into the following error:
==========
IPA Error 907: NetworkError
cannot connect to 
'https://REDACTED-1.ipa.REDACTED.com:443/ca/rest/account/login': [SSL: 
TLSV1_ALERT_UNKNOWN_CA] tlsv1 alert unknown ca (_ssl.c:2508)
==========

As I then start digging into cli-mode to attempt to understand where 
things are unhappy, I run into similar troubles with the server 
attempting to talk to itself and not being very happy about it.

==========
chris@REDACTED-1:~$ ipa ca-find
------------
1 CA matched
------------
   Name: ipa
   Description: IPA CA
   Authority ID: 8acca54b-64d7-44bf-b8f7-59316213cfb6
   Subject DN: CN=Certificate Authority,O=IPA.REDACTED.COM
   Issuer DN: CN=Certificate Authority,O=IPA.REDACTED.COM
----------------------------
Number of entries returned 1
----------------------------
chris@REDACTED-1:~$ ipa ca-show
Name: ipa
ipa: ERROR: cannot connect to 
'https://REDACTED-1.ipa.REDACTED.com:443/ca/rest/account/login': [SSL: 
TLSV1_ALERT_UNKNOWN_CA] tlsv1 alert unknown ca (_ssl.c:2508)
==========

Verifying with 'openssl s_client' returns the valid and non-expired LE 
cert-chain.

==========
chris@REDACTED-1:~$ openssl s_client REDACTED-1.ipa.REDACTED.com:443
CONNECTED(00000003)
depth=2 O = Digital Signature Trust Co., CN = DST Root CA X3
verify return:1
depth=1 C = US, O = Let's Encrypt, CN = R3
verify return:1
depth=0 CN = REDACTED-1.ipa.REDACTED.com
verify return:1
---
Certificate chain
  0 s:CN = REDACTED-1.ipa.REDACTED.com
    i:C = US, O = Let's Encrypt, CN = R3
  1 s:C = US, O = Let's Encrypt, CN = R3
    i:O = Digital Signature Trust Co., CN = DST Root CA X3
---
...<output-truncated>...
---
SSL handshake has read 3046 bytes and written 413 bytes
Verification: OK
---
New, TLSv1.3, Cipher is TLS_AES_256_GCM_SHA384
Server public key is 2048 bit
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
Early data was not sent
Verify return code: 0 (ok)
...<output-truncated>...
==========

Can anyone please hit me with some clue-bat as to where I can read to 
understand how to get IPA to love itself?  I'm suspecting it's likely 
some certificate inclusion/exception that I need to add so that API 
calls and the ipa command itself will actually respect the LE cert-chain?

Any hints would be greatly appreciated.

Thanks,
-Chris

-- 
Node-Nine, Inc.
chris(a)node-nine.com
619.354.6463

[Florence Renaud]

Hi,

when the let's encrypt certificates were installed, did you run
ipa-cacert-manage install on one of the nodes + ipa-certupdate on *all the
IPA machines*? It's important to run ipa-certupdate on all the
server/replicas/clients in order to install the CA everywhere.

flo

On Sat, Jun 12, 2021 at 2:19 AM Chris Moody via FreeIPA-users <
freeipa-users(a)lists.fedorahosted.org&gt; wrote:

>  Hello folks.
>
>  Hopefully I'm just missing something face-palm level obvious, but I am
>  running into some trouble when interfacing with my CA functionality on an
>  IPA server cluster.  My attempts at scouring all my saved prior-comms from
>  the mailing-list as well as several search-engines are not enchanting me
>  with much clue.
>
>  It appears that my need for the LetsEncrypt certs for the user-facing
>  Web-UI and LDAPs components are causing IPA to dis-trust itself.
>
>  ===
>  4-node cluster - Ubuntu19.10
>  (all nodes currently fully updated/patched via the official Ubuntu repos)
>  ===
>  ipa --version
>  VERSION: 4.8.1, API_VERSION: 2.233
>  ===
>  running letsencrypt certificates successfully for HTTPs & LDAPs
>  connectivity
>  ===
>
>  These 4-nodes are all happily running and replicating betwixt each other.
>  LDAPs is functioning great and many linux systems are able to all join as
>  freeipa-clients.  Users and groups are replicating and being used elegantly
>  for many LDAP-based authentication/authorization needs.
>
>  Overall, for these nodes, life is good.
>
>
>  Where I'm running into trouble is in finally wanting to leverage
>  certificate issuance on a per-user basis.  End goal is integrating things
>  like yubikeys, user-cert auth, and so on.
>
>
>  In the UI, when I enter a user's account and select Actions->New
>  Certificate, I am able to successfully issue the couple prompted 'certutil'
>  commands to generate the user's CSR.  I then paste in the contents of the
>  CSR and hit 'Issue' and run into the following error:
>  ==========
>  IPA Error 907: NetworkError
>  cannot connect to '
>  https://REDACTED-1.ipa.REDACTED.com:443/ca/rest/account/login';: [SSL:
>  TLSV1_ALERT_UNKNOWN_CA] tlsv1 alert unknown ca (_ssl.c:2508)
>  ==========
>
>  As I then start digging into cli-mode to attempt to understand where
>  things are unhappy, I run into similar troubles with the server attempting
>  to talk to itself and not being very happy about it.
>
>  ==========
>  chris@REDACTED-1:~$ ipa ca-find
>  ------------
>  1 CA matched
>  ------------
>    Name: ipa
>    Description: IPA CA
>    Authority ID: 8acca54b-64d7-44bf-b8f7-59316213cfb6
>    Subject DN: CN=Certificate Authority,O=IPA.REDACTED.COM
>    Issuer DN: CN=Certificate Authority,O=IPA.REDACTED.COM
>  ----------------------------
>  Number of entries returned 1
>  ----------------------------
>  chris@REDACTED-1:~$ ipa ca-show
>  Name: ipa
>  ipa: ERROR: cannot connect to '
>  https://REDACTED-1.ipa.REDACTED.com:443/ca/rest/account/login';: [SSL:
>  TLSV1_ALERT_UNKNOWN_CA] tlsv1 alert unknown ca (_ssl.c:2508)
>  ==========
>
>  Verifying with 'openssl s_client' returns the valid and non-expired LE
>  cert-chain.
>
>  ==========
>  chris@REDACTED-1:~$ openssl s_client REDACTED-1.ipa.REDACTED.com:443
>  CONNECTED(00000003)
>  depth=2 O = Digital Signature Trust Co., CN = DST Root CA X3
>  verify return:1
>  depth=1 C = US, O = Let's Encrypt, CN = R3
>  verify return:1
>  depth=0 CN = REDACTED-1.ipa.REDACTED.com
>  verify return:1
>  ---
>  Certificate chain
>   0 s:CN = REDACTED-1.ipa.REDACTED.com
>     i:C = US, O = Let's Encrypt, CN = R3
>   1 s:C = US, O = Let's Encrypt, CN = R3
>     i:O = Digital Signature Trust Co., CN = DST Root CA X3
>  ---
>  ...<output-truncated>...
>  ---
>  SSL handshake has read 3046 bytes and written 413 bytes
>  Verification: OK
>  ---
>  New, TLSv1.3, Cipher is TLS_AES_256_GCM_SHA384
>  Server public key is 2048 bit
>  Secure Renegotiation IS NOT supported
>  Compression: NONE
>  Expansion: NONE
>  No ALPN negotiated
>  Early data was not sent
>  Verify return code: 0 (ok)
>  ...<output-truncated>...
>  ==========
>
>  Can anyone please hit me with some clue-bat as to where I can read to
>  understand how to get IPA to love itself?  I'm suspecting it's likely some
>  certificate inclusion/exception that I need to add so that API calls and
>  the ipa command itself will actually respect the LE cert-chain?
>
>  Any hints would be greatly appreciated.
>
>  Thanks,
>  -Chris
>
>  --
>  Node-Nine, Inc.chris(a)node-nine.com
>  619.354.6463
>
>  _______________________________________________
>  FreeIPA-users mailing list -- freeipa-users(a)lists.fedorahosted.org
>  To unsubscribe send an email to freeipa-users-leave(a)lists.fedorahosted.org
>  Fedora Code of Conduct:
>  https://docs.fedoraproject.org/en-US/project/code-of-conduct/
>  List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
>  List Archives:
>  https://lists.fedorahosted.org/archives/list/[e-mail]...
>  Do not reply to spam on the list, report it:
>  https://pagure.io/fedora-infrastructure
>

[Chris Moody]

Apologies for the belated response - took me a bit to verify across all 
clients.

When I installed the LE certs on each replica/server, I performed the 
following:
=====(the privkey & fullchain files provided by LE)=====
ipa-server-certinstall -w -d privkey.pem fullchain.pem
&
/usr/sbin/ipa-certupdate
=====

I have verified via 'openssl s_client -connect' that both https & ldaps 
are serving the proper LE certificates across all IPA servers.

I have also now iterated across every ipa-client installation and run 
'ipa-certupdate' as well.  All the /etc/ssl/certs/ca-certificates.crt 
files on each and every system have the LE certs and are current.

Unfortunately, the 'Actions->New Certificate' process I mentioned is 
still giving me identical behavior.

I followed the exact steps in the NewCert dialogue from one of the 
validated-current ipa-clients:
=====
# Create a certificate database or use an existing one. To create a new 
database:
|# certutil -N -d <database path>|
# Create a CSR with subject /CN=<uid>,O=<realm>/, for example:
|# certutil -R -d <database path> -a -g <key size> -s 
'CN=chris,O=IPA.REDACTED.COM'|


=====
IPA Error 907: NetworkError
cannot connect to 
'https://REDACTED-1.ipa.REDACTED.com:443/ca/rest/account/login': [SSL: 
TLSV1_ALERT_UNKNOWN_CA] tlsv1 alert unknown ca (_ssl.c:2508)
=====

-Chris


On 6/12/21 3:55 AM, Florence Renaud wrote:
>  Hi,
>
>  when the let's encrypt certificates were installed, did you run 
>  ipa-cacert-manage install on one of the nodes + ipa-certupdate on _all 
>  the IPA machines_? It's important to run ipa-certupdate on all the 
>  server/replicas/clients in order to install the CA everywhere.
>
>  flo
>
>  On Sat, Jun 12, 2021 at 2:19 AM Chris Moody via FreeIPA-users 
>  <freeipa-users(a)lists.fedorahosted.org 
>  <mailto:[e-mail]>> wrote:
>
>      Hello folks.
>
>      Hopefully I'm just missing something face-palm level obvious, but
>      I am running into some trouble when interfacing with my CA
>      functionality on an IPA server cluster.  My attempts at scouring
>      all my saved prior-comms from the mailing-list as well as several
>      search-engines are not enchanting me with much clue.
>
>      It appears that my need for the LetsEncrypt certs for the
>      user-facing Web-UI and LDAPs components are causing IPA to
>      dis-trust itself.
>
>      ===
>      4-node cluster - Ubuntu19.10
>      (all nodes currently fully updated/patched via the official Ubuntu
>      repos)
>      ===
>      ipa --version
>      VERSION: 4.8.1, API_VERSION: 2.233
>      ===
>      running letsencrypt certificates successfully for HTTPs & LDAPs
>      connectivity
>      ===
>
>      These 4-nodes are all happily running and replicating betwixt each
>      other.  LDAPs is functioning great and many linux systems are able
>      to all join as freeipa-clients. Users and groups are replicating
>      and being used elegantly for many LDAP-based
>      authentication/authorization needs.
>
>      Overall, for these nodes, life is good.
>
>
>      Where I'm running into trouble is in finally wanting to leverage
>      certificate issuance on a per-user basis.  End goal is integrating
>      things like yubikeys, user-cert auth, and so on.
>
>
>      In the UI, when I enter a user's account and select Actions->New
>      Certificate, I am able to successfully issue the couple prompted
>      'certutil' commands to generate the user's CSR.  I then paste in
>      the contents of the CSR and hit 'Issue' and run into the following
>      error:
>      ==========
>      IPA Error 907: NetworkError
>      cannot connect to
>      'https://REDACTED-1.ipa.REDACTED.com:443/ca/rest/account/login
>      <https://REDACTED-1.ipa.REDACTED.com:443/ca/rest/account/login>';:
>      [SSL: TLSV1_ALERT_UNKNOWN_CA] tlsv1 alert unknown ca (_ssl.c:2508)
>      ==========
>
>      As I then start digging into cli-mode to attempt to understand
>      where things are unhappy, I run into similar troubles with the
>      server attempting to talk to itself and not being very happy about it.
>
>      ==========
>      chris@REDACTED-1:~$ ipa ca-find
>      ------------
>      1 CA matched
>      ------------
>        Name: ipa
>        Description: IPA CA
>        Authority ID: 8acca54b-64d7-44bf-b8f7-59316213cfb6
>        Subject DN: CN=Certificate Authority,O=IPA.REDACTED.COM
>      <http://IPA.REDACTED.COM>
>        Issuer DN: CN=Certificate Authority,O=IPA.REDACTED.COM
>      <http://IPA.REDACTED.COM>
>      ----------------------------
>      Number of entries returned 1
>      ----------------------------
>      chris@REDACTED-1:~$ ipa ca-show
>      Name: ipa
>      ipa: ERROR: cannot connect to
>      'https://REDACTED-1.ipa.REDACTED.com:443/ca/rest/account/login
>      <https://REDACTED-1.ipa.REDACTED.com:443/ca/rest/account/login>';:
>      [SSL: TLSV1_ALERT_UNKNOWN_CA] tlsv1 alert unknown ca (_ssl.c:2508)
>      ==========
>
>      Verifying with 'openssl s_client' returns the valid and
>      non-expired LE cert-chain.
>
>      ==========
>      chris@REDACTED-1:~$ openssl s_client
>      REDACTED-1.ipa.REDACTED.com:443
>      <http://REDACTED-1.ipa.REDACTED.com:443>
>      CONNECTED(00000003)
>      depth=2 O = Digital Signature Trust Co., CN = DST Root CA X3
>      verify return:1
>      depth=1 C = US, O = Let's Encrypt, CN = R3
>      verify return:1
>      depth=0 CN = REDACTED-1.ipa.REDACTED.com
>      <http://REDACTED-1.ipa.REDACTED.com>
>      verify return:1
>      ---
>      Certificate chain
>       0 s:CN = REDACTED-1.ipa.REDACTED.com
>      <http://REDACTED-1.ipa.REDACTED.com>
>         i:C = US, O = Let's Encrypt, CN = R3
>       1 s:C = US, O = Let's Encrypt, CN = R3
>         i:O = Digital Signature Trust Co., CN = DST Root CA X3
>      ---
>      ...<output-truncated>...
>      ---
>      SSL handshake has read 3046 bytes and written 413 bytes
>      Verification: OK
>      ---
>      New, TLSv1.3, Cipher is TLS_AES_256_GCM_SHA384
>      Server public key is 2048 bit
>      Secure Renegotiation IS NOT supported
>      Compression: NONE
>      Expansion: NONE
>      No ALPN negotiated
>      Early data was not sent
>      Verify return code: 0 (ok)
>      ...<output-truncated>...
>      ==========
>
>      Can anyone please hit me with some clue-bat as to where I can read
>      to understand how to get IPA to love itself? I'm suspecting it's
>      likely some certificate inclusion/exception that I need to add so
>      that API calls and the ipa command itself will actually respect
>      the LE cert-chain?
>
>      Any hints would be greatly appreciated.
>
>      Thanks,
>      -Chris
>
>      -- 
>      Node-Nine, Inc.
>      chris(a)node-nine.com  <mailto:[e-mail]>
>      619.354.6463
>
>      _______________________________________________
>      FreeIPA-users mailing list -- freeipa-users(a)lists.fedorahosted.org
>      <mailto:[e-mail]>
>      To unsubscribe send an email to
>      freeipa-users-leave(a)lists.fedorahosted.org
>      <mailto:[e-mail]>
>      Fedora Code of Conduct:
>      https://docs.fedoraproject.org/en-US/project/code-of-conduct/
>      <https://docs.fedoraproject.org/en-US/project/code-of-conduct/>
>      List Guidelines:
>      https://fedoraproject.org/wiki/Mailing_list_guidelines
>      <https://fedoraproject.org/wiki/Mailing_list_guidelines>
>      List Archives:
>      https://lists.fedorahosted.org/archives/list/[e-mail]...
>     
> <https://lists.fedorahosted.org/archives/list/[e-mail]...
>      Do not reply to spam on the list, report it:
>      https://pagure.io/fedora-infrastructure
>      <https://pagure.io/fedora-infrastructure>
>  
-- 
Node-Nine, Inc.
chris(a)node-nine.com
619.354.6463

[Chris Moody]

Just found some additional possible clues in the apache error.log

=====
[Tue Jun 15 17:11:34.636290 2021] [:warn] [pid 31831:tid 
139703600768768] [client 2001:470:8af9:255::10:47920] failed to set 
perms (3140) on file (/run/ipa/ccaches/chris(a)IPA.NODE-NINE.COM)!, 
referer: https://REDACTED-1.ipa.REDACTED.com/ipa/ui/
[Tue Jun 15 17:11:34.674975 2021] [ssl:error] [pid 31830:tid 
139703550412544] [client 2604:a880:2:d1::36:4001:58500] AH02039: 
Certificate Verification: Error (19): self signed certificate in 
certificate chain
[Tue Jun 15 17:11:34.675088 2021] [ssl:error] [pid 31830:tid 
139703550412544] [client 2604:a880:2:d1::36:4001:58500] AH02261: 
Re-negotiation handshake failed
[Tue Jun 15 17:11:34.675111 2021] [ssl:error] [pid 31830:tid 
139703550412544] SSL Library Error: error:1417C086:SSL 
routines:tls_process_client_certificate:certificate verify failed
[Tue Jun 15 17:11:34.675884 2021] [wsgi:error] [pid 31828:tid 
139703722125056] [remote 2001:470:8af9:255::10:47920] ipa: INFO: 
[jsonserver_session] chris(a)IPA.REDACTED.COM: cert_request('-----BEGIN 
NEW CERTIFICATE 
REQUEST-----\\nMIIEcTCCAlkCAQAwLDEaMBgGA1UEChMRSVBBLk5PREUtTklORS5DT00xDjAMBgNV\\nBAMTBWNocmlzMIICIjANBgkqhkiG9w0BAQEFAAOCAg8AMIICCgKCAgEA8QR2JbFR\\nE7S7/XAk9V1bNEvvXxB65MpLnIu6gLnfXr8yz0xcTjvEDmKAukS7u+GhwnVfQvwS\\nRjlexS5leIZfMFU59R5K6ubzM5e3Qy07xQU40+8jLS2o3SUo7CvY+TcgTmfJWiK4\\nlWlG70LcMy6VbBVf+nQNoenryPDK2Y94kG3ydcKjGLZsaJ69s1OaSXxZY4esEhk/\\nFJs+odjMQRguS5sbUmfpSy3FTJsvuy4Sge2X5HsHOCKM2bWMcDqkRGI428toET0i\\n+FIXlvroqJQ7OL/RmwVcYOFMwMWN5Ne3g0ECZUrOWRNGt+TfMmcUxfDaw1kl/QqA\\nIYtv0xBszOC9ECSAak9pUuhabn616jI0teZE1+4trXc1ZLwBKiF+Ev2F7wFLdoPK\\nTyikms8lzQ/GLj9c61NieXsDRJLU3ZMjhQkcfbKKp5R4fp6UPUuJ8MdDzNDiLyuB\\nkgyRLRmF7NBFGvdEV9javdvAuSQz/Wa2ReGRWhI4hj8OgYZaFrNdWbgIRSZUDy3f\\ngDt8NoS7ouoD8vGQ93OvvaUfqgMhs57qZbSZNTgoLKcDFwGyT4oqfA25wzQhzlXU\\nV+q6JLpyz2J+d2CI4B/7/Udh44YT9LiLn9y84QNBaD3qJWrLuYrZYk9hb5HIuJf7\\nXDSh7zHmSed1BrMn/BnzKpxwl201P/Oy/5cCAwEAAaAAMA0GCSqGSIb3DQEBCwUA\\nA4ICAQBhFLpWtZXDhpnmfRu1tkfVQEuGMhG0MdIC1NXAUMs0EoBXKA0/Ak3xCN9c\\nbbvZtBBOktqIV5bxV+d4X0RHLGYh2NGezWHS5gkA92xzmw9gyUKHRQpDmIV9IicH\\nUCSFwL5xO5DFUC2XXshHFgHqpi3kYxCvfcX+wvrmJwjxsbv7raMVsZlN/6w2Z4/p\\nS1VYakUZnXXBl8x3TbK4yZ+mvRX6RjQOU8oCvZMAGns/IgjTL++4O59XThV7VaAa\\nIromeDGnS9klR/hx7BBy7UureQT/aIBpFczjaU5qPBJxkrFi7K+f3vdms9PZOfKv\\n4eowQhanJwoxhkSE11sVmrQQ9+pmrTXHLgO8IFRWAonnM0La31WQ+uBD9vF8iAGS\\neMk+CvEGV6u2UwZph4P6t/KCCGT89BMnJHTRYmyMulx3Ko4jJDMV9v3nm/Ls75ti\\abcdefghijklmnophPfch6I7wJEp+t7egdgrd5jbj4m4lDOT9v9msknlOXoUu\\nibGzaylvac6xhtggDVdz/OIQS7l0jNZE0t0w5ZgEP2fkUlCP6ZpBBL7hloBIzv28...REDACTED\\n-----END

NEW CERTIFICATE REQUEST-----', cacn='ipa', principal='chris', 
version='2.233'): NetworkError
=====

-Chris



On 6/15/21 5:09 PM, Chris Moody via FreeIPA-users wrote:
>  Apologies for the belated response - took me a bit to verify across 
>  all clients.
>
>  When I installed the LE certs on each replica/server, I performed the 
>  following:
>  =====(the privkey & fullchain files provided by LE)=====
>  ipa-server-certinstall -w -d privkey.pem fullchain.pem
>  &
>  /usr/sbin/ipa-certupdate
>  =====
>
>  I have verified via 'openssl s_client -connect' that both https & 
>  ldaps are serving the proper LE certificates across all IPA servers.
>
>  I have also now iterated across every ipa-client installation and run 
>  'ipa-certupdate' as well.  All the /etc/ssl/certs/ca-certificates.crt 
>  files on each and every system have the LE certs and are current.
>
>  Unfortunately, the 'Actions->New Certificate' process I mentioned is 
>  still giving me identical behavior.
>
>  I followed the exact steps in the NewCert dialogue from one of the 
>  validated-current ipa-clients:
>  =====
>  # Create a certificate database or use an existing one. To create a new  
>  database:
>  |# certutil -N -d <database path>|
>  # Create a CSR with subject /CN=<uid>,O=<realm>/, for example:
>  |# certutil -R -d <database path> -a -g <key size> -s 
>  'CN=chris,O=IPA.REDACTED.COM'|
>
>
>  =====
>  IPA Error 907: NetworkError
>  cannot connect to 
>  'https://REDACTED-1.ipa.REDACTED.com:443/ca/rest/account/login': [SSL: 
>  TLSV1_ALERT_UNKNOWN_CA] tlsv1 alert unknown ca (_ssl.c:2508)
>  =====
>
>  -Chris
>
>
>  On 6/12/21 3:55 AM, Florence Renaud wrote:
> > Hi,
> >
> > when the let's encrypt certificates were installed, did you run 
> > ipa-cacert-manage install on one of the nodes + ipa-certupdate on 
> > _all the IPA machines_? It's important to run ipa-certupdate on all 
> > the server/replicas/clients in order to install the CA everywhere.
> >
> > flo
> >
> > On Sat, Jun 12, 2021 at 2:19 AM Chris Moody via FreeIPA-users 
> > <freeipa-users(a)lists.fedorahosted.org 
> > <mailto:[e-mail]>> wrote:
> >
> >     Hello folks.
> >
> >     Hopefully I'm just missing something face-palm level obvious, but
> >     I am running into some trouble when interfacing with my CA
> >     functionality on an IPA server cluster.  My attempts at scouring
> >     all my saved prior-comms from the mailing-list as well as several
> >     search-engines are not enchanting me with much clue.
> >
> >     It appears that my need for the LetsEncrypt certs for the
> >     user-facing Web-UI and LDAPs components are causing IPA to
> >     dis-trust itself.
> >
> >     ===
> >     4-node cluster - Ubuntu19.10
> >     (all nodes currently fully updated/patched via the official
> >     Ubuntu repos)
> >     ===
> >     ipa --version
> >     VERSION: 4.8.1, API_VERSION: 2.233
> >     ===
> >     running letsencrypt certificates successfully for HTTPs & LDAPs
> >     connectivity
> >     ===
> >
> >     These 4-nodes are all happily running and replicating betwixt
> >     each other.  LDAPs is functioning great and many linux systems
> >     are able to all join as freeipa-clients. Users and groups are
> >     replicating and being used elegantly for many LDAP-based
> >     authentication/authorization needs.
> >
> >     Overall, for these nodes, life is good.
> >
> >
> >     Where I'm running into trouble is in finally wanting to leverage
> >     certificate issuance on a per-user basis.  End goal is
> >     integrating things like yubikeys, user-cert auth, and so on.
> >
> >
> >     In the UI, when I enter a user's account and select Actions->New
> >     Certificate, I am able to successfully issue the couple prompted
> >     'certutil' commands to generate the user's CSR.  I then paste 
in
> >     the contents of the CSR and hit 'Issue' and run into
> the
> >     following error:
> >     ==========
> >     IPA Error 907: NetworkError
> >     cannot connect to
> >     'https://REDACTED-1.ipa.REDACTED.com:443/ca/rest/account/login
> >     <https://REDACTED-1.ipa.REDACTED.com:443/ca/rest/account/login>';:
> >     [SSL: TLSV1_ALERT_UNKNOWN_CA] tlsv1 alert unknown ca (_ssl.c:2508)
> >     ==========
> >
> >     As I then start digging into cli-mode to attempt to understand
> >     where things are unhappy, I run into similar troubles with the
> >     server attempting to talk to itself and not being very happy
> >     about it.
> >
> >     ==========
> >     chris@REDACTED-1:~$ ipa ca-find
> >     ------------
> >     1 CA matched
> >     ------------
> >       Name: ipa
> >       Description: IPA CA
> >       Authority ID: 8acca54b-64d7-44bf-b8f7-59316213cfb6
> >       Subject DN: CN=Certificate Authority,O=IPA.REDACTED.COM
> >     <http://IPA.REDACTED.COM>
> >       Issuer DN: CN=Certificate Authority,O=IPA.REDACTED.COM
> >     <http://IPA.REDACTED.COM>
> >     ----------------------------
> >     Number of entries returned 1
> >     ----------------------------
> >     chris@REDACTED-1:~$ ipa ca-show
> >     Name: ipa
> >     ipa: ERROR: cannot connect to
> >     'https://REDACTED-1.ipa.REDACTED.com:443/ca/rest/account/login
> >     <https://REDACTED-1.ipa.REDACTED.com:443/ca/rest/account/login>';:
> >     [SSL: TLSV1_ALERT_UNKNOWN_CA] tlsv1 alert unknown ca (_ssl.c:2508)
> >     ==========
> >
> >     Verifying with 'openssl s_client' returns the valid and
> >     non-expired LE cert-chain.
> >
> >     ==========
> >     chris@REDACTED-1:~$ openssl s_client
> >     REDACTED-1.ipa.REDACTED.com:443
> >     <http://REDACTED-1.ipa.REDACTED.com:443>
> >     CONNECTED(00000003)
> >     depth=2 O = Digital Signature Trust Co., CN = DST Root CA X3
> >     verify return:1
> >     depth=1 C = US, O = Let's Encrypt, CN = R3
> >     verify return:1
> >     depth=0 CN = REDACTED-1.ipa.REDACTED.com
> >     <http://REDACTED-1.ipa.REDACTED.com>
> >     verify return:1
> >     ---
> >     Certificate chain
> >      0 s:CN = REDACTED-1.ipa.REDACTED.com
> >     <http://REDACTED-1.ipa.REDACTED.com>
> >        i:C = US, O = Let's Encrypt, CN = R3
> >      1 s:C = US, O = Let's Encrypt, CN = R3
> >        i:O = Digital Signature Trust Co., CN = DST Root CA X3
> >     ---
> >     ...<output-truncated>...
> >     ---
> >     SSL handshake has read 3046 bytes and written 413 bytes
> >     Verification: OK
> >     ---
> >     New, TLSv1.3, Cipher is TLS_AES_256_GCM_SHA384
> >     Server public key is 2048 bit
> >     Secure Renegotiation IS NOT supported
> >     Compression: NONE
> >     Expansion: NONE
> >     No ALPN negotiated
> >     Early data was not sent
> >     Verify return code: 0 (ok)
> >     ...<output-truncated>...
> >     ==========
> >
> >     Can anyone please hit me with some clue-bat as to where I can
> >     read to understand how to get IPA to love itself?  I'm suspecting
> >     it's likely some certificate inclusion/exception that I need to
> >     add so that API calls and the ipa command itself will actually
> >     respect the LE cert-chain?
> >
> >     Any hints would be greatly appreciated.
> >
> >     Thanks,
> >     -Chris
> >
> >     -- 
> >     REDACTED, Inc.
> >     chris(a)REDACTED.com  <mailto:[e-mail]>
> >     619.354.6463
> >
> >     _______________________________________________
> >     FreeIPA-users mailing list --
> >     freeipa-users(a)lists.fedorahosted.org
> >     <mailto:[e-mail]>
> >     To unsubscribe send an email to
> >     freeipa-users-leave(a)lists.fedorahosted.org
> >     <mailto:[e-mail]>
> >     Fedora Code of Conduct:
> >     https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> >     <https://docs.fedoraproject.org/en-US/project/code-of-conduct/>
> >     List Guidelines:
> >     https://fedoraproject.org/wiki/Mailing_list_guidelines
> >     <https://fedoraproject.org/wiki/Mailing_list_guidelines>
> >     List Archives:
> >    
> https://lists.fedorahosted.org/archives/list/[e-mail]...
> >    
> <https://lists.fedorahosted.org/archives/list/[e-mail]...
> >     Do not reply to spam on the list, report it:
> >     https://pagure.io/fedora-infrastructure
> >     <https://pagure.io/fedora-infrastructure>
> >
>
>  -- 
>  REDACTED, Inc.
>  chris(a)REDACTED.com
>  619.354.6463
>
>  _______________________________________________
>  FreeIPA-users mailing list -- freeipa-users(a)lists.fedorahosted.org
>  To unsubscribe send an email to freeipa-users-leave(a)lists.fedorahosted.org
>  Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
>  List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
>  List Archives:
> https://lists.fedorahosted.org/archives/list/[e-mail]...
>  Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
-- 
REDACTED, Inc.
chris(a)REDACTED.com
619.354.6463

[Rob Crittenden]

The error suggests that your IPA server doesn't trust its own CA
certificate.

Does ipa-cacert-manage list include the IPA CA?

BTW the new certificate steps are unrelated. This affects all CA requests.

rob

Chris Moody via FreeIPA-users wrote:
>  Just found some additional possible clues in the apache error.log
>  
>  =====
>  [Tue Jun 15 17:11:34.636290 2021] [:warn] [pid 31831:tid
>  139703600768768] [client 2001:470:8af9:255::10:47920] failed to set
>  perms (3140) on file (/run/ipa/ccaches/chris(a)IPA.NODE-NINE.COM)!,
>  referer: https://REDACTED-1.ipa.REDACTED.com/ipa/ui/
>  [Tue Jun 15 17:11:34.674975 2021] [ssl:error] [pid 31830:tid
>  139703550412544] [client 2604:a880:2:d1::36:4001:58500] AH02039:
>  Certificate Verification: Error (19): self signed certificate in
>  certificate chain
>  [Tue Jun 15 17:11:34.675088 2021] [ssl:error] [pid 31830:tid
>  139703550412544] [client 2604:a880:2:d1::36:4001:58500] AH02261:
>  Re-negotiation handshake failed
>  [Tue Jun 15 17:11:34.675111 2021] [ssl:error] [pid 31830:tid
>  139703550412544] SSL Library Error: error:1417C086:SSL
>  routines:tls_process_client_certificate:certificate verify failed
>  [Tue Jun 15 17:11:34.675884 2021] [wsgi:error] [pid 31828:tid
>  139703722125056] [remote 2001:470:8af9:255::10:47920] ipa: INFO:
>  [jsonserver_session] chris(a)IPA.REDACTED.COM: cert_request('-----BEGIN
>  NEW CERTIFICATE
>
> REQUEST-----\\nMIIEcTCCAlkCAQAwLDEaMBgGA1UEChMRSVBBLk5PREUtTklORS5DT00xDjAMBgNV\\nBAMTBWNocmlzMIICIjANBgkqhkiG9w0BAQEFAAOCAg8AMIICCgKCAgEA8QR2JbFR\\nE7S7/XAk9V1bNEvvXxB65MpLnIu6gLnfXr8yz0xcTjvEDmKAukS7u+GhwnVfQvwS\\nRjlexS5leIZfMFU59R5K6ubzM5e3Qy07xQU40+8jLS2o3SUo7CvY+TcgTmfJWiK4\\nlWlG70LcMy6VbBVf+nQNoenryPDK2Y94kG3ydcKjGLZsaJ69s1OaSXxZY4esEhk/\\nFJs+odjMQRguS5sbUmfpSy3FTJsvuy4Sge2X5HsHOCKM2bWMcDqkRGI428toET0i\\n+FIXlvroqJQ7OL/RmwVcYOFMwMWN5Ne3g0ECZUrOWRNGt+TfMmcUxfDaw1kl/QqA\\nIYtv0xBszOC9ECSAak9pUuhabn616jI0teZE1+4trXc1ZLwBKiF+Ev2F7wFLdoPK\\nTyikms8lzQ/GLj9c61NieXsDRJLU3ZMjhQkcfbKKp5R4fp6UPUuJ8MdDzNDiLyuB\\nkgyRLRmF7NBFGvdEV9javdvAuSQz/Wa2ReGRWhI4hj8OgYZaFrNdWbgIRSZUDy3f\\ngDt8NoS7ouoD8vGQ93OvvaUfqgMhs57qZbSZNTgoLKcDFwGyT4oqfA25wzQhzlXU\\nV+q6JLpyz2J+d2CI4B/7/Udh44YT9LiLn9y84QNBaD3qJWrLuYrZYk9hb5HIuJf7\\nXDSh7zHmSed1BrMn/BnzKpxwl201P/Oy/5cCAwEAAaAAMA0GCSqGSIb3DQEBCwUA\\nA4ICAQBhFLpWtZXDhpnmfRu1tkfVQEuGMhG0MdIC1NXAUMs0EoBXKA0/Ak3xCN9c\\nbbvZtBBOktqIV5bxV+d4X0RHLGYh2NGezWHS5gkA92xzmw9gyUKHRQpDmIV9IicH\\nUCSFwL5xO5DFUC2XXshHFgHqpi3kYxCvfcX+wvrmJwjxsbv7raMVsZlN/6w2Z4/p\\nS1VYakUZnXXBl8x3TbK4yZ+mvRX6RjQOU8oCvZMAGns/IgjTL++4O59XThV7VaAa\\nIromeDGnS9klR/hx7BBy7UureQT/aIBpFczjaU5qPBJxkrFi7K+f3vdms9PZOfKv\\n4eowQhanJwoxhkSE11sVmrQQ9+pmrTXHLgO8IFRWAonnM0La31WQ+uBD9vF8iAGS\\neMk+CvEGV6u2UwZph4P6t/KCCGT89BMnJHTRYmyMulx3Ko4jJDMV9v3nm/Ls75ti\\abcdefghijklmnophPfch6I7wJEp+t7egdgrd5jbj4m4lDOT9v9msknlOXoUu\\nibGzaylvac6xhtggDVdz/OIQS7l0jNZE0t0w5ZgEP2fkUlCP6ZpBBL7hloBIzv28...REDACTED\\n-----END
>  NEW CERTIFICATE REQUEST-----', cacn='ipa', principal='chris',
>  version='2.233'): NetworkError
>  =====
>  
>  -Chris
>  
>  
>  
>  On 6/15/21 5:09 PM, Chris Moody via FreeIPA-users wrote:
> > Apologies for the belated response - took me a bit to verify across
> > all clients.
> >
> > When I installed the LE certs on each replica/server, I performed the
> > following:
> > =====(the privkey & fullchain files provided by LE)=====
> > ipa-server-certinstall -w -d privkey.pem fullchain.pem
> > &
> > /usr/sbin/ipa-certupdate
> > =====
> >
> > I have verified via 'openssl s_client -connect' that both https &
> > ldaps are serving the proper LE certificates across all IPA servers.
> >
> > I have also now iterated across every ipa-client installation and run
> > 'ipa-certupdate' as well.  All the /etc/ssl/certs/ca-certificates.crt
> > files on each and every system have the LE certs and are current.
> >
> > Unfortunately, the 'Actions->New Certificate' process I mentioned is
> > still giving me identical behavior.
> >
> > I followed the exact steps in the NewCert dialogue from one of the
> > validated-current ipa-clients:
> > =====
> > # Create a certificate database or use an existing one. To create a new
> > database:
> > |# certutil -N -d <database path>|
> > # Create a CSR with subject /CN=<uid>,O=<realm>/, for example:
> > |# certutil -R -d <database path> -a -g <key size> -s
> > 'CN=chris,O=IPA.REDACTED.COM'|
> >
> >
> > =====
> > IPA Error 907: NetworkError
> > cannot connect to
> > 'https://REDACTED-1.ipa.REDACTED.com:443/ca/rest/account/login': [SSL:
> > TLSV1_ALERT_UNKNOWN_CA] tlsv1 alert unknown ca (_ssl.c:2508)
> > =====
> >
> > -Chris
> >
> >
> > On 6/12/21 3:55 AM, Florence Renaud wrote:
> >> Hi,
> >>
> >> when the let's encrypt certificates were installed, did you run
> >> ipa-cacert-manage install on one of the nodes + ipa-certupdate on
> >> _all the IPA machines_? It's important to run ipa-certupdate on all
> >> the server/replicas/clients in order to install the CA everywhere.
> >>
> >> flo
> >>
> >> On Sat, Jun 12, 2021 at 2:19 AM Chris Moody via FreeIPA-users
> >> <freeipa-users(a)lists.fedorahosted.org
> >> <mailto:[e-mail]>> wrote:
> >>
> >>     Hello folks.
> >>
> >>     Hopefully I'm just missing something face-palm level obvious, but
> >>     I am running into some trouble when interfacing with my CA
> >>     functionality on an IPA server cluster.  My attempts at scouring
> >>     all my saved prior-comms from the mailing-list as well as several
> >>     search-engines are not enchanting me with much clue.
> >>
> >>     It appears that my need for the LetsEncrypt certs for the
> >>     user-facing Web-UI and LDAPs components are causing IPA to
> >>     dis-trust itself.
> >>
> >>     ===
> >>     4-node cluster - Ubuntu19.10
> >>     (all nodes currently fully updated/patched via the official
> >>     Ubuntu repos)
> >>     ===
> >>     ipa --version
> >>     VERSION: 4.8.1, API_VERSION: 2.233
> >>     ===
> >>     running letsencrypt certificates successfully for HTTPs & LDAPs
> >>     connectivity
> >>     ===
> >>
> >>     These 4-nodes are all happily running and replicating betwixt
> >>     each other.  LDAPs is functioning great and many linux systems
> >>     are able to all join as freeipa-clients.  Users and groups are
> >>     replicating and being used elegantly for many LDAP-based
> >>     authentication/authorization needs.
> >>
> >>     Overall, for these nodes, life is good.
> >>
> >>
> >>     Where I'm running into trouble is in finally wanting to leverage
> >>     certificate issuance on a per-user basis.  End goal is
> >>     integrating things like yubikeys, user-cert auth, and so on.
> >>
> >>
> >>     In the UI, when I enter a user's account and select Actions->New
> >>     Certificate, I am able to successfully issue the couple prompted
> >>     'certutil' commands to generate the user's CSR.  I then paste
> in
> >>     the contents of the CSR and hit 'Issue' and run into the
> >>     following error:
> >>     ==========
> >>     IPA Error 907: NetworkError
> >>     cannot connect to
> >>     'https://REDACTED-1.ipa.REDACTED.com:443/ca/rest/account/login':
> >>     [SSL: TLSV1_ALERT_UNKNOWN_CA] tlsv1 alert unknown ca (_ssl.c:2508)
> >>     ==========
> >>
> >>     As I then start digging into cli-mode to attempt to understand
> >>     where things are unhappy, I run into similar troubles with the
> >>     server attempting to talk to itself and not being very happy
> >>     about it.
> >>
> >>     ==========
> >>     chris@REDACTED-1:~$ ipa ca-find
> >>     ------------
> >>     1 CA matched
> >>     ------------
> >>       Name: ipa
> >>       Description: IPA CA
> >>       Authority ID: 8acca54b-64d7-44bf-b8f7-59316213cfb6
> >>       Subject DN: CN=Certificate Authority,O=IPA.REDACTED.COM
> >>     <http://IPA.REDACTED.COM>
> >>       Issuer DN: CN=Certificate Authority,O=IPA.REDACTED.COM
> >>     <http://IPA.REDACTED.COM>
> >>     ----------------------------
> >>     Number of entries returned 1
> >>     ----------------------------
> >>     chris@REDACTED-1:~$ ipa ca-show
> >>     Name: ipa
> >>     ipa: ERROR: cannot connect to
> >>     'https://REDACTED-1.ipa.REDACTED.com:443/ca/rest/account/login':
> >>     [SSL: TLSV1_ALERT_UNKNOWN_CA] tlsv1 alert unknown ca (_ssl.c:2508)
> >>     ==========
> >>
> >>     Verifying with 'openssl s_client' returns the valid and
> >>     non-expired LE cert-chain.
> >>
> >>     ==========
> >>     chris@REDACTED-1:~$ openssl s_client
> >>     REDACTED-1.ipa.REDACTED.com:443
> >>     <http://REDACTED-1.ipa.REDACTED.com:443>
> >>     CONNECTED(00000003)
> >>     depth=2 O = Digital Signature Trust Co., CN = DST Root CA X3
> >>     verify return:1
> >>     depth=1 C = US, O = Let's Encrypt, CN = R3
> >>     verify return:1
> >>     depth=0 CN = REDACTED-1.ipa.REDACTED.com
> >>     <http://REDACTED-1.ipa.REDACTED.com>
> >>     verify return:1
> >>     ---
> >>     Certificate chain
> >>      0 s:CN = REDACTED-1.ipa.REDACTED.com
> >>     <http://REDACTED-1.ipa.REDACTED.com>
> >>        i:C = US, O = Let's Encrypt, CN = R3
> >>      1 s:C = US, O = Let's Encrypt, CN = R3
> >>        i:O = Digital Signature Trust Co., CN = DST Root CA X3
> >>     ---
> >>     ...<output-truncated>...
> >>     ---
> >>     SSL handshake has read 3046 bytes and written 413 bytes
> >>     Verification: OK
> >>     ---
> >>     New, TLSv1.3, Cipher is TLS_AES_256_GCM_SHA384
> >>     Server public key is 2048 bit
> >>     Secure Renegotiation IS NOT supported
> >>     Compression: NONE
> >>     Expansion: NONE
> >>     No ALPN negotiated
> >>     Early data was not sent
> >>     Verify return code: 0 (ok)
> >>     ...<output-truncated>...
> >>     ==========
> >>
> >>     Can anyone please hit me with some clue-bat as to where I can
> >>     read to understand how to get IPA to love itself?  I'm suspecting
> >>     it's likely some certificate inclusion/exception that I need to
> >>     add so that API calls and the ipa command itself will actually
> >>     respect the LE cert-chain?
> >>
> >>     Any hints would be greatly appreciated.
> >>
> >>     Thanks,
> >>     -Chris
> >>
> >>     -- 
> >>     REDACTED, Inc.
> >>     chris(a)REDACTED.com <mailto:[e-mail]>
> >>     619.354.6463
> >>
> >>     _______________________________________________
> >>     FreeIPA-users mailing list --
> >>     freeipa-users(a)lists.fedorahosted.org
> >>     <mailto:[e-mail]>
> >>     To unsubscribe send an email to
> >>     freeipa-users-leave(a)lists.fedorahosted.org
> >>     <mailto:[e-mail]>
> >>     Fedora Code of Conduct:
> >>     https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> >>     List Guidelines:
> >>     https://fedoraproject.org/wiki/Mailing_list_guidelines
> >>     List Archives:
> >>    
> https://lists.fedorahosted.org/archives/list/[e-mail]...
> >>     Do not reply to spam on the list, report it:
> >>     https://pagure.io/fedora-infrastructure
> >>
> >
> > -- 
> > REDACTED, Inc.
> > chris(a)REDACTED.com
> > 619.354.6463
> >
> > _______________________________________________
> > FreeIPA-users mailing list -- freeipa-users(a)lists.fedorahosted.org
> > To unsubscribe send an email to freeipa-users-leave(a)lists.fedorahosted.org
> > Fedora Code of Conduct:
> https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> > List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> > List Archives:
> https://lists.fedorahosted.org/archives/list/[e-mail]...
> > Do not reply to spam on the list, report it:
> https://pagure.io/fedora-infrastructure
>  
>  -- 
>  REDACTED, Inc.
>  chris(a)REDACTED.com
>  619.354.6463
>  
>  
>  _______________________________________________
>  FreeIPA-users mailing list -- freeipa-users(a)lists.fedorahosted.org
>  To unsubscribe send an email to freeipa-users-leave(a)lists.fedorahosted.org
>  Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
>  List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
>  List Archives:
> https://lists.fedorahosted.org/archives/list/[e-mail]...
>  Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
>  
>

Did you install the LE Root CA's first?

CERTS=("isrgrootx1.pem" "isrg-root-x2.pem"
"lets-encrypt-r3.pem" 
"lets-encrypt-e1.pem" "lets-encrypt-r4.pem"
"lets-encrypt-e2.pem")

for CERT in "${CERTS[@]}"
do
   wget -O "/etc/ssl/$CERT" "https://letsencrypt.org/certs/$CERT"
   ipa-cacert-manage install "/etc/ssl/$CERT"
done

I had to do the above before I was able to import my LE certs.

On 2021-06-16 16:26, Rob Crittenden via FreeIPA-users wrote:
>  The error suggests that your IPA server doesn't trust its own CA
>  certificate.
>  
>  Does ipa-cacert-manage list include the IPA CA?
>  
>  BTW the new certificate steps are unrelated. This affects all CA 
>  requests.
>  
>  rob
>  
>  Chris Moody via FreeIPA-users wrote:
> > Just found some additional possible clues in the apache error.log
> > 
> > =====
> > [Tue Jun 15 17:11:34.636290 2021] [:warn] [pid 31831:tid
> > 139703600768768] [client 2001:470:8af9:255::10:47920] failed to set
> > perms (3140) on file (/run/ipa/ccaches/chris(a)IPA.NODE-NINE.COM)!,
> > referer: https://REDACTED-1.ipa.REDACTED.com/ipa/ui/
> > [Tue Jun 15 17:11:34.674975 2021] [ssl:error] [pid 31830:tid
> > 139703550412544] [client 2604:a880:2:d1::36:4001:58500] AH02039:
> > Certificate Verification: Error (19): self signed certificate in
> > certificate chain
> > [Tue Jun 15 17:11:34.675088 2021] [ssl:error] [pid 31830:tid
> > 139703550412544] [client 2604:a880:2:d1::36:4001:58500] AH02261:
> > Re-negotiation handshake failed
> > [Tue Jun 15 17:11:34.675111 2021] [ssl:error] [pid 31830:tid
> > 139703550412544] SSL Library Error: error:1417C086:SSL
> > routines:tls_process_client_certificate:certificate verify failed
> > [Tue Jun 15 17:11:34.675884 2021] [wsgi:error] [pid 31828:tid
> > 139703722125056] [remote 2001:470:8af9:255::10:47920] ipa: INFO:
> > [jsonserver_session] chris(a)IPA.REDACTED.COM: cert_request('-----BEGIN
> > NEW CERTIFICATE
> >
REQUEST-----\\nMIIEcTCCAlkCAQAwLDEaMBgGA1UEChMRSVBBLk5PREUtTklORS5DT00xDjAMBgNV\\nBAMTBWNocmlzMIICIjANBgkqhkiG9w0BAQEFAAOCAg8AMIICCgKCAgEA8QR2JbFR\\nE7S7/XAk9V1bNEvvXxB65MpLnIu6gLnfXr8yz0xcTjvEDmKAukS7u+GhwnVfQvwS\\nRjlexS5leIZfMFU59R5K6ubzM5e3Qy07xQU40+8jLS2o3SUo7CvY+TcgTmfJWiK4\\nlWlG70LcMy6VbBVf+nQNoenryPDK2Y94kG3ydcKjGLZsaJ69s1OaSXxZY4esEhk/\\nFJs+odjMQRguS5sbUmfpSy3FTJsvuy4Sge2X5HsHOCKM2bWMcDqkRGI428toET0i\\n+FIXlvroqJQ7OL/RmwVcYOFMwMWN5Ne3g0ECZUrOWRNGt+TfMmcUxfDaw1kl/QqA\\nIYtv0xBszOC9ECSAak9pUuhabn616jI0teZE1+4trXc1ZLwBKiF+Ev2F7wFLdoPK\\nTyikms8lzQ/GLj9c61NieXsDRJLU3ZMjhQkcfbKKp5R4fp6UPUuJ8MdDzNDiLyuB\\nkgyRLRmF7NBFGvdEV9javdvAuSQz/Wa2ReGRWhI4hj8OgYZaFrNdWbgIRSZUDy3f\\ngDt8NoS7ouoD8vGQ93OvvaUfqgMhs57qZbSZNTgoLKcDFwGyT4oqfA25wzQhzlXU\\nV+q6JLpyz2J+d2CI4B/7/Udh44YT9LiLn9y84QNBaD3qJWrLuYrZYk9hb5HIuJf7\\nXDSh7zHmSed1BrMn/BnzKpxwl201P/Oy/5cCAwEAAaAAMA0GCSqGSIb3DQEBCwUA\\nA4ICAQBhFLpWtZXDhpnmfRu1tkfVQEuGMhG0MdIC1NXAUMs0EoBXKA0/Ak3xCN9c\\nbbvZtBBOktqIV5bxV+d4X0RHLGYh2NGezWHS5gkA92xzm
w9gyUKHRQpDmIV9IicH\\nUCSFwL5xO5DFUC2XXshHFgHqpi3kYxCvfcX+wvrmJwjxsbv7raMVsZlN/6w2Z4/p\\nS1VYakUZnXXBl8x3TbK4yZ+mvRX6RjQOU8oCvZMAGns/IgjTL++4O59XThV7VaAa\\nIromeDGnS9klR/hx7BBy7UureQT/aIBpFczjaU5qPBJxkrFi7K+f3vdms9PZOfKv\\n4eowQhanJwoxhkSE11sVmrQQ9+pmrTXHLgO8IFRWAonnM0La31WQ+uBD9vF8iAGS\\neMk+CvEGV6u2UwZph4P6t/KCCGT89BMnJHTRYmyMulx3Ko4jJDMV9v3nm/Ls75ti\\abcdefghijklmnophPfch6I7wJEp+t7egdgrd5jbj4m4lDOT9v9msknlOXoUu\\nibGzaylvac6xhtggDVdz/OIQS7l0jNZE0t0w5ZgEP2fkUlCP6ZpBBL7hloBIzv28...REDACTED\\n-----END
>> NEW CERTIFICATE REQUEST-----', cacn='ipa',
principal='chris',
>> version='2.233'): NetworkError
>> =====
>> 
>> -Chris
>> 
>> 
>> 
>> On 6/15/21 5:09 PM, Chris Moody via FreeIPA-users wrote:
>>> Apologies for the belated response - took me a bit to verify across
>>> all clients.
>>> 
>>> When I installed the LE certs on each replica/server, I performed the
>>> following:
>>> =====(the privkey & fullchain files provided by LE)=====
>>> ipa-server-certinstall -w -d privkey.pem fullchain.pem
>>> &
>>> /usr/sbin/ipa-certupdate
>>> =====
>>> 
>>> I have verified via 'openssl s_client -connect' that both https
&
>>> ldaps are serving the proper LE certificates across all IPA servers.
>>> 
>>> I have also now iterated across every ipa-client installation and run
>>> 'ipa-certupdate' as well.  All the
/etc/ssl/certs/ca-certificates.crt
>>> files on each and every system have the LE certs and are current.
>>> 
>>> Unfortunately, the 'Actions->New Certificate' process I mentioned
is
>>> still giving me identical behavior.
>>> 
>>> I followed the exact steps in the NewCert dialogue from one of the
>>> validated-current ipa-clients:
>>> =====
>>> # Create a certificate database or use an existing one. To create a 
>>> new
>>> database:
>>> |# certutil -N -d <database path>|
>>> # Create a CSR with subject /CN=<uid>,O=<realm>/, for example:
>>> |# certutil -R -d <database path> -a -g <key size> -s
>>> 'CN=chris,O=IPA.REDACTED.COM'|
>>> 
>>> 
>>> =====
>>> IPA Error 907: NetworkError
>>> cannot connect to
>>> 'https://REDACTED-1.ipa.REDACTED.com:443/ca/rest/account/login': 
>>> [SSL:
>>> TLSV1_ALERT_UNKNOWN_CA] tlsv1 alert unknown ca (_ssl.c:2508)
>>> =====
>>> 
>>> -Chris
>>> 
>>> 
>>> On 6/12/21 3:55 AM, Florence Renaud wrote:
>>>> Hi,
>>>> 
>>>> when the let's encrypt certificates were installed, did you run
>>>> ipa-cacert-manage install on one of the nodes + ipa-certupdate on
>>>> _all the IPA machines_? It's important to run ipa-certupdate on all
>>>> the server/replicas/clients in order to install the CA everywhere.
>>>> 
>>>> flo
>>>> 
>>>> On Sat, Jun 12, 2021 at 2:19 AM Chris Moody via FreeIPA-users
>>>> <freeipa-users(a)lists.fedorahosted.org
>>>> <mailto:[e-mail]>> wrote:
>>>> 
>>>>     Hello folks.
>>>> 
>>>>     Hopefully I'm just missing something face-palm level obvious, 
>>>> but
>>>>     I am running into some trouble when interfacing with my CA
>>>>     functionality on an IPA server cluster.  My attempts at scouring
>>>>     all my saved prior-comms from the mailing-list as well as 
>>>> several
>>>>     search-engines are not enchanting me with much clue.
>>>> 
>>>>     It appears that my need for the LetsEncrypt certs for the
>>>>     user-facing Web-UI and LDAPs components are causing IPA to
>>>>     dis-trust itself.
>>>> 
>>>>     ===
>>>>     4-node cluster - Ubuntu19.10
>>>>     (all nodes currently fully updated/patched via the official
>>>>     Ubuntu repos)
>>>>     ===
>>>>     ipa --version
>>>>     VERSION: 4.8.1, API_VERSION: 2.233
>>>>     ===
>>>>     running letsencrypt certificates successfully for HTTPs & LDAPs
>>>>     connectivity
>>>>     ===
>>>> 
>>>>     These 4-nodes are all happily running and replicating betwixt
>>>>     each other.  LDAPs is functioning great and many linux systems
>>>>     are able to all join as freeipa-clients.  Users and groups are
>>>>     replicating and being used elegantly for many LDAP-based
>>>>     authentication/authorization needs.
>>>> 
>>>>     Overall, for these nodes, life is good.
>>>> 
>>>> 
>>>>     Where I'm running into trouble is in finally wanting to leverage
>>>>     certificate issuance on a per-user basis.  End goal is
>>>>     integrating things like yubikeys, user-cert auth, and so on.
>>>> 
>>>> 
>>>>     In the UI, when I enter a user's account and select
Actions->New
>>>>     Certificate, I am able to successfully issue the couple prompted
>>>>     'certutil' commands to generate the user's CSR.  I then
paste in
>>>>     the contents of the CSR and hit 'Issue' and run into the
>>>>     following error:
>>>>     ==========
>>>>     IPA Error 907: NetworkError
>>>>     cannot connect to
>>>>    
'https://REDACTED-1.ipa.REDACTED.com:443/ca/rest/account/login':
>>>>     [SSL: TLSV1_ALERT_UNKNOWN_CA] tlsv1 alert unknown ca 
>>>> (_ssl.c:2508)
>>>>     ==========
>>>> 
>>>>     As I then start digging into cli-mode to attempt to understand
>>>>     where things are unhappy, I run into similar troubles with the
>>>>     server attempting to talk to itself and not being very happy
>>>>     about it.
>>>> 
>>>>     ==========
>>>>     chris@REDACTED-1:~$ ipa ca-find
>>>>     ------------
>>>>     1 CA matched
>>>>     ------------
>>>>       Name: ipa
>>>>       Description: IPA CA
>>>>       Authority ID: 8acca54b-64d7-44bf-b8f7-59316213cfb6
>>>>       Subject DN: CN=Certificate Authority,O=IPA.REDACTED.COM
>>>>     <http://IPA.REDACTED.COM>
>>>>       Issuer DN: CN=Certificate Authority,O=IPA.REDACTED.COM
>>>>     <http://IPA.REDACTED.COM>
>>>>     ----------------------------
>>>>     Number of entries returned 1
>>>>     ----------------------------
>>>>     chris@REDACTED-1:~$ ipa ca-show
>>>>     Name: ipa
>>>>     ipa: ERROR: cannot connect to
>>>>    
'https://REDACTED-1.ipa.REDACTED.com:443/ca/rest/account/login':
>>>>     [SSL: TLSV1_ALERT_UNKNOWN_CA] tlsv1 alert unknown ca 
>>>> (_ssl.c:2508)
>>>>     ==========
>>>> 
>>>>     Verifying with 'openssl s_client' returns the valid and
>>>>     non-expired LE cert-chain.
>>>> 
>>>>     ==========
>>>>     chris@REDACTED-1:~$ openssl s_client
>>>>     REDACTED-1.ipa.REDACTED.com:443
>>>>     <http://REDACTED-1.ipa.REDACTED.com:443>
>>>>     CONNECTED(00000003)
>>>>     depth=2 O = Digital Signature Trust Co., CN = DST Root CA X3
>>>>     verify return:1
>>>>     depth=1 C = US, O = Let's Encrypt, CN = R3
>>>>     verify return:1
>>>>     depth=0 CN = REDACTED-1.ipa.REDACTED.com
>>>>     <http://REDACTED-1.ipa.REDACTED.com>
>>>>     verify return:1
>>>>     ---
>>>>     Certificate chain
>>>>      0 s:CN = REDACTED-1.ipa.REDACTED.com
>>>>     <http://REDACTED-1.ipa.REDACTED.com>
>>>>        i:C = US, O = Let's Encrypt, CN = R3
>>>>      1 s:C = US, O = Let's Encrypt, CN = R3
>>>>        i:O = Digital Signature Trust Co., CN = DST Root CA X3
>>>>     ---
>>>>     ...<output-truncated>...
>>>>     ---
>>>>     SSL handshake has read 3046 bytes and written 413 bytes
>>>>     Verification: OK
>>>>     ---
>>>>     New, TLSv1.3, Cipher is TLS_AES_256_GCM_SHA384
>>>>     Server public key is 2048 bit
>>>>     Secure Renegotiation IS NOT supported
>>>>     Compression: NONE
>>>>     Expansion: NONE
>>>>     No ALPN negotiated
>>>>     Early data was not sent
>>>>     Verify return code: 0 (ok)
>>>>     ...<output-truncated>...
>>>>     ==========
>>>> 
>>>>     Can anyone please hit me with some clue-bat as to where I can
>>>>     read to understand how to get IPA to love itself?  I'm 
>>>> suspecting
>>>>     it's likely some certificate inclusion/exception that I need to
>>>>     add so that API calls and the ipa command itself will actually
>>>>     respect the LE cert-chain?
>>>> 
>>>>     Any hints would be greatly appreciated.
>>>> 
>>>>     Thanks,
>>>>     -Chris
>>>> 
>>>>     --
>>>>     REDACTED, Inc.
>>>>     chris(a)REDACTED.com <mailto:[e-mail]>
>>>>     619.354.6463
>>>> 
>>>>     _______________________________________________
>>>>     FreeIPA-users mailing list --
>>>>     freeipa-users(a)lists.fedorahosted.org
>>>>     <mailto:[e-mail]>
>>>>     To unsubscribe send an email to
>>>>     freeipa-users-leave(a)lists.fedorahosted.org
>>>>     <mailto:[e-mail]>
>>>>     Fedora Code of Conduct:
>>>>     https://docs.fedoraproject.org/en-US/project/code-of-conduct/
>>>>     List Guidelines:
>>>>     https://fedoraproject.org/wiki/Mailing_list_guidelines
>>>>     List Archives:
>>>>     
>>>>
https://lists.fedorahosted.org/archives/list/[e-mail]...
>>>>     Do not reply to spam on the list, report it:
>>>>     https://pagure.io/fedora-infrastructure
>>>> 
>>> 
>>> --
>>> REDACTED, Inc.
>>> chris(a)REDACTED.com
>>> 619.354.6463
>>> 
>>> _______________________________________________
>>> FreeIPA-users mailing list -- freeipa-users(a)lists.fedorahosted.org
>>> To unsubscribe send an email to 
>>> freeipa-users-leave(a)lists.fedorahosted.org
>>> Fedora Code of Conduct: 
>>> https://docs.fedoraproject.org/en-US/project/code-of-conduct/
>>> List Guidelines: 
>>> https://fedoraproject.org/wiki/Mailing_list_guidelines
>>> List Archives: 
>>>
https://lists.fedorahosted.org/archives/list/[e-mail]...
>>> Do not reply to spam on the list, report it: 
>>> https://pagure.io/fedora-infrastructure
>> 
>> --
>> REDACTED, Inc.
>> chris(a)REDACTED.com
>> 619.354.6463
>> 
>> 
>> _______________________________________________
>> FreeIPA-users mailing list -- freeipa-users(a)lists.fedorahosted.org
>> To unsubscribe send an email to 
>> freeipa-users-leave(a)lists.fedorahosted.org
>> Fedora Code of Conduct: 
>> https://docs.fedoraproject.org/en-US/project/code-of-conduct/
>> List Guidelines: 
>> https://fedoraproject.org/wiki/Mailing_list_guidelines
>> List Archives: 
>>
https://lists.fedorahosted.org/archives/list/[e-mail]...
>> Do not reply to spam on the list, report it: 
>> https://pagure.io/fedora-infrastructure
>> 
> 
> _______________________________________________
> FreeIPA-users mailing list -- freeipa-users(a)lists.fedorahosted.org
> To unsubscribe send an email to 
> freeipa-users-leave(a)lists.fedorahosted.org
> Fedora Code of Conduct:
> https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives:
> https://lists.fedorahosted.org/archives/list/[e-mail]...
> Do not reply to spam on the list, report it:
> https://pagure.io/fedora-infrastructure

[Chris Moody]

That was kinda my belief thus far as well that the hosts were not 
trusting themselves - not 100% sure how things got here though.  I have 
a hunch it might be related to the initial deployment and the prior 
admin using an outdated method to install/manage/renew the LE-certificates.

=====
# ipa-cacert-manage list
IPA.XXXYYY.COM IPA CA
IPA.XXXYYY.COM IPA CA
DSTRootCAX3
letsencryptx3
isrgrootx1
lets-encrypt-r3-cross-signed
The ipa-cacert-manage command was successful
=====

How would I go about forcing re-installation of the host's own CA 
certificate to ensure it's trust?

Also, since these nodes are not running on an RPM-based distro, the 
typical cert-store locations I have seen on other systems are not in the 
same location(s) so I'm not sure totally sure every location to point 
certutil to be able to examine each cert-store in depth as well (if that 
might help diagnose further).  I ask because I believe these were 
initially built and then had the 
"https://github.com/freeipa/freeipa-letsencrypt/" project used to 
initially deploy the LE-certs - prior to the ipa-cacert-manage command 
being the official path toward installing/managing these external 
certificates.  I know because this code had been git-pulled onto these 
nodes, but it obviously doesn't work properly since this git project 
manipulates the paths below directly instead of managing via the 
ipa-cacert-manage command.

ex>
/etc/httpd/alias/ (<=== not on these systems)
/etc/pki/pki-tomcat/alias/
/etc/ipa/nssdb/
/etc/dirsrv/slapd-IPA-XXXYYY-COM/


Checking that project's git page now though, I see their readme now 
mentions /var/lib/ipa/certs/, where I just noticed cacert.pem.

=====
/var/lib/ipa/certs# openssl x509 -text -noout -in cacert.pem
Certificate:
     Data:
         Version: 3 (0x2)
         Serial Number: 1 (0x1)
         Signature Algorithm: sha256WithRSAEncryption
         Issuer: O = IPA.XXXYYY.COM, 
CN = Certificate Authority
         Validity
             Not Before: Apr 16 20:45:46 2020 GMT
             Not After : Apr 16 20:45:46 2040 GMT
         Subject: O = IPA.XXXYYY.COM, CN = Certificate Authority
...
=====
so I believe I might have just located the IPA CA cert in case I need to 
re-install it.



To the following question, I have the following LE-related certs 
installed.  And yes, I did run into issues a couple months back when 
LE 
moved to the new certs on their end so had to import the new authority 
certs to get the LE host certs to update & import.  The LE certificates 
are functioning and verify for both slapd and apache/tomcat.

=====
DSTRootCAX3.pem  LetsEncryptAuthorityX3.pem  isrgrootx1.pem 
lets-encrypt-r3-cross-signed.pem
=====

Thank you all so much for the assistance through all this.

-Chris


On 6/16/21 1:26 PM, Rob Crittenden via FreeIPA-users wrote:
>  The error suggests that your IPA server doesn't trust its own CA
>  certificate.
>
>  Does ipa-cacert-manage list include the IPA CA?
>
>  BTW the new certificate steps are unrelated. This affects all CA requests.
>
>  rob
>
>  Chris Moody via FreeIPA-users wrote:
> > Just found some additional possible clues in the apache error.log
> >
> > =====
> > [Tue Jun 15 17:11:34.636290 2021] [:warn] [pid 31831:tid
> > 139703600768768] [client 2001:470:8af9:255::10:47920] failed to set
> > perms (3140) on file (/run/ipa/ccaches/chris(a)IPA.NODE-NINE.COM)!,
> > referer: https://REDACTED-1.ipa.REDACTED.com/ipa/ui/
> > [Tue Jun 15 17:11:34.674975 2021] [ssl:error] [pid 31830:tid
> > 139703550412544] [client 2604:a880:2:d1::36:4001:58500] AH02039:
> > Certificate Verification: Error (19): self signed certificate in
> > certificate chain
> > [Tue Jun 15 17:11:34.675088 2021] [ssl:error] [pid 31830:tid
> > 139703550412544] [client 2604:a880:2:d1::36:4001:58500] AH02261:
> > Re-negotiation handshake failed
> > [Tue Jun 15 17:11:34.675111 2021] [ssl:error] [pid 31830:tid
> > 139703550412544] SSL Library Error: error:1417C086:SSL
> > routines:tls_process_client_certificate:certificate verify failed
> > [Tue Jun 15 17:11:34.675884 2021] [wsgi:error] [pid 31828:tid
> > 139703722125056] [remote 2001:470:8af9:255::10:47920] ipa: INFO:
> > [jsonserver_session] chris(a)IPA.REDACTED.COM: cert_request('-----BEGIN
> > NEW CERTIFICATE
> >
> REQUEST-----\\nMIIEcTCCAlkCAQAwLDEaMBgGA1UEChMRSVBBLk5PREUtTklORS5DT00xDjAMBgNV\\nBAMTBWNocmlzMIICIjANBgkqhkiG9w0BAQEFAAOCAg8AMIICCgKCAgEA8QR2JbFR\\nE7S7/XAk9V1bNEvvXxB65MpLnIu6gLnfXr8yz0xcTjvEDmKAukS7u+GhwnVfQvwS\\nRjlexS5leIZfMFU59R5K6ubzM5e3Qy07xQU40+8jLS2o3SUo7CvY+TcgTmfJWiK4\\nlWlG70LcMy6VbBVf+nQNoenryPDK2Y94kG3ydcKjGLZsaJ69s1OaSXxZY4esEhk/\\nFJs+odjMQRguS5sbUmfpSy3FTJsvuy4Sge2X5HsHOCKM2bWMcDqkRGI428toET0i\\n+FIXlvroqJQ7OL/RmwVcYOFMwMWN5Ne3g0ECZUrOWRNGt+TfMmcUxfDaw1kl/QqA\\nIYtv0xBszOC9ECSAak9pUuhabn616jI0teZE1+4trXc1ZLwBKiF+Ev2F7wFLdoPK\\nTyikms8lzQ/GLj9c61NieXsDRJLU3ZMjhQkcfbKKp5R4fp6UPUuJ8MdDzNDiLyuB\\nkgyRLRmF7NBFGvdEV9javdvAuSQz/Wa2ReGRWhI4hj8OgYZaFrNdWbgIRSZUDy3f\\ngDt8NoS7ouoD8vGQ93OvvaUfqgMhs57qZbSZNTgoLKcDFwGyT4oqfA25wzQhzlXU\\nV+q6JLpyz2J+d2CI4B/7/Udh44YT9LiLn9y84QNBaD3qJWrLuYrZYk9hb5HIuJf7\\nXDSh7zHmSed1BrMn/BnzKpxwl201P/Oy/5cCAwEAAaAAMA0GCSqGSIb3DQEBCwUA\\nA4ICAQBhFLpWtZXDhpnmfRu1tkfVQEuGMhG0MdIC1NXAUMs0EoBXKA0/Ak3xCN9c\\nbbvZtBBOktqIV5bxV+d4X0RHLGYh2NGezWHS5gkA92xzmw9gyUKHRQpDmIV9IicH\\nUCSFwL5xO5DFUC2XXshHFgHqpi3kYxCvfcX+wvrmJwjxsbv7raMVsZlN/6w2Z4/p\\nS1VYakUZnXXBl8x3TbK4yZ+mvRX6RjQOU8oCvZMAGns/IgjTL++4O59XThV7VaAa\\nIromeDGnS9klR/hx7BBy7UureQT/aIBpFczjaU5qPBJxkrFi7K+f3vdms9PZOfKv\\n4eowQhanJwoxhkSE11sVmrQQ9+pmrTXHLgO8IFRWAonnM0La31WQ+uBD9vF8iAGS\\neMk+CvEGV6u2UwZph4P6t/KCCGT89BMnJHTRYmyMulx3Ko4jJDMV9v3nm/Ls75ti\\abcdefghijklmnophPfch6I7wJEp+t7egdgrd5jbj4m4lDOT9v9msknlOXoUu\\nibGzaylvac6xhtggDVdz/OIQS7l0jNZE0t0w5ZgEP2fkUlCP6ZpBBL7hloBIzv28...REDACTED\\n-----END
> > NEW CERTIFICATE REQUEST-----', cacn='ipa', principal='chris',
> > version='2.233'): NetworkError
> > =====
> >
> > -Chris
> >
> >
> >
> > On 6/15/21 5:09 PM, Chris Moody via FreeIPA-users wrote:
> >> Apologies for the belated response - took me a bit to verify across
> >> all clients.
> >>
> >> When I installed the LE certs on each replica/server, I performed the
> >> following:
> >> =====(the privkey & fullchain files provided by LE)=====
> >> ipa-server-certinstall -w -d privkey.pem fullchain.pem
> >> &
> >> /usr/sbin/ipa-certupdate
> >> =====
> >>
> >> I have verified via 'openssl s_client -connect' that both https &
> >> ldaps are serving the proper LE certificates across all IPA servers.
> >>
> >> I have also now iterated across every ipa-client installation and run
> >> 'ipa-certupdate' as well.  All the /etc/ssl/certs/ca-certificates.crt
> >> files on each and every system have the LE certs and are current.
> >>
> >> Unfortunately, the 'Actions->New Certificate' process I mentioned is
> >> still giving me identical behavior.
> >>
> >> I followed the exact steps in the NewCert dialogue from one of the
> >> validated-current ipa-clients:
> >> =====
> >> # Create a certificate database or use an existing one. To create a new
> >> database:
> >> |# certutil -N -d <database path>|
> >> # Create a CSR with subject /CN=<uid>,O=<realm>/, for example:
> >> |# certutil -R -d <database path> -a -g <key size> -s
> >> 'CN=chris,O=IPA.REDACTED.COM'|
> >>
> >>
> >> =====
> >> IPA Error 907: NetworkError
> >> cannot connect to
> >> 'https://REDACTED-1.ipa.REDACTED.com:443/ca/rest/account/login': [SSL:
> >> TLSV1_ALERT_UNKNOWN_CA] tlsv1 alert unknown ca (_ssl.c:2508)
> >> =====
> >>
> >> -Chris
> >>
> >>
> >> On 6/12/21 3:55 AM, Florence Renaud wrote:
> >>> Hi,
> >>>
> >>> when the let's encrypt certificates were installed, did you run
> >>> ipa-cacert-manage install on one of the nodes + ipa-certupdate on
> >>> _all the IPA machines_? It's important to run ipa-certupdate on all
> >>> the server/replicas/clients in order to install the CA everywhere.
> >>>
> >>> flo
> >>>
> >>> On Sat, Jun 12, 2021 at 2:19 AM Chris Moody via FreeIPA-users
> >>> <freeipa-users(a)lists.fedorahosted.org
> >>> <mailto:[e-mail]>> wrote:
> >>>
> >>>      Hello folks.
> >>>
> >>>      Hopefully I'm just missing something face-palm level obvious, but
> >>>      I am running into some trouble when interfacing with my CA
> >>>      functionality on an IPA server cluster.  My attempts at scouring
> >>>      all my saved prior-comms from the mailing-list as well as several
> >>>      search-engines are not enchanting me with much clue.
> >>>
> >>>      It appears that my need for the LetsEncrypt certs for the
> >>>      user-facing Web-UI and LDAPs components are causing IPA to
> >>>      dis-trust itself.
> >>>
> >>>      ===
> >>>      4-node cluster - Ubuntu19.10
> >>>      (all nodes currently fully updated/patched via the official
> >>>      Ubuntu repos)
> >>>      ===
> >>>      ipa --version
> >>>      VERSION: 4.8.1, API_VERSION: 2.233
> >>>      ===
> >>>      running letsencrypt certificates successfully for HTTPs & LDAPs
> >>>      connectivity
> >>>      ===
> >>>
> >>>      These 4-nodes are all happily running and replicating betwixt
> >>>      each other.  LDAPs is functioning great and many linux systems
> >>>      are able to all join as freeipa-clients.  Users and groups  
are
> >>>      replicating and being used elegantly for many
> LDAP-based
> >>>      authentication/authorization needs.
> >>>
> >>>      Overall, for these nodes, life is good.
> >>>
> >>>
> >>>      Where I'm running into trouble is in finally wanting to leverage
> >>>      certificate issuance on a per-user basis.  End goal is
> >>>      integrating things like yubikeys, user-cert auth, and so on.
> >>>
> >>>
> >>>      In the UI, when I enter a user's account and select Actions->New
> >>>      Certificate, I am able to successfully issue the couple prompted
> >>>      'certutil' commands to generate the user's CSR.  I then
> paste in
> >>>      the contents of the CSR and hit 'Issue' and run into the
> >>>      following error:
> >>>      ==========
> >>>      IPA Error 907: NetworkError
> >>>      cannot connect to
> >>>     
> 'https://REDACTED-1.ipa.REDACTED.com:443/ca/rest/account/login':
> >>>      [SSL: TLSV1_ALERT_UNKNOWN_CA] tlsv1 alert unknown ca (_ssl.c:2508)
> >>>      ==========
> >>>
> >>>      As I then start digging into cli-mode to attempt to understand
> >>>      where things are unhappy, I run into similar troubles with the
> >>>      server attempting to talk to itself and not being very happy
> >>>      about it.
> >>>
> >>>      ==========
> >>>      chris@REDACTED-1:~$ ipa ca-find
> >>>      ------------
> >>>      1 CA matched
> >>>      ------------
> >>>        Name: ipa
> >>>        Description: IPA CA
> >>>        Authority ID: 8acca54b-64d7-44bf-b8f7-59316213cfb6
> >>>        Subject DN: CN=Certificate Authority,O=IPA.REDACTED.COM
> >>>      <http://IPA.REDACTED.COM>
> >>>        Issuer DN: CN=Certificate Authority,O=IPA.REDACTED.COM
> >>>      <http://IPA.REDACTED.COM>
> >>>      ----------------------------
> >>>      Number of entries returned 1
> >>>      ----------------------------
> >>>      chris@REDACTED-1:~$ ipa ca-show
> >>>      Name: ipa
> >>>      ipa: ERROR: cannot connect to
> >>>     
> 'https://REDACTED-1.ipa.REDACTED.com:443/ca/rest/account/login':
> >>>      [SSL: TLSV1_ALERT_UNKNOWN_CA] tlsv1 alert unknown ca (_ssl.c:2508)
> >>>      ==========
> >>>
> >>>      Verifying with 'openssl s_client' returns the valid and
> >>>      non-expired LE cert-chain.
> >>>
> >>>      ==========
> >>>      chris@REDACTED-1:~$ openssl s_client
> >>>      REDACTED-1.ipa.REDACTED.com:443
> >>>      <http://REDACTED-1.ipa.REDACTED.com:443>
> >>>      CONNECTED(00000003)
> >>>      depth=2 O = Digital Signature Trust Co., CN = DST Root CA  
X3
> >>>      verify return:1
> >>>      depth=1 C = US, O = Let's Encrypt, CN = R3
> >>>      verify return:1
> >>>      depth=0 CN = REDACTED-1.ipa.REDACTED.com
> >>>      <http://REDACTED-1.ipa.REDACTED.com>
> >>>      verify return:1
> >>>      ---
> >>>      Certificate chain
> >>>       0 s:CN = REDACTED-1.ipa.REDACTED.com
> >>>      <http://REDACTED-1.ipa.REDACTED.com>
> >>>         i:C = US, O = Let's Encrypt, CN = R3
> >>>       1 s:C = US, O = Let's Encrypt, CN = R3
> >>>         i:O = Digital Signature Trust Co., CN = DST Root CA X3
> >>>      ---
> >>>      ...<output-truncated>...
> >>>      ---
> >>>      SSL handshake has read 3046 bytes and written 413 bytes
> >>>      Verification: OK
> >>>      ---
> >>>      New, TLSv1.3, Cipher is TLS_AES_256_GCM_SHA384
> >>>      Server public key is 2048 bit
> >>>      Secure Renegotiation IS NOT supported
> >>>      Compression: NONE
> >>>      Expansion: NONE
> >>>      No ALPN negotiated
> >>>      Early data was not sent
> >>>      Verify return code: 0 (ok)
> >>>      ...<output-truncated>...
> >>>      ==========
> >>>
> >>>      Can anyone please hit me with some clue-bat as to where I can
> >>>      read to understand how to get IPA to love itself?  I'm suspecting
> >>>      it's likely some certificate inclusion/exception that I need to
> >>>      add so that API calls and the ipa command itself will actually
> >>>      respect the LE cert-chain?
> >>>
> >>>      Any hints would be greatly appreciated.
> >>>
> >>>      Thanks,
> >>>      -Chris
> >>>
> >>>      --
> >>>      REDACTED, Inc.
> >>>      chris(a)REDACTED.com <mailto:[e-mail]>
> >>>      619.354.6463
> >>>
> >>>      _______________________________________________
> >>>      FreeIPA-users mailing list --
> >>>      freeipa-users(a)lists.fedorahosted.org
> >>>      <mailto:[e-mail]>
> >>>      To unsubscribe send an email to
> >>>      freeipa-users-leave(a)lists.fedorahosted.org
> >>>      <mailto:[e-mail]>
> >>>      Fedora Code of Conduct:
> >>>      https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> >>>      List Guidelines:
> >>>      https://fedoraproject.org/wiki/Mailing_list_guidelines
> >>>      List Archives:
> >>>     
> https://lists.fedorahosted.org/archives/list/[e-mail]...
> >>>      Do not reply to spam on the list, report it:
> >>>      https://pagure.io/fedora-infrastructure
> >>>
> >> -- 
> >> REDACTED, Inc.
> >> chris(a)REDACTED.com
> >> 619.354.6463
> >>
> >> _______________________________________________
> >> FreeIPA-users mailing list -- freeipa-users(a)lists.fedorahosted.org
> >> To unsubscribe send an email to freeipa-users-leave(a)lists.fedorahosted.org
> >> Fedora Code of Conduct:
> https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> >> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> >> List Archives:
> https://lists.fedorahosted.org/archives/list/[e-mail]...
> >> Do not reply to spam on the list, report it:
> https://pagure.io/fedora-infrastructure
> > -- 
> > REDACTED, Inc.
> > chris(a)REDACTED.com
> > 619.354.6463
> >
> >
> > _______________________________________________
> > FreeIPA-users mailing list -- freeipa-users(a)lists.fedorahosted.org
> > To unsubscribe send an email to freeipa-users-leave(a)lists.fedorahosted.org
> > Fedora Code of Conduct:
> https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> > List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> > List Archives:
> https://lists.fedorahosted.org/archives/list/[e-mail]...
> > Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
> >
>  _______________________________________________
>  FreeIPA-users mailing list -- freeipa-users(a)lists.fedorahosted.org
>  To unsubscribe send an email to freeipa-users-leave(a)lists.fedorahosted.org
>  Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
>  List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
>  List Archives:
> https://lists.fedorahosted.org/archives/list/[e-mail]...
>  Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
-- 
Node-Nine, Inc.
chris(a)node-nine.com
619.354.6463

[Chris Moody]

Finally had a chance to circle back and work on this further.

Based on my prior output of:
=====
# ipa-cacert-manage list
*IPA.REDACTED.COM IPA CA**
**IPA.REDACTED.COM IPA CA*
DSTRootCAX3
letsencryptx3
isrgrootx1
lets-encrypt-r3-cross-signed
The ipa-cacert-manage command was successful
=====
which does show the IPA CA certificate as being recognized and 
installed...and the manpage for the command references:
...CA certificate of the IPA CA (NSS database nickname: "caSigningCert 
cert-pki-ca")...

I was also able to see that dogtag implies that the IPA CA component(s) 
are installed/recognized/not-expired:
==========
REDACTED-1:~# getcert list
Number of certificates and requests being tracked: 10.
Request ID '20200416204629':
     status: MONITORING
     stuck: no
     key pair storage: type=FILE,location='/var/lib/ipa/ra-agent.key'
     certificate: type=FILE,location='/var/lib/ipa/ra-agent.pem'
     CA: dogtag-ipa-ca-renew-agent
     issuer: CN=Certificate Authority,O=IPA.REDACTED.COM
     subject: CN=IPA RA,O=IPA.REDACTED.COM
     expires: 2022-04-06 13:46:30 PDT
     key usage: digitalSignature,keyEncipherment,dataEncipherment
     eku: id-kp-clientAuth
     pre-save command: /usr/lib/ipa/certmonger/renew_ra_cert_pre
     post-save command: /usr/lib/ipa/certmonger/renew_ra_cert
     track: yes
     auto-renew: yes
Request ID '20200416204717':
     status: MONITORING
     stuck: no
     key pair storage: 
type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='auditSigningCert 
cert-pki-ca',token='NSS Certificate DB',pin set
     certificate: 
type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='auditSigningCert 
cert-pki-ca',token='NSS Certificate DB'
     CA: dogtag-ipa-ca-renew-agent
     issuer: CN=Certificate Authority,O=IPA.REDACTED.COM
     subject: CN=CA Audit,O=IPA.REDACTED.COM
     expires: 2022-04-06 13:45:48 PDT
     key usage: digitalSignature,nonRepudiation
     pre-save command: /usr/lib/ipa/certmonger/stop_pkicad
     post-save command: /usr/lib/ipa/certmonger/renew_ca_cert 
"auditSigningCert cert-pki-ca"
     track: yes
     auto-renew: yes
Request ID '20200416204718':
     status: MONITORING
     stuck: no
     key pair storage: 
type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='ocspSigningCert 
cert-pki-ca',token='NSS Certificate DB',pin set
     certificate: 
type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='ocspSigningCert 
cert-pki-ca',token='NSS Certificate DB'
     CA: dogtag-ipa-ca-renew-agent
     issuer: CN=Certificate Authority,O=IPA.REDACTED.COM
     subject: CN=OCSP Subsystem,O=IPA.REDACTED.COM
     expires: 2022-04-06 13:45:47 PDT
     eku: id-kp-OCSPSigning
     pre-save command: /usr/lib/ipa/certmonger/stop_pkicad
     post-save command: /usr/lib/ipa/certmonger/renew_ca_cert 
"ocspSigningCert cert-pki-ca"
     track: yes
     auto-renew: yes
Request ID '20200416204719':
     status: MONITORING
     stuck: no
     key pair storage: 
type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='subsystemCert 
cert-pki-ca',token='NSS Certificate DB',pin set
     certificate: 
type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='subsystemCert 
cert-pki-ca',token='NSS Certificate DB'
     CA: dogtag-ipa-ca-renew-agent
     issuer: CN=Certificate Authority,O=IPA.REDACTED.COM
     subject: CN=CA Subsystem,O=IPA.REDACTED.COM
     expires: 2022-04-06 13:45:47 PDT
     key usage: 
digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment
     eku: id-kp-clientAuth
     pre-save command: /usr/lib/ipa/certmonger/stop_pkicad
     post-save command: /usr/lib/ipa/certmonger/renew_ca_cert 
"subsystemCert cert-pki-ca"
     track: yes
     auto-renew: yes
Request ID '20200416204720':
     status: MONITORING
     stuck: no
     key pair storage: 
*type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='caSigningCert 
cert-pki-ca'*,token='NSS Certificate DB',pin set
     certificate: 
type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='caSigningCert 
cert-pki-ca',token='NSS Certificate DB'
     CA: dogtag-ipa-ca-renew-agent
     issuer: CN=Certificate Authority,O=IPA.REDACTED.COM
     subject: CN=Certificate Authority,O=IPA.REDACTED.COM
     expires: 2040-06-26 14:20:56 PDT
     key usage: digitalSignature,nonRepudiation,keyCertSign,cRLSign
     pre-save command: /usr/lib/ipa/certmonger/stop_pkicad
     post-save command: /usr/lib/ipa/certmonger/renew_ca_cert 
"caSigningCert cert-pki-ca"
     track: yes
     auto-renew: yes
Request ID '20200416204721':
     status: MONITORING
     stuck: no
     key pair storage: 
type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='Server-Cert 
cert-pki-ca',token='NSS Certificate DB',pin set
     certificate: 
type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='Server-Cert 
cert-pki-ca',token='NSS Certificate DB'
     CA: dogtag-ipa-ca-renew-agent
     issuer: CN=Certificate Authority,O=IPA.REDACTED.COM
     subject: CN=REDACTED-1.ipa.REDACTED.com,O=IPA.REDACTED.COM
     expires: 2022-04-06 13:45:47 PDT
     dns: REDACTED-1.ipa.REDACTED.com
     key usage: digitalSignature,keyEncipherment,dataEncipherment
     eku: id-kp-serverAuth
     pre-save command: /usr/lib/ipa/certmonger/stop_pkicad
     post-save command: /usr/lib/ipa/certmonger/renew_ca_cert 
"Server-Cert cert-pki-ca"
     track: yes
     auto-renew: yes
Request ID '20200416204854':
     status: MONITORING
     stuck: no
     key pair storage: type=FILE,location='/var/lib/ipa/certs/kdc.key'
     certificate: type=FILE,location='/var/lib/ipa/certs/kdc.crt'
     CA: IPA
     issuer: CN=Certificate Authority,O=IPA.REDACTED.COM
     subject: CN=REDACTED-1.ipa.REDACTED.com,O=IPA.REDACTED.COM
     expires: 2022-04-17 13:48:54 PDT
     principal name: krbtgt/IPA.REDACTED.COM(a)IPA.REDACTED.COM
     key usage: 
digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment
     eku: id-kp-serverAuth,id-pkinit-KPKdc
     pre-save command:
     post-save command: /usr/lib/ipa/certmonger/renew_kdc_cert
     track: yes
     auto-renew: yes
Request ID '20200416205655':
     status: MONITORING
     stuck: no
     key pair storage: 
type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='auditSigningCert 
cert-pki-kra',token='NSS Certificate DB',pin set
     certificate: 
type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='auditSigningCert 
cert-pki-kra',token='NSS Certificate DB'
     CA: dogtag-ipa-ca-renew-agent
     issuer: CN=Certificate Authority,O=IPA.REDACTED.COM
     subject: CN=KRA Audit,O=IPA.REDACTED.COM
     expires: 2022-04-06 13:53:14 PDT
     key usage: digitalSignature,nonRepudiation
     pre-save command: /usr/lib/ipa/certmonger/stop_pkicad
     post-save command: /usr/lib/ipa/certmonger/renew_ca_cert 
"auditSigningCert cert-pki-kra"
     track: yes
     auto-renew: yes
Request ID '20200416205656':
     status: MONITORING
     stuck: no
     key pair storage: 
type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='transportCert 
cert-pki-kra',token='NSS Certificate DB',pin set
     certificate: 
type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='transportCert 
cert-pki-kra',token='NSS Certificate DB'
     CA: dogtag-ipa-ca-renew-agent
     issuer: CN=Certificate Authority,O=IPA.REDACTED.COM
     subject: CN=KRA Transport Certificate,O=IPA.REDACTED.COM
     expires: 2022-04-06 13:53:13 PDT
     key usage: 
digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment
     eku: id-kp-clientAuth
     pre-save command: /usr/lib/ipa/certmonger/stop_pkicad
     post-save command: /usr/lib/ipa/certmonger/renew_ca_cert 
"transportCert cert-pki-kra"
     track: yes
     auto-renew: yes
Request ID '20200416205657':
     status: MONITORING
     stuck: no
     key pair storage: 
type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='storageCert 
cert-pki-kra',token='NSS Certificate DB',pin set
     certificate: 
type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='storageCert 
cert-pki-kra',token='NSS Certificate DB'
     CA: dogtag-ipa-ca-renew-agent
     issuer: CN=Certificate Authority,O=IPA.REDACTED.COM
     subject: CN=KRA Storage Certificate,O=IPA.REDACTED.COM
     expires: 2022-04-06 13:53:13 PDT
     key usage: 
digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment
     eku: id-kp-clientAuth
     pre-save command: /usr/lib/ipa/certmonger/stop_pkicad
     post-save command: /usr/lib/ipa/certmonger/renew_ca_cert 
"storageCert cert-pki-kra"
     track: yes
     auto-renew: yes
=====

Where else should I be looking to try and understand/debug why the 
server is rejecting itś own connection to itself?  From my (albeit 
limited) understanding thus far, all the requisite components are 
present and accounted for, no?

Do my apache logs of the following give any hints to anyone as to what 
isn´t being trusted?
=====
[Tue Jun 15 17:11:34.674975 2021] [ssl:error] [pid 31830:tid 
139703550412544] [client 2604:XXX::36:4001:58500] AH02039: Certificate 
Verification: Error (19): self signed certificate in certificate chain
[Tue Jun 15 17:11:34.675088 2021] [ssl:error] [pid 31830:tid 
139703550412544] [client 2604:XXX::36:4001:58500] AH02261: 
Re-negotiation handshake failed
[Tue Jun 15 17:11:34.675111 2021] [ssl:error] [pid 31830:tid 
139703550412544] SSL Library Error: error:1417C086:SSL 
routines:tls_process_client_certificate:certificate verify failed
=====

-Chris


On 6/16/21 3:32 PM, Chris Moody via FreeIPA-users wrote:
>  That was kinda my belief thus far as well that the hosts were not 
>  trusting themselves - not 100% sure how things got here though.  I 
>  have a hunch it might be related to the initial deployment and the 
>  prior admin using an outdated method to install/manage/renew the 
>  LE-certificates.
>
>  =====
>  # ipa-cacert-manage list
>  IPA.REDACTED.COM IPA CA
>  IPA.REDACTED.COM IPA CA
>  DSTRootCAX3
>  letsencryptx3
>  isrgrootx1
>  lets-encrypt-r3-cross-signed
>  The ipa-cacert-manage command was successful
>  =====
>
>  How would I go about forcing re-installation of the host's own CA 
>  certificate to ensure it's trust?
>
>  Also, since these nodes are not running on an RPM-based distro, the 
>  typical cert-store locations I have seen on other systems are not in 
>  the same location(s) so I'm not sure totally sure every location to 
>  point certutil to be able to examine each cert-store in depth as well 
>  (if that might help diagnose further).  I ask because I believe these 
>  were initially built and then had the 
>  "https://github.com/freeipa/freeipa-letsencrypt/" project used to 
>  initially deploy the LE-certs - prior to the ipa-cacert-manage command 
>  being the official path toward installing/managing these external 
>  certificates.  I know because this code had been git-pulled onto these 
>  nodes, but it obviously doesn't work properly since this git project 
>  manipulates the paths below directly instead of managing via the 
>  ipa-cacert-manage command.
>
>  ex>
>  /etc/httpd/alias/ (<=== not on these systems)
>  /etc/pki/pki-tomcat/alias/
>  /etc/ipa/nssdb/
>  /etc/dirsrv/slapd-IPA-REDACTED-COM/
>
>
>  Checking that project's git page now though, I see their readme now 
>  mentions /var/lib/ipa/certs/, where I just noticed cacert.pem.
>
>  =====
>  /var/lib/ipa/certs# openssl x509 -text -noout -in cacert.pem
>  Certificate:
>      Data:
>          Version: 3 (0x2)
>          Serial Number: 1 (0x1)
>          Signature Algorithm: sha256WithRSAEncryption
>          Issuer: O = IPA.REDACTED.COM, CN = Certificate Authority
>          Validity
>              Not Before: Apr 16 20:45:46 2020 GMT
>              Not After : Apr 16 20:45:46 2040 GMT
>          Subject: O = IPA.REDACTED.COM, CN = Certificate Authority
>  ...
>  =====
>  so I believe I might have just located the IPA CA cert in case I need 
>  to re-install it.
>
>
>
>  To the following question, I have the following LE-related certs 
>  installed.  And yes, I did run into issues a couple months back when 
>  LE moved to the new certs on their end so had to import the new 
>  authority certs to get the LE host certs to update & import.  The LE 
>  certificates are functioning and verify for both slapd and apache/tomcat.
>
>  =====
>  DSTRootCAX3.pem  LetsEncryptAuthorityX3.pem  isrgrootx1.pem 
>  lets-encrypt-r3-cross-signed.pem
>  =====
>
>  Thank you all so much for the assistance through all this.
>
>  -Chris
>
>
>  On 6/16/21 1:26 PM, Rob Crittenden via FreeIPA-users wrote:
> > The error suggests that your
> > IPA server doesn't trust its own CA
> > certificate.
> >
> > Does ipa-cacert-manage list include the IPA CA?
> >
> > BTW the new certificate steps are unrelated. This affects all CA requests.
> >
> > rob

[Florence Renaud]

Hi,

it seems the error happens when you run commands that require communication
between IPA framework and the Certificate Server (like ipa ca-show). The
workflow is the following:
1. the client (= the command "ipa ca-show") is a python process that
communicates with httpd on the secure port. It seems this part is OK (ipa
ca-find returns successfully).
2. the IPA framework is a wsgi app running inside httpd. The handling of
"ipa ca-show" requires the framework to communicate with Dogtag, which is
running inside pki-tomcat.The communication happens over a secure port with
authentication based on the RA certificate. This communication is not
working, probably because httpd doesn't trust the CA that issued Dogtag's
server cert.

I think you need to check where httpd is getting its list of trusted CAs
when it's acting as a client of Dogtag server. The code is using
api.env.tls_ca_cert which is /etc/ipa/ca.crt on rhel/fedora (you can check
with "ipa env tls_ca_cert" to find the value on your server) but may be
different on ubuntu.

flo

On Thu, Jun 24, 2021 at 8:49 PM Chris Moody via FreeIPA-users <
freeipa-users(a)lists.fedorahosted.org&gt; wrote:

>  Finally had a chance to circle back and work on this further.
>
>  Based on my prior output of:
>  =====
>  # ipa-cacert-manage list
>  *IPA.REDACTED.COM <http://IPA.REDACTED.COM> IPA CA*
>  * IPA.REDACTED.COM <http://IPA.REDACTED.COM> IPA CA*
>  DSTRootCAX3
>  letsencryptx3
>  isrgrootx1
>  lets-encrypt-r3-cross-signed
>  The ipa-cacert-manage command was successful
>  =====
>  which does show the IPA CA certificate as being recognized and
>  installed...and the manpage for the command references:
>  ...CA certificate of the IPA CA (NSS database nickname: "caSigningCert
>  cert-pki-ca")...
>
>  I was also able to see that dogtag implies that the IPA CA component(s)
>  are installed/recognized/not-expired:
>  ==========
>  REDACTED-1:~# getcert list
>  Number of certificates and requests being tracked: 10.
>  Request ID '20200416204629':
>      status: MONITORING
>      stuck: no
>      key pair storage: type=FILE,location='/var/lib/ipa/ra-agent.key'
>      certificate: type=FILE,location='/var/lib/ipa/ra-agent.pem'
>      CA: dogtag-ipa-ca-renew-agent
>      issuer: CN=Certificate Authority,O=IPA.REDACTED.COM
>      subject: CN=IPA RA,O=IPA.REDACTED.COM
>      expires: 2022-04-06 13:46:30 PDT
>      key usage: digitalSignature,keyEncipherment,dataEncipherment
>      eku: id-kp-clientAuth
>      pre-save command: /usr/lib/ipa/certmonger/renew_ra_cert_pre
>      post-save command: /usr/lib/ipa/certmonger/renew_ra_cert
>      track: yes
>      auto-renew: yes
>  Request ID '20200416204717':
>      status: MONITORING
>      stuck: no
>      key pair storage:
>  type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='auditSigningCert
>  cert-pki-ca',token='NSS Certificate DB',pin set
>      certificate:
>  type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='auditSigningCert
>  cert-pki-ca',token='NSS Certificate DB'
>      CA: dogtag-ipa-ca-renew-agent
>      issuer: CN=Certificate Authority,O=IPA.REDACTED.COM
>      subject: CN=CA Audit,O=IPA.REDACTED.COM
>      expires: 2022-04-06 13:45:48 PDT
>      key usage: digitalSignature,nonRepudiation
>      pre-save command: /usr/lib/ipa/certmonger/stop_pkicad
>      post-save command: /usr/lib/ipa/certmonger/renew_ca_cert
>  "auditSigningCert cert-pki-ca"
>      track: yes
>      auto-renew: yes
>  Request ID '20200416204718':
>      status: MONITORING
>      stuck: no
>      key pair storage:
>  type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='ocspSigningCert
>  cert-pki-ca',token='NSS Certificate DB',pin set
>      certificate:
>  type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='ocspSigningCert
>  cert-pki-ca',token='NSS Certificate DB'
>      CA: dogtag-ipa-ca-renew-agent
>      issuer: CN=Certificate Authority,O=IPA.REDACTED.COM
>      subject: CN=OCSP Subsystem,O=IPA.REDACTED.COM
>      expires: 2022-04-06 13:45:47 PDT
>      eku: id-kp-OCSPSigning
>      pre-save command: /usr/lib/ipa/certmonger/stop_pkicad
>      post-save command: /usr/lib/ipa/certmonger/renew_ca_cert
>  "ocspSigningCert cert-pki-ca"
>      track: yes
>      auto-renew: yes
>  Request ID '20200416204719':
>      status: MONITORING
>      stuck: no
>      key pair storage:
>  type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='subsystemCert
>  cert-pki-ca',token='NSS Certificate DB',pin set
>      certificate:
>  type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='subsystemCert
>  cert-pki-ca',token='NSS Certificate DB'
>      CA: dogtag-ipa-ca-renew-agent
>      issuer: CN=Certificate Authority,O=IPA.REDACTED.COM
>      subject: CN=CA Subsystem,O=IPA.REDACTED.COM
>      expires: 2022-04-06 13:45:47 PDT
>      key usage:
>  digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment
>      eku: id-kp-clientAuth
>      pre-save command: /usr/lib/ipa/certmonger/stop_pkicad
>      post-save command: /usr/lib/ipa/certmonger/renew_ca_cert
>  "subsystemCert cert-pki-ca"
>      track: yes
>      auto-renew: yes
>  Request ID '20200416204720':
>      status: MONITORING
>      stuck: no
>      key pair storage:
> *type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='caSigningCert
>  cert-pki-ca'*,token='NSS Certificate DB',pin set
>      certificate:
>  type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='caSigningCert
>  cert-pki-ca',token='NSS Certificate DB'
>      CA: dogtag-ipa-ca-renew-agent
>      issuer: CN=Certificate Authority,O=IPA.REDACTED.COM
>      subject: CN=Certificate Authority,O=IPA.REDACTED.COM
>      expires: 2040-06-26 14:20:56 PDT
>      key usage: digitalSignature,nonRepudiation,keyCertSign,cRLSign
>      pre-save command: /usr/lib/ipa/certmonger/stop_pkicad
>      post-save command: /usr/lib/ipa/certmonger/renew_ca_cert
>  "caSigningCert cert-pki-ca"
>      track: yes
>      auto-renew: yes
>  Request ID '20200416204721':
>      status: MONITORING
>      stuck: no
>      key pair storage:
>  type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='Server-Cert
>  cert-pki-ca',token='NSS Certificate DB',pin set
>      certificate:
>  type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='Server-Cert
>  cert-pki-ca',token='NSS Certificate DB'
>      CA: dogtag-ipa-ca-renew-agent
>      issuer: CN=Certificate Authority,O=IPA.REDACTED.COM
>      subject: CN=REDACTED-1.ipa.REDACTED.com,O=IPA.REDACTED.COM
>      expires: 2022-04-06 13:45:47 PDT
>      dns: REDACTED-1.ipa.REDACTED.com
>      key usage: digitalSignature,keyEncipherment,dataEncipherment
>      eku: id-kp-serverAuth
>      pre-save command: /usr/lib/ipa/certmonger/stop_pkicad
>      post-save command: /usr/lib/ipa/certmonger/renew_ca_cert "Server-Cert
>  cert-pki-ca"
>      track: yes
>      auto-renew: yes
>  Request ID '20200416204854':
>      status: MONITORING
>      stuck: no
>      key pair storage: type=FILE,location='/var/lib/ipa/certs/kdc.key'
>      certificate: type=FILE,location='/var/lib/ipa/certs/kdc.crt'
>      CA: IPA
>      issuer: CN=Certificate Authority,O=IPA.REDACTED.COM
>      subject: CN=REDACTED-1.ipa.REDACTED.com,O=IPA.REDACTED.COM
>      expires: 2022-04-17 13:48:54 PDT
>      principal name: krbtgt/IPA.REDACTED.COM(a)IPA.REDACTED.COM
>      key usage:
>  digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment
>      eku: id-kp-serverAuth,id-pkinit-KPKdc
>      pre-save command:
>      post-save command: /usr/lib/ipa/certmonger/renew_kdc_cert
>      track: yes
>      auto-renew: yes
>  Request ID '20200416205655':
>      status: MONITORING
>      stuck: no
>      key pair storage:
>  type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='auditSigningCert
>  cert-pki-kra',token='NSS Certificate DB',pin set
>      certificate:
>  type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='auditSigningCert
>  cert-pki-kra',token='NSS Certificate DB'
>      CA: dogtag-ipa-ca-renew-agent
>      issuer: CN=Certificate Authority,O=IPA.REDACTED.COM
>      subject: CN=KRA Audit,O=IPA.REDACTED.COM
>      expires: 2022-04-06 13:53:14 PDT
>      key usage: digitalSignature,nonRepudiation
>      pre-save command: /usr/lib/ipa/certmonger/stop_pkicad
>      post-save command: /usr/lib/ipa/certmonger/renew_ca_cert
>  "auditSigningCert cert-pki-kra"
>      track: yes
>      auto-renew: yes
>  Request ID '20200416205656':
>      status: MONITORING
>      stuck: no
>      key pair storage:
>  type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='transportCert
>  cert-pki-kra',token='NSS Certificate DB',pin set
>      certificate:
>  type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='transportCert
>  cert-pki-kra',token='NSS Certificate DB'
>      CA: dogtag-ipa-ca-renew-agent
>      issuer: CN=Certificate Authority,O=IPA.REDACTED.COM
>      subject: CN=KRA Transport Certificate,O=IPA.REDACTED.COM
>      expires: 2022-04-06 13:53:13 PDT
>      key usage:
>  digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment
>      eku: id-kp-clientAuth
>      pre-save command: /usr/lib/ipa/certmonger/stop_pkicad
>      post-save command: /usr/lib/ipa/certmonger/renew_ca_cert
>  "transportCert cert-pki-kra"
>      track: yes
>      auto-renew: yes
>  Request ID '20200416205657':
>      status: MONITORING
>      stuck: no
>      key pair storage:
>  type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='storageCert
>  cert-pki-kra',token='NSS Certificate DB',pin set
>      certificate:
>  type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='storageCert
>  cert-pki-kra',token='NSS Certificate DB'
>      CA: dogtag-ipa-ca-renew-agent
>      issuer: CN=Certificate Authority,O=IPA.REDACTED.COM
>      subject: CN=KRA Storage Certificate,O=IPA.REDACTED.COM
>      expires: 2022-04-06 13:53:13 PDT
>      key usage:
>  digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment
>      eku: id-kp-clientAuth
>      pre-save command: /usr/lib/ipa/certmonger/stop_pkicad
>      post-save command: /usr/lib/ipa/certmonger/renew_ca_cert "storageCert
>  cert-pki-kra"
>      track: yes
>      auto-renew: yes
>  =====
>
>  Where else should I be looking to try and understand/debug why the server
>  is rejecting itś own connection to itself?  From my (albeit limited)
>  understanding thus far, all the requisite components are present and
>  accounted for, no?
>
>  Do my apache logs of the following give any hints to anyone as to what
>  isn´t being trusted?
>  =====
>  [Tue Jun 15 17:11:34.674975 2021] [ssl:error] [pid 31830:tid
>  139703550412544] [client 2604:XXX::36:4001:58500] AH02039: Certificate
>  Verification: Error (19): self signed certificate in certificate chain
>  [Tue Jun 15 17:11:34.675088 2021] [ssl:error] [pid 31830:tid
>  139703550412544] [client 2604:XXX::36:4001:58500] AH02261: Re-negotiation
>  handshake failed
>  [Tue Jun 15 17:11:34.675111 2021] [ssl:error] [pid 31830:tid
>  139703550412544] SSL Library Error: error:1417C086:SSL
>  routines:tls_process_client_certificate:certificate verify failed
>  =====
>
>  -Chris
>
>
>  On 6/16/21 3:32 PM, Chris Moody via FreeIPA-users wrote:
>
>  That was kinda my belief thus far as well that the hosts were not trusting
>  themselves - not 100% sure how things got here though.  I have a hunch it
>  might be related to the initial deployment and the prior admin using an
>  outdated method to install/manage/renew the LE-certificates.
>
>  =====
>  # ipa-cacert-manage list
>  IPA.REDACTED.COM IPA CA
>  IPA.REDACTED.COM IPA CA
>  DSTRootCAX3
>  letsencryptx3
>  isrgrootx1
>  lets-encrypt-r3-cross-signed
>  The ipa-cacert-manage command was successful
>  =====
>
>  How would I go about forcing re-installation of the host's own CA
>  certificate to ensure it's trust?
>
>  Also, since these nodes are not running on an RPM-based distro, the
>  typical cert-store locations I have seen on other systems are not in the
>  same location(s) so I'm not sure totally sure every location to point
>  certutil to be able to examine each cert-store in depth as well (if that
>  might help diagnose further).  I ask because I believe these were initially
>  built and then had the "https://github.com/freeipa/freeipa-letsencrypt/"
>  <https://github.com/freeipa/freeipa-letsencrypt/> project used to
>  initially deploy the LE-certs - prior to the ipa-cacert-manage command
>  being the official path toward installing/managing these external
>  certificates.  I know because this code had been git-pulled onto these
>  nodes, but it obviously doesn't work properly since this git project
>  manipulates the paths below directly instead of managing via the
>  ipa-cacert-manage command.
>
>  ex>
>  /etc/httpd/alias/ (<=== not on these systems)
>  /etc/pki/pki-tomcat/alias/
>  /etc/ipa/nssdb/
>  /etc/dirsrv/slapd-IPA-REDACTED-COM/
>
>
>  Checking that project's git page now though, I see their readme now
>  mentions /var/lib/ipa/certs/, where I just noticed cacert.pem.
>
>  =====
>  /var/lib/ipa/certs# openssl x509 -text -noout -in cacert.pem
>  Certificate:
>      Data:
>          Version: 3 (0x2)
>          Serial Number: 1 (0x1)
>          Signature Algorithm: sha256WithRSAEncryption
>          Issuer: O = IPA.REDACTED.COM, CN = Certificate Authority
>          Validity
>              Not Before: Apr 16 20:45:46 2020 GMT
>              Not After : Apr 16 20:45:46 2040 GMT
>          Subject: O = IPA.REDACTED.COM, CN = Certificate Authority
>  ...
>  =====
>  so I believe I might have just located the IPA CA cert in case I need to
>  re-install it.
>
>
>
>  To the following question, I have the following LE-related certs
>  installed.  And yes, I did run into issues a couple months back when LE
>  moved to the new certs on their end so had to import the new authority
>  certs to get the LE host certs to update & import.  The LE certificates are
>  functioning and verify for both slapd and apache/tomcat.
>
>  =====
>  DSTRootCAX3.pem  LetsEncryptAuthorityX3.pem  isrgrootx1.pem
>  lets-encrypt-r3-cross-signed.pem
>  =====
>
>  Thank you all so much for the assistance through all this.
>
>  -Chris
>
>
>  On 6/16/21 1:26 PM, Rob Crittenden via FreeIPA-users wrote:
>
>  The error suggests that your
>  IPA server doesn't trust its own CA
>  certificate.
>
>  Does ipa-cacert-manage list include the IPA CA?
>
>  BTW the new certificate steps are unrelated. This affects all CA requests.
>
>  rob
>
>
>  _______________________________________________
>  FreeIPA-users mailing list -- freeipa-users(a)lists.fedorahosted.org
>  To unsubscribe send an email to freeipa-users-leave(a)lists.fedorahosted.org
>  Fedora Code of Conduct:
>  https://docs.fedoraproject.org/en-US/project/code-of-conduct/
>  List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
>  List Archives:
>  https://lists.fedorahosted.org/archives/list/[e-mail]...
>  Do not reply to spam on the list, report it:
>  https://pagure.io/fedora-infrastructure
>