ERR - slapi_ldap_bind - Could not send bind request for id [(anon)] authentication mechanism [EXTERNAL]: error -1 (Can't contact LDAP server), system error 0 (no error), network error 0

[Graham Leggett]

Hi all,

We have a long standing 389ds master LDAP server that was found to be unable to contact
it’s slaves. Most specifically, the slaves show nothing in their logs about any kind of
connection, while the master is logging this:

[12/Nov/2019:21:39:47.212715697 +0000] - ERR - slapi_ldap_bind - Could not send bind
request for id [(anon)] authentication mechanism [EXTERNAL]: error -1 (Can't contact
LDAP server), system error 0 (no error), network error 0 (Unknown error, host
“ldap01:636”)

Key is "system error 0 (no error)”, which leaves us stumped. The error is obviously
“success”.

Has anyone seen this kind of thing before?

This is 389ds running on CentOS7 as follows:

389-ds-base-1.3.9.1-10.el7.x86_64

Regards,
Graham
—

[Mark Reynolds]

On 11/12/19 4:47 PM, Graham Leggett wrote:
>  Hi all,
>
>  We have a long standing 389ds master LDAP server that was found to be unable to contact
> it’s slaves. Most specifically, the slaves show nothing in their logs about any kind of
> connection, while the master is logging this:
>
>  [12/Nov/2019:21:39:47.212715697 +0000] - ERR - slapi_ldap_bind - Could not send bind
> request for id [(anon)] authentication mechanism [EXTERNAL]: error -1 (Can't contact
> LDAP server), system error 0 (no error), network error 0 (Unknown error, host
> “ldap01:636”) 
What is the bind method of the agreement?  SSLCLIENTAUTH?  The problem 
is that the ID is anonymous (anon).  So it's not binding correctly to 
the consumer.   What do you have for these attributes in the replication 
agreement:


This is what I have:

dn: cn=blah,cn=replica,cn=dc\3Dexample\2Cdc\3Dcom,cn=mapping tree,cn=config

nsDS5ReplicaBindMethod: sslclientauth
nsDS5ReplicaTransportInfo: LDAPS
nsDS5ReplicaBindDN: cn=replication manager,cn=config


Mark

>  Key is "system error 0 (no error)”, which leaves us stumped. The error is obviously
> “success”.
>
>  Has anyone seen this kind of thing before?
>
>  This is 389ds running on CentOS7 as follows:
>
>  389-ds-base-1.3.9.1-10.el7.x86_64
>
>  Regards,
>  Graham
>  —
>
>
>  _______________________________________________
>  389-users mailing list -- 389-users(a)lists.fedoraproject.org
>  To unsubscribe send an email to 389-users-leave(a)lists.fedoraproject.org
>  Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
>  List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
>  List Archives:
> https://lists.fedoraproject.org/archives/list/[e-mail]... 
-- 

389 Directory Server Development Team

[William Brown]

>  On 13 Nov 2019, at 08:13, Mark Reynolds <mreynolds(a)redhat.com&gt;
> wrote:
>  
>  
>  On 11/12/19 4:47 PM, Graham Leggett wrote:
> > Hi all,
> > 
> > We have a long standing 389ds master LDAP server that was found to be unable to
> contact it’s slaves. Most specifically, the slaves show nothing in their logs about any
> kind of connection, while the master is logging this:
> > 
> > [12/Nov/2019:21:39:47.212715697 +0000] - ERR - slapi_ldap_bind - Could not send bind
> request for id [(anon)] authentication mechanism [EXTERNAL]: error -1 (Can't contact
> LDAP server), system error 0 (no error), network error 0 (Unknown error, host
> “ldap01:636”)
> > 
>  What is the bind method of the agreement?  SSLCLIENTAUTH?  The problem is that the ID is
> anonymous (anon).  So it's not binding correctly to the consumer.   What do you have
> for these attributes in the replication agreement: 
Hmmm, ldap01:636 also seems like a bad hostname too? 

>  
>  
>  
>  This is what I have:
>  
>  dn: cn=blah,cn=replica,cn=dc\3Dexample\2Cdc\3Dcom,cn=mapping tree,cn=config
>  
>  nsDS5ReplicaBindMethod: sslclientauth
>  nsDS5ReplicaTransportInfo: LDAPS
>  nsDS5ReplicaBindDN: cn=replication manager,cn=config
>  
>  
>  
>  Mark
>  
> > Key is "system error 0 (no error)”, which leaves us stumped. The error is
> obviously “success”.
> > 
> > Has anyone seen this kind of thing before?
> > 
> > This is 389ds running on CentOS7 as follows:
> > 
> > 389-ds-base-1.3.9.1-10.el7.x86_64
> > 
> > Regards,
> > Graham
> > —
> > 
> > 
> > 
> > 
> > _______________________________________________
> > 389-users mailing list -- 
> > 389-users(a)lists.fedoraproject.org
> > 
> > To unsubscribe send an email to 
> > 389-users-leave(a)lists.fedoraproject.org
> > 
> > Fedora Code of Conduct: 
> > https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> > 
> > List Guidelines: 
> > https://fedoraproject.org/wiki/Mailing_list_guidelines
> > 
> > List Archives: 
> > https://lists.fedoraproject.org/archives/list/[e-mail]...
>  -- 
>  
>  389 Directory Server Development Team
>  
>  _______________________________________________
>  389-users mailing list -- 389-users(a)lists.fedoraproject.org
>  To unsubscribe send an email to 389-users-leave(a)lists.fedoraproject.org
>  Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
>  List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
>  List Archives:
> https://lists.fedoraproject.org/archives/list/[e-mail]... 
—
Sincerely,

William Brown

Senior Software Engineer, 389 Directory Server
SUSE Labs

[Graham Leggett]

On 13 Nov 2019, at 00:13, Mark Reynolds <mreynolds(a)redhat.com&gt; wrote:
> > We have a long standing 389ds master LDAP server that was found
> to be unable to contact it’s slaves. Most specifically, the slaves show nothing in their
> logs about any kind of connection, while the master is logging this:
> > 
> > [12/Nov/2019:21:39:47.212715697 +0000] - ERR - slapi_ldap_bind - Could not send bind
> request for id [(anon)] authentication mechanism [EXTERNAL]: error -1 (Can't contact
> LDAP server), system error 0 (no error), network error 0 (Unknown error, host
> “ldap01:636”)
>  What is the bind method of the agreement?  SSLCLIENTAUTH?  The problem is that the ID is
> anonymous (anon).  So it's not binding correctly to the consumer.   What do you have
> for these attributes in the replication agreement:
>   
More of the problem was picked up by wireshark - the 389ds LDAP slave is telling
the 389ds LDAP master that the 389ds LDAP slave does not recognise the CA:

Transmission Control Protocol, Src Port: 636, Dst Port: 53994, Seq: 5462, Ack: 2279, Len:
7
Transport Layer Security
    TLSv1.2 Record Layer: Alert (Level: Fatal, Description: Unknown CA)
        Content Type: Alert (21)
        Version: TLS 1.2 (0x0303)
        Length: 2
        Alert Message
            Level: Fatal (2)
            Description: Unknown CA (48)

(The certificates are privately generated, and have been in place since 2016, and are all
still valid.)

This in in turn caused because the 389ds LDAP master has for some reason decided to not
pass the full certificate chain across to the slave (intermediates are involved) and the
slave is quite correctly saying unknown CA.

Does anyone know why 389ds would suddenly stop sending the full certificate chain while
replicating?

It also looks like the error handling in 389ds SSL is broken - if the slave sent “unknown
CA" to the master, the master needs to log that fact, and not report the error as
“success”.

Regards,
Graham
—

[William Brown]

>  On 13 Nov 2019, at 09:34, Graham Leggett <minfrin(a)sharp.fm&gt;
> wrote:
>  
>  On 13 Nov 2019, at 00:13, Mark Reynolds <mreynolds(a)redhat.com&gt; wrote:
> >> We have a long standing 389ds master LDAP server that was found to be unable to
> contact it’s slaves. Most specifically, the slaves show nothing in their logs about any
> kind of connection, while the master is logging this:
> >> 
> >> [12/Nov/2019:21:39:47.212715697 +0000] - ERR - slapi_ldap_bind - Could not send
> bind request for id [(anon)] authentication mechanism [EXTERNAL]: error -1 (Can't
> contact LDAP server), system error 0 (no error), network error 0 (Unknown error, host
> “ldap01:636”)
> >> 
> > What is the bind method of the agreement?  SSLCLIENTAUTH?  The problem is that the ID
> is anonymous (anon).  So it's not binding correctly to the consumer.   What do you
> have for these attributes in the replication agreement:
> > 
>  More of the problem was picked up by wireshark - the 389ds LDAP slave is telling the
> 389ds LDAP master that the 389ds LDAP slave does not recognise the CA:
>  
>  Transmission Control Protocol, Src Port: 636, Dst Port: 53994, Seq: 5462, Ack: 2279, Len:
> 7
>  Transport Layer Security
>      TLSv1.2 Record Layer: Alert (Level: Fatal, Description: Unknown CA)
>          Content Type: Alert (21)
>          Version: TLS 1.2 (0x0303)
>          Length: 2
>          Alert Message
>              Level: Fatal (2)
>              Description: Unknown CA (48)
>  
>  (The certificates are privately generated, and have been in place since 2016, and are all
> still valid.)
>  
>  This in in turn caused because the 389ds LDAP master has for some reason decided to not
> pass the full certificate chain across to the slave (intermediates are involved) and the
> slave is quite correctly saying unknown CA.
>  
>  Does anyone know why 389ds would suddenly stop sending the full certificate chain while
> replicating?
>  
>  It also looks like the error handling in 389ds SSL is broken - if the slave sent “unknown
> CA" to the master, the master needs to log that fact, and not report the error as
> “success”. 
We'll need to see the output of certutil -L -d /etc/dirsrv/slapd-<instance>/
from both the master and replica servers please.

In a TLS auth process the client doesn't send it's CA - if you get unknown CA
it's most likely the replica has either had the CA and it's chain members expire,
or they are not marked as trusted for client auth. So that's why I'd like to see
the certutil output please. 


>  
>  Regards,
>  Graham
>  —
>  
>  _______________________________________________
>  389-users mailing list -- 389-users(a)lists.fedoraproject.org
>  To unsubscribe send an email to 389-users-leave(a)lists.fedoraproject.org
>  Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
>  List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
>  List Archives:
> https://lists.fedoraproject.org/archives/list/[e-mail]... 
—
Sincerely,

William Brown

Senior Software Engineer, 389 Directory Server
SUSE Labs

[Graham Leggett]

On 13 Nov 2019, at 01:37, William Brown <wbrown(a)suse.de&gt; wrote:

> > Does anyone know why 389ds would suddenly stop sending the full
> certificate chain while replicating?
> > 
> > It also looks like the error handling in 389ds SSL is broken - if the slave sent
> “unknown CA" to the master, the master needs to log that fact, and not report the
> error as “success”.
>  
>  We'll need to see the output of certutil -L -d /etc/dirsrv/slapd-<instance>/
> from both the master and replica servers please.
>  
>  In a TLS auth process the client doesn't send it's CA - if you get unknown CA
> it's most likely the replica has either had the CA and it's chain members expire,
> or they are not marked as trusted for client auth. So that's why I'd like to see
> the certutil output please.  
I discovered the same problem had been reported in OpenLDAP:
https://www.centos.org/forums/viewtopic.php?t=67042

This in turn is caused by a regression in NSS, where it is no longer sufficient to have a
trusted root certificate, you now need all intermediate certificates marked as trusted as
well.

Making the following change to the intermediate certs fixed the problem:

[root@ldap01 ~]# certutil -L -d /etc/dirsrv/slapd-hg

Certificate Nickname                                         Trust Attributes
                                                             SSL,S/MIME,JAR/XPI

intermediateB                                 ,,   
intermediateA                                   ,,   
rootrootroot                                    CT,C,C
ldap01                                     u,u,u
[root@ldap01 ~]# certutil -M -d /etc/dirsrv/slapd-hg -t "CT,C,C" -n
"intermediateA"
[root@ldap01 ~]# certutil -M -d /etc/dirsrv/slapd-hg -t "CT,C,C" -n
"intermediateB"
[root@ldap01 ~]# certutil -L -d /etc/dirsrv/slapd-hg

Certificate Nickname                                         Trust Attributes
                                                             SSL,S/MIME,JAR/XPI

intermediateA                                   CT,C,C
intermediateB                               CT,C,C
rootrootroot                                    CT,C,C
ldap01                                          u,u,u

Raised the bug here: https://bugzilla.redhat.com/show_bug.cgi?id=1771979

Regards,
Graham
—

[William Brown]

>  On 13 Nov 2019, at 20:29, Graham Leggett <minfrin(a)sharp.fm&gt;
> wrote:
>  
>  On 13 Nov 2019, at 01:37, William Brown <wbrown(a)suse.de&gt; wrote:
>  
> >> Does anyone know why 389ds would suddenly stop sending the full certificate chain
> while replicating?
> >> 
> >> It also looks like the error handling in 389ds SSL is broken - if the slave sent
> “unknown CA" to the master, the master needs to log that fact, and not report the
> error as “success”.
> > 
> > We'll need to see the output of certutil -L -d
> /etc/dirsrv/slapd-<instance>/ from both the master and replica servers please.
> > 
> > In a TLS auth process the client doesn't send it's CA - if you get unknown CA
> it's most likely the replica has either had the CA and it's chain members expire,
> or they are not marked as trusted for client auth. So that's why I'd like to see
> the certutil output please. 
>  
>  I discovered the same problem had been reported in OpenLDAP:
> https://www.centos.org/forums/viewtopic.php?t=67042
>  
>  This in turn is caused by a regression in NSS, where it is no longer sufficient to have a
> trusted root certificate, you now need all intermediate certificates marked as trusted as
> well.
>  
>  Making the following change to the intermediate certs fixed the problem:
>  
>  [root@ldap01 ~]# certutil -L -d /etc/dirsrv/slapd-hg
>  
>  Certificate Nickname                                         Trust Attributes
>                                                              SSL,S/MIME,JAR/XPI
>  
>  intermediateB                                 ,,   
>  intermediateA                                   ,,   
>  rootrootroot                                    CT,C,C
>  ldap01                                     u,u,u
>  [root@ldap01 ~]# certutil -M -d /etc/dirsrv/slapd-hg -t "CT,C,C" -n
> "intermediateA"
>  [root@ldap01 ~]# certutil -M -d /etc/dirsrv/slapd-hg -t "CT,C,C" -n
> "intermediateB"
>  [root@ldap01 ~]# certutil -L -d /etc/dirsrv/slapd-hg
>  
>  Certificate Nickname                                         Trust Attributes
>                                                              SSL,S/MIME,JAR/XPI
>  
>  intermediateA                                   CT,C,C
>  intermediateB                               CT,C,C
>  rootrootroot                                    CT,C,C
>  ldap01                                          u,u,u
>  
>  Raised the bug here: https://bugzilla.redhat.com/show_bug.cgi?id=1771979 
Awesome work, thanks for following up on this! 

>  
>  Regards,
>  Graham
>  —
>  
>  _______________________________________________
>  389-users mailing list -- 389-users(a)lists.fedoraproject.org
>  To unsubscribe send an email to 389-users-leave(a)lists.fedoraproject.org
>  Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
>  List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
>  List Archives:
> https://lists.fedoraproject.org/archives/list/[e-mail]... 
—
Sincerely,

William Brown

Senior Software Engineer, 389 Directory Server
SUSE Labs

[Graham Leggett]

On 13 Nov 2019, at 12:29, Graham Leggett <minfrin(a)sharp.fm&gt; wrote:

>  On 13 Nov 2019, at 01:37, William Brown <wbrown(a)suse.de&gt;
> wrote:
>  
> >> Does anyone know why 389ds would suddenly stop sending the full certificate chain
> while replicating?
> >> 
> >> It also looks like the error handling in 389ds SSL is broken - if the slave sent
> “unknown CA" to the master, the master needs to log that fact, and not report the
> error as “success”.
> > 
> > We'll need to see the output of certutil -L -d
> /etc/dirsrv/slapd-<instance>/ from both the master and replica servers please.
> > 
> > In a TLS auth process the client doesn't send it's CA - if you get unknown CA
> it's most likely the replica has either had the CA and it's chain members expire,
> or they are not marked as trusted for client auth. So that's why I'd like to see
> the certutil output please. 
>  
>  I discovered the same problem had been reported in OpenLDAP:
> https://www.centos.org/forums/viewtopic.php?t=67042
>  
>  This in turn is caused by a regression in NSS, where it is no longer sufficient to have a
> trusted root certificate, you now need all intermediate certificates marked as trusted as
> well.
>  
>  Making the following change to the intermediate certs fixed the problem:
>  
>  [root@ldap01 ~]# certutil -L -d /etc/dirsrv/slapd-hg
>  
>  Certificate Nickname                                         Trust Attributes
>                                                              SSL,S/MIME,JAR/XPI
>  
>  intermediateB                                 ,,   
>  intermediateA                                   ,,   
>  rootrootroot                                    CT,C,C
>  ldap01                                     u,u,u
>  [root@ldap01 ~]# certutil -M -d /etc/dirsrv/slapd-hg -t "CT,C,C" -n
> "intermediateA"
>  [root@ldap01 ~]# certutil -M -d /etc/dirsrv/slapd-hg -t "CT,C,C" -n
> "intermediateB"
>  [root@ldap01 ~]# certutil -L -d /etc/dirsrv/slapd-hg
>  
>  Certificate Nickname                                         Trust Attributes
>                                                              SSL,S/MIME,JAR/XPI
>  
>  intermediateA                                   CT,C,C
>  intermediateB                               CT,C,C
>  rootrootroot                                    CT,C,C
>  ldap01                                          u,u,u
>  
>  Raised the bug here: https://bugzilla.redhat.com/show_bug.cgi?id=1771979
> <https://bugzilla.redhat.com/show_bug.cgi?id=1771979> 
Coming back to this one - got to the bottom of this while investigating something else
that wasn’t working.

This wasn’t a regression in NSS, but rather a regression in the openldap libraries shipped
by RHEL7.5 and above.

For reasons that I haven’t found, there was an architecture change made half way through
the RHEL7 lifecycle where openldap was linked to openssl instead of NSS.

Openldap's NSS support and openldap’s openssl support differ in a fundamental way -
with NSS, when openldap makes an SSL connection intermediate certificates are filled in by
the client side as normal. With openssl, when openldap makes an SSL connection
intermediate certificates are ignored, and the connection breaks.

The hack workaround above fixes this because openldap’s openssl support expects you to
place intermediate certs in your trusted certificate store. As soon as you mark the
intermediates as trusted in NSS, the hack workaround in 389ds that makes replication
sort-of work bound to two different crypto libraries exports trusted certs across into the
ca certificate list passed to openldap. Openldap then finds the intermediates and things
work.

Fundamentally there are two bugs:

- https://bugzilla.redhat.com/show_bug.cgi?id=1898924
<https://bugzilla.redhat.com/show_bug.cgi?id=1898924>

- An architectural change half way through the lifecycle of what is supposed to be a
stable OS.

Regards,
Graham
—

[William Brown]

>  On 19 Nov 2020, at 20:34, Graham Leggett <minfrin(a)sharp.fm&gt;
> wrote:
>  
>  On 13 Nov 2019, at 12:29, Graham Leggett <minfrin(a)sharp.fm&gt; wrote:
>  
> > On 13 Nov 2019, at 01:37, William Brown <wbrown(a)suse.de&gt; wrote:
> > 
> >>> Does anyone know why 389ds would suddenly stop sending the full certificate
> chain while replicating?
> >>> 
> >>> It also looks like the error handling in 389ds SSL is broken - if the slave
> sent “unknown CA" to the master, the master needs to log that fact, and not report
> the error as “success”.
> >> 
> >> We'll need to see the output of certutil -L -d
> /etc/dirsrv/slapd-<instance>/ from both the master and replica servers please.
> >> 
> >> In a TLS auth process the client doesn't send it's CA - if you get
> unknown CA it's most likely the replica has either had the CA and it's chain
> members expire, or they are not marked as trusted for client auth. So that's why
> I'd like to see the certutil output please. 
> > 
> > I discovered the same problem had been reported in OpenLDAP:
> https://www.centos.org/forums/viewtopic.php?t=67042
> > 
> > This in turn is caused by a regression in NSS, where it is no longer sufficient to
> have a trusted root certificate, you now need all intermediate certificates marked as
> trusted as well.
> > 
> > Making the following change to the intermediate certs fixed the problem:
> > 
> > [root@ldap01 ~]# certutil -L -d /etc/dirsrv/slapd-hg
> > 
> > Certificate Nickname                                         Trust Attributes
> >                                                             SSL,S/MIME,JAR/XPI
> > 
> > intermediateB                                 ,,   
> > intermediateA                                   ,,   
> > rootrootroot                                    CT,C,C
> > ldap01                                     u,u,u
> > [root@ldap01 ~]# certutil -M -d /etc/dirsrv/slapd-hg -t "CT,C,C" -n
> "intermediateA"
> > [root@ldap01 ~]# certutil -M -d /etc/dirsrv/slapd-hg -t "CT,C,C" -n
> "intermediateB"
> > [root@ldap01 ~]# certutil -L -d /etc/dirsrv/slapd-hg
> > 
> > Certificate Nickname                                         Trust Attributes
> >                                                             SSL,S/MIME,JAR/XPI
> > 
> > intermediateA                                   CT,C,C
> > intermediateB                               CT,C,C
> > rootrootroot                                    CT,C,C
> > ldap01                                          u,u,u
> > 
> > Raised the bug here: https://bugzilla.redhat.com/show_bug.cgi?id=1771979
>  
>  Coming back to this one - got to the bottom of this while investigating something else
> that wasn’t working.
>  
>  This wasn’t a regression in NSS, but rather a regression in the openldap libraries
> shipped by RHEL7.5 and above.
>  
>  For reasons that I haven’t found, there was an architecture change made half way through
> the RHEL7 lifecycle where openldap was linked to openssl instead of NSS.
>  
>  Openldap's NSS support and openldap’s openssl support differ in a fundamental way -
> with NSS, when openldap makes an SSL connection intermediate certificates are filled in by
> the client side as normal. With openssl, when openldap makes an SSL connection
> intermediate certificates are ignored, and the connection breaks.
>  
>  The hack workaround above fixes this because openldap’s openssl support expects you to
> place intermediate certs in your trusted certificate store. As soon as you mark the
> intermediates as trusted in NSS, the hack workaround in 389ds that makes replication
> sort-of work bound to two different crypto libraries exports trusted certs across into the
> ca certificate list passed to openldap. Openldap then finds the intermediates and things
> work.
>  
>  Fundamentally there are two bugs:
>  
>  - https://bugzilla.redhat.com/show_bug.cgi?id=1898924
>  
>  - An architectural change half way through the lifecycle of what is supposed to be a
> stable OS. 
I seem to remember this change (this was at a time I worked at RH). If memory serves
correctly, OpenLDAP upstream removed/deprecated their NSS support. This was making it much
much harder to apply fixes for issues both stability and technical, so moving to OpenSSL
was the "best move" for customer support.

Even internally to 389-ds which has to link to OpenLDAP for some outbound client
operations, it internally swapped from NSS to OpenSSL for this as well, which involves
extracting some certificates into temporary stores for OpenLDAP client to use. It's
quite fun to put it mildly.

There are very good reasons why those decisions were made, and that was very carefully
managed too. It was a lot of work and ultimately, it did make the OpenLDAP client library
better for our maintainers and many consumers, but as you have noticed, these are complex
systems, designed and built in a time that preceded deep testing of things. I believe our
client TLS auth tests were made *after* the OpenLDAP to OpenSSL switch was made, which
could be why this was not noticed.

Anyway, I hope that this gives some more context to why this happened. 

—
Sincerely,

William Brown

Senior Software Engineer, 389 Directory Server
SUSE Labs, Australia

[Graham Leggett]

On 19 Nov 2020, at 10:34, Graham Leggett <minfrin(a)sharp.fm&gt; wrote:

> > Raised the bug here:
> https://bugzilla.redhat.com/show_bug.cgi?id=1771979
>  
>  Coming back to this one - got to the bottom of this while investigating something else
> that wasn’t working.
>  
>  This wasn’t a regression in NSS, but rather a regression in the openldap libraries
> shipped by RHEL7.5 and above.
>  
>  For reasons that I haven’t found, there was an architecture change made half way through
> the RHEL7 lifecycle where openldap was linked to openssl instead of NSS.
>  
>  Openldap's NSS support and openldap’s openssl support differ in a fundamental way -
> with NSS, when openldap makes an SSL connection intermediate certificates are filled in by
> the client side as normal. With openssl, when openldap makes an SSL connection
> intermediate certificates are ignored, and the connection breaks.
>  
>  The hack workaround above fixes this because openldap’s openssl support expects you to
> place intermediate certs in your trusted certificate store. As soon as you mark the
> intermediates as trusted in NSS, the hack workaround in 389ds that makes replication
> sort-of work bound to two different crypto libraries exports trusted certs across into the
> ca certificate list passed to openldap. Openldap then finds the intermediates and things
> work.
>  
>  Fundamentally there are two bugs:
>  
>  - https://bugzilla.redhat.com/show_bug.cgi?id=1898924
>  
>  - An architectural change half way through the lifecycle of what is supposed to be a
> stable OS. 
End of 2023, the bug is still present in RHEL9:

[11/Dec/2023:23:02:09.510906411 +0000] - ERR - slapi_ldap_bind - Could not send bind
request for id [(anon)] authentication mechanism [EXTERNAL]: error -1 (Can't contact
LDAP server), system error -5987 (Invalid function argument.), network error 0 (Unknown
error, host “ldap2.example.com:636")

This time, the workaround of forcing the intermediate certificates to be marked trusted no
longer works. We now get a low level complaint about a certificate verification failure.
The error message doesn’t tell us which certificate failed, but this message is an openssl
message.

[11/Dec/2023:19:45:28.115134273 +0000] - ERR - NSMMReplicationPlugin - bind_and_check_pwp
- agmt=“cn=ldap2" (thor:636) - Replication bind with EXTERNAL auth failed: LDAP error
-1 (Can't contact LDAP server) (error:0A000086:SSL routines::certificate verify failed
(self-signed certificate in certificate chain))

There are no self-signed certificates being used, they are certs issued by public CAs,
which like all public CAs, have intermediate certs.

The bugs I raised in 2020 were all abandoned and closed.

Regards,
Graham
—