On 13 Nov 2019, at 01:37, William Brown <wbrown@suse.de> wrote:

Does anyone know why 389ds would suddenly stop sending the full certificate chain while replicating?

It also looks like the error handling in 389ds SSL is broken - if the slave sent “unknown CA" to the master, the master needs to log that fact, and not report the error as “success”.

We'll need to see the output of certutil -L -d /etc/dirsrv/slapd-<instance>/ from both the master and replica servers please.

In a TLS auth process the client doesn't send it's CA - if you get unknown CA it's most likely the replica has either had the CA and it's chain members expire, or they are not marked as trusted for client auth. So that's why I'd like to see the certutil output please.

I discovered the same problem had been reported in OpenLDAP: https://www.centos.org/forums/viewtopic.php?t=67042

This in turn is caused by a regression in NSS, where it is no longer sufficient to have a trusted root certificate, you now need all intermediate certificates marked as trusted as well.

Making the following change to the intermediate certs fixed the problem:

[root@ldap01 ~]# certutil -L -d /etc/dirsrv/slapd-hg

Certificate Nickname                                         Trust Attributes
                                                            SSL,S/MIME,JAR/XPI

intermediateB                                 ,,
intermediateA                                   ,,
rootrootroot                                    CT,C,C
ldap01                                     u,u,u
[root@ldap01 ~]# certutil -M -d /etc/dirsrv/slapd-hg -t "CT,C,C" -n "intermediateA"
[root@ldap01 ~]# certutil -M -d /etc/dirsrv/slapd-hg -t "CT,C,C" -n "intermediateB"
[root@ldap01 ~]# certutil -L -d /etc/dirsrv/slapd-hg

Certificate Nickname                                         Trust Attributes
                                                            SSL,S/MIME,JAR/XPI

intermediateA                                   CT,C,C
intermediateB                               CT,C,C
rootrootroot                                    CT,C,C
ldap01                                          u,u,u

Raised the bug here: https://bugzilla.redhat.com/show_bug.cgi?id=1771979

Coming back to this one - got to the bottom of this while investigating something else that wasn’t working.

This wasn’t a regression in NSS, but rather a regression in the openldap libraries shipped by RHEL7.5 and above.

For reasons that I haven’t found, there was an architecture change made half way through the RHEL7 lifecycle where openldap was linked to openssl instead of NSS.

Openldap's NSS support and openldap’s openssl support differ in a fundamental way - with NSS, when openldap makes an SSL connection intermediate certificates are filled in by the client side as normal. With openssl, when openldap makes an SSL connection intermediate certificates are ignored, and the connection breaks.

The hack workaround above fixes this because openldap’s openssl support expects you to place intermediate certs in your trusted certificate store. As soon as you mark the intermediates as trusted in NSS, the hack workaround in 389ds that makes replication sort-of work bound to two different crypto libraries exports trusted certs across into the ca certificate list passed to openldap. Openldap then finds the intermediates and things work.

Fundamentally there are two bugs:

- https://bugzilla.redhat.com/show_bug.cgi?id=1898924

- An architectural change half way through the lifecycle of what is supposed to be a stable OS.

Regards,

Graham

—