Hi - When our primary DNS is unreachable, SSSD with LDAP breaks, or is incredibly slow. I've traced it to the fact that several of the LDAP timeout values are 6 seconds. This is not long enough, because the default DNS timeout failover is 5 seconds. Incoming SSH connections are impossible with increasing the LDAP timeout value. I'm not sure yet which is the critical setting, but I've increased the following from 6 seconds to 30:
ldap_network_timeout = 30 ldap_opt_timeout = 30 ldap_search_timeout = 30
What I don't understand is, even with this increase, while it allows SSH logins to work, it still takes about 1 1/2 minutes to login to the computer. The logs show that every LDAP connection takes about 5 seconds to complete. That seems to imply to me that the code doing an DNS Lookup to the primary DNS server, rather than using the local DNS cache. See below. I've not traced the code, to see if in fact this is the case. But whatever the reason, LDAP connections are taking a long time, with the primary DNS server being down:
(Mon Aug 13 15:43:46 2012) [sssd[be[PSFC]]] [sdap_ldap_connect_callback_add] (0x1000): New LDAP connection to [ldaps://ForestDnsZones.psfc.mit.edu/DC=ForestDnsZones,DC=psfc,DC=mit,DC=edu] with fd [32]. (Mon Aug 13 15:43:51 2012) [sssd[be[PSFC]]] [sdap_ldap_connect_callback_add] (0x1000): New LDAP connection to [ldaps://DomainDnsZones.psfc.mit.edu/DC=DomainDnsZones,DC=psfc,DC=mit,DC=edu] with fd [33]. (Mon Aug 13 15:43:56 2012) [sssd[be[PSFC]]] [sdap_ldap_connect_callback_add] (0x1000): New LDAP connection to [ldaps://psfc.mit.edu/CN=Configuration,DC=psfc,DC=mit,DC=edu] with fd [34]. (Mon Aug 13 15:43:56 2012) [sssd[be[PSFC]]] [sdap_rebind_proc] (0x1000): Successfully bind to [ldaps://DomainDnsZones.psfc.mit.edu/DC=DomainDnsZones,DC=psfc,DC=mit,DC=edu]. (Mon Aug 13 15:43:56 2012) [sssd[be[PSFC]]] [sdap_rebind_proc] (0x1000): Successfully bind to [ldaps://ForestDnsZones.psfc.mit.edu/DC=ForestDnsZones,DC=psfc,DC=mit,DC=edu].
This is true, even though I have enumeration turned on, with long cached timeouts:
enum_cache_timeout = 86400 ldap_purge_cache_timeout = 0 ldap_enumeration_refresh_timeout = 300
I was under the impression that enumeration prevented blocking, and used cache values, rather than waiting for LDAP requests to finish.
Any comments, or opinions on how to fix this problem? Thanks.
- Mark
Mark London wrote:
Hi - When our primary DNS is unreachable, SSSD with LDAP breaks, or is incredibly slow. I've traced it to the fact that several of the LDAP timeout values are 6 seconds. This is not long enough, because the default DNS timeout failover is 5 seconds. Incoming SSH connections are impossible without increasing the LDAP timeout value. I'm not sure yet which is the critical setting, but I've increased the following from 6 seconds to 30:
Hi - First, I had a typo in the above paragraph of my previous email. I mean to say "impossible without increasing". Instead I said "impossible with increasing".
Secondly, I discovered that SSSD has an internal resolver, and found the dns_resolver_timeout variable. I changed it from 5 seconds 1 second, but that didn't help. I still see 5 second delays when sdap_ldap_connect_callback_add is called. It would be nice if the internal resolver had a cache! Any other suggestions? I'll be happy to hack the code, if someone could give me any idea of what needs to be fixed. This situation has occurred several times over the past few months, causing major problems. Thanks.
- Mark
On Mon, Aug 13, 2012 at 10:16:49PM -0400, Mark London wrote:
Mark London wrote:
Hi - When our primary DNS is unreachable, SSSD with LDAP breaks, or is incredibly slow. I've traced it to the fact that several of the LDAP timeout values are 6 seconds. This is not long enough, because the default DNS timeout failover is 5 seconds. Incoming SSH connections are impossible without increasing the LDAP timeout value. I'm not sure yet which is the critical setting, but I've increased the following from 6 seconds to 30:
Hi - First, I had a typo in the above paragraph of my previous email. I mean to say "impossible without increasing". Instead I said "impossible with increasing".
Secondly, I discovered that SSSD has an internal resolver, and found the dns_resolver_timeout variable. I changed it from 5 seconds 1 second, but that didn't help. I still see 5 second delays when sdap_ldap_connect_callback_add is called. It would be nice if the internal resolver had a cache! Any other suggestions? I'll be happy to hack the code, if someone could give me any idea of what needs to be fixed. This situation has occurred several times over the past few months, causing major problems. Thanks.
I would recommend turning off the referral support: ldap_referrals = false
That should get rid of many reconnection attempts. We don't control name resolution for referred servers and I suspect the resolution is done internally in libldap.
On Tue, 2012-08-14 at 10:33 +0200, Jakub Hrozek wrote:
On Mon, Aug 13, 2012 at 10:16:49PM -0400, Mark London wrote:
Mark London wrote:
Hi - When our primary DNS is unreachable, SSSD with LDAP breaks, or is incredibly slow. I've traced it to the fact that several of the LDAP timeout values are 6 seconds. This is not long enough, because the default DNS timeout failover is 5 seconds. Incoming SSH connections are impossible without increasing the LDAP timeout value. I'm not sure yet which is the critical setting, but I've increased the following from 6 seconds to 30:
Hi - First, I had a typo in the above paragraph of my previous email. I mean to say "impossible without increasing". Instead I said "impossible with increasing".
Secondly, I discovered that SSSD has an internal resolver, and found the dns_resolver_timeout variable. I changed it from 5 seconds 1 second, but that didn't help. I still see 5 second delays when sdap_ldap_connect_callback_add is called. It would be nice if the internal resolver had a cache! Any other suggestions? I'll be happy to hack the code, if someone could give me any idea of what needs to be fixed. This situation has occurred several times over the past few months, causing major problems. Thanks.
I would recommend turning off the referral support: ldap_referrals = false
That should get rid of many reconnection attempts. We don't control name resolution for referred servers and I suspect the resolution is done internally in libldap.
For a little more detail to anyone else who stumbles across this thread: SSSD doesn't process referral chasing itself. We instead rely on a feature built into the openldap client libraries that automatically chases referrals and then calls an SSSD callback to perform the bind to the new referred server. Unfortunately, as time has passed we've become more and more aware that openldap's internal referral chasing is both unreliable and in some cases painfully slow.
The second contributing factor here is that Active Directory misuses the referral concept entirely; it returns three (usually) unnecessary referrals for every subtree search request. What this means is that every lookup against Active Directory (with ldap_referrals = True) results in a minimum of four LDAP round-trips. Three of these are only ever useful in the case where you are performing partial replication (e.g. you're talking to an office AD server but want to get information about a user not replicated to this server).
Finally, as Jakub says, when performing internal referrals, the openldap client libraries do not provide a callback mechanism to us for performing DNS resolution. They just call the libc mechanism gethostbyname2() internally (IIRC). So they cannot take advantage of any caching that we might do for DNS. However, they *could* take advantage of dnsmasq caching if it's set up on the system.
tl;dr Active Directory is bad about referrals, so unless you use partial replication, set ldap_referrals = False.
sssd-devel@lists.fedorahosted.org
-
Jakub Hrozek -
Mark London -
Stephen Gallagher