My solution to keep an update cache of all LDAP entries. Is there a better way?

[Mark London]

Here is my solution to have a persistant uptodate local cache of all 
ldap entries, so as to avoid very long delays when a user issues a 
command that causes a large number of LDAP lookups, i.e. by doing a "ls 
-l /home":

 enumerate = true
 enum_cache_timeout = 86400
 ldap_purge_cache_timeout = 0
 ldap_enumeration_refresh_timeout = 300

I set the cache timeout to be 24 hours, and do an enumerate every 5 minutes.

What I would like to know, is why such long delays (i.e. minutes) occurs 
when doing an "ls -l /home".  Is it because it has to write out each 
entry into the local database?   Just curious. :)  Thanks.

Mark London

[Simo Sorce]

On Sat, 2012-06-30 at 08:52 -0400, Mark London wrote:
>  Here is my solution to have a persistant uptodate local cache of all
>
>  ldap entries, so as to avoid very long delays when a user issues a 
>  command that causes a large number of LDAP lookups, i.e. by doing a "ls 
>  -l /home":
>  
>   enumerate = true
>   enum_cache_timeout = 86400
>   ldap_purge_cache_timeout = 0
>   ldap_enumeration_refresh_timeout = 300
>  
>  I set the cache timeout to be 24 hours, and do an enumerate every 5 minutes.
>  
>  What I would like to know, is why such long delays (i.e. minutes) occurs 
>  when doing an "ls -l /home".  Is it because it has to write out each 
>  entry into the local database?   Just curious. :)  Thanks. 
It depends on the version of sssd. In master we have a shared memory
based client that will speed up local lookups by many orders of
magnitude. Until you have that you need to send a request and wait for a
reply from the sssd_nss process for each getXX() call. If you pile many
that adds up in latency and context switches, so it can still take some
time if you need to do a lot of getXXuid/gid calls.


Simo.

-- 
Simo Sorce * Red Hat, Inc * New York

[Stephen Gallagher]

On Sat, 2012-06-30 at 08:52 -0400, Mark London wrote:
>  Here is my solution to have a persistant uptodate local cache of all
>
>  ldap entries, so as to avoid very long delays when a user issues a 
>  command that causes a large number of LDAP lookups, i.e. by doing a "ls 
>  -l /home":
>  
>   enumerate = true
>   enum_cache_timeout = 86400
>   ldap_purge_cache_timeout = 0
>   ldap_enumeration_refresh_timeout = 300
>  
>  I set the cache timeout to be 24 hours, and do an enumerate every 5 minutes.
>  
>  What I would like to know, is why such long delays (i.e. minutes) occurs 
>  when doing an "ls -l /home".  Is it because it has to write out each 
>  entry into the local database?   Just curious. :)  Thanks. 


How many users are we talking about, and are they all represented
in /home?

The initial enumeration can take up to a few minutes the first time
(after that we only update on differences). Once the full enumeration
has been completed, it should take about 0.02 seconds per lookup against
the local cache. On ordinary lookups, there are no writes going on.
Writes to the cache only occur on cache misses.