enumerate=false still enumerates a lot of users/groups

[GOLLSCHEWSKY, Tim]

Hi SSSD Users.

I'm trying to increase the performance of my user's logins, we have a medium sized
Active Direcctory.

According to the man page, the enumerate directive:

   enumerate (bool)
              Determines if a domain can be enumerated. This parameter can have one of the
following values:
              TRUE = Users and groups are enumerated
              FALSE = No enumerations for this domain

However when I start sssd with no cache and simulate an initgroups, it still seems to
enumerate many
many groups and user accounts.

I'm running sssd v1.8.4:

  # pkill sssd
  # pgrep sssd
  # pwd
  /apps/sssd-1.8.4
  # rm -f var/lib/sss/db/*
  # grep enumerate /etc/sssd/sssd.conf
  enumerate = FALSE
  # grep ldap_access /etc/sssd/sssd.conf
  ldap_access_filter =
memberOf=cn=xxxgroup,ou=yyyOU,ou=zzzOU,ou=Groups,dc=aaa,dc=bbb,dc=ccc
  # sbin/sssd -c /etc/sssd/sssd.conf
  # su - myuser -c "groups | wc"
        1     193    1181
  # strings var/lib/sss/db/cache_AAA.BBB.CCC.ldb | grep OU=Groups,DC=aaa,DC=bbb,DC=ccc |
sort -u | wc -l
  522
  # strings var/lib/sss/db/cache_AAA.BBB.CCC.ldb | grep OU=Accounts,DC=aaa,DC=bbb,DC=ccc |
sort -u | wc -l
  1938

Sorry for my use of strings and sort -u, I don't know a better way to interrogate the
cache.

Why does it still enumerate so many users and groups (that are not me, and not in my
ldap_access_filter) when I log in?  Even when
I have disabled domain enumeration?

Regards,

Tim.

________________________________

This e-mail is sent by Suncorp Group Limited ABN 66 145 290 124 or one of its related
entities "Suncorp".
Suncorp may be contacted at Level 18, 36 Wickham Terrace, Brisbane or on 13 11 55 or at
suncorp.com.au.
The content of this e-mail is the view of the sender or stated author and does not
necessarily reflect the view of Suncorp. The content, including attachments, is a
confidential communication between Suncorp and the intended recipient. If you are not the
intended recipient, any use, interference with, disclosure or copying of this e-mail,
including attachments, is unauthorised and expressly prohibited. If you have received this
e-mail in error please contact the sender immediately and delete the e-mail and any
attachments from your system.

[Jakub Hrozek]

On Thu, May 31, 2012 at 06:50:55AM +0000, GOLLSCHEWSKY, Tim wrote:
>  Hi SSSD Users.
>  
>  I'm trying to increase the performance of my user's logins, we have a medium
> sized Active Direcctory.
>  
>  According to the man page, the enumerate directive:
>  
>     enumerate (bool)
>                Determines if a domain can be enumerated. This parameter can have one of
> the following values:
>                TRUE = Users and groups are enumerated
>                FALSE = No enumerations for this domain
>   
I think there is some confusion with the terminology the SSSD uses.
"enumerate=true" for SSSD means "download all data in the directory
periodically".

>  However when I start sssd with no cache and simulate an initgroups,
> it still seems to enumerate many
>  many groups and user accounts.
>   
I assume that you simulated initgroups with id -G or logging in?

>  I'm running sssd v1.8.4:
>  
>    # pkill sssd
>    # pgrep sssd
>    # pwd
>    /apps/sssd-1.8.4
>    # rm -f var/lib/sss/db/*
>    # grep enumerate /etc/sssd/sssd.conf
>    enumerate = FALSE
>    # grep ldap_access /etc/sssd/sssd.conf
>    ldap_access_filter =
> memberOf=cn=xxxgroup,ou=yyyOU,ou=zzzOU,ou=Groups,dc=aaa,dc=bbb,dc=ccc 
The access filter only tells the SSSD who can log in, it doesn't have
effect on the data downloaded.

Since SSSD 1.7, the SSSD allows specifying filter as part of the search
base. For example, to only download users that are members of a particular
group:

ldap_user_search_base =
OU=Accounts,DC=aaa,DC=bbb,DC=ccc?subtree?(memberOf=cn=xxxgroup,ou=yyyOU,ou=zzzOU,ou=Groups,dc=aaa,dc=bbb,dc=ccc)

The format is as per RFC2254. See man sssd-ldap for more details.

>    # sbin/sssd -c /etc/sssd/sssd.conf
>    # su - myuser -c "groups | wc"
>          1     193    1181
>    # strings var/lib/sss/db/cache_AAA.BBB.CCC.ldb | grep OU=Groups,DC=aaa,DC=bbb,DC=ccc |
> sort -u | wc -l
>    522
>    # strings var/lib/sss/db/cache_AAA.BBB.CCC.ldb | grep OU=Accounts,DC=aaa,DC=bbb,DC=ccc
> | sort -u | wc -l
>    1938
>  
>  Sorry for my use of strings and sort -u, I don't know a better way to interrogate the
> cache. 
To poke at the cache, install the ldb-tools package and use the
"ldbsearch" utility. It works much like ldapsearch:

ldbsearch -H /var/lib/sss/db/cache_AAA.BBB.CCC.ldb 'objectclass=user'

>  
>  Why does it still enumerate so many users and groups (that are not me, and not in my
> ldap_access_filter) when I log in?  Even when
>  I have disabled domain enumeration?
>  
>  Regards,
>  
>  Tim.

[GOLLSCHEWSKY, Tim]

Hi Jakub,

>  Since SSSD 1.7, the SSSD allows specifying filter as part of the
>  search base. For example, to only download users that are members of a particular
> group:
>
>  ldap_user_search_base =
>  OU=Accounts,DC=aaa,DC=bbb,DC=ccc?subtree?(memberOf=cn=xxxgroup,ou=yyyO
>  U,ou=zzzOU,ou=Groups,dc=aaa,dc=bbb,dc=ccc)
>  
I've been having a play around with this. Strangely it doesn't seem to limit
much... and I can't work out why.  :/

If I have the ldap_user_search_base set at the OU level, it gets this many user records:

  # grep ^ldap_user_search_base /etc/sssd/sssd.conf
  ldap_user_search_base = ou=Accounts,dc=aaa,dc=bbb,dc=ccc
  # ldbsearch -H var/lib/sss/db/cache_AAA.BBB.CCC.ldb 'objectclass=user' | grep
^gecos | wc -l
  asq: Unable to register control with rootdse!
  1939

But if I put in the memberOf=cn= in there, it gets:

  # grep ^ldap_user_search_base /etc/sssd/sssd.conf
  ldap_user_search_base =
ou=Accounts,dc=aaa,dc=bbb,dc=ccc?subtree?(memberOf=cn=mygroup,ou=UnixGroups,ou=Groups,ou=aaa,ou=bbb,ou=ccc)
  # ldbsearch -H var/lib/sss/db/cache_AAA.BBB.CCC.ldb 'objectclass=user' | grep
^gecos | wc -l
  asq: Unable to register control with rootdse!
  1938

So not a lot of difference.  Just 1 record less.

>  > I'll ensure all my testing from now on is an actual ssh
> login and not the "groups" command.
>  I suspect the reason is that SSSD tries to be smart here and differentiates between
> initgroups operation performed via the NSS interface (id -G) and initgroups performed
> during auth via the PAM stack.
>  The initgroups that comes from the NSS can be returned from the cache for speed's
> sake. On the other hand, the PAM responder only keeps a very short-lived in-memory cache -
> see the pam_id_timeout option in sssd.conf, defaults to 5 seconds. Its purpose is mainly
> to only contact the server once even if the login program performs multiple initgroups
> calls during login - SSH is one example, it performs several initgroups operations during
> various stages of the PAM conversation and even from different processes.
>  
>  *Maybe* we could do more heuristics if the group and user base DNs
> are
>  specified and don't overlap, something similar to what's proposed in
>  https://fedorahosted.org/sssd/ticket/1319 ?
>  
If I'm interpreting this correctly (that we can determine if an object is a
"user" record based on the DN and not have to do a lookup), and if we can also
do this for AD as well as IPA, I think this enhancement would provide a truly remarkable
performance improvement for AD-based sites like mine using RFC2307bis with a medium(to
large) sized directory.

>  > This goes back to my original question.  When I have enumerate
> disabled, why does SSSD still look up all this fake user information in LDAP?  And is
> there any way to disable it from doing so?
>
>  Hm, this is interesting. I'm not seeing this in my test setup -- I tested with the
> 1.8 branch, with ldap_group_nesting_level=0 against a RFC2307bis server. Would you mind
> sharing a snippet from your logs?
>  
Sure.

There is probably a better way to do this.  :)  But this is how I'm doing it: turning
on debug, and grepping for ldap_search:

  sbin/sssd -c /etc/sssd/sssd.conf -i -d8 2>&1 | fgrep calling\ ldap_search_ext
> /tmp/searches.txt

Note in this example I'm using a cleared cache with nesting = 0 and enumerate = false
and the following user search base.
The login took 31 seconds.

ldap_user_search_base =
ou=Accounts,dc=aaa,dc=bbb,dc=ccc?subtree?(memberOf=CN=mygroup,OU=UnixGroups,OU=Groups,DC=aaa,DC=bbb,DC=ccc)

# wc -l /tmp/searches.txt
2980 /tmp/searches.txt
# grep objectclass=user /tmp/searches.txt | wc -l
2397

"mygroup" contains 581 users.

This is what some of the searches look like:

# grep objectclass=user /tmp/searches.txt | head
(Mon Jun  4 16:25:11 2012) [sssd[be[AAA.BBB.CCC]]] [sdap_get_generic_ext_step] (0x0400):
calling ldap_search_ext with
[(&(&(msSFU30Name=myuser)(objectclass=user))(memberOf=CN=mygroup,OU=UnixGroups,OU=Groups,DC=aaa,DC=bbb,DC=ccc))][ou=Accounts,dc=aaa,dc=corp,dc=ccc].
(Mon Jun  4 16:25:11 2012) [sssd[be[AAA.BBB.CCC]]] [sdap_get_generic_ext_step] (0x0400):
calling ldap_search_ext with
[(&(&(msSFU30Name=myuser)(objectclass=user))(memberOf=CN=mygroup,OU=UnixGroups,OU=Groups,DC=aaa,DC=bbb,DC=ccc))][ou=Accounts,dc=aaa,dc=corp,dc=ccc].
(Mon Jun  4 16:25:12 2012) [sssd[be[AAA.BBB.CCC]]] [sdap_get_generic_ext_step] (0x0400):
calling ldap_search_ext with [(objectclass=user)][CN=someuser1,OU=Admin
Accounts,OU=Accounts,DC=aaa,DC=bbb,DC=ccc].
(Mon Jun  4 16:25:12 2012) [sssd[be[AAA.BBB.CCC]]] [sdap_get_generic_ext_step] (0x0400):
calling ldap_search_ext with [(objectclass=user)][CN=someuser2,OU=Admin
Accounts,OU=Accounts,DC=aaa,DC=bbb,DC=ccc].
(Mon Jun  4 16:25:12 2012) [sssd[be[AAA.BBB.CCC]]] [sdap_get_generic_ext_step] (0x0400):
calling ldap_search_ext with [(objectclass=user)][CN=someuser3,OU=Admin
Accounts,OU=Accounts,DC=aaa,DC=bbb,DC=ccc].
(Mon Jun  4 16:25:12 2012) [sssd[be[AAA.BBB.CCC]]] [sdap_get_generic_ext_step] (0x0400):
calling ldap_search_ext with [(objectclass=user)][CN=someuser4,OU=Admin
Accounts,OU=Accounts,DC=aaa,DC=bbb,DC=ccc].
(Mon Jun  4 16:25:12 2012) [sssd[be[AAA.BBB.CCC]]] [sdap_get_generic_ext_step] (0x0400):
calling ldap_search_ext with [(objectclass=user)][CN=someuser5,OU=Admin
Accounts,OU=Accounts,DC=aaa,DC=bbb,DC=ccc].
(Mon Jun  4 16:25:12 2012) [sssd[be[AAA.BBB.CCC]]] [sdap_get_generic_ext_step] (0x0400):
calling ldap_search_ext with [(objectclass=user)][CN=someuser6,OU=Admin
Accounts,OU=Accounts,DC=aaa,DC=bbb,DC=ccc].
(Mon Jun  4 16:25:12 2012) [sssd[be[AAA.BBB.CCC]]] [sdap_get_generic_ext_step] (0x0400):
calling ldap_search_ext with
[(objectclass=user)][CN=someuser7,OU=Users,OU=Accounts,DC=aaa,DC=bbb,DC=ccc].
(Mon Jun  4 16:25:12 2012) [sssd[be[AAA.BBB.CCC]]] [sdap_get_generic_ext_step] (0x0400):
calling ldap_search_ext with
[(objectclass=user)][CN=someuser8,OU=Users,OU=Accounts,DC=aaa,DC=bbb,DC=ccc].

If more sanitised debug output would be useful, please let me know.

Cheers,
Tim.

________________________________

This e-mail is sent by Suncorp Group Limited ABN 66 145 290 124 or one of its related
entities "Suncorp".
Suncorp may be contacted at Level 18, 36 Wickham Terrace, Brisbane or on 13 11 55 or at
suncorp.com.au.
The content of this e-mail is the view of the sender or stated author and does not
necessarily reflect the view of Suncorp. The content, including attachments, is a
confidential communication between Suncorp and the intended recipient. If you are not the
intended recipient, any use, interference with, disclosure or copying of this e-mail,
including attachments, is unauthorised and expressly prohibited. If you have received this
e-mail in error please contact the sender immediately and delete the e-mail and any
attachments from your system.

[Stephen Gallagher]

On Thu, 2012-05-31 at 06:50 +0000, GOLLSCHEWSKY, Tim wrote:
>  Hi SSSD Users.
>  
>  I'm trying to increase the performance of my user's logins, we have a medium
> sized Active Direcctory.
>  
>  According to the man page, the enumerate directive:
>  
>     enumerate (bool)
>                Determines if a domain can be enumerated. This parameter can have one of
> the following values:
>                TRUE = Users and groups are enumerated
>                FALSE = No enumerations for this domain
>  
>  However when I start sssd with no cache and simulate an initgroups, it still seems to
> enumerate many
>  many groups and user accounts.
>  
>  I'm running sssd v1.8.4:
>  
>    # pkill sssd
>    # pgrep sssd
>    # pwd
>    /apps/sssd-1.8.4
>    # rm -f var/lib/sss/db/*
>    # grep enumerate /etc/sssd/sssd.conf
>    enumerate = FALSE
>    # grep ldap_access /etc/sssd/sssd.conf
>    ldap_access_filter =
> memberOf=cn=xxxgroup,ou=yyyOU,ou=zzzOU,ou=Groups,dc=aaa,dc=bbb,dc=ccc
>    # sbin/sssd -c /etc/sssd/sssd.conf
>    # su - myuser -c "groups | wc"
>          1     193    1181
>    # strings var/lib/sss/db/cache_AAA.BBB.CCC.ldb | grep OU=Groups,DC=aaa,DC=bbb,DC=ccc |
> sort -u | wc -l
>    522
>    # strings var/lib/sss/db/cache_AAA.BBB.CCC.ldb | grep OU=Accounts,DC=aaa,DC=bbb,DC=ccc
> | sort -u | wc -l
>    1938
>  
>  Sorry for my use of strings and sort -u, I don't know a better way to interrogate the
> cache.
>  
>  Why does it still enumerate so many users and groups (that are not me, and not in my
> ldap_access_filter) when I log in?  Even when
>  I have disabled domain enumeration? 

As Jakub suggested, your investigation is slightly flawed. I'm guessing
your version of "simulating an initgroups" is by running 'id username'.
This is actually different from initgroups. What this does is an
initgroups() call followed by a loop to look up every group that the
user is a member of.

The net effect of this is that by doing this, we're also doing a lookup
of all the users in those groups (we don't have a choice in this,
because RFC2307bis servers can have other groups as a member and we
cannot know which we're dealing with until we request it).

Most of what you're seeing in the 'strings' are actually what we call
"fake" users though. They're users we've saved a minimal set of data
about in the cache so that we can maintain our member/memberOf linkages
so that group lookups work properly.

Also, you get many more results when grepping on the search base there
because it will appear in multiple places among groups and users. Groups
will have it listed as the 'originalDN' attribute, but every user that
is a member of those groups will also have an 'originalMemberOf' entry
pointing at that subtree as well. This is also the reason you're seeing
groups that your initial user doesn't belong to. It's showing up in the
originalMemberOf attribue of these fake users.

So if you want to get a better view of what is happening, use the
ldb-tools as Jakub said. Also, if you want to test JUST initgroups(),
use the 'id -G' command instead of the bare 'id' command. This will skip
the individual group lookups and save you a lot of trouble.

[GOLLSCHEWSKY, Tim]

Hi Jakub and Stephen,

Thanks for your responses.

Stephen wrote:
> > Why does it still enumerate so many users and groups (that are
> not
> > me, and not in my ldap_access_filter) when I log in?  Even when I have disabled
> domain enumeration?
>
>
> As Jakub suggested, your investigation is slightly flawed. I'm guessing your version
> of "simulating an initgroups" is by running 'id username'.
> This is actually different from initgroups. What this does is an
> initgroups() call followed by a loop to look up every group that the user is a member of.
>  
OK, thanks. I see now I've been using the wrong method to test this.

You guys have suggested "id -G" or actually logging in.  "id -G"
doesnt seem to have the same effect on SSSD that actually logging in has.  "id
-G" returns quite quickly, whereas a normal ssh login takes a very long time as SSSD
goes nuts performing user lookups on our AD (for hundreds of users that aren't even
logging into the server at the time).

I'll ensure all my testing from now on is an actual ssh login and not the
"groups" command.

> The net effect of this is that by doing this, we're also doing a
> lookup of all the users in those groups (we don't have a choice in this, because
> RFC2307bis servers can have other groups as a member and we cannot know which we're
> dealing with until we request it).
>  
Yes this is what I'm seeing.  But is this also the case when I have disabled
nesting?

  # grep nesting /etc/sssd/sssd.conf
  ldap_group_nesting_level = 0

All our LDAP groups in the Unix OU have been flattened specifically to negate the need for
nested searching and (hopefully) increase performance.

> Most of what you're seeing in the 'strings' are actually
> what we call "fake" users though. They're users we've saved a minimal
> set of data about in the cache so that we can maintain our member/memberOf linkages so
> that group lookups work properly.
>  
OK thanks, that makes sense.  So when you get the results from a group search, you create
"fake" entries for those users.

Can I ask why SSSD still performs an LDAP lookup for each of these "fake" users?
Since none of these users are actually logging in at the time, it seems
to penalise the user who is logging in because sshd(& pam) blocks waiting for a
response from SSSD.

I'm just installed ldb-tools (this is going to make interrogating the cache much
easier!) and I can see fullname and gecos information for almost 2000 users after I clear
the cache log in with my login user.

  # pkill sssd
  # rm -f var/lib/sss/db/*
  # sbin/sssd -c /etc/sssd/sssd.conf
  # ssh myuser@0 date
  Fri Jun  1 10:48:56 EST 2012
  # ldbsearch -H var/lib/sss/db/cache_AAA.BBB.CCC.ldb 'objectclass=user' | grep
^gecos | wc -l
  asq: Unable to register control with rootdse!
  1937

I can confirm SSSD is doing ldap user lookups for all the "fake" users by
turning on debug.

This goes back to my original question.  When I have enumerate disabled, why does SSSD
still look up all this fake user information in LDAP?  And is there any way to disable it
from doing so?

Thanks for your help.

Tim.

________________________________

This e-mail is sent by Suncorp Group Limited ABN 66 145 290 124 or one of its related
entities "Suncorp".
Suncorp may be contacted at Level 18, 36 Wickham Terrace, Brisbane or on 13 11 55 or at
suncorp.com.au.
The content of this e-mail is the view of the sender or stated author and does not
necessarily reflect the view of Suncorp. The content, including attachments, is a
confidential communication between Suncorp and the intended recipient. If you are not the
intended recipient, any use, interference with, disclosure or copying of this e-mail,
including attachments, is unauthorised and expressly prohibited. If you have received this
e-mail in error please contact the sender immediately and delete the e-mail and any
attachments from your system.

[Jakub Hrozek]

On Fri, Jun 01, 2012 at 12:53:36AM +0000, GOLLSCHEWSKY, Tim wrote:
>  Hi Jakub and Stephen,
>  
>  Thanks for your responses.
>  
>  Stephen wrote:
>  >> Why does it still enumerate so many users and groups (that are not
>  >> me, and not in my ldap_access_filter) when I log in?  Even when I have disabled
> domain enumeration?
>  >
>  >
>  >As Jakub suggested, your investigation is slightly flawed. I'm guessing your
> version of "simulating an initgroups" is by running 'id username'.
>  >This is actually different from initgroups. What this does is an
>  >initgroups() call followed by a loop to look up every group that the user is a member
> of.
>  
>  OK, thanks. I see now I've been using the wrong method to test this.
>  
>  You guys have suggested "id -G" or actually logging in.  "id -G"
> doesnt seem to have the same effect on SSSD that actually logging in has.  "id
> -G" returns quite quickly, whereas a normal ssh login takes a very long time as SSSD
> goes nuts performing user lookups on our AD (for hundreds of users that aren't even
> logging into the server at the time).
>  
>  I'll ensure all my testing from now on is an actual ssh login and not the
> "groups" command. 
I suspect the reason is that SSSD tries to be smart here and
differentiates between initgroups operation performed via the NSS
interface (id -G) and initgroups performed during auth via the PAM
stack.

The initgroups that comes from the NSS can be returned from the cache
for speed's sake. On the other hand, the PAM responder only keeps a very
short-lived in-memory cache - see the pam_id_timeout option in sssd.conf,
defaults to 5 seconds. Its purpose is mainly to only contact the server
once even if the login program performs multiple initgroups calls during
login - SSH is one example, it performs several initgroups operations during
various stages of the PAM conversation and even from different processes.

>  
>  >The net effect of this is that by doing this, we're also doing a lookup of all
> the users in those groups (we don't have a choice in this, because RFC2307bis servers
> can have other groups as a member and we cannot know which we're dealing with until we
> request it).
>  
>  Yes this is what I'm seeing.  But is this also the case when I have disabled
> nesting?
>  
>    # grep nesting /etc/sssd/sssd.conf
>    ldap_group_nesting_level = 0
>  
>  All our LDAP groups in the Unix OU have been flattened specifically to negate the need
> for nested searching and (hopefully) increase performance. 
Still, the SSSD needs to check the DNs to see if the DN points to a user
entry or not. In theory, the group members can be other objects than
users.

*Maybe* we could do more heuristics if the group and user base DNs are
specified and don't overlap, something similar to what's proposed in
https://fedorahosted.org/sssd/ticket/1319 ?

>  
>  >Most of what you're seeing in the 'strings' are actually what we call
> "fake" users though. They're users we've saved a minimal set of data
> about in the cache so that we can maintain our member/memberOf linkages so that group
> lookups work properly.
>  
>  OK thanks, that makes sense.  So when you get the results from a group search, you create
> "fake" entries for those users.
>  
>  Can I ask why SSSD still performs an LDAP lookup for each of these "fake"
> users? Since none of these users are actually logging in at the time, it seems
>  to penalise the user who is logging in because sshd(& pam) blocks waiting for a
> response from SSSD.
>  
>  I'm just installed ldb-tools (this is going to make interrogating the cache much
> easier!) and I can see fullname and gecos information for almost 2000 users after I clear
> the cache log in with my login user.
>  
>    # pkill sssd
>    # rm -f var/lib/sss/db/*
>    # sbin/sssd -c /etc/sssd/sssd.conf
>    # ssh myuser@0 date
>    Fri Jun  1 10:48:56 EST 2012
>    # ldbsearch -H var/lib/sss/db/cache_AAA.BBB.CCC.ldb 'objectclass=user' | grep
> ^gecos | wc -l
>    asq: Unable to register control with rootdse!
>    1937
>  
>  I can confirm SSSD is doing ldap user lookups for all the "fake" users by
> turning on debug.
>  
>  This goes back to my original question.  When I have enumerate disabled, why does SSSD
> still look up all this fake user information in LDAP?  And is there any way to disable it
> from doing so?
>  
>  Thanks for your help.
>  
>  Tim. 
Hm, this is interesting. I'm not seeing this in my test setup -- I
tested with the 1.8 branch, with ldap_group_nesting_level=0 against a
RFC2307bis server. Would you mind sharing a snippet from your logs?

[Dmitri Pal]

On 06/01/2012 03:11 AM, Jakub Hrozek wrote:
>  On Fri, Jun 01, 2012 at 12:53:36AM +0000, GOLLSCHEWSKY, Tim wrote:
> > Hi Jakub and Stephen,
> >
> > Thanks for your responses.
> >
> > Stephen wrote:
> >>> Why does it still enumerate so many users and groups (that are not
> >>> me, and not in my ldap_access_filter) when I log in?  Even when I have
> disabled domain enumeration?
> >>
> >> As Jakub suggested, your investigation is slightly flawed. I'm guessing your
> version of "simulating an initgroups" is by running 'id username'.
> >> This is actually different from initgroups. What this does is an
> >> initgroups() call followed by a loop to look up every group that the user is a
> member of.
> > OK, thanks. I see now I've been using the wrong method to test this.
> >
> > You guys have suggested "id -G" or actually logging in.  "id -G"
> doesnt seem to have the same effect on SSSD that actually logging in has.  "id
> -G" returns quite quickly, whereas a normal ssh login takes a very long time as SSSD
> goes nuts performing user lookups on our AD (for hundreds of users that aren't even
> logging into the server at the time).
> >
> > I'll ensure all my testing from now on is an actual ssh login and not the
> "groups" command.
>  I suspect the reason is that SSSD tries to be smart here and
>  differentiates between initgroups operation performed via the NSS
>  interface (id -G) and initgroups performed during auth via the PAM
>  stack.
>
>  The initgroups that comes from the NSS can be returned from the cache
>  for speed's sake. On the other hand, the PAM responder only keeps a very
>  short-lived in-memory cache - see the pam_id_timeout option in sssd.conf,
>  defaults to 5 seconds. Its purpose is mainly to only contact the server
>  once even if the login program performs multiple initgroups calls during
>  login - SSH is one example, it performs several initgroups operations during
>  various stages of the PAM conversation and even from different processes.
>
> >> The net effect of this is that by doing this, we're also doing a lookup of
> all the users in those groups (we don't have a choice in this, because RFC2307bis
> servers can have other groups as a member and we cannot know which we're dealing with
> until we request it).
> > Yes this is what I'm seeing.  But is this also the case when I have disabled
> nesting?
> >
> >   # grep nesting /etc/sssd/sssd.conf
> >   ldap_group_nesting_level = 0
> >
> > All our LDAP groups in the Unix OU have been flattened specifically to negate the
> need for nested searching and (hopefully) increase performance.
>  Still, the SSSD needs to check the DNs to see if the DN points to a user
>  entry or not. In theory, the group members can be other objects than
>  users.
>
>  *Maybe* we could do more heuristics if the group and user base DNs are
>  specified and don't overlap, something similar to what's proposed in
>  https://fedorahosted.org/sssd/ticket/1319 ?
>  
Are you saying that no when we get a set of the DNs in the memberOf we
do not know whether the entry is a user or a group so we have to fetch
entries to see if the entry is a user or a group?
IMO with there should be a way to regex it in SSSD and fetch only in
cases when it is not possible to determine if the member is a user or a
group. Use of the regex will help with other 2307 bis servers not just
IPA. We need to think about AD too.
> >> Most of what you're seeing in the 'strings' are
> actually what we call "fake" users though. They're users we've saved a
> minimal set of data about in the cache so that we can maintain our member/memberOf
> linkages so that group lookups work properly.
> > OK thanks, that makes sense.  So when you get the results from a group search, you
> create "fake" entries for those users.
> >
> > Can I ask why SSSD still performs an LDAP lookup for each of these "fake"
> users? Since none of these users are actually logging in at the time, it seems
> > to penalise the user who is logging in because sshd(& pam) blocks waiting for a
> response from SSSD.
> >
> > I'm just installed ldb-tools (this is going to make interrogating the cache much
> easier!) and I can see fullname and gecos information for almost 2000 users after I clear
> the cache log in with my login user.
> >
> >   # pkill sssd
> >   # rm -f var/lib/sss/db/*
> >   # sbin/sssd -c /etc/sssd/sssd.conf
> >   # ssh myuser@0 date
> >   Fri Jun  1 10:48:56 EST 2012
> >   # ldbsearch -H var/lib/sss/db/cache_AAA.BBB.CCC.ldb 'objectclass=user' |
> grep ^gecos | wc -l
> >   asq: Unable to register control with rootdse!
> >   1937
> >
> > I can confirm SSSD is doing ldap user lookups for all the "fake" users by
> turning on debug.
> >
> > This goes back to my original question.  When I have enumerate disabled, why does
> SSSD still look up all this fake user information in LDAP?  And is there any way to
> disable it from doing so?
> >
> > Thanks for your help.
> >
> > Tim.
>  Hm, this is interesting. I'm not seeing this in my test setup -- I
>  tested with the 1.8 branch, with ldap_group_nesting_level=0 against a
>  RFC2307bis server. Would you mind sharing a snippet from your logs?
>  _______________________________________________
>  sssd-users mailing list
>  sssd-users(a)lists.fedorahosted.org
>  https://fedorahosted.org/mailman/listinfo/sssd-users 

-- 
Thank you,
Dmitri Pal

Sr. Engineering Manager IPA project,
Red Hat Inc.


-------------------------------
Looking to carve out IT costs?
www.redhat.com/carveoutcosts/

[Simo Sorce]

On Sat, 2012-06-02 at 14:28 -0400, Dmitri Pal wrote:
>  On 06/01/2012 03:11 AM, Jakub Hrozek wrote:
>  > On Fri, Jun 01, 2012 at 12:53:36AM +0000, GOLLSCHEWSKY, Tim wrote:
>  >> Hi Jakub and Stephen,
>  >>
>  >> Thanks for your responses.
>  >>
>  >> Stephen wrote:
>  >>>> Why does it still enumerate so many users and groups (that are not
>  >>>> me, and not in my ldap_access_filter) when I log in?  Even when I have
> disabled domain enumeration?
>  >>>
>  >>> As Jakub suggested, your investigation is slightly flawed. I'm guessing
> your version of "simulating an initgroups" is by running 'id username'.
>  >>> This is actually different from initgroups. What this does is an
>  >>> initgroups() call followed by a loop to look up every group that the user is
> a member of.
>  >> OK, thanks. I see now I've been using the wrong method to test this.
>  >>
>  >> You guys have suggested "id -G" or actually logging in.  "id
> -G" doesnt seem to have the same effect on SSSD that actually logging in has. 
> "id -G" returns quite quickly, whereas a normal ssh login takes a very long time
> as SSSD goes nuts performing user lookups on our AD (for hundreds of users that aren't
> even logging into the server at the time).
>  >>
>  >> I'll ensure all my testing from now on is an actual ssh login and not the
> "groups" command.
>  > I suspect the reason is that SSSD tries to be smart here and
>  > differentiates between initgroups operation performed via the NSS
>  > interface (id -G) and initgroups performed during auth via the PAM
>  > stack.
>  >
>  > The initgroups that comes from the NSS can be returned from the cache
>  > for speed's sake. On the other hand, the PAM responder only keeps a very
>  > short-lived in-memory cache - see the pam_id_timeout option in sssd.conf,
>  > defaults to 5 seconds. Its purpose is mainly to only contact the server
>  > once even if the login program performs multiple initgroups calls during
>  > login - SSH is one example, it performs several initgroups operations during
>  > various stages of the PAM conversation and even from different processes.
>  >
>  >>> The net effect of this is that by doing this, we're also doing a lookup
> of all the users in those groups (we don't have a choice in this, because RFC2307bis
> servers can have other groups as a member and we cannot know which we're dealing with
> until we request it).
>  >> Yes this is what I'm seeing.  But is this also the case when I have disabled
> nesting?
>  >>
>  >>   # grep nesting /etc/sssd/sssd.conf
>  >>   ldap_group_nesting_level = 0
>  >>
>  >> All our LDAP groups in the Unix OU have been flattened specifically to negate
> the need for nested searching and (hopefully) increase performance.
>  > Still, the SSSD needs to check the DNs to see if the DN points to a user
>  > entry or not. In theory, the group members can be other objects than
>  > users.
>  >
>  > *Maybe* we could do more heuristics if the group and user base DNs are
>  > specified and don't overlap, something similar to what's proposed in
>  > https://fedorahosted.org/sssd/ticket/1319 ?
>  >
>  
>  Are you saying that no when we get a set of the DNs in the memberOf we
>  do not know whether the entry is a user or a group so we have to fetch
>  entries to see if the entry is a user or a group? 
Yes, that's the case, especially with AD, as Users and groups can be
stored in the same OU, and there is nothing in the DN that can tell you
what is what.

>  IMO with there should be a way to regex it in SSSD and fetch only in
>  cases when it is not possible to determine if the member is a user or a
>  group. Use of the regex will help with other 2307 bis servers not just
>  IPA. We need to think about AD too. 
We can do it for IPA, as long as we force a scheme where we have
something in the DN that makes the distinction clear (right now we have
both the RDN and part of the DN to hint us, we just need to implement
the code to read the smarts instead of blindly searching the LDAp
Directory).
Unfortunately nothing in AD allows you to do that. However, we should
probably smarten up SSSD and make it check the *local* database first.
We keep the originalDN attribute so we can search locally to find out if
we already know what is the object.

We should also always use the MS-PAC for initgroups when using AD (if
Kerberos auth is used), and not search the LDAP directory until some app
really asks a for a getgrgid() op, IMO.

Simo.

-- 
Simo Sorce * Red Hat, Inc * New York

[Jakub Hrozek]

On Sat, Jun 02, 2012 at 02:28:57PM -0400, Dmitri Pal wrote:
>  On 06/01/2012 03:11 AM, Jakub Hrozek wrote:
>  > On Fri, Jun 01, 2012 at 12:53:36AM +0000, GOLLSCHEWSKY, Tim wrote:
>  >> Hi Jakub and Stephen,
>  >>
>  >> Thanks for your responses.
>  >>
>  >> Stephen wrote:
>  >>>> Why does it still enumerate so many users and groups (that are not
>  >>>> me, and not in my ldap_access_filter) when I log in?  Even when I have
> disabled domain enumeration?
>  >>>
>  >>> As Jakub suggested, your investigation is slightly flawed. I'm guessing
> your version of "simulating an initgroups" is by running 'id username'.
>  >>> This is actually different from initgroups. What this does is an
>  >>> initgroups() call followed by a loop to look up every group that the user is
> a member of.
>  >> OK, thanks. I see now I've been using the wrong method to test this.
>  >>
>  >> You guys have suggested "id -G" or actually logging in.  "id
> -G" doesnt seem to have the same effect on SSSD that actually logging in has. 
> "id -G" returns quite quickly, whereas a normal ssh login takes a very long time
> as SSSD goes nuts performing user lookups on our AD (for hundreds of users that aren't
> even logging into the server at the time).
>  >>
>  >> I'll ensure all my testing from now on is an actual ssh login and not the
> "groups" command.
>  > I suspect the reason is that SSSD tries to be smart here and
>  > differentiates between initgroups operation performed via the NSS
>  > interface (id -G) and initgroups performed during auth via the PAM
>  > stack.
>  >
>  > The initgroups that comes from the NSS can be returned from the cache
>  > for speed's sake. On the other hand, the PAM responder only keeps a very
>  > short-lived in-memory cache - see the pam_id_timeout option in sssd.conf,
>  > defaults to 5 seconds. Its purpose is mainly to only contact the server
>  > once even if the login program performs multiple initgroups calls during
>  > login - SSH is one example, it performs several initgroups operations during
>  > various stages of the PAM conversation and even from different processes.
>  >
>  >>> The net effect of this is that by doing this, we're also doing a lookup
> of all the users in those groups (we don't have a choice in this, because RFC2307bis
> servers can have other groups as a member and we cannot know which we're dealing with
> until we request it).
>  >> Yes this is what I'm seeing.  But is this also the case when I have disabled
> nesting?
>  >>
>  >>   # grep nesting /etc/sssd/sssd.conf
>  >>   ldap_group_nesting_level = 0
>  >>
>  >> All our LDAP groups in the Unix OU have been flattened specifically to negate
> the need for nested searching and (hopefully) increase performance.
>  > Still, the SSSD needs to check the DNs to see if the DN points to a user
>  > entry or not. In theory, the group members can be other objects than
>  > users.
>  >
>  > *Maybe* we could do more heuristics if the group and user base DNs are
>  > specified and don't overlap, something similar to what's proposed in
>  > https://fedorahosted.org/sssd/ticket/1319 ?
>  >
>  
>  Are you saying that no when we get a set of the DNs in the memberOf we
>  do not know whether the entry is a user or a group so we have to fetch
>  entries to see if the entry is a user or a group?
>  IMO with there should be a way to regex it in SSSD and fetch only in
>  cases when it is not possible to determine if the member is a user or a
>  group. Use of the regex will help with other 2307 bis servers not just
>  IPA. We need to think about AD too. 
That's only 100% doable with tightly-controlled environments such as IPA
where you know for certain that a particular DN points to user or group
containers.

In the generic case, the DN might simply look like
"cn=name,ou=accounts,dc=example,dc=com" and then only a lookup can tell
if it's a user or a group.

What I was telling Tim earlier was that we might want to do the
LDAP-search based heuristics also in cases when the group and user
containers are distinct, such as "ou=Users,dc=example,dc=com" and
"ou=Groups,dc=example,dc=com".

[GOLLSCHEWSKY, Tim]

>  In the generic case, the DN might simply look like
> "cn=name,ou=accounts,dc=example,dc=com" and then only a lookup can tell if
> it's a user or a group.
>  What I was telling Tim earlier was that we might want to do the LDAP-search based
> heuristics also in cases when the group and user containers are distinct, > such as
> "ou=Users,dc=example,dc=com" and "ou=Groups,dc=example,dc=com".
>  
Just to add to this, I think a lot of organisations with largeish AD directories would
have their set up like this (Accounts and Groups in separate OUs).  It's just too
messy to do it otherwise.

It sounds like this happens by default with IPA, if there was some way we could enable
this for AD as well (perhaps with a config option that specified the OUs for Accounts and
Groups), this would save SSSD from doing the ldap searches just to distinguish if the
record is a user or group, and would be a huge win for performance.

Cheers,
Tim.

________________________________

This e-mail is sent by Suncorp Group Limited ABN 66 145 290 124 or one of its related
entities "Suncorp".
Suncorp may be contacted at Level 18, 36 Wickham Terrace, Brisbane or on 13 11 55 or at
suncorp.com.au.
The content of this e-mail is the view of the sender or stated author and does not
necessarily reflect the view of Suncorp. The content, including attachments, is a
confidential communication between Suncorp and the intended recipient. If you are not the
intended recipient, any use, interference with, disclosure or copying of this e-mail,
including attachments, is unauthorised and expressly prohibited. If you have received this
e-mail in error please contact the sender immediately and delete the e-mail and any
attachments from your system.

[Stephen Gallagher]

On Tue, 2012-06-05 at 03:21 +0000, GOLLSCHEWSKY, Tim wrote:
>  > In the generic case, the DN might simply look like
>  "cn=name,ou=accounts,dc=example,dc=com" and then only a lookup can tell
>  if it's a user or a group. > What I was telling Tim earlier was that we
>  might want to do the LDAP-search based heuristics also in cases when
>  the group and user containers are distinct, > such as
>  "ou=Users,dc=example,dc=com" and "ou=Groups,dc=example,dc=com".
>  
>  Just to add to this, I think a lot of organisations with largeish AD
>  directories would have their set up like this (Accounts and Groups in
>  separate OUs).  It's just too messy to do it otherwise.
>  
>  It sounds like this happens by default with IPA, if there was some way
>  we could enable this for AD as well (perhaps with a config option that
>  specified the OUs for Accounts and Groups), this would save SSSD from
>  doing the ldap searches just to distinguish if the record is a user or
>  group, and would be a huge win for performance. 
Well, as Jakub suggested, one thing that might be possible is to examine
the user and group search bases[*] to see if there is zero overlap. If
this is the case, then we can optimize by checking the member DN against
those bases. However, if there is any overlap (e.g. they are both set to
the generic ldap_search_base) then we have to take the slower approach.

I've opened https://fedorahosted.org/sssd/ticket/1366 to track this
enhancement.

[*] The catch here is that we support multiple search bases for users
and groups, so this means we would need to ensure that there was no
overlap among ANY of them.

[Sigbjorn Lie]

>  The net effect of this is that by doing this, we're also doing a
> lookup
>  of all the users in those groups (we don't have a choice in this, because RFC2307bis
> servers can
>  have other groups as a member and we cannot know which we're dealing with until we
> request it).
>  

Would the information in the link below provide any help for looking up all members of a
group,
and all a user group memberships in a single LDAP lookup?

I used the filter
"(memberof:1.2.840.113556.1.4.1941:=(cn=Group1,OU=groupsOU,DC=x))" to find all
users of a group, including users that we're member of that group through membership
of another
group some time back.

There seem to be a similar filter for looking up all groups the user is a member of in a
single
lookup: "(member:1.2.840.113556.1.4.1941:=(cn=user1,cn=users,DC=x))".


See this link for further reference:
http://msdn.microsoft.com/en-us/library/windows/desktop/aa746475%28v=vs.8...


Rgds,
Siggi

[Simo Sorce]

On Tue, 2012-06-05 at 16:51 +0200, Sigbjorn Lie wrote:
>  > The net effect of this is that by doing this, we're also
> doing a lookup
>  > of all the users in those groups (we don't have a choice in this, because
> RFC2307bis servers can
>  > have other groups as a member and we cannot know which we're dealing with until
> we request it).
>  >
>  
>  
>  Would the information in the link below provide any help for looking up all members of a
> group,
>  and all a user group memberships in a single LDAP lookup?
>  
>  I used the filter
> "(memberof:1.2.840.113556.1.4.1941:=(cn=Group1,OU=groupsOU,DC=x))" to find all
>  users of a group, including users that we're member of that group through membership
> of another
>  group some time back.
>  
>  There seem to be a similar filter for looking up all groups the user is a member of in a
> single
>  lookup: "(member:1.2.840.113556.1.4.1941:=(cn=user1,cn=users,DC=x))".
>  
>  
>  See this link for further reference:
>  http://msdn.microsoft.com/en-us/library/windows/desktop/aa746475%28v=vs.8...
Thanks a lot for uncovering this Siggi, it will definitely be useful.

Simo.

-- 
Simo Sorce * Red Hat, Inc * New York

[Stephen Gallagher]

On Tue, 2012-06-05 at 11:14 -0400, Simo Sorce wrote:
>  On Tue, 2012-06-05 at 16:51 +0200, Sigbjorn Lie wrote:
>  > > The net effect of this is that by doing this, we're also doing a lookup
>  > > of all the users in those groups (we don't have a choice in this, because
> RFC2307bis servers can
>  > > have other groups as a member and we cannot know which we're dealing with
> until we request it).
>  > >
>  > 
>  > 
>  > Would the information in the link below provide any help for looking up all members
> of a group,
>  > and all a user group memberships in a single LDAP lookup?
>  > 
>  > I used the filter
> "(memberof:1.2.840.113556.1.4.1941:=(cn=Group1,OU=groupsOU,DC=x))" to find all
>  > users of a group, including users that we're member of that group through
> membership of another
>  > group some time back.
>  > 
>  > There seem to be a similar filter for looking up all groups the user is a member of
> in a single
>  > lookup: "(member:1.2.840.113556.1.4.1941:=(cn=user1,cn=users,DC=x))".
>  > 
>  > 
>  > See this link for further reference:
>  > http://msdn.microsoft.com/en-us/library/windows/desktop/aa746475%28v=vs.8...
>  
>  Thanks a lot for uncovering this Siggi, it will definitely be useful. 

I agree, this will be very handy. I've opened
https://fedorahosted.org/sssd/ticket/1367 to track its inclusion. I'll
be proposing this for 1.9.0 beta 3, I think.