missing secondary groups that have Global scope

[Jeffrey Chung]

Hello all.  We’re noticing an issue where at times the id command does not return a complete list of the user’s secondary groups. In our Linux environment we use both Universal and Global groups and it’s always only the Global groups that are missing.  From troubleshooting we’re certain this issue comes up when a Global Catalog that is not in the same domain of the missing Global groups is used to query for group membership.  From my understanding, in Active Directory only Universal group memberships are replicated to every Global Catalog.

We have a single forest with multiple domains.  The forest root domain is empty and only use for administration.  We have a child domain for resource like users, groups and computers.

contoso.com – empty forest root 
corp.contoso.com – child domain of root, users, groups and computer objects live here.  SSSD Linux are joined to this domain.

Every Domain Controller in the forest are Global Catalogs, so we have GCs in both contoso.com and corp.contoso.com.

Here’s an example of when secondary groups are missing.
We have a user [e-mail] and is a direct member of the following corp.contoso.com groups.
group1 (Universal scope)
group2 (Global scope)

On the SSSD host we run the following command.
% id john

Only john’s primary group and group1 are returned.  From the SSSD debug logs, we only see this happen when the query is hitting a Global Catalog (port 3268) in the root domain contoso.com.  If the query hits a GC in corp.contoso.com the correct groups are returned.

The LDAP filter we see from the logs is this.
(&(member=CN=john,CN=Users,DC=corp,DC=contoso,DC=com)(objectClass=posixgroup)(cn=*))

Which will not return Global groups because the Global Catalog Domain Controllers in contoso.com only have membership data for Universal groups.

We noticed this issue on sssd-1.16.1, we have hosts running an older version sssd-1.11.5.1 that don’t have this issue because the group membership query is using the LDAP port 389 not GC port 3268. On both versions we have “ad_enable_gc” set to true.

Is this a bug we’re hitting, or we have SSSD configured incorrectly?  Should SSSD be group scope aware and handle this situation correctly?

We can think of a couple work-arounds, but rather not implement it if possible.
1. Set “ad_enable_gc” to false – this will probably cause some inefficiencies for SSSD
2. Convert the Global groups to Universal – we have a lot of groups and this will take some time to vette out to avoid any negative impact to our environment

Here is an example of our sssd.conf.

[sssd]
config_file_version = 2
debug_level = 9
services = nss,pam
domains = corp.contoso.com

[nss]
debug_level = 9
filter_users = root
filter_groups = root
filter_users_in_groups = false
timeout = 60

[pam]
debug_level = 9
reconnection_retries = 3
timeout = 60

[domain/corp.contoso.com]
ad_gpo_access_control = disabled
debug_level = 9
ldap_user_object_class = posixaccount
ldap_user_home_directory = unixhomedirectory
ldap_group_object_class = posixgroup
ldap_group_name = cn
ldap_group_member = member
ldap_group_nesting_level = 0
ldap_use_tokengroups = false
cache_credentials = true
id_provider = ad
auth_provider = ad
chpass_provider = ad
access_provider = ad
dns_discovery_domain = corp.contoso.com
dyndns_update = false
ldap_id_mapping = false
ad_enable_gc = true
ldap_search_base = dc=corp,dc=contoso,dc=com?subtree?
ad_maximum_machine_account_password_age = 0
use_fully_qualified_names = false
krb5_ccachedir = /tmp
krb5_ccname_template = FILE:%d/krb5cc_%U
krb5_renew_interval = 60m
timeout = 60

Thanks in advance

-Jeff

[Sumit Bose]

Am Tue, Dec 20, 2022 at 06:55:58PM -0000 schrieb Jeffrey Chung:
> Hello all.  We’re noticing an issue where at times the id command does not return a complete list of the user’s secondary groups. In our Linux environment we use both Universal and Global groups and it’s always only the Global groups that are missing.  From troubleshooting we’re certain this issue comes up when a Global Catalog that is not in the same domain of the missing Global groups is used to query for group membership.  From my understanding, in Active Directory only Universal group memberships are replicated to every Global Catalog.
>
> We have a single forest with multiple domains.  The forest root domain is empty and only use for administration.  We have a child domain for resource like users, groups and computers.

Hi,

is there a reason you are using 'ldap_use_tokengroups = false'? Have you
tried if 'ldap_use_tokengroups = true' is working more reliable?

bye,
Sumit
> contoso.com – empty forest root 
> corp.contoso.com – child domain of root, users, groups and computer objects live here.  SSSD Linux are joined to this domain.
>
> Every Domain Controller in the forest are Global Catalogs, so we have GCs in both contoso.com and corp.contoso.com.
>
> Here’s an example of when secondary groups are missing.
> We have a user [e-mail] and is a direct member of the following corp.contoso.com groups.
> group1 (Universal scope)
> group2 (Global scope)
>
> On the SSSD host we run the following command.
> % id john
>
> Only john’s primary group and group1 are returned.  From the SSSD debug logs, we only see this happen when the query is hitting a Global Catalog (port 3268) in the root domain contoso.com.  If the query hits a GC in corp.contoso.com the correct groups are returned.
>
> The LDAP filter we see from the logs is this.
> (&(member=CN=john,CN=Users,DC=corp,DC=contoso,DC=com)(objectClass=posixgroup)(cn=*))
>
> Which will not return Global groups because the Global Catalog Domain Controllers in contoso.com only have membership data for Universal groups.
>
> We noticed this issue on sssd-1.16.1, we have hosts running an older version sssd-1.11.5.1 that don’t have this issue because the group membership query is using the LDAP port 389 not GC port 3268. On both versions we have “ad_enable_gc” set to true.
>
> Is this a bug we’re hitting, or we have SSSD configured incorrectly?  Should SSSD be group scope aware and handle this situation correctly?
>
> We can think of a couple work-arounds, but rather not implement it if possible.
> Set “ad_enable_gc” to false – this will probably cause some inefficiencies for SSSD
> Convert the Global groups to Universal – we have a lot of groups and this will take some time to vette out to avoid any negative impact to our environment
>
> Here is an example of our sssd.conf.
>
> [sssd]
> config_file_version = 2
> debug_level = 9
> services = nss,pam
> domains = corp.contoso.com
>
> [nss]
> debug_level = 9
> filter_users = root
> filter_groups = root
> filter_users_in_groups = false
> timeout = 60
>
> [pam]
> debug_level = 9
> reconnection_retries = 3
> timeout = 60
>
> [domain/corp.contoso.com]
> ad_gpo_access_control = disabled
> debug_level = 9
> ldap_user_object_class = posixaccount
> ldap_user_home_directory = unixhomedirectory
> ldap_group_object_class = posixgroup
> ldap_group_name = cn
> ldap_group_member = member
> ldap_group_nesting_level = 0
> ldap_use_tokengroups = false
> cache_credentials = true
> id_provider = ad
> auth_provider = ad
> chpass_provider = ad
> access_provider = ad
> dns_discovery_domain = corp.contoso.com
> dyndns_update = false
> ldap_id_mapping = false
> ad_enable_gc = true
> ldap_search_base = dc=corp,dc=contoso,dc=com?subtree?
> ad_maximum_machine_account_password_age = 0
> use_fully_qualified_names = false
> krb5_ccachedir = /tmp
> krb5_ccname_template = FILE:%d/krb5cc_%U
> krb5_renew_interval = 60m
> timeout = 60
>
> Thanks in advance
>
> -Jeff
> _______________________________________________
> sssd-users mailing list -- [e-mail]
> To unsubscribe send an email to [e-mail]
> Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives: https://lists.fedorahosted.org/archives/list/[e-mail].o...
> Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue

[Jeffrey Chung]

The primary reason we disabled tokenGroups is because our sssd logs were filling up with 'Unable to resolve SID S-1-5-21-XXX-XXX-XXX-XXX - will try next sid.' entries.  We found a work-around from this doc.
https://pagure.io/SSSD/sssd/issue/2914

In our environment not all of our AD groups are POSIX enabled, so I think that's why we see a lot of those log entries.

I just tested enabling tokenGroups and that seem to have solved the issue.  I'm seeing the LDAP query (port 389) going to a domain controller from the same domain as the user.

Is enabling tokenGroups the recommended configuration when using the AD provider? The one thing I read is querying for tokenGroups is an expensive operation on the domain controllers and care should be taken when scaling this to larger environments. 
https://learn.microsoft.com/en-us/windows/win32/adschema/a-tokengroups
Any insight into this?  Is SSSD more efficient with tokenGroups enabled versus not?

-Jeff

[Spike White]

Jeffrey,

I'm told that the alternative to tokengroups would be recursive LDAP
queries.  Which would be expensive for the clients, particularly with
heavily-nested subgroups.

Prior to us using tokengroups, we tried to limit the cost of these LDAP
queries by limiting the LDAP query depth to false.

Interestingly,  we also formerly used to have problems with global AD group
memberships not being properly reflected.  Our solution was to promote any
such global groups up to universal groups and then the membership was seen.

But for performance reasons, we started using tokengroups.  We haven't seen
this global group problem in a while.  Interesting (and totally obscure!!)
that it's related to use of tokengroups.

BTW, if tokengroups is an expensive operation on AD DCs, seems like your AD
team needs to size your AD DCs appropriately.  Doesn't seem reasonable to
throttle clients, due to some DC hardware.

Spike

On Wed, Dec 21, 2022 at 5:30 PM Jeffrey Chung  wrote:
> The primary reason we disabled tokenGroups is because our sssd logs were
> filling up with 'Unable to resolve SID S-1-5-21-XXX-XXX-XXX-XXX - will try
> next sid.' entries.  We found a work-around from this doc.
> https://pagure.io/SSSD/sssd/issue/2914
>
> In our environment not all of our AD groups are POSIX enabled, so I think
> that's why we see a lot of those log entries.
>
> I just tested enabling tokenGroups and that seem to have solved the
> issue.  I'm seeing the LDAP query (port 389) going to a domain controller
> from the same domain as the user.
>
> Is enabling tokenGroups the recommended configuration when using the AD
> provider? The one thing I read is querying for tokenGroups is an expensive
> operation on the domain controllers and care should be taken when scaling
> this to larger environments.
> https://learn.microsoft.com/en-us/windows/win32/adschema/a-tokengroups
> Any insight into this?  Is SSSD more efficient with tokenGroups enabled
> versus not?
>
> -Jeff
> _______________________________________________
> sssd-users mailing list -- [e-mail]
> To unsubscribe send an email to [e-mail]
> Fedora Code of Conduct:
> https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives:
> https://lists.fedorahosted.org/archives/list/[e-mail].o...
> Do not reply to spam, report it:
> https://pagure.io/fedora-infrastructure/new_issue

[Jeffrey Chung]

Thanks for the reply Spike.  We will do some performance tests in our AD environment for this.

There are situations where tokenGroups should be disabled to get consistent results like changing the search base for groups.
https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/htm...

In this scenario with tokenGroups disabled we would still hit the same issue in my original post.  To me this seems to be a bug in sssd, it can't rely just on the GC to get back a complete list of groups a user is member of because you'll be missing other group scopes like Global and Domain Local.  Am I thinking about this wrong?

-Jeff

[Spike White]

Jeffrey,

Bear in mind I'm a Linux engineer.  (I speak regularly to our AD team).  As
I understand it, the domain-local memberships are housed in the local
domain, not the GC.

If you look at the output of 'sssctl domain-status <domain>', you will see
it references two DCs that it's bound to.  The local DC and the global
catalog server.

So for domain-local groups, it would consult the first DC and get back the
proper membership.  For universal groups, yes their membership is contained
in the GC, so it'd have to consult that second DC (the global catalog
server).

That leaves only the global groups.  and I think your reasoning is correct
for those.

To be honest, when we were transitioning to sssd from our previous AD
integration tool, we were under a time crunch.  So while we saw (same as
you) that not all global group memberships showed up, we didn't have time
to deep-dive into the actual culprit.  (I suspect it's exactly as you
state, because at that time we were doing 'tokengroups = false').

Since we were under a severe time crunch, we just promoted any global group
that got called out by the users.   We promoted that global group to a
univeral group.

I haven't seen a global group getting called out for missing membership in
over a year.  However, we have transitioned to 'tokengroups = true', so
that's likely why.

Spike

On Thu, Dec 22, 2022 at 2:53 PM Jeffrey Chung  wrote:
> Thanks for the reply Spike.  We will do some performance tests in our AD
> environment for this.
>
> There are situations where tokenGroups should be disabled to get
> consistent results like changing the search base for groups.
>
> https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/htm...
>
> In this scenario with tokenGroups disabled we would still hit the same
> issue in my original post.  To me this seems to be a bug in sssd, it can't
> rely just on the GC to get back a complete list of groups a user is member
> of because you'll be missing other group scopes like Global and Domain
> Local.  Am I thinking about this wrong?
>
> -Jeff
> _______________________________________________
> sssd-users mailing list -- [e-mail]
> To unsubscribe send an email to [e-mail]
> Fedora Code of Conduct:
> https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives:
> https://lists.fedorahosted.org/archives/list/[e-mail].o...
> Do not reply to spam, report it:
> https://pagure.io/fedora-infrastructure/new_issue

[Spike White]

Jeffrey,

Follow-on question.  How does your sssd configuration ever bind to a
nonlocal-domain global catalog server?

Are you relying on sssd's site-awareness to determine preferred AD DCs and
global catalog servers?   Or are you hard-coding global catalog servers in
your sssd.conf file?

If the former,  then I'd beat up on your maintainer for AD site and
services;  they have defined non-local global catalog servers for your site.

Spike

On Thu, Dec 22, 2022 at 4:14 PM Spike White  wrote:
> Jeffrey,
>
> Bear in mind I'm a Linux engineer.  (I speak regularly to our AD team).
> As I understand it, the domain-local memberships are housed in the local
> domain, not the GC.
>
> If you look at the output of 'sssctl domain-status <domain>', you will see
> it references two DCs that it's bound to.  The local DC and the global
> catalog server.
>
> So for domain-local groups, it would consult the first DC and get back the
> proper membership.  For universal groups, yes their membership is contained
> in the GC, so it'd have to consult that second DC (the global catalog
> server).
>
> That leaves only the global groups.  and I think your reasoning is correct
> for those.
>
> To be honest, when we were transitioning to sssd from our previous AD
> integration tool, we were under a time crunch.  So while we saw (same as
> you) that not all global group memberships showed up, we didn't have time
> to deep-dive into the actual culprit.  (I suspect it's exactly as you
> state, because at that time we were doing 'tokengroups = false').
>
> Since we were under a severe time crunch, we just promoted any global
> group that got called out by the users.   We promoted that global group to
> a univeral group.
>
> I haven't seen a global group getting called out for missing membership in
> over a year.  However, we have transitioned to 'tokengroups = true', so
> that's likely why.
>
> Spike
>
> On Thu, Dec 22, 2022 at 2:53 PM Jeffrey Chung  wrote:
> > Thanks for the reply Spike.  We will do some performance tests in our AD
> > environment for this.
> >
> > There are situations where tokenGroups should be disabled to get
> > consistent results like changing the search base for groups.
> >
> > https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/htm...
> >
> > In this scenario with tokenGroups disabled we would still hit the same
> > issue in my original post.  To me this seems to be a bug in sssd, it can't
> > rely just on the GC to get back a complete list of groups a user is member
> > of because you'll be missing other group scopes like Global and Domain
> > Local.  Am I thinking about this wrong?
> >
> > -Jeff
> > _______________________________________________
> > sssd-users mailing list -- [e-mail]
> > To unsubscribe send an email to [e-mail]
> > Fedora Code of Conduct:
> > https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> > List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> > List Archives:
> > https://lists.fedorahosted.org/archives/list/[e-mail].o...
> > Do not reply to spam, report it:
> > https://pagure.io/fedora-infrastructure/new_issue

[Jeffrey Chung]

Yes, we rely on sssd's site-awareness, no hard-coding.  I don't believe we have any misconfiguration in AD sites and services.
For example we have a site name SiteA and within this site we have a DC from the forest root and a DC from the child domain.

SiteA
   Root-DC1 (contoso.com)
   Corp-DC1 (corp.contoso.com)

Both Root-DC1 and Corp-DC1 are GCs in the same forest. I don't see anything wrong with having GCs from multiple domains in the same site, in fact I think this would be more efficient for clients when it need to communicate with forest root DCs.

When sssd does the discovery both Root-DC1 and Corp-DC1 will be returned in the list of available GCs, so there could be a chance sssd uses Root-DC1 for the GC queries and when that happens Global secondary groups will be missing.

We have work-arounds for this issue in our environment like using TokenGroups, but I just wanted to see if we think this is a bug then it should be fixed in future releases.

-Jeff