Is there a way to restrict nss group membership searches for local users to only /etc/group?

[Spike White]

All,

Successful sssd consumer here.

Have an app team running Hadoop.  They're getting these performance errors
in their app.

This is from their app logs.

ddlflhdm201.us.company.com
WARN June 15, 2023 10:08 AM Groups Potential performance problem:
getGroups(user=hbase) took 58022 milliseconds
ddlflhdm304.us.company.com
WARN June 15, 2023 8:38 AM Groups Potential performance problem:
getGroups(user=mapred) took 39962 milliseconds
ddlplhdm505.us.company.com
WARN June 16, 2023 5:37 AM Groups Potential performance problem:
getGroups(user=hdfs) took 62437 milliseconds.
ddlplhdm305.us.company.com
WARN June 15, 2023 9:36 PM Groups Potential performance problem:
getGroups(user=hdfs) took 58017 milliseconds

ddlplhdm505.us.company.com
WARN June 10, 2023 8:01 AM Groups Potential performance problem:
getGroups(user=yarn) took 58017 milliseconds.


Interestingly, all these users are local users created by the Hadoop
package installation.  (hbase, mapred, hdfs, yarn are all local users).


These local users are members only of local groups.  So theoretically the
search for group memberships should be lightning quick as the groups are
found only in /etc/group.


But since my /etc/nsswitch.conf looks like this:


passwd:     files systemd sss

group:      files systemd sss

netgroup:   files sss

automount:  files sss

services:   files

shadow:     files sss

hosts:      files dns myhostname


It not only searches local files for group membership, it searches AD as
well.  And doesn't find it, as there is no such AD user or uid.
 Eventually timing out.


In /var/log/messages we see this around the same time frame as


Jun 15 10:08:09 ddlflhdm201 sssd[be[amer.company.com]]: GSSAPI Error: An
invalid name was supplied (Success)

Is there a way to restrict these particular local app accounts from having
their group memberships looked up in AD?

I.e., my [nss] section looks like this today:

[nss]
#debug_level = 9
filter_groups = root
filter_users = root
#override_homedir = /home/%u

Can I do this?

[nss]
filter_users = root,  hbase, mapred, hdfs, yarn

And would I want to set filter_users_in_groups = false?

Spike White

[Alexey Tikhonov]

Hi,

On Thu, Jun 22, 2023 at 4:47 PM Spike White <spikewhitetx(a)gmail.com&gt; wrote:

>  All,
>
>  Successful sssd consumer here.
>
>  Have an app team running Hadoop.  They're getting these performance errors
>  in their app.
>
>  This is from their app logs.
>
>  ddlflhdm201.us.company.com
>  WARN June 15, 2023 10:08 AM Groups Potential performance problem:
>  getGroups(user=hbase) took 58022 milliseconds
>  
Does it use `getgrouplist()` under the hood?


>  ddlflhdm304.us.company.com
>  WARN June 15, 2023 8:38 AM Groups Potential performance problem:
>  getGroups(user=mapred) took 39962 milliseconds
>  ddlplhdm505.us.company.com
>  WARN June 16, 2023 5:37 AM Groups Potential performance problem:
>  getGroups(user=hdfs) took 62437 milliseconds.
>  ddlplhdm305.us.company.com
>  WARN June 15, 2023 9:36 PM Groups Potential performance problem:
>  getGroups(user=hdfs) took 58017 milliseconds
>
>  ddlplhdm505.us.company.com
>  WARN June 10, 2023 8:01 AM Groups Potential performance problem:
>  getGroups(user=yarn) took 58017 milliseconds.
>
>
>  Interestingly, all these users are local users created by the Hadoop
>  package installation.  (hbase, mapred, hdfs, yarn are all local users).
>
>
>  These local users are members only of local groups.  So theoretically the
>  search for group memberships should be lightning quick as the groups are
>  found only in /etc/group.
>
>
>  But since my /etc/nsswitch.conf looks like this:
>
>
>  passwd:     files systemd sss
>
>  group:      files systemd sss
>
>  netgroup:   files sss
>
>  automount:  files sss
>
>  services:   files
>
>  shadow:     files sss
>
>  hosts:      files dns myhostname
>  
You might consider specifying 'initgroups' explicitly and using something
like '[SUCCESS=return]' between databases to avoid going to sss altogether
(but see below)

>  It not only searches local files for group membership, it searches AD as
>  well.  And doesn't find it, as there is no such AD user or uid.
>   Eventually timing out.
>
>
>  In /var/log/messages we see this around the same time frame as
>
>
>  Jun 15 10:08:09 ddlflhdm201 sssd[be[amer.company.com]]: GSSAPI Error: An
>  invalid name was supplied (Success)
>  
Is sssd-ad properly configured at this host?
Initgroups lookup for unknown users should be fast.



>  Is there a way to restrict these particular local app accounts from having
>  their group memberships looked up in AD?
>
>  I.e., my [nss] section looks like this today:
>
>  [nss]
>  #debug_level = 9
>  filter_groups = root
>  filter_users = root
>  #override_homedir = /home/%u
>
>  Can I do this?
>
>  [nss]
>  filter_users = root,  hbase, mapred, hdfs, yarn
>
>  And would I want to set filter_users_in_groups = false?
>
>  Spike White
>
>
>  _______________________________________________
>  sssd-users mailing list -- sssd-users(a)lists.fedorahosted.org
>  To unsubscribe send an email to sssd-users-leave(a)lists.fedorahosted.org
>  Fedora Code of Conduct:
>  https://docs.fedoraproject.org/en-US/project/code-of-conduct/
>  List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
>  List Archives:
>  https://lists.fedorahosted.org/archives/list/[e-mail]...
>  Do not reply to spam, report it:
>  https://pagure.io/fedora-infrastructure/new_issue
>

[Spike White]

Alexey,

Thanks for the suggestion.

This is a commercial application.  Cloudera's hadoop implementation.  No
idea if they use getgrouplist() under the hood.  I can ask our Cloudera
tech engineer, but I doubt he knows either.

sssd-ad seems to work pretty well on these servers, except for these
occasional hiccups.  It's over 150 nodes in this hadoop cluster, with only
these occasional hiccups.  Been AD integrated for >1 yr and app team happy
(except for one annoying occasional cross-domain caching issue.)

In particular, sssd finds its AD site so it's binding to AD DCs local to
this datacenter.  (I checked that first thing.)

These sporadic hiccups I discuss earlier seem limited to local users only,
looking up their group memberships.

Would the /etc/nsswitch.conf file syntax be:

     initgroups:  files [SUCCESS=return] systemd sss

It seems this would apply for all accounts that were members of local
groups and not just these specific accounts.  (Not sure that's a problem
for this particular app team, but I've seen AD accounts added in /etc/group
as members of local groups.)

To me, "filter_users" in sssd.conf where I specify the specific accounts
would be safer.

Spike

On Thu, Jun 22, 2023 at 10:44 AM Alexey Tikhonov <atikhono(a)redhat.com&gt;
wrote:

>  Hi,
>
>  On Thu, Jun 22, 2023 at 4:47 PM Spike White <spikewhitetx(a)gmail.com&gt;
>  wrote:
>
> > All,
> >
> > Successful sssd consumer here.
> >
> > Have an app team running Hadoop.  They're getting these performance
> > errors in their app.
> >
> > This is from their app logs.
> >
> > ddlflhdm201.us.company.com
> > WARN June 15, 2023 10:08 AM Groups Potential performance problem:
> > getGroups(user=hbase) took 58022 milliseconds
> >
>
>  Does it use `getgrouplist()` under the hood?
>
>
> > ddlflhdm304.us.company.com
> > WARN June 15, 2023 8:38 AM Groups Potential performance problem:
> > getGroups(user=mapred) took 39962 milliseconds
> > ddlplhdm505.us.company.com
> > WARN June 16, 2023 5:37 AM Groups Potential performance problem:
> > getGroups(user=hdfs) took 62437 milliseconds.
> > ddlplhdm305.us.company.com
> > WARN June 15, 2023 9:36 PM Groups Potential performance problem:
> > getGroups(user=hdfs) took 58017 milliseconds
> >
> > ddlplhdm505.us.company.com
> > WARN June 10, 2023 8:01 AM Groups Potential performance problem:
> > getGroups(user=yarn) took 58017 milliseconds.
> >
> >
> > Interestingly, all these users are local users created by the Hadoop
> > package installation.  (hbase, mapred, hdfs, yarn are all local users).
> >
> >
> > These local users are members only of local groups.  So theoretically the
> > search for group memberships should be lightning quick as the groups are
> > found only in /etc/group.
> >
> >
> > But since my /etc/nsswitch.conf looks like this:
> >
> >
> > passwd:     files systemd sss
> >
> > group:      files systemd sss
> >
> > netgroup:   files sss
> >
> > automount:  files sss
> >
> > services:   files
> >
> > shadow:     files sss
> >
> > hosts:      files dns myhostname
> >
>
>  You might consider specifying 'initgroups' explicitly and using something
>  like '[SUCCESS=return]' between databases to avoid going to sss altogether
>  (but see below)
>
> >
> >
> > It not only searches local files for group membership, it searches AD as
> > well.  And doesn't find it, as there is no such AD user or uid.
> >  Eventually timing out.
> >
> >
> > In /var/log/messages we see this around the same time frame as
> >
> >
> > Jun 15 10:08:09 ddlflhdm201 sssd[be[amer.company.com]]: GSSAPI Error: An
> > invalid name was supplied (Success)
> >
>
>  Is sssd-ad properly configured at this host?
>  Initgroups lookup for unknown users should be fast.
>
>
>
> >
> > Is there a way to restrict these particular local app accounts from
> > having their group memberships looked up in AD?
> >
> > I.e., my [nss] section looks like this today:
> >
> > [nss]
> > #debug_level = 9
> > filter_groups = root
> > filter_users = root
> > #override_homedir = /home/%u
> >
> > Can I do this?
> >
> > [nss]
> > filter_users = root,  hbase, mapred, hdfs, yarn
> >
> > And would I want to set filter_users_in_groups = false?
> >
> > Spike White
> >
> >
> > _______________________________________________
> > sssd-users mailing list -- sssd-users(a)lists.fedorahosted.org
> > To unsubscribe send an email to sssd-users-leave(a)lists.fedorahosted.org
> > Fedora Code of Conduct:
> > https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> > List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> > List Archives:
> > https://lists.fedorahosted.org/archives/list/[e-mail]...
> > Do not reply to spam, report it:
> > https://pagure.io/fedora-infrastructure/new_issue
> >
>  _______________________________________________
>  sssd-users mailing list -- sssd-users(a)lists.fedorahosted.org
>  To unsubscribe send an email to sssd-users-leave(a)lists.fedorahosted.org
>  Fedora Code of Conduct:
>  https://docs.fedoraproject.org/en-US/project/code-of-conduct/
>  List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
>  List Archives:
>  https://lists.fedorahosted.org/archives/list/[e-mail]...
>  Do not reply to spam, report it:
>  https://pagure.io/fedora-infrastructure/new_issue
>

[Alexey Tikhonov]

Hi,


On Thu, Jun 22, 2023 at 7:52 PM Spike White <spikewhitetx(a)gmail.com&gt; wrote:

>  Alexey,
>
>  Thanks for the suggestion.
>
>  This is a commercial application.  Cloudera's hadoop implementation.  No
>  idea if they use getgrouplist() under the hood.  I can ask our Cloudera
>  tech engineer, but I doubt he knows either.
>
>  sssd-ad seems to work pretty well on these servers, except for these
>  occasional hiccups.  It's over 150 nodes in this hadoop cluster, with only
>  these occasional hiccups.  Been AD integrated for >1 yr and app team happy
>  (except for one annoying occasional cross-domain caching issue.)
>
>  In particular, sssd finds its AD site so it's binding to AD DCs local to
>  this datacenter.  (I checked that first thing.)
>
>  These sporadic hiccups I discuss earlier seem limited to local users only,
>  looking up their group memberships.
>
>  Would the /etc/nsswitch.conf file syntax be:
>
>       initgroups:  files [SUCCESS=return] systemd sss
>
>  It seems this would apply for all accounts that were members of local
>  groups and not just these specific accounts.  (Not sure that's a problem
>  for this particular app team, but I've seen AD accounts added in /etc/group
>  as members of local groups.)
>
>  To me, "filter_users" in sssd.conf where I specify the specific accounts
>  would be safer.
>  
Then filtering sounds as a way to go.

If you are courageous you can also try to read SSSD logs to figure out the
reason for those "sporadic hiccups".
Latest version, around sssd-2.8+, should make it easier to track a specific
lookup across a set of logs.



>  Spike
>
>  On Thu, Jun 22, 2023 at 10:44 AM Alexey Tikhonov <atikhono(a)redhat.com&gt;
>  wrote:
>
> > Hi,
> >
> > On Thu, Jun 22, 2023 at 4:47 PM Spike White <spikewhitetx(a)gmail.com&gt;
> > wrote:
> >
> >> All,
> >>
> >> Successful sssd consumer here.
> >>
> >> Have an app team running Hadoop.  They're getting these performance
> >> errors in their app.
> >>
> >> This is from their app logs.
> >>
> >> ddlflhdm201.us.company.com
> >> WARN June 15, 2023 10:08 AM Groups Potential performance problem:
> >> getGroups(user=hbase) took 58022 milliseconds
> >>
> >
> > Does it use `getgrouplist()` under the hood?
> >
> >
> >> ddlflhdm304.us.company.com
> >> WARN June 15, 2023 8:38 AM Groups Potential performance problem:
> >> getGroups(user=mapred) took 39962 milliseconds
> >> ddlplhdm505.us.company.com
> >> WARN June 16, 2023 5:37 AM Groups Potential performance problem:
> >> getGroups(user=hdfs) took 62437 milliseconds.
> >> ddlplhdm305.us.company.com
> >> WARN June 15, 2023 9:36 PM Groups Potential performance problem:
> >> getGroups(user=hdfs) took 58017 milliseconds
> >>
> >> ddlplhdm505.us.company.com
> >> WARN June 10, 2023 8:01 AM Groups Potential performance problem:
> >> getGroups(user=yarn) took 58017 milliseconds.
> >>
> >>
> >> Interestingly, all these users are local users created by the Hadoop
> >> package installation.  (hbase, mapred, hdfs, yarn are all local users).
> >>
> >>
> >> These local users are members only of local groups.  So theoretically
> >> the search for group memberships should be lightning quick as the groups
> >> are found only in /etc/group.
> >>
> >>
> >> But since my /etc/nsswitch.conf looks like this:
> >>
> >>
> >> passwd:     files systemd sss
> >>
> >> group:      files systemd sss
> >>
> >> netgroup:   files sss
> >>
> >> automount:  files sss
> >>
> >> services:   files
> >>
> >> shadow:     files sss
> >>
> >> hosts:      files dns myhostname
> >>
> >
> > You might consider specifying 'initgroups' explicitly and using something
> > like '[SUCCESS=return]' between databases to avoid going to sss altogether
> > (but see below)
> >
> >>
> >>
> >> It not only searches local files for group membership, it searches AD as
> >> well.  And doesn't find it, as there is no such AD user or uid.
> >>  Eventually timing out.
> >>
> >>
> >> In /var/log/messages we see this around the same time frame as
> >>
> >>
> >> Jun 15 10:08:09 ddlflhdm201 sssd[be[amer.company.com]]: GSSAPI Error:
> >> An invalid name was supplied (Success)
> >>
> >
> > Is sssd-ad properly configured at this host?
> > Initgroups lookup for unknown users should be fast.
> >
> >
> >
> >>
> >> Is there a way to restrict these particular local app accounts from
> >> having their group memberships looked up in AD?
> >>
> >> I.e., my [nss] section looks like this today:
> >>
> >> [nss]
> >> #debug_level = 9
> >> filter_groups = root
> >> filter_users = root
> >> #override_homedir = /home/%u
> >>
> >> Can I do this?
> >>
> >> [nss]
> >> filter_users = root,  hbase, mapred, hdfs, yarn
> >>
> >> And would I want to set filter_users_in_groups = false?
> >>
> >> Spike White
> >>
> >>
> >> _______________________________________________
> >> sssd-users mailing list -- sssd-users(a)lists.fedorahosted.org
> >> To unsubscribe send an email to sssd-users-leave(a)lists.fedorahosted.org
> >> Fedora Code of Conduct:
> >> https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> >> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> >> List Archives:
> >> https://lists.fedorahosted.org/archives/list/[e-mail]...
> >> Do not reply to spam, report it:
> >> https://pagure.io/fedora-infrastructure/new_issue
> >>
> > _______________________________________________
> > sssd-users mailing list -- sssd-users(a)lists.fedorahosted.org
> > To unsubscribe send an email to sssd-users-leave(a)lists.fedorahosted.org
> > Fedora Code of Conduct:
> > https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> > List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> > List Archives:
> > https://lists.fedorahosted.org/archives/list/[e-mail]...
> > Do not reply to spam, report it:
> > https://pagure.io/fedora-infrastructure/new_issue
> >
>  _______________________________________________
>  sssd-users mailing list -- sssd-users(a)lists.fedorahosted.org
>  To unsubscribe send an email to sssd-users-leave(a)lists.fedorahosted.org
>  Fedora Code of Conduct:
>  https://docs.fedoraproject.org/en-US/project/code-of-conduct/
>  List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
>  List Archives:
>  https://lists.fedorahosted.org/archives/list/[e-mail]...
>  Do not reply to spam, report it:
>  https://pagure.io/fedora-infrastructure/new_issue
>

[Spike White]

Appreciate the insight.  These are production RHEL7 servers, which I see
are based on sssd-1.16.5-xxx.  As in anything production, I am risk-adverse.

I'll just filter and move on.  Appreciate your help.

Spike

On Fri, Jun 23, 2023 at 9:02 AM Alexey Tikhonov <atikhono(a)redhat.com&gt; wrote:

>  Hi,
>
>
>  On Thu, Jun 22, 2023 at 7:52 PM Spike White <spikewhitetx(a)gmail.com&gt;
>  wrote:
>
> > Alexey,
> >
> > Thanks for the suggestion.
> >
> > This is a commercial application.  Cloudera's hadoop implementation.  No
> > idea if they use getgrouplist() under the hood.  I can ask our Cloudera
> > tech engineer, but I doubt he knows either.
> >
> > sssd-ad seems to work pretty well on these servers, except for these
> > occasional hiccups.  It's over 150 nodes in this hadoop cluster, with only
> > these occasional hiccups.  Been AD integrated for >1 yr and app team happy
> > (except for one annoying occasional cross-domain caching issue.)
> >
> > In particular, sssd finds its AD site so it's binding to AD DCs local to
> > this datacenter.  (I checked that first thing.)
> >
> > These sporadic hiccups I discuss earlier seem limited to local users
> > only, looking up their group memberships.
> >
> > Would the /etc/nsswitch.conf file syntax be:
> >
> >      initgroups:  files [SUCCESS=return] systemd sss
> >
> > It seems this would apply for all accounts that were members of local
> > groups and not just these specific accounts.  (Not sure that's a problem
> > for this particular app team, but I've seen AD accounts added in /etc/group
> > as members of local groups.)
> >
> > To me, "filter_users" in sssd.conf where I specify the specific accounts
> > would be safer.
> >
>
>  Then filtering sounds as a way to go.
>
>  If you are courageous you can also try to read SSSD logs to figure out the
>  reason for those "sporadic hiccups".
>  Latest version, around sssd-2.8+, should make it easier to track a
>  specific lookup across a set of logs.
>
>
>
> >
> > Spike
> >
> > On Thu, Jun 22, 2023 at 10:44 AM Alexey Tikhonov <atikhono(a)redhat.com&gt;
> > wrote:
> >
> >> Hi,
> >>
> >> On Thu, Jun 22, 2023 at 4:47 PM Spike White <spikewhitetx(a)gmail.com&gt;
> >> wrote:
> >>
> >>> All,
> >>>
> >>> Successful sssd consumer here.
> >>>
> >>> Have an app team running Hadoop.  They're getting these performance
> >>> errors in their app.
> >>>
> >>> This is from their app logs.
> >>>
> >>> ddlflhdm201.us.company.com
> >>> WARN June 15, 2023 10:08 AM Groups Potential performance problem:
> >>> getGroups(user=hbase) took 58022 milliseconds
> >>>
> >>
> >> Does it use `getgrouplist()` under the hood?
> >>
> >>
> >>> ddlflhdm304.us.company.com
> >>> WARN June 15, 2023 8:38 AM Groups Potential performance problem:
> >>> getGroups(user=mapred) took 39962 milliseconds
> >>> ddlplhdm505.us.company.com
> >>> WARN June 16, 2023 5:37 AM Groups Potential performance problem:
> >>> getGroups(user=hdfs) took 62437 milliseconds.
> >>> ddlplhdm305.us.company.com
> >>> WARN June 15, 2023 9:36 PM Groups Potential performance problem:
> >>> getGroups(user=hdfs) took 58017 milliseconds
> >>>
> >>> ddlplhdm505.us.company.com
> >>> WARN June 10, 2023 8:01 AM Groups Potential performance problem:
> >>> getGroups(user=yarn) took 58017 milliseconds.
> >>>
> >>>
> >>> Interestingly, all these users are local users created by the Hadoop
> >>> package installation.  (hbase, mapred, hdfs, yarn are all local users).
> >>>
> >>>
> >>> These local users are members only of local groups.  So theoretically
> >>> the search for group memberships should be lightning quick as the groups
> >>> are found only in /etc/group.
> >>>
> >>>
> >>> But since my /etc/nsswitch.conf looks like this:
> >>>
> >>>
> >>> passwd:     files systemd sss
> >>>
> >>> group:      files systemd sss
> >>>
> >>> netgroup:   files sss
> >>>
> >>> automount:  files sss
> >>>
> >>> services:   files
> >>>
> >>> shadow:     files sss
> >>>
> >>> hosts:      files dns myhostname
> >>>
> >>
> >> You might consider specifying 'initgroups' explicitly and using
> >> something like '[SUCCESS=return]' between databases to avoid going to
> sss
> >> altogether (but see below)
> >>
> >>>
> >>>
> >>> It not only searches local files for group membership, it searches AD
> >>> as well.  And doesn't find it, as there is no such AD user or uid.
> >>>  Eventually timing out.
> >>>
> >>>
> >>> In /var/log/messages we see this around the same time frame as
> >>>
> >>>
> >>> Jun 15 10:08:09 ddlflhdm201 sssd[be[amer.company.com]]: GSSAPI Error:
> >>> An invalid name was supplied (Success)
> >>>
> >>
> >> Is sssd-ad properly configured at this host?
> >> Initgroups lookup for unknown users should be fast.
> >>
> >>
> >>
> >>>
> >>> Is there a way to restrict these particular local app accounts from
> >>> having their group memberships looked up in AD?
> >>>
> >>> I.e., my [nss] section looks like this today:
> >>>
> >>> [nss]
> >>> #debug_level = 9
> >>> filter_groups = root
> >>> filter_users = root
> >>> #override_homedir = /home/%u
> >>>
> >>> Can I do this?
> >>>
> >>> [nss]
> >>> filter_users = root,  hbase, mapred, hdfs, yarn
> >>>
> >>> And would I want to set filter_users_in_groups = false?
> >>>
> >>> Spike White
> >>>
> >>>
> >>> _______________________________________________
> >>> sssd-users mailing list -- sssd-users(a)lists.fedorahosted.org
> >>> To unsubscribe send an email to sssd-users-leave(a)lists.fedorahosted.org
> >>> Fedora Code of Conduct:
> >>> https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> >>> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> >>> List Archives:
> >>>
> https://lists.fedorahosted.org/archives/list/[e-mail]...
> >>> Do not reply to spam, report it:
> >>> https://pagure.io/fedora-infrastructure/new_issue
> >>>
> >> _______________________________________________
> >> sssd-users mailing list -- sssd-users(a)lists.fedorahosted.org
> >> To unsubscribe send an email to sssd-users-leave(a)lists.fedorahosted.org
> >> Fedora Code of Conduct:
> >> https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> >> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> >> List Archives:
> >> https://lists.fedorahosted.org/archives/list/[e-mail]...
> >> Do not reply to spam, report it:
> >> https://pagure.io/fedora-infrastructure/new_issue
> >>
> > _______________________________________________
> > sssd-users mailing list -- sssd-users(a)lists.fedorahosted.org
> > To unsubscribe send an email to sssd-users-leave(a)lists.fedorahosted.org
> > Fedora Code of Conduct:
> > https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> > List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> > List Archives:
> > https://lists.fedorahosted.org/archives/list/[e-mail]...
> > Do not reply to spam, report it:
> > https://pagure.io/fedora-infrastructure/new_issue
> >
>  _______________________________________________
>  sssd-users mailing list -- sssd-users(a)lists.fedorahosted.org
>  To unsubscribe send an email to sssd-users-leave(a)lists.fedorahosted.org
>  Fedora Code of Conduct:
>  https://docs.fedoraproject.org/en-US/project/code-of-conduct/
>  List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
>  List Archives:
>  https://lists.fedorahosted.org/archives/list/[e-mail]...
>  Do not reply to spam, report it:
>  https://pagure.io/fedora-infrastructure/new_issue
>

[Spike White]

All,

Turns out this particular application connects to kerberos (AD in our env)
directly.  So the errors that show up in their app logs:

ddlflhdm201.us.company.com
WARN June 15, 2023 10:08 AM Groups Potential performance problem:
getGroups(user=hbase) took 58022 milliseconds
ddlflhdm304.us.company.com
WARN June 15, 2023 8:38 AM Groups Potential performance problem:
getGroups(user=mapred) took 39962 milliseconds
ddlplhdm505.us.company.com
WARN June 16, 2023 5:37 AM Groups Potential performance problem:
getGroups(user=hdfs) took 62437 milliseconds.
ddlplhdm305.us.company.com
WARN June 15, 2023 9:36 PM Groups Potential performance problem:
getGroups(user=hdfs) took 58017 milliseconds
ddlplhdm505.us.company.com
WARN June 10, 2023 8:01 AM Groups Potential performance problem:
getGroups(user=yarn) took 58017 milliseconds.

arise from their kerberos client implementation.  Not from sssd.  The
insights about filtering on local users might be useful for the app team
when communicating back with their app vendor.  But ultimately, it's
between this app team and their app vendor.  sssd is in no way involved in
this.

Sorry for the confusion, but the app team was blaming sssd for this
slowness.  Only on exhaustive research did they determine their app is
contacting kerberos directly.

Spike



On Fri, Jun 23, 2023 at 11:07 AM Spike White <spikewhitetx(a)gmail.com&gt; wrote:

>  Appreciate the insight.  These are production RHEL7 servers, which I
> see
>  are based on sssd-1.16.5-xxx.  As in anything production, I am risk-adverse.
>
>  I'll just filter and move on.  Appreciate your help.
>
>  Spike
>
>  On Fri, Jun 23, 2023 at 9:02 AM Alexey Tikhonov <atikhono(a)redhat.com&gt;
>  wrote:
>
> > Hi,
> >
> >
> > On Thu, Jun 22, 2023 at 7:52 PM Spike White <spikewhitetx(a)gmail.com&gt;
> > wrote:
> >
> >> Alexey,
> >>
> >> Thanks for the suggestion.
> >>
> >> This is a commercial application.  Cloudera's hadoop implementation.  No
> >> idea if they use getgrouplist() under the hood.  I can ask our Cloudera
> >> tech engineer, but I doubt he knows either.
> >>
> >> sssd-ad seems to work pretty well on these servers, except for these
> >> occasional hiccups.  It's over 150 nodes in this hadoop cluster, with only
> >> these occasional hiccups.  Been AD integrated for >1 yr and app team happy
> >> (except for one annoying occasional cross-domain caching issue.)
> >>
> >> In particular, sssd finds its AD site so it's binding to AD DCs local to
> >> this datacenter.  (I checked that first thing.)
> >>
> >> These sporadic hiccups I discuss earlier seem limited to local users
> >> only, looking up their group memberships.
> >>
> >> Would the /etc/nsswitch.conf file syntax be:
> >>
> >>      initgroups:  files [SUCCESS=return] systemd sss
> >>
> >> It seems this would apply for all accounts that were members of local
> >> groups and not just these specific accounts.  (Not sure that's a problem
> >> for this particular app team, but I've seen AD accounts added in /etc/group
> >> as members of local groups.)
> >>
> >> To me, "filter_users" in sssd.conf where I specify the specific
> accounts
> >> would be safer.
> >>
> >
> > Then filtering sounds as a way to go.
> >
> > If you are courageous you can also try to read SSSD logs to figure out
> > the reason for those "sporadic hiccups".
> > Latest version, around sssd-2.8+, should make it easier to track a
> > specific lookup across a set of logs.
> >
> >
> >
> >>
> >> Spike
> >>
> >> On Thu, Jun 22, 2023 at 10:44 AM Alexey Tikhonov <atikhono(a)redhat.com&gt;
> >> wrote:
> >>
> >>> Hi,
> >>>
> >>> On Thu, Jun 22, 2023 at 4:47 PM Spike White <spikewhitetx(a)gmail.com&gt;
> >>> wrote:
> >>>
> >>>> All,
> >>>>
> >>>> Successful sssd consumer here.
> >>>>
> >>>> Have an app team running Hadoop.  They're getting these performance
> >>>> errors in their app.
> >>>>
> >>>> This is from their app logs.
> >>>>
> >>>> ddlflhdm201.us.company.com
> >>>> WARN June 15, 2023 10:08 AM Groups Potential performance problem:
> >>>> getGroups(user=hbase) took 58022 milliseconds
> >>>>
> >>>
> >>> Does it use `getgrouplist()` under the hood?
> >>>
> >>>
> >>>> ddlflhdm304.us.company.com
> >>>> WARN June 15, 2023 8:38 AM Groups Potential performance problem:
> >>>> getGroups(user=mapred) took 39962 milliseconds
> >>>> ddlplhdm505.us.company.com
> >>>> WARN June 16, 2023 5:37 AM Groups Potential performance problem:
> >>>> getGroups(user=hdfs) took 62437 milliseconds.
> >>>> ddlplhdm305.us.company.com
> >>>> WARN June 15, 2023 9:36 PM Groups Potential performance problem:
> >>>> getGroups(user=hdfs) took 58017 milliseconds
> >>>>
> >>>> ddlplhdm505.us.company.com
> >>>> WARN June 10, 2023 8:01 AM Groups Potential performance problem:
> >>>> getGroups(user=yarn) took 58017 milliseconds.
> >>>>
> >>>>
> >>>> Interestingly, all these users are local users created by the Hadoop
> >>>> package installation.  (hbase, mapred, hdfs, yarn are all local users).
> >>>>
> >>>>
> >>>> These local users are members only of local groups.  So theoretically
> >>>> the search for group memberships should be lightning quick as the groups
> >>>> are found only in /etc/group.
> >>>>
> >>>>
> >>>> But since my /etc/nsswitch.conf looks like this:
> >>>>
> >>>>
> >>>> passwd:     files systemd sss
> >>>>
> >>>> group:      files systemd sss
> >>>>
> >>>> netgroup:   files sss
> >>>>
> >>>> automount:  files sss
> >>>>
> >>>> services:   files
> >>>>
> >>>> shadow:     files sss
> >>>>
> >>>> hosts:      files dns myhostname
> >>>>
> >>>
> >>> You might consider specifying 'initgroups' explicitly and using
> >>> something like '[SUCCESS=return]' between databases to avoid going to
> sss
> >>> altogether (but see below)
> >>>
> >>>>
> >>>>
> >>>> It not only searches local files for group membership, it searches AD
> >>>> as well.  And doesn't find it, as there is no such AD user or uid.
> >>>>  Eventually timing out.
> >>>>
> >>>>
> >>>> In /var/log/messages we see this around the same time frame as
> >>>>
> >>>>
> >>>> Jun 15 10:08:09 ddlflhdm201 sssd[be[amer.company.com]]: GSSAPI Error:
> >>>> An invalid name was supplied (Success)
> >>>>
> >>>
> >>> Is sssd-ad properly configured at this host?
> >>> Initgroups lookup for unknown users should be fast.
> >>>
> >>>
> >>>
> >>>>
> >>>> Is there a way to restrict these particular local app accounts from
> >>>> having their group memberships looked up in AD?
> >>>>
> >>>> I.e., my [nss] section looks like this today:
> >>>>
> >>>> [nss]
> >>>> #debug_level = 9
> >>>> filter_groups = root
> >>>> filter_users = root
> >>>> #override_homedir = /home/%u
> >>>>
> >>>> Can I do this?
> >>>>
> >>>> [nss]
> >>>> filter_users = root,  hbase, mapred, hdfs, yarn
> >>>>
> >>>> And would I want to set filter_users_in_groups = false?
> >>>>
> >>>> Spike White
> >>>>
> >>>>
> >>>> _______________________________________________
> >>>> sssd-users mailing list -- sssd-users(a)lists.fedorahosted.org
> >>>> To unsubscribe send an email to
> >>>> sssd-users-leave(a)lists.fedorahosted.org
> >>>> Fedora Code of Conduct:
> >>>> https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> >>>> List Guidelines:
> >>>> https://fedoraproject.org/wiki/Mailing_list_guidelines
> >>>> List Archives:
> >>>>
> https://lists.fedorahosted.org/archives/list/[e-mail]...
> >>>> Do not reply to spam, report it:
> >>>> https://pagure.io/fedora-infrastructure/new_issue
> >>>>
> >>> _______________________________________________
> >>> sssd-users mailing list -- sssd-users(a)lists.fedorahosted.org
> >>> To unsubscribe send an email to sssd-users-leave(a)lists.fedorahosted.org
> >>> Fedora Code of Conduct:
> >>> https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> >>> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> >>> List Archives:
> >>>
> https://lists.fedorahosted.org/archives/list/[e-mail]...
> >>> Do not reply to spam, report it:
> >>> https://pagure.io/fedora-infrastructure/new_issue
> >>>
> >> _______________________________________________
> >> sssd-users mailing list -- sssd-users(a)lists.fedorahosted.org
> >> To unsubscribe send an email to sssd-users-leave(a)lists.fedorahosted.org
> >> Fedora Code of Conduct:
> >> https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> >> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> >> List Archives:
> >> https://lists.fedorahosted.org/archives/list/[e-mail]...
> >> Do not reply to spam, report it:
> >> https://pagure.io/fedora-infrastructure/new_issue
> >>
> > _______________________________________________
> > sssd-users mailing list -- sssd-users(a)lists.fedorahosted.org
> > To unsubscribe send an email to sssd-users-leave(a)lists.fedorahosted.org
> > Fedora Code of Conduct:
> > https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> > List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> > List Archives:
> > https://lists.fedorahosted.org/archives/list/[e-mail]...
> > Do not reply to spam, report it:
> > https://pagure.io/fedora-infrastructure/new_issue
> >
>