memory cache for initgroups

[Jakub Hrozek]

Hi,

we had short discussion on $SUBJECT with Simo on IRC already, but there
are multiple people involved from multiple timezones, so I think a mailing
list thread would be better trackable.

Can we add another memory cache file to SSSD, that would track
initgroups/getgrouplist results for the NSS responder? I realize initgroups
is a bit different operation than getpw{uid,nam} and getgr{gid,nam} but
what if the new memcache was only used by the NSS responder and at the
same time invalidated when initgroups is initiated by the PAM responder
to ensure the memcache is up-to-date?

I already know about two projects that would benefit from faster initgroups
operation - GlusterFS uses getgrouplist() quite a lot with some setups
and also in the IPA server mode, the SSSD running on the server becomes
a bit of a bottlenec for some operations.

Are there any technical reasons against a new memcache file that would
cache initgroups?

[Simo Sorce]

On Mon, 3 Nov 2014 13:57:08 +0100
Jakub Hrozek <jhrozek(a)redhat.com&gt; wrote:

>  Hi,
>  
>  we had short discussion on $SUBJECT with Simo on IRC already, but
>  there are multiple people involved from multiple timezones, so I
>  think a mailing list thread would be better trackable.
>  
>  Can we add another memory cache file to SSSD, that would track
>  initgroups/getgrouplist results for the NSS responder? I realize
>  initgroups is a bit different operation than getpw{uid,nam} and
>  getgr{gid,nam} but what if the new memcache was only used by the NSS
>  responder and at the same time invalidated when initgroups is
>  initiated by the PAM responder to ensure the memcache is up-to-date? 
Can you describe the use case before jumping into a proposed solution ?

>  I already know about two projects that would benefit from faster
>  initgroups operation - GlusterFS uses getgrouplist() quite a lot with
>  some setups and also in the IPA server mode, the SSSD running on the
>  server becomes a bit of a bottlenec for some operations. 
Is the problem related to accessing the main ldb cache, or the fact we
also go out and contact the IdM server ?

>  Are there any technical reasons against a new memcache file that
> would
>  cache initgroups? 
The main reason would be potential inconsistency between this cache and
the users and groups caches, but I do not think that would be really a
big deal if there is a pretty good reason for the cache.

Do we need to insure all groups are resolvable in the groups cache ?
Or is it ok if gids returned by the getgrouplist are not immediately
available in the groups cache ?
What about the user ?

Simo.

-- 
Simo Sorce * Red Hat, Inc * New York

[Jakub Hrozek]

On Mon, Nov 03, 2014 at 08:53:06AM -0500, Simo Sorce wrote:
>  On Mon, 3 Nov 2014 13:57:08 +0100
>  Jakub Hrozek <jhrozek(a)redhat.com&gt; wrote:
>  
>  > Hi,
>  > 
>  > we had short discussion on $SUBJECT with Simo on IRC already, but
>  > there are multiple people involved from multiple timezones, so I
>  > think a mailing list thread would be better trackable.
>  > 
>  > Can we add another memory cache file to SSSD, that would track
>  > initgroups/getgrouplist results for the NSS responder? I realize
>  > initgroups is a bit different operation than getpw{uid,nam} and
>  > getgr{gid,nam} but what if the new memcache was only used by the NSS
>  > responder and at the same time invalidated when initgroups is
>  > initiated by the PAM responder to ensure the memcache is up-to-date?
>  
>  Can you describe the use case before jumping into a proposed solution ? 
Many getgrouplist() or initgroups() calls in a quick succession. One
user is GlusterFS -- I'm not quite sure what the reason is there, maybe
Vijay can elaborate.

The other use-case is slapi-nis that calls getgrouplist() -- Alexander
would explain better here, I only know that slapi-nis uses
getgrouplist() during memberUID processing, but I'm fuzzy on the
details.

>  
>  > I already know about two projects that would benefit from faster
>  > initgroups operation - GlusterFS uses getgrouplist() quite a lot with
>  > some setups and also in the IPA server mode, the SSSD running on the
>  > server becomes a bit of a bottlenec for some operations.
>  
>  Is the problem related to accessing the main ldb cache, or the fact we
>  also go out and contact the IdM server ? 
Mostly the cache. In particular, with the GlusterFS case, there was a
huge filesystem directory that triggered 4000+ getgrouplist() calls, one
per a file. Even though the getgrouplist() calls were for fewer than 10
users, we would hit LDB every time and that produced quite a load on the
SSSD.

>  
>  > Are there any technical reasons against a new memcache file that would
>  > cache initgroups?
>  
>  The main reason would be potential inconsistency between this cache and
>  the users and groups caches, but I do not think that would be really a
>  big deal if there is a pretty good reason for the cache. 
Hm, I see, then we have the same information duplicated in two caches,
just from a different angle. However, for cases where precision really
matters (access control), we don't use these caches anyway.

Could we mitigate this problem by invalidating the memcache when group
membership changes at all during a backend operation? I think changes to
LDAP object are so rare, that invalidating the memcache (which is quite
shortlived anyway) would be worth it.

>  
>  Do we need to insure all groups are resolvable in the groups cache ? 
I don't think so, if the group is not in the groups cache, then we just
hit the responder.

>  Or is it ok if gids returned by the getgrouplist are not immediately
>  available in the groups cache ?
>  What about the user ? 
I *think* the user could also be left out. If the application also needs
user data, it simply calls getpwnam -- like id(1) does. getgrouplist is
a different interface with different outputs.

>  
>  Simo.
>  
>  -- 
>  Simo Sorce * Red Hat, Inc * New York

[Jakub Hrozek]

On Mon, Nov 03, 2014 at 03:41:43PM +0100, Jakub Hrozek wrote:
>  On Mon, Nov 03, 2014 at 08:53:06AM -0500, Simo Sorce wrote:
>  > On Mon, 3 Nov 2014 13:57:08 +0100
>  > Jakub Hrozek <jhrozek(a)redhat.com&gt; wrote:
>  > 
>  > > Hi,
>  > > 
>  > > we had short discussion on $SUBJECT with Simo on IRC already, but
>  > > there are multiple people involved from multiple timezones, so I
>  > > think a mailing list thread would be better trackable.
>  > > 
>  > > Can we add another memory cache file to SSSD, that would track
>  > > initgroups/getgrouplist results for the NSS responder? I realize
>  > > initgroups is a bit different operation than getpw{uid,nam} and
>  > > getgr{gid,nam} but what if the new memcache was only used by the NSS
>  > > responder and at the same time invalidated when initgroups is
>  > > initiated by the PAM responder to ensure the memcache is up-to-date?
>  > 
>  > Can you describe the use case before jumping into a proposed solution ?
>  
>  Many getgrouplist() or initgroups() calls in a quick succession. One
>  user is GlusterFS -- I'm not quite sure what the reason is there, maybe
>  Vijay can elaborate. 
Sorry, I forgot to CC Vijay.

>  
>  The other use-case is slapi-nis that calls getgrouplist() -- Alexander
>  would explain better here, I only know that slapi-nis uses
>  getgrouplist() during memberUID processing, but I'm fuzzy on the
>  details.
>  
>  > 
>  > > I already know about two projects that would benefit from faster
>  > > initgroups operation - GlusterFS uses getgrouplist() quite a lot with
>  > > some setups and also in the IPA server mode, the SSSD running on the
>  > > server becomes a bit of a bottlenec for some operations.
>  > 
>  > Is the problem related to accessing the main ldb cache, or the fact we
>  > also go out and contact the IdM server ?
>  
>  Mostly the cache. In particular, with the GlusterFS case, there was a
>  huge filesystem directory that triggered 4000+ getgrouplist() calls, one
>  per a file. Even though the getgrouplist() calls were for fewer than 10
>  users, we would hit LDB every time and that produced quite a load on the
>  SSSD.
>  
>  > 
>  > > Are there any technical reasons against a new memcache file that would
>  > > cache initgroups?
>  > 
>  > The main reason would be potential inconsistency between this cache and
>  > the users and groups caches, but I do not think that would be really a
>  > big deal if there is a pretty good reason for the cache.
>  
>  Hm, I see, then we have the same information duplicated in two caches,
>  just from a different angle. However, for cases where precision really
>  matters (access control), we don't use these caches anyway.
>  
>  Could we mitigate this problem by invalidating the memcache when group
>  membership changes at all during a backend operation? I think changes to
>  LDAP object are so rare, that invalidating the memcache (which is quite
>  shortlived anyway) would be worth it.
>  
>  > 
>  > Do we need to insure all groups are resolvable in the groups cache ?
>  
>  I don't think so, if the group is not in the groups cache, then we just
>  hit the responder.
>  
>  > Or is it ok if gids returned by the getgrouplist are not immediately
>  > available in the groups cache ?
>  > What about the user ?
>  
>  I *think* the user could also be left out. If the application also needs
>  user data, it simply calls getpwnam -- like id(1) does. getgrouplist is
>  a different interface with different outputs.
>  
>  > 
>  > Simo.
>  > 
>  > -- 
>  > Simo Sorce * Red Hat, Inc * New York

[Vijay Bellur]

On 11/03/2014 08:12 PM, Jakub Hrozek wrote:
>  On Mon, Nov 03, 2014 at 03:41:43PM +0100, Jakub Hrozek wrote:
> > On Mon, Nov 03, 2014 at 08:53:06AM -0500, Simo Sorce wrote:
> >> On Mon, 3 Nov 2014 13:57:08 +0100
> >> Jakub Hrozek <jhrozek(a)redhat.com&gt; wrote:
> >>
> >>> Hi,
> >>>
> >>> we had short discussion on $SUBJECT with Simo on IRC already, but
> >>> there are multiple people involved from multiple timezones, so I
> >>> think a mailing list thread would be better trackable.
> >>>
> >>> Can we add another memory cache file to SSSD, that would track
> >>> initgroups/getgrouplist results for the NSS responder? I realize
> >>> initgroups is a bit different operation than getpw{uid,nam} and
> >>> getgr{gid,nam} but what if the new memcache was only used by the NSS
> >>> responder and at the same time invalidated when initgroups is
> >>> initiated by the PAM responder to ensure the memcache is up-to-date?
> >>
> >> Can you describe the use case before jumping into a proposed solution ?
> >
> > Many getgrouplist() or initgroups() calls in a quick succession. One
> > user is GlusterFS -- I'm not quite sure what the reason is there, maybe
> > Vijay can elaborate.
>  
GlusterFS server invokes getgrouplist() to identify gids associated with 
an user on whose behalf a rpc request has been sent over the wire. There 
is a gid caching layer in GlusterFS and getgrouplist() does get called 
only if there is a gid cache miss. In the worst case, getgrouplist() can 
be invoked for every rpc request that GlusterFS receives and that seems 
to be the case in a deployment where we found that sssd was being busy. 
I am not certain about the sequence of operations that can cause the 
cache to be missed.

Adding Niels who is more familiar with the gid resolution & caching 
features in GlusterFS.

Thanks,
Vijay


>  Sorry, I forgot to CC Vijay.
>
> >
> > The other use-case is slapi-nis that calls getgrouplist() -- Alexander
> > would explain better here, I only know that slapi-nis uses
> > getgrouplist() during memberUID processing, but I'm fuzzy on the
> > details.
> >
> >>
> >>> I already know about two projects that would benefit from faster
> >>> initgroups operation - GlusterFS uses getgrouplist() quite a lot with
> >>> some setups and also in the IPA server mode, the SSSD running on the
> >>> server becomes a bit of a bottlenec for some operations.
> >>
> >> Is the problem related to accessing the main ldb cache, or the fact we
> >> also go out and contact the IdM server ?
> >
> > Mostly the cache. In particular, with the GlusterFS case, there was a
> > huge filesystem directory that triggered 4000+ getgrouplist() calls, one
> > per a file. Even though the getgrouplist() calls were for fewer than 10
> > users, we would hit LDB every time and that produced quite a load on the
> > SSSD.
> >
> >>
> >>> Are there any technical reasons against a new memcache file that would
> >>> cache initgroups?
> >>
> >> The main reason would be potential inconsistency between this cache and
> >> the users and groups caches, but I do not think that would be really a
> >> big deal if there is a pretty good reason for the cache.
> >
> > Hm, I see, then we have the same information duplicated in two caches,
> > just from a different angle. However, for cases where precision really
> > matters (access control), we don't use these caches anyway.
> >
> > Could we mitigate this problem by invalidating the memcache when group
> > membership changes at all during a backend operation? I think changes to
> > LDAP object are so rare, that invalidating the memcache (which is quite
> > shortlived anyway) would be worth it.
> >
> >>
> >> Do we need to insure all groups are resolvable in the groups cache ?
> >
> > I don't think so, if the group is not in the groups cache, then we just
> > hit the responder.
> >
> >> Or is it ok if gids returned by the getgrouplist are not immediately
> >> available in the groups cache ?
> >> What about the user ?
> >
> > I *think* the user could also be left out. If the application also needs
> > user data, it simply calls getpwnam -- like id(1) does. getgrouplist is
> > a different interface with different outputs.
> >
> >>
> >> Simo.
> >>
> >> --
> >> Simo Sorce * Red Hat, Inc * New York

[Niels de Vos]

On Thu, Nov 06, 2014 at 11:45:18PM +0530, Vijay Bellur wrote:
>  On 11/03/2014 08:12 PM, Jakub Hrozek wrote:
>  >On Mon, Nov 03, 2014 at 03:41:43PM +0100, Jakub Hrozek wrote:
>  >>On Mon, Nov 03, 2014 at 08:53:06AM -0500, Simo Sorce wrote:
>  >>>On Mon, 3 Nov 2014 13:57:08 +0100
>  >>>Jakub Hrozek <jhrozek(a)redhat.com&gt; wrote:
>  >>>
>  >>>>Hi,
>  >>>>
>  >>>>we had short discussion on $SUBJECT with Simo on IRC already, but
>  >>>>there are multiple people involved from multiple timezones, so I
>  >>>>think a mailing list thread would be better trackable.
>  >>>>
>  >>>>Can we add another memory cache file to SSSD, that would track
>  >>>>initgroups/getgrouplist results for the NSS responder? I realize
>  >>>>initgroups is a bit different operation than getpw{uid,nam} and
>  >>>>getgr{gid,nam} but what if the new memcache was only used by the NSS
>  >>>>responder and at the same time invalidated when initgroups is
>  >>>>initiated by the PAM responder to ensure the memcache is up-to-date?
>  >>>
>  >>>Can you describe the use case before jumping into a proposed solution ?
>  >>
>  >>Many getgrouplist() or initgroups() calls in a quick succession. One
>  >>user is GlusterFS -- I'm not quite sure what the reason is there, maybe
>  >>Vijay can elaborate.
>  >
>  
>  GlusterFS server invokes getgrouplist() to identify gids associated with an
>  user on whose behalf a rpc request has been sent over the wire. There is a
>  gid caching layer in GlusterFS and getgrouplist() does get called only if
>  there is a gid cache miss. In the worst case, getgrouplist() can be invoked
>  for every rpc request that GlusterFS receives and that seems to be the case
>  in a deployment where we found that sssd was being busy. I am not certain
>  about the sequence of operations that can cause the cache to be missed.
>  
>  Adding Niels who is more familiar with the gid resolution & caching features
>  in GlusterFS. 
Just to add some background information on the getgrouplist(). GlusterFS
uses several processes that can call getgrouplist():
- NFS-server, a single process per system
- brick, a process per exported filesystem/directory, potentally several
  per system

  [Here, a Gluster environment has many systems (vm/physical). Each
   system normally runs the NFS-server, and a number of brick processes.
   The layout of the volume is important, but it is very common to have
   one or more distributed volumes that use multiple bricks on the same
   system (and many other systems).]

The need for resolving the groups of a user comes in when users belong
to many groups. The RPC protocols can not carry a huge list of groups,
so the resolving can be done on the server side when the protocol hits
its limits (> 16 for NFS, approx. > 93 for GlusterFS).

Upon using a Gluster volume, certain operations are sent to all the
bricks (i.e. some directory related operations). I can imagine that
a network share which is used by many users, trigger many getgrouplist()
calls in different brick processes at the (almost) same time.

For reference, the usage of getgrouplist() in the brick process can be
found here:
-
https://github.com/gluster/glusterfs/blob/master/xlators/protocol/server/...

The gid_resolve() function get called in case the brick process should
resolve the groups (and ignore the list of groups from the protocol). It
uses the gidcache functions from a private library:
- https://github.com/gluster/glusterfs/blob/master/libglusterfs/src/gidcache.h
- https://github.com/gluster/glusterfs/blob/master/libglusterfs/src/gidcache.c

The default time for the gidcache to expire is 2 seconds. Users should
be able to configure this to 30 seconds (or anything else) with:

    # gluster volume set <VOLUME> server.gid-timeout 30


I think this should explain the use-case sufficiently, but let me know
if there are any remaining questions. It might well be possible to make
this code more sssd friendly. I'm sure that we as Gluster developers are
open to any suggestions.

Thanks,
Niels

>  
>  Thanks,
>  Vijay
>  
>  
>  >Sorry, I forgot to CC Vijay.
>  >
>  >>
>  >>The other use-case is slapi-nis that calls getgrouplist() -- Alexander
>  >>would explain better here, I only know that slapi-nis uses
>  >>getgrouplist() during memberUID processing, but I'm fuzzy on the
>  >>details.
>  >>
>  >>>
>  >>>>I already know about two projects that would benefit from faster
>  >>>>initgroups operation - GlusterFS uses getgrouplist() quite a lot with
>  >>>>some setups and also in the IPA server mode, the SSSD running on the
>  >>>>server becomes a bit of a bottlenec for some operations.
>  >>>
>  >>>Is the problem related to accessing the main ldb cache, or the fact we
>  >>>also go out and contact the IdM server ?
>  >>
>  >>Mostly the cache. In particular, with the GlusterFS case, there was a
>  >>huge filesystem directory that triggered 4000+ getgrouplist() calls, one
>  >>per a file. Even though the getgrouplist() calls were for fewer than 10
>  >>users, we would hit LDB every time and that produced quite a load on the
>  >>SSSD.
>  >>
>  >>>
>  >>>>Are there any technical reasons against a new memcache file that would
>  >>>>cache initgroups?
>  >>>
>  >>>The main reason would be potential inconsistency between this cache and
>  >>>the users and groups caches, but I do not think that would be really a
>  >>>big deal if there is a pretty good reason for the cache.
>  >>
>  >>Hm, I see, then we have the same information duplicated in two caches,
>  >>just from a different angle. However, for cases where precision really
>  >>matters (access control), we don't use these caches anyway.
>  >>
>  >>Could we mitigate this problem by invalidating the memcache when group
>  >>membership changes at all during a backend operation? I think changes to
>  >>LDAP object are so rare, that invalidating the memcache (which is quite
>  >>shortlived anyway) would be worth it.
>  >>
>  >>>
>  >>>Do we need to insure all groups are resolvable in the groups cache ?
>  >>
>  >>I don't think so, if the group is not in the groups cache, then we just
>  >>hit the responder.
>  >>
>  >>>Or is it ok if gids returned by the getgrouplist are not immediately
>  >>>available in the groups cache ?
>  >>>What about the user ?
>  >>
>  >>I *think* the user could also be left out. If the application also needs
>  >>user data, it simply calls getpwnam -- like id(1) does. getgrouplist is
>  >>a different interface with different outputs.
>  >>
>  >>>
>  >>>Simo.
>  >>>
>  >>>--
>  >>>Simo Sorce * Red Hat, Inc * New York
>

[Simo Sorce]

On Thu, 6 Nov 2014 22:02:29 +0100
Niels de Vos <ndevos(a)redhat.com&gt; wrote:

>  On Thu, Nov 06, 2014 at 11:45:18PM +0530, Vijay Bellur wrote:
>  > On 11/03/2014 08:12 PM, Jakub Hrozek wrote:
>  > >On Mon, Nov 03, 2014 at 03:41:43PM +0100, Jakub Hrozek wrote:
>  > >>On Mon, Nov 03, 2014 at 08:53:06AM -0500, Simo Sorce wrote:
>  > >>>On Mon, 3 Nov 2014 13:57:08 +0100
>  > >>>Jakub Hrozek <jhrozek(a)redhat.com&gt; wrote:
>  > >>>
>  > >>>>Hi,
>  > >>>>
>  > >>>>we had short discussion on $SUBJECT with Simo on IRC already,
>  > >>>>but there are multiple people involved from multiple timezones,
>  > >>>>so I think a mailing list thread would be better trackable.
>  > >>>>
>  > >>>>Can we add another memory cache file to SSSD, that would track
>  > >>>>initgroups/getgrouplist results for the NSS responder? I realize
>  > >>>>initgroups is a bit different operation than getpw{uid,nam} and
>  > >>>>getgr{gid,nam} but what if the new memcache was only used by
>  > >>>>the NSS responder and at the same time invalidated when
>  > >>>>initgroups is initiated by the PAM responder to ensure the
>  > >>>>memcache is up-to-date?
>  > >>>
>  > >>>Can you describe the use case before jumping into a proposed
>  > >>>solution ?
>  > >>
>  > >>Many getgrouplist() or initgroups() calls in a quick succession.
>  > >>One user is GlusterFS -- I'm not quite sure what the reason is
>  > >>there, maybe Vijay can elaborate.
>  > >
>  > 
>  > GlusterFS server invokes getgrouplist() to identify gids associated
>  > with an user on whose behalf a rpc request has been sent over the
>  > wire. There is a gid caching layer in GlusterFS and getgrouplist()
>  > does get called only if there is a gid cache miss. In the worst
>  > case, getgrouplist() can be invoked for every rpc request that
>  > GlusterFS receives and that seems to be the case in a deployment
>  > where we found that sssd was being busy. I am not certain about the
>  > sequence of operations that can cause the cache to be missed.
>  > 
>  > Adding Niels who is more familiar with the gid resolution & caching
>  > features in GlusterFS.
>  
>  Just to add some background information on the getgrouplist().
>  GlusterFS uses several processes that can call getgrouplist():
>  - NFS-server, a single process per system
>  - brick, a process per exported filesystem/directory, potentally
>  several per system
>  
>    [Here, a Gluster environment has many systems (vm/physical). Each
>     system normally runs the NFS-server, and a number of brick
>  processes. The layout of the volume is important, but it is very
>  common to have one or more distributed volumes that use multiple
>  bricks on the same system (and many other systems).]
>  
>  The need for resolving the groups of a user comes in when users belong
>  to many groups. The RPC protocols can not carry a huge list of groups,
>  so the resolving can be done on the server side when the protocol hits
>  its limits (> 16 for NFS, approx. > 93 for GlusterFS).
>  
>  Upon using a Gluster volume, certain operations are sent to all the
>  bricks (i.e. some directory related operations). I can imagine that
>  a network share which is used by many users, trigger many
>  getgrouplist() calls in different brick processes at the (almost)
>  same time.
>  
>  For reference, the usage of getgrouplist() in the brick process can be
>  found here:
>  -
>
> https://github.com/gluster/glusterfs/blob/master/xlators/protocol/server/...
>  
>  The gid_resolve() function get called in case the brick process should
>  resolve the groups (and ignore the list of groups from the protocol).
>  It uses the gidcache functions from a private library:
>  -
>  https://github.com/gluster/glusterfs/blob/master/libglusterfs/src/gidcache.h
>  -
>  https://github.com/gluster/glusterfs/blob/master/libglusterfs/src/gidcache.c
>  
>  The default time for the gidcache to expire is 2 seconds. Users should
>  be able to configure this to 30 seconds (or anything else) with:
>  
>      # gluster volume set <VOLUME> server.gid-timeout 30
>  
>  
>  I think this should explain the use-case sufficiently, but let me know
>  if there are any remaining questions. It might well be possible to
>  make this code more sssd friendly. I'm sure that we as Gluster
>  developers are open to any suggestions. 

TBH this looks a little bit strange, other filesystems (as well as the
kernel) create a credentials token when a user first authenticate and
keep these credentials attached to the user session for the duration.
Why does GlusterFS keeps hammering the system requesting the same
information again and again ?

Keep in mind that the use of getgrouplist() is an inherently costly
operation. Even adding caches, the system cannot cache for long because
it needs to return updated results eventually. Only the application
know when a user session terminates and/or the list needs to be
refreshed, so "caching" for this type of operation should be done
mostly on the application side.

Simo.

-- 
Simo Sorce * Red Hat, Inc * New York

[Niels de Vos]

On Thu, Nov 06, 2014 at 05:32:53PM -0500, Simo Sorce wrote:
>  On Thu, 6 Nov 2014 22:02:29 +0100
>  Niels de Vos <ndevos(a)redhat.com&gt; wrote:
>  
>  > On Thu, Nov 06, 2014 at 11:45:18PM +0530, Vijay Bellur wrote:
>  > > On 11/03/2014 08:12 PM, Jakub Hrozek wrote:
>  > > >On Mon, Nov 03, 2014 at 03:41:43PM +0100, Jakub Hrozek wrote:
>  > > >>On Mon, Nov 03, 2014 at 08:53:06AM -0500, Simo Sorce wrote:
>  > > >>>On Mon, 3 Nov 2014 13:57:08 +0100
>  > > >>>Jakub Hrozek <jhrozek(a)redhat.com&gt; wrote:
>  > > >>>
>  > > >>>>Hi,
>  > > >>>>
>  > > >>>>we had short discussion on $SUBJECT with Simo on IRC already,
>  > > >>>>but there are multiple people involved from multiple
> timezones,
>  > > >>>>so I think a mailing list thread would be better trackable.
>  > > >>>>
>  > > >>>>Can we add another memory cache file to SSSD, that would track
>  > > >>>>initgroups/getgrouplist results for the NSS responder? I
> realize
>  > > >>>>initgroups is a bit different operation than getpw{uid,nam}
> and
>  > > >>>>getgr{gid,nam} but what if the new memcache was only used by
>  > > >>>>the NSS responder and at the same time invalidated when
>  > > >>>>initgroups is initiated by the PAM responder to ensure the
>  > > >>>>memcache is up-to-date?
>  > > >>>
>  > > >>>Can you describe the use case before jumping into a proposed
>  > > >>>solution ?
>  > > >>
>  > > >>Many getgrouplist() or initgroups() calls in a quick succession.
>  > > >>One user is GlusterFS -- I'm not quite sure what the reason is
>  > > >>there, maybe Vijay can elaborate.
>  > > >
>  > > 
>  > > GlusterFS server invokes getgrouplist() to identify gids associated
>  > > with an user on whose behalf a rpc request has been sent over the
>  > > wire. There is a gid caching layer in GlusterFS and getgrouplist()
>  > > does get called only if there is a gid cache miss. In the worst
>  > > case, getgrouplist() can be invoked for every rpc request that
>  > > GlusterFS receives and that seems to be the case in a deployment
>  > > where we found that sssd was being busy. I am not certain about the
>  > > sequence of operations that can cause the cache to be missed.
>  > > 
>  > > Adding Niels who is more familiar with the gid resolution & caching
>  > > features in GlusterFS.
>  > 
>  > Just to add some background information on the getgrouplist().
>  > GlusterFS uses several processes that can call getgrouplist():
>  > - NFS-server, a single process per system
>  > - brick, a process per exported filesystem/directory, potentally
>  > several per system
>  > 
>  >   [Here, a Gluster environment has many systems (vm/physical). Each
>  >    system normally runs the NFS-server, and a number of brick
>  > processes. The layout of the volume is important, but it is very
>  > common to have one or more distributed volumes that use multiple
>  > bricks on the same system (and many other systems).]
>  > 
>  > The need for resolving the groups of a user comes in when users belong
>  > to many groups. The RPC protocols can not carry a huge list of groups,
>  > so the resolving can be done on the server side when the protocol hits
>  > its limits (> 16 for NFS, approx. > 93 for GlusterFS).
>  > 
>  > Upon using a Gluster volume, certain operations are sent to all the
>  > bricks (i.e. some directory related operations). I can imagine that
>  > a network share which is used by many users, trigger many
>  > getgrouplist() calls in different brick processes at the (almost)
>  > same time.
>  > 
>  > For reference, the usage of getgrouplist() in the brick process can be
>  > found here:
>  > -
>  >
> https://github.com/gluster/glusterfs/blob/master/xlators/protocol/server/...
>  > 
>  > The gid_resolve() function get called in case the brick process should
>  > resolve the groups (and ignore the list of groups from the protocol).
>  > It uses the gidcache functions from a private library:
>  > -
>  > https://github.com/gluster/glusterfs/blob/master/libglusterfs/src/gidcache.h
>  > -
>  > https://github.com/gluster/glusterfs/blob/master/libglusterfs/src/gidcache.c
>  > 
>  > The default time for the gidcache to expire is 2 seconds. Users should
>  > be able to configure this to 30 seconds (or anything else) with:
>  > 
>  >     # gluster volume set <VOLUME> server.gid-timeout 30
>  > 
>  > 
>  > I think this should explain the use-case sufficiently, but let me know
>  > if there are any remaining questions. It might well be possible to
>  > make this code more sssd friendly. I'm sure that we as Gluster
>  > developers are open to any suggestions.
>  
>  
>  TBH this looks a little bit strange, other filesystems (as well as the
>  kernel) create a credentials token when a user first authenticate and
>  keep these credentials attached to the user session for the duration.
>  Why does GlusterFS keeps hammering the system requesting the same
>  information again and again ? 
The GlusterFS protocol itself is very much stateless, similar to NFSv3.
We need all the groups of the user on the server-side (brick) to allow
the backing filesystem (mostly XFS) perform the permission checking. In
the current GlusterFS protocol, there is no user authentication. (Well,
there has been work done on adding support for SSL, maybe that could be
used for tracking sessions on a per-client, not user, basis.)

Just for clarity, a GlusterFS client (like a fuse-mount, or the
samba/vfs_glusterfs module) is used by many different users. The client
builds the connection to the volume. After that, all users with access
to the fuse-mount or samba-share are using the same client connection.

By default the client sends a list of groups in each RPC request, and
the server-side trusts the list the client provides. However, for
environments where these lists are too small to hold all the groups,
there is an option to do the group resolving on the server side. This is
the "server.manage-gids" volume option, which acts very much like the
"rpc.mountd --manage-gids" functionality for NFS.

>  Keep in mind that the use of getgrouplist() is an inherently costly
>  operation. Even adding caches, the system cannot cache for long because
>  it needs to return updated results eventually. Only the application
>  know when a user session terminates and/or the list needs to be
>  refreshed, so "caching" for this type of operation should be done
>  mostly on the application side. 
I assume that your "application side" here is the brick process that
runs on the same system as sssd. As mentioned above, the brick processes
do cache the result of getgrouplist(). It may well be possible that the
default expiry of 2 seconds is too short for many environments. But
users can change that timeout easily with the "server.gid-timeout"
volume option.

> From my understanding of this thread, we (the Gluster Community) have
two things to do:

1. Clearly document side-effects that can be caused by enabling the
   "server.manage-gids" option, and suggest increasing the
   "server.gid-timeout" value (maybe change the default?).

2. Think about improving the GlusterFS protocol(s) and introduce some
   kind of credentials token that is linked with the groups of a user.
   Token expiry should invalidate the group-cache. One option would be
   to use Kerberos like NFS (RPCSEC_GSS).


Does this all make sense to others too? I'm adding gluster-devel@ to CC
so that others can chime in and this topic won't be forgotton.

Thanks,
Niels

[Jakub Hrozek]

On Fri, Nov 07, 2014 at 09:59:32AM +0100, Niels de Vos wrote:
>  On Thu, Nov 06, 2014 at 05:32:53PM -0500, Simo Sorce wrote:
>  > On Thu, 6 Nov 2014 22:02:29 +0100
>  > Niels de Vos <ndevos(a)redhat.com&gt; wrote:
>  > 
>  > > On Thu, Nov 06, 2014 at 11:45:18PM +0530, Vijay Bellur wrote:
>  > > > On 11/03/2014 08:12 PM, Jakub Hrozek wrote:
>  > > > >On Mon, Nov 03, 2014 at 03:41:43PM +0100, Jakub Hrozek wrote:
>  > > > >>On Mon, Nov 03, 2014 at 08:53:06AM -0500, Simo Sorce wrote:
>  > > > >>>On Mon, 3 Nov 2014 13:57:08 +0100
>  > > > >>>Jakub Hrozek <jhrozek(a)redhat.com&gt; wrote:
>  > > > >>>
>  > > > >>>>Hi,
>  > > > >>>>
>  > > > >>>>we had short discussion on $SUBJECT with Simo on IRC
> already,
>  > > > >>>>but there are multiple people involved from multiple
> timezones,
>  > > > >>>>so I think a mailing list thread would be better
> trackable.
>  > > > >>>>
>  > > > >>>>Can we add another memory cache file to SSSD, that would
> track
>  > > > >>>>initgroups/getgrouplist results for the NSS responder? I
> realize
>  > > > >>>>initgroups is a bit different operation than
> getpw{uid,nam} and
>  > > > >>>>getgr{gid,nam} but what if the new memcache was only used
> by
>  > > > >>>>the NSS responder and at the same time invalidated when
>  > > > >>>>initgroups is initiated by the PAM responder to ensure
> the
>  > > > >>>>memcache is up-to-date?
>  > > > >>>
>  > > > >>>Can you describe the use case before jumping into a proposed
>  > > > >>>solution ?
>  > > > >>
>  > > > >>Many getgrouplist() or initgroups() calls in a quick succession.
>  > > > >>One user is GlusterFS -- I'm not quite sure what the reason
> is
>  > > > >>there, maybe Vijay can elaborate.
>  > > > >
>  > > > 
>  > > > GlusterFS server invokes getgrouplist() to identify gids associated
>  > > > with an user on whose behalf a rpc request has been sent over the
>  > > > wire. There is a gid caching layer in GlusterFS and getgrouplist()
>  > > > does get called only if there is a gid cache miss. In the worst
>  > > > case, getgrouplist() can be invoked for every rpc request that
>  > > > GlusterFS receives and that seems to be the case in a deployment
>  > > > where we found that sssd was being busy. I am not certain about the
>  > > > sequence of operations that can cause the cache to be missed.
>  > > > 
>  > > > Adding Niels who is more familiar with the gid resolution & caching
>  > > > features in GlusterFS.
>  > > 
>  > > Just to add some background information on the getgrouplist().
>  > > GlusterFS uses several processes that can call getgrouplist():
>  > > - NFS-server, a single process per system
>  > > - brick, a process per exported filesystem/directory, potentally
>  > > several per system
>  > > 
>  > >   [Here, a Gluster environment has many systems (vm/physical). Each
>  > >    system normally runs the NFS-server, and a number of brick
>  > > processes. The layout of the volume is important, but it is very
>  > > common to have one or more distributed volumes that use multiple
>  > > bricks on the same system (and many other systems).]
>  > > 
>  > > The need for resolving the groups of a user comes in when users belong
>  > > to many groups. The RPC protocols can not carry a huge list of groups,
>  > > so the resolving can be done on the server side when the protocol hits
>  > > its limits (> 16 for NFS, approx. > 93 for GlusterFS).
>  > > 
>  > > Upon using a Gluster volume, certain operations are sent to all the
>  > > bricks (i.e. some directory related operations). I can imagine that
>  > > a network share which is used by many users, trigger many
>  > > getgrouplist() calls in different brick processes at the (almost)
>  > > same time.
>  > > 
>  > > For reference, the usage of getgrouplist() in the brick process can be
>  > > found here:
>  > > -
>  > >
> https://github.com/gluster/glusterfs/blob/master/xlators/protocol/server/...
>  > > 
>  > > The gid_resolve() function get called in case the brick process should
>  > > resolve the groups (and ignore the list of groups from the protocol).
>  > > It uses the gidcache functions from a private library:
>  > > -
>  > > https://github.com/gluster/glusterfs/blob/master/libglusterfs/src/gidcache.h
>  > > -
>  > > https://github.com/gluster/glusterfs/blob/master/libglusterfs/src/gidcache.c
>  > > 
>  > > The default time for the gidcache to expire is 2 seconds. Users should
>  > > be able to configure this to 30 seconds (or anything else) with:
>  > > 
>  > >     # gluster volume set <VOLUME> server.gid-timeout 30
>  > > 
>  > > 
>  > > I think this should explain the use-case sufficiently, but let me know
>  > > if there are any remaining questions. It might well be possible to
>  > > make this code more sssd friendly. I'm sure that we as Gluster
>  > > developers are open to any suggestions.
>  > 
>  > 
>  > TBH this looks a little bit strange, other filesystems (as well as the
>  > kernel) create a credentials token when a user first authenticate and
>  > keep these credentials attached to the user session for the duration.
>  > Why does GlusterFS keeps hammering the system requesting the same
>  > information again and again ?
>  
>  The GlusterFS protocol itself is very much stateless, similar to NFSv3.
>  We need all the groups of the user on the server-side (brick) to allow
>  the backing filesystem (mostly XFS) perform the permission checking. In
>  the current GlusterFS protocol, there is no user authentication. (Well,
>  there has been work done on adding support for SSL, maybe that could be
>  used for tracking sessions on a per-client, not user, basis.)
>  
>  Just for clarity, a GlusterFS client (like a fuse-mount, or the
>  samba/vfs_glusterfs module) is used by many different users. The client
>  builds the connection to the volume. After that, all users with access
>  to the fuse-mount or samba-share are using the same client connection.
>  
>  By default the client sends a list of groups in each RPC request, and
>  the server-side trusts the list the client provides. However, for
>  environments where these lists are too small to hold all the groups,
>  there is an option to do the group resolving on the server side. This is
>  the "server.manage-gids" volume option, which acts very much like the
>  "rpc.mountd --manage-gids" functionality for NFS.
>  
>  > Keep in mind that the use of getgrouplist() is an inherently costly
>  > operation. Even adding caches, the system cannot cache for long because
>  > it needs to return updated results eventually. Only the application
>  > know when a user session terminates and/or the list needs to be
>  > refreshed, so "caching" for this type of operation should be done
>  > mostly on the application side.
>  
>  I assume that your "application side" here is the brick process that
>  runs on the same system as sssd. As mentioned above, the brick processes
>  do cache the result of getgrouplist(). It may well be possible that the
>  default expiry of 2 seconds is too short for many environments. But
>  users can change that timeout easily with the "server.gid-timeout"
>  volume option. 
I guess that might be a viable option to work around the problem for the
user who initially reported it, but it also doesn't align with what I
saw in the logs..the sssd_nss logs showed 4000 initgroup requests over
two minutes from maybe about 10 users..

>  
>  From my understanding of this thread, we (the Gluster Community) have
>  two things to do:
>  
>  1. Clearly document side-effects that can be caused by enabling the
>     "server.manage-gids" option, and suggest increasing the
>     "server.gid-timeout" value (maybe change the default?).
>  
>  2. Think about improving the GlusterFS protocol(s) and introduce some
>     kind of credentials token that is linked with the groups of a user.
>     Token expiry should invalidate the group-cache. One option would be
>     to use Kerberos like NFS (RPCSEC_GSS).
>  
>  
>  Does this all make sense to others too? I'm adding gluster-devel@ to CC
>  so that others can chime in and this topic won't be forgotton.
>  
>  Thanks,
>  Niels 
And on the SSSD side, we need to think about an initgroups cache. So far
I filed ticket https://fedorahosted.org/sssd/ticket/2485 listing the two
options Simo outlined earlier.

GlusterFS is not the only project that requested faster initgroups
caching, Alexander's slapi-nis would also benefit from the new cache
(Although with slapi-nis we also have a bit conflicting RFE to stop
using NSS interfaces and go to SSSD directly, but that's something for
us to solve..)

[Lukas Slebodnik]

On (07/11/14 10:13), Jakub Hrozek wrote:
> On Fri, Nov 07, 2014 at 09:59:32AM +0100, Niels de Vos wrote:
> > On Thu, Nov 06, 2014 at 05:32:53PM -0500, Simo Sorce wrote:
> > > On Thu, 6 Nov 2014 22:02:29 +0100
> > > Niels de Vos <ndevos(a)redhat.com&gt; wrote:
> > > 
> > > > On Thu, Nov 06, 2014 at 11:45:18PM +0530, Vijay Bellur wrote:
> > > > > On 11/03/2014 08:12 PM, Jakub Hrozek wrote:
> > > > > >On Mon, Nov 03, 2014 at 03:41:43PM +0100, Jakub Hrozek wrote:
> > > > > >>On Mon, Nov 03, 2014 at 08:53:06AM -0500, Simo Sorce wrote:
> > > > > >>>On Mon, 3 Nov 2014 13:57:08 +0100
> > > > > >>>Jakub Hrozek <jhrozek(a)redhat.com&gt; wrote:
> > > > > >>>
> > > > > >>>>Hi,
> > > > > >>>>
> > > > > >>>>we had short discussion on $SUBJECT with Simo on IRC
> already,
> > > > > >>>>but there are multiple people involved from multiple
> timezones,
> > > > > >>>>so I think a mailing list thread would be better
> trackable.
> > > > > >>>>
> > > > > >>>>Can we add another memory cache file to SSSD, that
> would track
> > > > > >>>>initgroups/getgrouplist results for the NSS responder?
> I realize
> > > > > >>>>initgroups is a bit different operation than
> getpw{uid,nam} and
> > > > > >>>>getgr{gid,nam} but what if the new memcache was only
> used by
> > > > > >>>>the NSS responder and at the same time invalidated
> when
> > > > > >>>>initgroups is initiated by the PAM responder to ensure
> the
> > > > > >>>>memcache is up-to-date?
> > > > > >>>
> > > > > >>>Can you describe the use case before jumping into a
> proposed
> > > > > >>>solution ?
> > > > > >>
> > > > > >>Many getgrouplist() or initgroups() calls in a quick
> succession.
> > > > > >>One user is GlusterFS -- I'm not quite sure what the
> reason is
> > > > > >>there, maybe Vijay can elaborate.
> > > > > >
> > > > > 
> > > > > GlusterFS server invokes getgrouplist() to identify gids associated
> > > > > with an user on whose behalf a rpc request has been sent over the
> > > > > wire. There is a gid caching layer in GlusterFS and getgrouplist()
> > > > > does get called only if there is a gid cache miss. In the worst
> > > > > case, getgrouplist() can be invoked for every rpc request that
> > > > > GlusterFS receives and that seems to be the case in a deployment
> > > > > where we found that sssd was being busy. I am not certain about the
> > > > > sequence of operations that can cause the cache to be missed.
> > > > > 
> > > > > Adding Niels who is more familiar with the gid resolution &
> caching
> > > > > features in GlusterFS.
> > > > 
> > > > Just to add some background information on the getgrouplist().
> > > > GlusterFS uses several processes that can call getgrouplist():
> > > > - NFS-server, a single process per system
> > > > - brick, a process per exported filesystem/directory, potentally
> > > > several per system
> > > > 
> > > >   [Here, a Gluster environment has many systems (vm/physical). Each
> > > >    system normally runs the NFS-server, and a number of brick
> > > > processes. The layout of the volume is important, but it is very
> > > > common to have one or more distributed volumes that use multiple
> > > > bricks on the same system (and many other systems).]
> > > > 
> > > > The need for resolving the groups of a user comes in when users belong
> > > > to many groups. The RPC protocols can not carry a huge list of groups,
> > > > so the resolving can be done on the server side when the protocol hits
> > > > its limits (> 16 for NFS, approx. > 93 for GlusterFS).
> > > > 
> > > > Upon using a Gluster volume, certain operations are sent to all the
> > > > bricks (i.e. some directory related operations). I can imagine that
> > > > a network share which is used by many users, trigger many
> > > > getgrouplist() calls in different brick processes at the (almost)
> > > > same time.
> > > > 
> > > > For reference, the usage of getgrouplist() in the brick process can be
> > > > found here:
> > > > -
> > > >
> https://github.com/gluster/glusterfs/blob/master/xlators/protocol/server/...
> > > > 
> > > > The gid_resolve() function get called in case the brick process should
> > > > resolve the groups (and ignore the list of groups from the protocol).
> > > > It uses the gidcache functions from a private library:
> > > > -
> > > >
> https://github.com/gluster/glusterfs/blob/master/libglusterfs/src/gidcache.h
> > > > -
> > > >
> https://github.com/gluster/glusterfs/blob/master/libglusterfs/src/gidcache.c
> > > > 
> > > > The default time for the gidcache to expire is 2 seconds. Users should
> > > > be able to configure this to 30 seconds (or anything else) with:
> > > > 
> > > >     # gluster volume set <VOLUME> server.gid-timeout 30
> > > > 
> > > > 
> > > > I think this should explain the use-case sufficiently, but let me know
> > > > if there are any remaining questions. It might well be possible to
> > > > make this code more sssd friendly. I'm sure that we as Gluster
> > > > developers are open to any suggestions.
> > > 
> > > 
> > > TBH this looks a little bit strange, other filesystems (as well as the
> > > kernel) create a credentials token when a user first authenticate and
> > > keep these credentials attached to the user session for the duration.
> > > Why does GlusterFS keeps hammering the system requesting the same
> > > information again and again ?
> > 
> > The GlusterFS protocol itself is very much stateless, similar to NFSv3.
> > We need all the groups of the user on the server-side (brick) to allow
> > the backing filesystem (mostly XFS) perform the permission checking. In
> > the current GlusterFS protocol, there is no user authentication. (Well,
> > there has been work done on adding support for SSL, maybe that could be
> > used for tracking sessions on a per-client, not user, basis.)
> > 
> > Just for clarity, a GlusterFS client (like a fuse-mount, or the
> > samba/vfs_glusterfs module) is used by many different users. The client
> > builds the connection to the volume. After that, all users with access
> > to the fuse-mount or samba-share are using the same client connection.
> > 
> > By default the client sends a list of groups in each RPC request, and
> > the server-side trusts the list the client provides. However, for
> > environments where these lists are too small to hold all the groups,
> > there is an option to do the group resolving on the server side. This is
> > the "server.manage-gids" volume option, which acts very much like the
> > "rpc.mountd --manage-gids" functionality for NFS.
> > 
> > > Keep in mind that the use of getgrouplist() is an inherently costly
> > > operation. Even adding caches, the system cannot cache for long because
> > > it needs to return updated results eventually. Only the application
> > > know when a user session terminates and/or the list needs to be
> > > refreshed, so "caching" for this type of operation should be done
> > > mostly on the application side.
> > 
> > I assume that your "application side" here is the brick process that
> > runs on the same system as sssd. As mentioned above, the brick processes
> > do cache the result of getgrouplist(). It may well be possible that the
> > default expiry of 2 seconds is too short for many environments. But
> > users can change that timeout easily with the "server.gid-timeout"
> > volume option.
>
> I guess that might be a viable option to work around the problem for the
> user who initially reported it, but it also doesn't align with what I
> saw in the logs..the sssd_nss logs showed 4000 initgroup requests over
> two minutes from maybe about 10 users..
>
> > 
> > From my understanding of this thread, we (the Gluster Community) have
> > two things to do:
> > 
> > 1. Clearly document side-effects that can be caused by enabling the
> >    "server.manage-gids" option, and suggest increasing the
> >    "server.gid-timeout" value (maybe change the default?).
> > 
> > 2. Think about improving the GlusterFS protocol(s) and introduce some
> >    kind of credentials token that is linked with the groups of a user.
> >    Token expiry should invalidate the group-cache. One option would be
> >    to use Kerberos like NFS (RPCSEC_GSS).
> > 
> > 
> > Does this all make sense to others too? I'm adding gluster-devel@ to CC
> > so that others can chime in and this topic won't be forgotton.
> > 
> > Thanks,
> > Niels
>
> And on the SSSD side, we need to think about an initgroups cache. So far
> I filed ticket https://fedorahosted.org/sssd/ticket/2485 listing the two
> options Simo outlined earlier.
>
> GlusterFS is not the only project that requested faster initgroups
> caching, Alexander's slapi-nis would also benefit from the new cache
> (Although with slapi-nis we also have a bit conflicting RFE to stop
> using NSS interfaces and go to SSSD directly, but that's something for
> us to solve..) 
memory cache is used in nss responder as well :-)

LS

[Jakub Hrozek]

On Fri, Nov 07, 2014 at 10:17:30AM +0100, Lukas Slebodnik wrote:
>  On (07/11/14 10:13), Jakub Hrozek wrote:
>  >On Fri, Nov 07, 2014 at 09:59:32AM +0100, Niels de Vos wrote:
>  >> On Thu, Nov 06, 2014 at 05:32:53PM -0500, Simo Sorce wrote:
>  >> > On Thu, 6 Nov 2014 22:02:29 +0100
>  >> > Niels de Vos <ndevos(a)redhat.com&gt; wrote:
>  >> > 
>  >> > > On Thu, Nov 06, 2014 at 11:45:18PM +0530, Vijay Bellur wrote:
>  >> > > > On 11/03/2014 08:12 PM, Jakub Hrozek wrote:
>  >> > > > >On Mon, Nov 03, 2014 at 03:41:43PM +0100, Jakub Hrozek
> wrote:
>  >> > > > >>On Mon, Nov 03, 2014 at 08:53:06AM -0500, Simo Sorce
> wrote:
>  >> > > > >>>On Mon, 3 Nov 2014 13:57:08 +0100
>  >> > > > >>>Jakub Hrozek <jhrozek(a)redhat.com&gt; wrote:
>  >> > > > >>>
>  >> > > > >>>>Hi,
>  >> > > > >>>>
>  >> > > > >>>>we had short discussion on $SUBJECT with Simo on
> IRC already,
>  >> > > > >>>>but there are multiple people involved from
> multiple timezones,
>  >> > > > >>>>so I think a mailing list thread would be better
> trackable.
>  >> > > > >>>>
>  >> > > > >>>>Can we add another memory cache file to SSSD,
> that would track
>  >> > > > >>>>initgroups/getgrouplist results for the NSS
> responder? I realize
>  >> > > > >>>>initgroups is a bit different operation than
> getpw{uid,nam} and
>  >> > > > >>>>getgr{gid,nam} but what if the new memcache was
> only used by
>  >> > > > >>>>the NSS responder and at the same time
> invalidated when
>  >> > > > >>>>initgroups is initiated by the PAM responder to
> ensure the
>  >> > > > >>>>memcache is up-to-date?
>  >> > > > >>>
>  >> > > > >>>Can you describe the use case before jumping into a
> proposed
>  >> > > > >>>solution ?
>  >> > > > >>
>  >> > > > >>Many getgrouplist() or initgroups() calls in a quick
> succession.
>  >> > > > >>One user is GlusterFS -- I'm not quite sure what the
> reason is
>  >> > > > >>there, maybe Vijay can elaborate.
>  >> > > > >
>  >> > > > 
>  >> > > > GlusterFS server invokes getgrouplist() to identify gids
> associated
>  >> > > > with an user on whose behalf a rpc request has been sent over
> the
>  >> > > > wire. There is a gid caching layer in GlusterFS and
> getgrouplist()
>  >> > > > does get called only if there is a gid cache miss. In the worst
>  >> > > > case, getgrouplist() can be invoked for every rpc request that
>  >> > > > GlusterFS receives and that seems to be the case in a deployment
>  >> > > > where we found that sssd was being busy. I am not certain about
> the
>  >> > > > sequence of operations that can cause the cache to be missed.
>  >> > > > 
>  >> > > > Adding Niels who is more familiar with the gid resolution &
> caching
>  >> > > > features in GlusterFS.
>  >> > > 
>  >> > > Just to add some background information on the getgrouplist().
>  >> > > GlusterFS uses several processes that can call getgrouplist():
>  >> > > - NFS-server, a single process per system
>  >> > > - brick, a process per exported filesystem/directory, potentally
>  >> > > several per system
>  >> > > 
>  >> > >   [Here, a Gluster environment has many systems (vm/physical). Each
>  >> > >    system normally runs the NFS-server, and a number of brick
>  >> > > processes. The layout of the volume is important, but it is very
>  >> > > common to have one or more distributed volumes that use multiple
>  >> > > bricks on the same system (and many other systems).]
>  >> > > 
>  >> > > The need for resolving the groups of a user comes in when users
> belong
>  >> > > to many groups. The RPC protocols can not carry a huge list of
> groups,
>  >> > > so the resolving can be done on the server side when the protocol
> hits
>  >> > > its limits (> 16 for NFS, approx. > 93 for GlusterFS).
>  >> > > 
>  >> > > Upon using a Gluster volume, certain operations are sent to all the
>  >> > > bricks (i.e. some directory related operations). I can imagine that
>  >> > > a network share which is used by many users, trigger many
>  >> > > getgrouplist() calls in different brick processes at the (almost)
>  >> > > same time.
>  >> > > 
>  >> > > For reference, the usage of getgrouplist() in the brick process can
> be
>  >> > > found here:
>  >> > > -
>  >> > >
> https://github.com/gluster/glusterfs/blob/master/xlators/protocol/server/...
>  >> > > 
>  >> > > The gid_resolve() function get called in case the brick process
> should
>  >> > > resolve the groups (and ignore the list of groups from the protocol).
>  >> > > It uses the gidcache functions from a private library:
>  >> > > -
>  >> > >
> https://github.com/gluster/glusterfs/blob/master/libglusterfs/src/gidcache.h
>  >> > > -
>  >> > >
> https://github.com/gluster/glusterfs/blob/master/libglusterfs/src/gidcache.c
>  >> > > 
>  >> > > The default time for the gidcache to expire is 2 seconds. Users
> should
>  >> > > be able to configure this to 30 seconds (or anything else) with:
>  >> > > 
>  >> > >     # gluster volume set <VOLUME> server.gid-timeout 30
>  >> > > 
>  >> > > 
>  >> > > I think this should explain the use-case sufficiently, but let me
> know
>  >> > > if there are any remaining questions. It might well be possible to
>  >> > > make this code more sssd friendly. I'm sure that we as Gluster
>  >> > > developers are open to any suggestions.
>  >> > 
>  >> > 
>  >> > TBH this looks a little bit strange, other filesystems (as well as the
>  >> > kernel) create a credentials token when a user first authenticate and
>  >> > keep these credentials attached to the user session for the duration.
>  >> > Why does GlusterFS keeps hammering the system requesting the same
>  >> > information again and again ?
>  >> 
>  >> The GlusterFS protocol itself is very much stateless, similar to NFSv3.
>  >> We need all the groups of the user on the server-side (brick) to allow
>  >> the backing filesystem (mostly XFS) perform the permission checking. In
>  >> the current GlusterFS protocol, there is no user authentication. (Well,
>  >> there has been work done on adding support for SSL, maybe that could be
>  >> used for tracking sessions on a per-client, not user, basis.)
>  >> 
>  >> Just for clarity, a GlusterFS client (like a fuse-mount, or the
>  >> samba/vfs_glusterfs module) is used by many different users. The client
>  >> builds the connection to the volume. After that, all users with access
>  >> to the fuse-mount or samba-share are using the same client connection.
>  >> 
>  >> By default the client sends a list of groups in each RPC request, and
>  >> the server-side trusts the list the client provides. However, for
>  >> environments where these lists are too small to hold all the groups,
>  >> there is an option to do the group resolving on the server side. This is
>  >> the "server.manage-gids" volume option, which acts very much like the
>  >> "rpc.mountd --manage-gids" functionality for NFS.
>  >> 
>  >> > Keep in mind that the use of getgrouplist() is an inherently costly
>  >> > operation. Even adding caches, the system cannot cache for long because
>  >> > it needs to return updated results eventually. Only the application
>  >> > know when a user session terminates and/or the list needs to be
>  >> > refreshed, so "caching" for this type of operation should be
> done
>  >> > mostly on the application side.
>  >> 
>  >> I assume that your "application side" here is the brick process that
>  >> runs on the same system as sssd. As mentioned above, the brick processes
>  >> do cache the result of getgrouplist(). It may well be possible that the
>  >> default expiry of 2 seconds is too short for many environments. But
>  >> users can change that timeout easily with the "server.gid-timeout"
>  >> volume option.
>  >
>  >I guess that might be a viable option to work around the problem for the
>  >user who initially reported it, but it also doesn't align with what I
>  >saw in the logs..the sssd_nss logs showed 4000 initgroup requests over
>  >two minutes from maybe about 10 users..
>  >
>  >> 
>  >> From my understanding of this thread, we (the Gluster Community) have
>  >> two things to do:
>  >> 
>  >> 1. Clearly document side-effects that can be caused by enabling the
>  >>    "server.manage-gids" option, and suggest increasing the
>  >>    "server.gid-timeout" value (maybe change the default?).
>  >> 
>  >> 2. Think about improving the GlusterFS protocol(s) and introduce some
>  >>    kind of credentials token that is linked with the groups of a user.
>  >>    Token expiry should invalidate the group-cache. One option would be
>  >>    to use Kerberos like NFS (RPCSEC_GSS).
>  >> 
>  >> 
>  >> Does this all make sense to others too? I'm adding gluster-devel@ to CC
>  >> so that others can chime in and this topic won't be forgotton.
>  >> 
>  >> Thanks,
>  >> Niels
>  >
>  >And on the SSSD side, we need to think about an initgroups cache. So far
>  >I filed ticket https://fedorahosted.org/sssd/ticket/2485 listing the two
>  >options Simo outlined earlier.
>  >
>  >GlusterFS is not the only project that requested faster initgroups
>  >caching, Alexander's slapi-nis would also benefit from the new cache
>  >(Although with slapi-nis we also have a bit conflicting RFE to stop
>  >using NSS interfaces and go to SSSD directly, but that's something for
>  >us to solve..)
>  memory cache is used in nss responder as well :-)
>  
>  LS 
The whole point of the fastcache is to not hit the responder. The
responder is a writer in this scenario, the clients are readers.

[Niels de Vos]

On Fri, Nov 07, 2014 at 10:13:03AM +0100, Jakub Hrozek wrote:
>  On Fri, Nov 07, 2014 at 09:59:32AM +0100, Niels de Vos wrote:
>  > On Thu, Nov 06, 2014 at 05:32:53PM -0500, Simo Sorce wrote:
>  > > On Thu, 6 Nov 2014 22:02:29 +0100
>  > > Niels de Vos <ndevos(a)redhat.com&gt; wrote:
>  > > 
>  > > > On Thu, Nov 06, 2014 at 11:45:18PM +0530, Vijay Bellur wrote:
>  > > > > On 11/03/2014 08:12 PM, Jakub Hrozek wrote:
>  > > > > >On Mon, Nov 03, 2014 at 03:41:43PM +0100, Jakub Hrozek wrote:
>  > > > > >>On Mon, Nov 03, 2014 at 08:53:06AM -0500, Simo Sorce wrote:
>  > > > > >>>On Mon, 3 Nov 2014 13:57:08 +0100
>  > > > > >>>Jakub Hrozek <jhrozek(a)redhat.com&gt; wrote:
>  > > > > >>>
>  > > > > >>>>Hi,
>  > > > > >>>>
>  > > > > >>>>we had short discussion on $SUBJECT with Simo on IRC
> already,
>  > > > > >>>>but there are multiple people involved from multiple
> timezones,
>  > > > > >>>>so I think a mailing list thread would be better
> trackable.
>  > > > > >>>>
>  > > > > >>>>Can we add another memory cache file to SSSD, that
> would track
>  > > > > >>>>initgroups/getgrouplist results for the NSS
> responder? I realize
>  > > > > >>>>initgroups is a bit different operation than
> getpw{uid,nam} and
>  > > > > >>>>getgr{gid,nam} but what if the new memcache was only
> used by
>  > > > > >>>>the NSS responder and at the same time invalidated
> when
>  > > > > >>>>initgroups is initiated by the PAM responder to
> ensure the
>  > > > > >>>>memcache is up-to-date?
>  > > > > >>>
>  > > > > >>>Can you describe the use case before jumping into a
> proposed
>  > > > > >>>solution ?
>  > > > > >>
>  > > > > >>Many getgrouplist() or initgroups() calls in a quick
> succession.
>  > > > > >>One user is GlusterFS -- I'm not quite sure what the
> reason is
>  > > > > >>there, maybe Vijay can elaborate.
>  > > > > >
>  > > > > 
>  > > > > GlusterFS server invokes getgrouplist() to identify gids associated
>  > > > > with an user on whose behalf a rpc request has been sent over the
>  > > > > wire. There is a gid caching layer in GlusterFS and getgrouplist()
>  > > > > does get called only if there is a gid cache miss. In the worst
>  > > > > case, getgrouplist() can be invoked for every rpc request that
>  > > > > GlusterFS receives and that seems to be the case in a deployment
>  > > > > where we found that sssd was being busy. I am not certain about the
>  > > > > sequence of operations that can cause the cache to be missed.
>  > > > > 
>  > > > > Adding Niels who is more familiar with the gid resolution &
> caching
>  > > > > features in GlusterFS.
>  > > > 
>  > > > Just to add some background information on the getgrouplist().
>  > > > GlusterFS uses several processes that can call getgrouplist():
>  > > > - NFS-server, a single process per system
>  > > > - brick, a process per exported filesystem/directory, potentally
>  > > > several per system
>  > > > 
>  > > >   [Here, a Gluster environment has many systems (vm/physical). Each
>  > > >    system normally runs the NFS-server, and a number of brick
>  > > > processes. The layout of the volume is important, but it is very
>  > > > common to have one or more distributed volumes that use multiple
>  > > > bricks on the same system (and many other systems).]
>  > > > 
>  > > > The need for resolving the groups of a user comes in when users belong
>  > > > to many groups. The RPC protocols can not carry a huge list of groups,
>  > > > so the resolving can be done on the server side when the protocol hits
>  > > > its limits (> 16 for NFS, approx. > 93 for GlusterFS).
>  > > > 
>  > > > Upon using a Gluster volume, certain operations are sent to all the
>  > > > bricks (i.e. some directory related operations). I can imagine that
>  > > > a network share which is used by many users, trigger many
>  > > > getgrouplist() calls in different brick processes at the (almost)
>  > > > same time.
>  > > > 
>  > > > For reference, the usage of getgrouplist() in the brick process can be
>  > > > found here:
>  > > > -
>  > > >
> https://github.com/gluster/glusterfs/blob/master/xlators/protocol/server/...
>  > > > 
>  > > > The gid_resolve() function get called in case the brick process should
>  > > > resolve the groups (and ignore the list of groups from the protocol).
>  > > > It uses the gidcache functions from a private library:
>  > > > -
>  > > >
> https://github.com/gluster/glusterfs/blob/master/libglusterfs/src/gidcache.h
>  > > > -
>  > > >
> https://github.com/gluster/glusterfs/blob/master/libglusterfs/src/gidcache.c
>  > > > 
>  > > > The default time for the gidcache to expire is 2 seconds. Users should
>  > > > be able to configure this to 30 seconds (or anything else) with:
>  > > > 
>  > > >     # gluster volume set <VOLUME> server.gid-timeout 30
>  > > > 
>  > > > 
>  > > > I think this should explain the use-case sufficiently, but let me know
>  > > > if there are any remaining questions. It might well be possible to
>  > > > make this code more sssd friendly. I'm sure that we as Gluster
>  > > > developers are open to any suggestions.
>  > > 
>  > > 
>  > > TBH this looks a little bit strange, other filesystems (as well as the
>  > > kernel) create a credentials token when a user first authenticate and
>  > > keep these credentials attached to the user session for the duration.
>  > > Why does GlusterFS keeps hammering the system requesting the same
>  > > information again and again ?
>  > 
>  > The GlusterFS protocol itself is very much stateless, similar to NFSv3.
>  > We need all the groups of the user on the server-side (brick) to allow
>  > the backing filesystem (mostly XFS) perform the permission checking. In
>  > the current GlusterFS protocol, there is no user authentication. (Well,
>  > there has been work done on adding support for SSL, maybe that could be
>  > used for tracking sessions on a per-client, not user, basis.)
>  > 
>  > Just for clarity, a GlusterFS client (like a fuse-mount, or the
>  > samba/vfs_glusterfs module) is used by many different users. The client
>  > builds the connection to the volume. After that, all users with access
>  > to the fuse-mount or samba-share are using the same client connection.
>  > 
>  > By default the client sends a list of groups in each RPC request, and
>  > the server-side trusts the list the client provides. However, for
>  > environments where these lists are too small to hold all the groups,
>  > there is an option to do the group resolving on the server side. This is
>  > the "server.manage-gids" volume option, which acts very much like the
>  > "rpc.mountd --manage-gids" functionality for NFS.
>  > 
>  > > Keep in mind that the use of getgrouplist() is an inherently costly
>  > > operation. Even adding caches, the system cannot cache for long because
>  > > it needs to return updated results eventually. Only the application
>  > > know when a user session terminates and/or the list needs to be
>  > > refreshed, so "caching" for this type of operation should be done
>  > > mostly on the application side.
>  > 
>  > I assume that your "application side" here is the brick process that
>  > runs on the same system as sssd. As mentioned above, the brick processes
>  > do cache the result of getgrouplist(). It may well be possible that the
>  > default expiry of 2 seconds is too short for many environments. But
>  > users can change that timeout easily with the "server.gid-timeout"
>  > volume option.
>  
>  I guess that might be a viable option to work around the problem for the
>  user who initially reported it, but it also doesn't align with what I
>  saw in the logs..the sssd_nss logs showed 4000 initgroup requests over
>  two minutes from maybe about 10 users.. 
That would be easily possible in case the server.gid-timeout is set to 0
and the server hosts multiple bricks for the same volume. Each brick
process will call getgrouplist() for each RPC call it receives. I can
imagine an explosion of requests like that when some multi-threaded I/O
is done. More details on the Gluster environment and workload where this
was observed would help.

It is also possible that our gid-cache has a bug and does not apply its
timeout at all. This would be concerning and requires some investigation
on the Gluster side. But again, we'll need to verify the settings of the
Gluster volume for this.

>  
>  > 
>  > From my understanding of this thread, we (the Gluster Community) have
>  > two things to do:
>  > 
>  > 1. Clearly document side-effects that can be caused by enabling the
>  >    "server.manage-gids" option, and suggest increasing the
>  >    "server.gid-timeout" value (maybe change the default?).
>  > 
>  > 2. Think about improving the GlusterFS protocol(s) and introduce some
>  >    kind of credentials token that is linked with the groups of a user.
>  >    Token expiry should invalidate the group-cache. One option would be
>  >    to use Kerberos like NFS (RPCSEC_GSS).
>  > 
>  > 
>  > Does this all make sense to others too? I'm adding gluster-devel@ to CC
>  > so that others can chime in and this topic won't be forgotton.
>  > 
>  > Thanks,
>  > Niels
>  
>  And on the SSSD side, we need to think about an initgroups cache. So far
>  I filed ticket https://fedorahosted.org/sssd/ticket/2485 listing the two
>  options Simo outlined earlier.
>  
>  GlusterFS is not the only project that requested faster initgroups
>  caching, Alexander's slapi-nis would also benefit from the new cache
>  (Although with slapi-nis we also have a bit conflicting RFE to stop
>  using NSS interfaces and go to SSSD directly, but that's something for
>  us to solve..) 
Okay, thanks. We'll be filing bugs for our changes when we discussed it
a little more amoung the Gluster developers.

Cheers,
Niels

[Alexander Bokovoy]

On Fri, 07 Nov 2014, Niels de Vos wrote:
> On Thu, Nov 06, 2014 at 05:32:53PM -0500, Simo Sorce wrote:
> > On Thu, 6 Nov 2014 22:02:29 +0100
> > Niels de Vos <ndevos(a)redhat.com&gt; wrote:
> >
> > > On Thu, Nov 06, 2014 at 11:45:18PM +0530, Vijay Bellur wrote:
> > > > On 11/03/2014 08:12 PM, Jakub Hrozek wrote:
> > > > >On Mon, Nov 03, 2014 at 03:41:43PM +0100, Jakub Hrozek wrote:
> > > > >>On Mon, Nov 03, 2014 at 08:53:06AM -0500, Simo Sorce wrote:
> > > > >>>On Mon, 3 Nov 2014 13:57:08 +0100
> > > > >>>Jakub Hrozek <jhrozek(a)redhat.com&gt; wrote:
> > > > >>>
> > > > >>>>Hi,
> > > > >>>>
> > > > >>>>we had short discussion on $SUBJECT with Simo on IRC
> already,
> > > > >>>>but there are multiple people involved from multiple
> timezones,
> > > > >>>>so I think a mailing list thread would be better
> trackable.
> > > > >>>>
> > > > >>>>Can we add another memory cache file to SSSD, that would
> track
> > > > >>>>initgroups/getgrouplist results for the NSS responder? I
> realize
> > > > >>>>initgroups is a bit different operation than getpw{uid,nam}
> and
> > > > >>>>getgr{gid,nam} but what if the new memcache was only used
> by
> > > > >>>>the NSS responder and at the same time invalidated when
> > > > >>>>initgroups is initiated by the PAM responder to ensure the
> > > > >>>>memcache is up-to-date?
> > > > >>>
> > > > >>>Can you describe the use case before jumping into a proposed
> > > > >>>solution ?
> > > > >>
> > > > >>Many getgrouplist() or initgroups() calls in a quick succession.
> > > > >>One user is GlusterFS -- I'm not quite sure what the reason is
> > > > >>there, maybe Vijay can elaborate.
> > > > >
> > > >
> > > > GlusterFS server invokes getgrouplist() to identify gids associated
> > > > with an user on whose behalf a rpc request has been sent over the
> > > > wire. There is a gid caching layer in GlusterFS and getgrouplist()
> > > > does get called only if there is a gid cache miss. In the worst
> > > > case, getgrouplist() can be invoked for every rpc request that
> > > > GlusterFS receives and that seems to be the case in a deployment
> > > > where we found that sssd was being busy. I am not certain about the
> > > > sequence of operations that can cause the cache to be missed.
> > > >
> > > > Adding Niels who is more familiar with the gid resolution & caching
> > > > features in GlusterFS.
> > >
> > > Just to add some background information on the getgrouplist().
> > > GlusterFS uses several processes that can call getgrouplist():
> > > - NFS-server, a single process per system
> > > - brick, a process per exported filesystem/directory, potentally
> > > several per system
> > >
> > >   [Here, a Gluster environment has many systems (vm/physical). Each
> > >    system normally runs the NFS-server, and a number of brick
> > > processes. The layout of the volume is important, but it is very
> > > common to have one or more distributed volumes that use multiple
> > > bricks on the same system (and many other systems).]
> > >
> > > The need for resolving the groups of a user comes in when users belong
> > > to many groups. The RPC protocols can not carry a huge list of groups,
> > > so the resolving can be done on the server side when the protocol hits
> > > its limits (> 16 for NFS, approx. > 93 for GlusterFS).
> > >
> > > Upon using a Gluster volume, certain operations are sent to all the
> > > bricks (i.e. some directory related operations). I can imagine that
> > > a network share which is used by many users, trigger many
> > > getgrouplist() calls in different brick processes at the (almost)
> > > same time.
> > >
> > > For reference, the usage of getgrouplist() in the brick process can be
> > > found here:
> > > -
> > >
> https://github.com/gluster/glusterfs/blob/master/xlators/protocol/server/...
> > >
> > > The gid_resolve() function get called in case the brick process should
> > > resolve the groups (and ignore the list of groups from the protocol).
> > > It uses the gidcache functions from a private library:
> > > -
> > > https://github.com/gluster/glusterfs/blob/master/libglusterfs/src/gidcache.h
> > > -
> > > https://github.com/gluster/glusterfs/blob/master/libglusterfs/src/gidcache.c
> > >
> > > The default time for the gidcache to expire is 2 seconds. Users should
> > > be able to configure this to 30 seconds (or anything else) with:
> > >
> > >     # gluster volume set <VOLUME> server.gid-timeout 30
> > >
> > >
> > > I think this should explain the use-case sufficiently, but let me know
> > > if there are any remaining questions. It might well be possible to
> > > make this code more sssd friendly. I'm sure that we as Gluster
> > > developers are open to any suggestions. 
What I can see is that also NFS
xlator does call it in nfs_fix_groups():
https://github.com/gluster/glusterfs/blob/master/xlators/nfs/server/src/n...
which is then used in nlm4_file_open_and_resume().

> > TBH this looks a little bit strange, other filesystems (as well
> as the
> > kernel) create a credentials token when a user first authenticate and
> > keep these credentials attached to the user session for the duration.
> > Why does GlusterFS keeps hammering the system requesting the same
> > information again and again ?
>
> The GlusterFS protocol itself is very much stateless, similar to NFSv3.
> We need all the groups of the user on the server-side (brick) to allow
> the backing filesystem (mostly XFS) perform the permission checking. In
> the current GlusterFS protocol, there is no user authentication. (Well,
> there has been work done on adding support for SSL, maybe that could be
> used for tracking sessions on a per-client, not user, basis.)
>
> Just for clarity, a GlusterFS client (like a fuse-mount, or the
> samba/vfs_glusterfs module) is used by many different users. The client
> builds the connection to the volume. After that, all users with access
> to the fuse-mount or samba-share are using the same client connection.
>
> By default the client sends a list of groups in each RPC request, and
> the server-side trusts the list the client provides. However, for
> environments where these lists are too small to hold all the groups,
> there is an option to do the group resolving on the server side. This is
> the "server.manage-gids" volume option, which acts very much like the
> "rpc.mountd --manage-gids" functionality for NFS. 
In case of complex group
membership fetching the group list might take
longer than default 2 seconds which is not that unusual for LDAP-backed
configurations with high network latency. Setting group membership
caching to higher threshold by default is reasonable, as well as tying
it to a client connection/authentication source. After all, group
membership for a particular user doesn't really change every two
seconds, or even every 30 seconds.

It would definitely help to have a hand from GlusterFS protocol to
allow client to hint the server side that as result of authentication,
user properties did change, thus a refresh is needed on the server in
case gid cache is involved. That could be a per RPC flag and at worst
client would be forcing current behavior of dropping gid cache content
too often. There seem to be already too much trust to GlusterFS client
by the server side.

> > Keep in mind that the use of getgrouplist() is an inherently
> costly
> > operation. Even adding caches, the system cannot cache for long because
> > it needs to return updated results eventually. Only the application
> > know when a user session terminates and/or the list needs to be
> > refreshed, so "caching" for this type of operation should be done
> > mostly on the application side.
>
> I assume that your "application side" here is the brick process that
> runs on the same system as sssd. As mentioned above, the brick processes
> do cache the result of getgrouplist(). It may well be possible that the
> default expiry of 2 seconds is too short for many environments. But
> users can change that timeout easily with the "server.gid-timeout"
> volume option.
>
> From my understanding of this thread, we (the Gluster Community) have
> two things to do:
>
> 1. Clearly document side-effects that can be caused by enabling the
>    "server.manage-gids" option, and suggest increasing the
>    "server.gid-timeout" value (maybe change the default?).
>
> 2. Think about improving the GlusterFS protocol(s) and introduce some
>    kind of credentials token that is linked with the groups of a user.
>    Token expiry should invalidate the group-cache. One option would be
>    to use Kerberos like NFS (RPCSEC_GSS).
>
>
> Does this all make sense to others too? I'm adding gluster-devel@ to CC
> so that others can chime in and this topic won't be forgotton. 
Yes, both these
approaches make sense. Changing default to something
more reasonable like several minutes would already help to reduce
contention on an nss provider -- I've encountered that nss_files also
behaves badly when too many requests come from parallel threads from the
same process and can easily lock itself up in serializing access to
/etc/passwd or /etc/groups.

Perhaps you can also add a third one, like I proposed above -- to allow
per-RPC flag that gives a hint from client to the server about use of
cached gids in case of server.manage-gids option is set on the server
side.

-- 
/ Alexander Bokovoy

[Niels de Vos]

On Fri, Nov 07, 2014 at 12:13:59PM +0200, Alexander Bokovoy wrote:
>  On Fri, 07 Nov 2014, Niels de Vos wrote:
>  >On Thu, Nov 06, 2014 at 05:32:53PM -0500, Simo Sorce wrote:
>  >>On Thu, 6 Nov 2014 22:02:29 +0100
>  >>Niels de Vos <ndevos(a)redhat.com&gt; wrote:
>  >>
>  >>> On Thu, Nov 06, 2014 at 11:45:18PM +0530, Vijay Bellur wrote:
>  >>> > On 11/03/2014 08:12 PM, Jakub Hrozek wrote:
>  >>> > >On Mon, Nov 03, 2014 at 03:41:43PM +0100, Jakub Hrozek wrote:
>  >>> > >>On Mon, Nov 03, 2014 at 08:53:06AM -0500, Simo Sorce wrote:
>  >>> > >>>On Mon, 3 Nov 2014 13:57:08 +0100
>  >>> > >>>Jakub Hrozek <jhrozek(a)redhat.com&gt; wrote:
>  >>> > >>>
>  >>> > >>>>Hi,
>  >>> > >>>>
>  >>> > >>>>we had short discussion on $SUBJECT with Simo on IRC
> already,
>  >>> > >>>>but there are multiple people involved from multiple
> timezones,
>  >>> > >>>>so I think a mailing list thread would be better
> trackable.
>  >>> > >>>>
>  >>> > >>>>Can we add another memory cache file to SSSD, that
> would track
>  >>> > >>>>initgroups/getgrouplist results for the NSS responder?
> I realize
>  >>> > >>>>initgroups is a bit different operation than
> getpw{uid,nam} and
>  >>> > >>>>getgr{gid,nam} but what if the new memcache was only
> used by
>  >>> > >>>>the NSS responder and at the same time invalidated
> when
>  >>> > >>>>initgroups is initiated by the PAM responder to ensure
> the
>  >>> > >>>>memcache is up-to-date?
>  >>> > >>>
>  >>> > >>>Can you describe the use case before jumping into a
> proposed
>  >>> > >>>solution ?
>  >>> > >>
>  >>> > >>Many getgrouplist() or initgroups() calls in a quick
> succession.
>  >>> > >>One user is GlusterFS -- I'm not quite sure what the reason
> is
>  >>> > >>there, maybe Vijay can elaborate.
>  >>> > >
>  >>> >
>  >>> > GlusterFS server invokes getgrouplist() to identify gids associated
>  >>> > with an user on whose behalf a rpc request has been sent over the
>  >>> > wire. There is a gid caching layer in GlusterFS and getgrouplist()
>  >>> > does get called only if there is a gid cache miss. In the worst
>  >>> > case, getgrouplist() can be invoked for every rpc request that
>  >>> > GlusterFS receives and that seems to be the case in a deployment
>  >>> > where we found that sssd was being busy. I am not certain about the
>  >>> > sequence of operations that can cause the cache to be missed.
>  >>> >
>  >>> > Adding Niels who is more familiar with the gid resolution &
> caching
>  >>> > features in GlusterFS.
>  >>>
>  >>> Just to add some background information on the getgrouplist().
>  >>> GlusterFS uses several processes that can call getgrouplist():
>  >>> - NFS-server, a single process per system
>  >>> - brick, a process per exported filesystem/directory, potentally
>  >>> several per system
>  >>>
>  >>>   [Here, a Gluster environment has many systems (vm/physical). Each
>  >>>    system normally runs the NFS-server, and a number of brick
>  >>> processes. The layout of the volume is important, but it is very
>  >>> common to have one or more distributed volumes that use multiple
>  >>> bricks on the same system (and many other systems).]
>  >>>
>  >>> The need for resolving the groups of a user comes in when users belong
>  >>> to many groups. The RPC protocols can not carry a huge list of groups,
>  >>> so the resolving can be done on the server side when the protocol hits
>  >>> its limits (> 16 for NFS, approx. > 93 for GlusterFS).
>  >>>
>  >>> Upon using a Gluster volume, certain operations are sent to all the
>  >>> bricks (i.e. some directory related operations). I can imagine that
>  >>> a network share which is used by many users, trigger many
>  >>> getgrouplist() calls in different brick processes at the (almost)
>  >>> same time.
>  >>>
>  >>> For reference, the usage of getgrouplist() in the brick process can be
>  >>> found here:
>  >>> -
>  >>>
> https://github.com/gluster/glusterfs/blob/master/xlators/protocol/server/...
>  >>>
>  >>> The gid_resolve() function get called in case the brick process should
>  >>> resolve the groups (and ignore the list of groups from the protocol).
>  >>> It uses the gidcache functions from a private library:
>  >>> -
>  >>>
> https://github.com/gluster/glusterfs/blob/master/libglusterfs/src/gidcache.h
>  >>> -
>  >>>
> https://github.com/gluster/glusterfs/blob/master/libglusterfs/src/gidcache.c
>  >>>
>  >>> The default time for the gidcache to expire is 2 seconds. Users should
>  >>> be able to configure this to 30 seconds (or anything else) with:
>  >>>
>  >>>     # gluster volume set <VOLUME> server.gid-timeout 30
>  >>>
>  >>>
>  >>> I think this should explain the use-case sufficiently, but let me know
>  >>> if there are any remaining questions. It might well be possible to
>  >>> make this code more sssd friendly. I'm sure that we as Gluster
>  >>> developers are open to any suggestions.
>  What I can see is that also NFS xlator does call it in nfs_fix_groups():
>  https://github.com/gluster/glusterfs/blob/master/xlators/nfs/server/src/n...
>  which is then used in nlm4_file_open_and_resume(). 
Yes. The NFS-server is a single process that all Gluster volumes use.
The group cache is shared more efficiently when compared to the group
cache that gets duplicated in all the brick processes. Also, this is
only used when a client system mounts a volume over NFS. The rest of the
described functionality is pretty much the same. It is possible to that
group cashing is used in both the Gluster/NFS-server and the brick
processes.

>  
>  >>TBH this looks a little bit strange, other filesystems (as well as the
>  >>kernel) create a credentials token when a user first authenticate and
>  >>keep these credentials attached to the user session for the duration.
>  >>Why does GlusterFS keeps hammering the system requesting the same
>  >>information again and again ?
>  >
>  >The GlusterFS protocol itself is very much stateless, similar to NFSv3.
>  >We need all the groups of the user on the server-side (brick) to allow
>  >the backing filesystem (mostly XFS) perform the permission checking. In
>  >the current GlusterFS protocol, there is no user authentication. (Well,
>  >there has been work done on adding support for SSL, maybe that could be
>  >used for tracking sessions on a per-client, not user, basis.)
>  >
>  >Just for clarity, a GlusterFS client (like a fuse-mount, or the
>  >samba/vfs_glusterfs module) is used by many different users. The client
>  >builds the connection to the volume. After that, all users with access
>  >to the fuse-mount or samba-share are using the same client connection.
>  >
>  >By default the client sends a list of groups in each RPC request, and
>  >the server-side trusts the list the client provides. However, for
>  >environments where these lists are too small to hold all the groups,
>  >there is an option to do the group resolving on the server side. This is
>  >the "server.manage-gids" volume option, which acts very much like the
>  >"rpc.mountd --manage-gids" functionality for NFS.
>  In case of complex group membership fetching the group list might take
>  longer than default 2 seconds which is not that unusual for LDAP-backed
>  configurations with high network latency. Setting group membership
>  caching to higher threshold by default is reasonable, as well as tying
>  it to a client connection/authentication source. After all, group
>  membership for a particular user doesn't really change every two
>  seconds, or even every 30 seconds. 
Yes, I agree. If you have a suggestion for changing the default of 2
seconds to something more practical, we'll consider that. Any advise on
that is welcome. Even a description of things to take into account when
a user wants to configure the timeout would be appreciated.

>  It would definitely help to have a hand from GlusterFS protocol to
>  allow client to hint the server side that as result of authentication,
>  user properties did change, thus a refresh is needed on the server in
>  case gid cache is involved. That could be a per RPC flag and at worst
>  client would be forcing current behavior of dropping gid cache content
>  too often. There seem to be already too much trust to GlusterFS client
>  by the server side. 
I think using RPCSEC_GSS could be used for that? When the credentials of
a user are invalidated, or have changed (?), the process verifying the
credentials (brick process) should invalidate/refresh the group cache
too.

The GlusterFS protocol does not support RPCSEC_GSS yet, but we can
definitely think about adding that if it has an advantage here. The
protocol changes can be done relatively easy:
  - We would probably have something like a virtual xattr that a client
    can send to drop its own group cache.

For me, detecting the change in groups on the client-side would need
more thinking. Hints? The glusterfs-fuse client is all user-space, and
can be extended with functionalities easily.

I do not think NFS-clients can communicate a need for refreshing the
group list, so that might still be an issue. Gluster/NFS only supports
RPC/AUTH_UNIX and NFSv3. In future, nfs-ganesha will mostly be used, and
I think we are more flexible there.

>  >>Keep in mind that the use of getgrouplist() is an inherently
> costly
>  >>operation. Even adding caches, the system cannot cache for long because
>  >>it needs to return updated results eventually. Only the application
>  >>know when a user session terminates and/or the list needs to be
>  >>refreshed, so "caching" for this type of operation should be done
>  >>mostly on the application side.
>  >
>  >I assume that your "application side" here is the brick process that
>  >runs on the same system as sssd. As mentioned above, the brick processes
>  >do cache the result of getgrouplist(). It may well be possible that the
>  >default expiry of 2 seconds is too short for many environments. But
>  >users can change that timeout easily with the "server.gid-timeout"
>  >volume option.
>  >
>  >From my understanding of this thread, we (the Gluster Community) have
>  >two things to do:
>  >
>  >1. Clearly document side-effects that can be caused by enabling the
>  >  "server.manage-gids" option, and suggest increasing the
>  >  "server.gid-timeout" value (maybe change the default?).
>  >
>  >2. Think about improving the GlusterFS protocol(s) and introduce some
>  >  kind of credentials token that is linked with the groups of a user.
>  >  Token expiry should invalidate the group-cache. One option would be
>  >  to use Kerberos like NFS (RPCSEC_GSS).
>  >
>  >
>  >Does this all make sense to others too? I'm adding gluster-devel@ to CC
>  >so that others can chime in and this topic won't be forgotton.
>  Yes, both these approaches make sense. Changing default to something
>  more reasonable like several minutes would already help to reduce
>  contention on an nss provider -- I've encountered that nss_files also
>  behaves badly when too many requests come from parallel threads from the
>  same process and can easily lock itself up in serializing access to
>  /etc/passwd or /etc/groups.
>  
>  Perhaps you can also add a third one, like I proposed above -- to allow
>  per-RPC flag that gives a hint from client to the server about use of
>  cached gids in case of server.manage-gids option is set on the server
>  side. 
Thanks for sharing your thoughts! It really helps to get ideas and
opinions from others. It's much appreciated.

Niels

[Simo Sorce]

On Fri, 7 Nov 2014 09:59:32 +0100
Niels de Vos <ndevos(a)redhat.com&gt; wrote:

>  On Thu, Nov 06, 2014 at 05:32:53PM -0500, Simo Sorce wrote:
>  > On Thu, 6 Nov 2014 22:02:29 +0100
>  > Niels de Vos <ndevos(a)redhat.com&gt; wrote:
>  > 
>  > > On Thu, Nov 06, 2014 at 11:45:18PM +0530, Vijay Bellur wrote:
>  > > > On 11/03/2014 08:12 PM, Jakub Hrozek wrote:
>  > > > >On Mon, Nov 03, 2014 at 03:41:43PM +0100, Jakub Hrozek wrote:
>  > > > >>On Mon, Nov 03, 2014 at 08:53:06AM -0500, Simo Sorce wrote:
>  > > > >>>On Mon, 3 Nov 2014 13:57:08 +0100
>  > > > >>>Jakub Hrozek <jhrozek(a)redhat.com&gt; wrote:
>  > > > >>>
>  > > > >>>>Hi,
>  > > > >>>>
>  > > > >>>>we had short discussion on $SUBJECT with Simo on IRC
>  > > > >>>>already, but there are multiple people involved from
>  > > > >>>>multiple timezones, so I think a mailing list thread
> would
>  > > > >>>>be better trackable.
>  > > > >>>>
>  > > > >>>>Can we add another memory cache file to SSSD, that would
>  > > > >>>>track initgroups/getgrouplist results for the NSS
>  > > > >>>>responder? I realize initgroups is a bit different
>  > > > >>>>operation than getpw{uid,nam} and getgr{gid,nam} but what
>  > > > >>>>if the new memcache was only used by the NSS responder
> and
>  > > > >>>>at the same time invalidated when initgroups is initiated
>  > > > >>>>by the PAM responder to ensure the memcache is
> up-to-date?
>  > > > >>>
>  > > > >>>Can you describe the use case before jumping into a proposed
>  > > > >>>solution ?
>  > > > >>
>  > > > >>Many getgrouplist() or initgroups() calls in a quick
>  > > > >>succession. One user is GlusterFS -- I'm not quite sure what
>  > > > >>the reason is there, maybe Vijay can elaborate.
>  > > > >
>  > > > 
>  > > > GlusterFS server invokes getgrouplist() to identify gids
>  > > > associated with an user on whose behalf a rpc request has been
>  > > > sent over the wire. There is a gid caching layer in GlusterFS
>  > > > and getgrouplist() does get called only if there is a gid cache
>  > > > miss. In the worst case, getgrouplist() can be invoked for
>  > > > every rpc request that GlusterFS receives and that seems to be
>  > > > the case in a deployment where we found that sssd was being
>  > > > busy. I am not certain about the sequence of operations that
>  > > > can cause the cache to be missed.
>  > > > 
>  > > > Adding Niels who is more familiar with the gid resolution &
>  > > > caching features in GlusterFS.
>  > > 
>  > > Just to add some background information on the getgrouplist().
>  > > GlusterFS uses several processes that can call getgrouplist():
>  > > - NFS-server, a single process per system
>  > > - brick, a process per exported filesystem/directory, potentally
>  > > several per system
>  > > 
>  > >   [Here, a Gluster environment has many systems (vm/physical).
>  > > Each system normally runs the NFS-server, and a number of brick
>  > > processes. The layout of the volume is important, but it is very
>  > > common to have one or more distributed volumes that use multiple
>  > > bricks on the same system (and many other systems).]
>  > > 
>  > > The need for resolving the groups of a user comes in when users
>  > > belong to many groups. The RPC protocols can not carry a huge
>  > > list of groups, so the resolving can be done on the server side
>  > > when the protocol hits its limits (> 16 for NFS, approx. > 93 for
>  > > GlusterFS).
>  > > 
>  > > Upon using a Gluster volume, certain operations are sent to all
>  > > the bricks (i.e. some directory related operations). I can
>  > > imagine that a network share which is used by many users, trigger
>  > > many getgrouplist() calls in different brick processes at the
>  > > (almost) same time.
>  > > 
>  > > For reference, the usage of getgrouplist() in the brick process
>  > > can be found here:
>  > > -
>  > >
> https://github.com/gluster/glusterfs/blob/master/xlators/protocol/server/...
>  > > 
>  > > The gid_resolve() function get called in case the brick process
>  > > should resolve the groups (and ignore the list of groups from the
>  > > protocol). It uses the gidcache functions from a private library:
>  > > -
>  > > https://github.com/gluster/glusterfs/blob/master/libglusterfs/src/gidcache.h
>  > > -
>  > > https://github.com/gluster/glusterfs/blob/master/libglusterfs/src/gidcache.c
>  > > 
>  > > The default time for the gidcache to expire is 2 seconds. Users
>  > > should be able to configure this to 30 seconds (or anything else)
>  > > with:
>  > > 
>  > >     # gluster volume set <VOLUME> server.gid-timeout 30
>  > > 
>  > > 
>  > > I think this should explain the use-case sufficiently, but let me
>  > > know if there are any remaining questions. It might well be
>  > > possible to make this code more sssd friendly. I'm sure that we
>  > > as Gluster developers are open to any suggestions.
>  > 
>  > 
>  > TBH this looks a little bit strange, other filesystems (as well as
>  > the kernel) create a credentials token when a user first
>  > authenticate and keep these credentials attached to the user
>  > session for the duration. Why does GlusterFS keeps hammering the
>  > system requesting the same information again and again ?
>  
>  The GlusterFS protocol itself is very much stateless, similar to
>  NFSv3. We need all the groups of the user on the server-side (brick)
>  to allow the backing filesystem (mostly XFS) perform the permission
>  checking. In the current GlusterFS protocol, there is no user
>  authentication. (Well, there has been work done on adding support for
>  SSL, maybe that could be used for tracking sessions on a per-client,
>  not user, basis.)
>  
>  Just for clarity, a GlusterFS client (like a fuse-mount, or the
>  samba/vfs_glusterfs module) is used by many different users. The
>  client builds the connection to the volume. After that, all users
>  with access to the fuse-mount or samba-share are using the same
>  client connection.
>  
>  By default the client sends a list of groups in each RPC request, and
>  the server-side trusts the list the client provides. However, for
>  environments where these lists are too small to hold all the groups,
>  there is an option to do the group resolving on the server side. This
>  is the "server.manage-gids" volume option, which acts very much like
>  the "rpc.mountd --manage-gids" functionality for NFS. 
Instead of sending a list of groups every time ... wouldn't it be
better to send a "session token" (a random 128bit uuid) and let the
bricks use this value to associate their cached lists ?

This way you can control how caching is done from the client side.

>  > Keep in mind that the use of getgrouplist() is an inherently
> costly
>  > operation. Even adding caches, the system cannot cache for long
>  > because it needs to return updated results eventually. Only the
>  > application know when a user session terminates and/or the list
>  > needs to be refreshed, so "caching" for this type of operation
>  > should be done mostly on the application side.
>  
>  I assume that your "application side" here is the brick process that
>  runs on the same system as sssd. As mentioned above, the brick
>  processes do cache the result of getgrouplist(). It may well be
>  possible that the default expiry of 2 seconds is too short for many
>  environments. But users can change that timeout easily with the
>  "server.gid-timeout" volume option. 
Well the problem is that, unless you know you have some sort of user
session, longer caches have the only effect of upsetting users that
have had their credentials just changed.

The way it *should* work, at least if you want posix compatibility[1],
is that once a user, on a client, start a session, is that his
credentials never change until the user logs out and logs back in.
Regardless of what happens in the identity management system (or the
passwd/group files).

>  From my understanding of this thread, we (the Gluster Community)
> have
>  two things to do:
>  
>  1. Clearly document side-effects that can be caused by enabling the
>     "server.manage-gids" option, and suggest increasing the
>     "server.gid-timeout" value (maybe change the default?).
>  
>  2. Think about improving the GlusterFS protocol(s) and introduce some
>     kind of credentials token that is linked with the groups of a user.
>     Token expiry should invalidate the group-cache. One option would be
>     to use Kerberos like NFS (RPCSEC_GSS). 
Using RPCSEC_GSS is one good way to tie a user to its credentials, as
said credentials are tied to the GSS context and never changed until
the context is destroyed. Using, in general, a token created on
"session establishment"[2] and used until valid would resolve a host of
issues and make your filesystem more posix compliant and predictable
when it comes to access control decisions.

>  Does this all make sense to others too? I'm adding gluster-devel@
> to
>  CC so that others can chime in and this topic won't be forgotton. 
It does.

Simo.



[1] IIRC Posix requires that the credentials set in the kernel at login
time are used throughout the lifetime of the process unchanged. This is
particularly important as a process may *intentionally* drop auxiliary
groups or even change its credentials set entirely (like root switching
to a different uid and arbitrary set of gids).
You may decide this is not something you want to care about for network
access, and in fact NFS + RPC_GSSSEC do es *not* do this, as it always
computes the credentials set on the server side at (GSSAPI) context
establishment time. Up to you to decide what semantics you want to
follow, but they should be at least predictable if at all possible.


[2] You will have to define what this means for GlusterFS, I can see
only a few constraints to make it useful.
- the session needs to be initiated by a specific client
- you need a way to either pass the information that a new session is
  being established or pass the credential set to the bricks
- you need to cache this session on the bricks side and you cannot
  discard it at will (yes, this means state needs to be kept)*
- if a client connects randomly to multiple bricks, it means this cache
  needs to be distributed and accessible to any brick anywhere that
  needs the information
- if state cannot be kept then you have no other option but to always
  re-transmit the whole credential token, as big as it may be (maximum
  size on a linux system would be 256K at the moment (1 32bit uid + 65k
  32bit gids).

* the reason you do not want to let each brick resolve the groups is
that you may end up with different bricks having a different list of
groups a uid is member of. This would lead to nasty, very hard to debug
access issues that admins would hate you for :)

-- 
Simo Sorce * Red Hat, Inc * New York

[Niels de Vos]

On Fri, Nov 07, 2014 at 08:42:50AM -0500, Simo Sorce wrote:
>  On Fri, 7 Nov 2014 09:59:32 +0100
>  Niels de Vos <ndevos(a)redhat.com&gt; wrote:
>  
>  > On Thu, Nov 06, 2014 at 05:32:53PM -0500, Simo Sorce wrote:
>  > > On Thu, 6 Nov 2014 22:02:29 +0100
>  > > Niels de Vos <ndevos(a)redhat.com&gt; wrote:
>  > > 
>  > > > On Thu, Nov 06, 2014 at 11:45:18PM +0530, Vijay Bellur wrote:
>  > > > > On 11/03/2014 08:12 PM, Jakub Hrozek wrote:
>  > > > > >On Mon, Nov 03, 2014 at 03:41:43PM +0100, Jakub Hrozek wrote:
>  > > > > >>On Mon, Nov 03, 2014 at 08:53:06AM -0500, Simo Sorce wrote:
>  > > > > >>>On Mon, 3 Nov 2014 13:57:08 +0100
>  > > > > >>>Jakub Hrozek <jhrozek(a)redhat.com&gt; wrote:
>  > > > > >>>
>  > > > > >>>>Hi,
>  > > > > >>>>
>  > > > > >>>>we had short discussion on $SUBJECT with Simo on IRC
>  > > > > >>>>already, but there are multiple people involved from
>  > > > > >>>>multiple timezones, so I think a mailing list thread
> would
>  > > > > >>>>be better trackable.
>  > > > > >>>>
>  > > > > >>>>Can we add another memory cache file to SSSD, that
> would
>  > > > > >>>>track initgroups/getgrouplist results for the NSS
>  > > > > >>>>responder? I realize initgroups is a bit different
>  > > > > >>>>operation than getpw{uid,nam} and getgr{gid,nam} but
> what
>  > > > > >>>>if the new memcache was only used by the NSS
> responder and
>  > > > > >>>>at the same time invalidated when initgroups is
> initiated
>  > > > > >>>>by the PAM responder to ensure the memcache is
> up-to-date?
>  > > > > >>>
>  > > > > >>>Can you describe the use case before jumping into a
> proposed
>  > > > > >>>solution ?
>  > > > > >>
>  > > > > >>Many getgrouplist() or initgroups() calls in a quick
>  > > > > >>succession. One user is GlusterFS -- I'm not quite sure
> what
>  > > > > >>the reason is there, maybe Vijay can elaborate.
>  > > > > >
>  > > > > 
>  > > > > GlusterFS server invokes getgrouplist() to identify gids
>  > > > > associated with an user on whose behalf a rpc request has been
>  > > > > sent over the wire. There is a gid caching layer in GlusterFS
>  > > > > and getgrouplist() does get called only if there is a gid cache
>  > > > > miss. In the worst case, getgrouplist() can be invoked for
>  > > > > every rpc request that GlusterFS receives and that seems to be
>  > > > > the case in a deployment where we found that sssd was being
>  > > > > busy. I am not certain about the sequence of operations that
>  > > > > can cause the cache to be missed.
>  > > > > 
>  > > > > Adding Niels who is more familiar with the gid resolution &
>  > > > > caching features in GlusterFS.
>  > > > 
>  > > > Just to add some background information on the getgrouplist().
>  > > > GlusterFS uses several processes that can call getgrouplist():
>  > > > - NFS-server, a single process per system
>  > > > - brick, a process per exported filesystem/directory, potentally
>  > > > several per system
>  > > > 
>  > > >   [Here, a Gluster environment has many systems (vm/physical).
>  > > > Each system normally runs the NFS-server, and a number of brick
>  > > > processes. The layout of the volume is important, but it is very
>  > > > common to have one or more distributed volumes that use multiple
>  > > > bricks on the same system (and many other systems).]
>  > > > 
>  > > > The need for resolving the groups of a user comes in when users
>  > > > belong to many groups. The RPC protocols can not carry a huge
>  > > > list of groups, so the resolving can be done on the server side
>  > > > when the protocol hits its limits (> 16 for NFS, approx. > 93 for
>  > > > GlusterFS).
>  > > > 
>  > > > Upon using a Gluster volume, certain operations are sent to all
>  > > > the bricks (i.e. some directory related operations). I can
>  > > > imagine that a network share which is used by many users, trigger
>  > > > many getgrouplist() calls in different brick processes at the
>  > > > (almost) same time.
>  > > > 
>  > > > For reference, the usage of getgrouplist() in the brick process
>  > > > can be found here:
>  > > > -
>  > > >
> https://github.com/gluster/glusterfs/blob/master/xlators/protocol/server/...
>  > > > 
>  > > > The gid_resolve() function get called in case the brick process
>  > > > should resolve the groups (and ignore the list of groups from the
>  > > > protocol). It uses the gidcache functions from a private library:
>  > > > -
>  > > >
> https://github.com/gluster/glusterfs/blob/master/libglusterfs/src/gidcache.h
>  > > > -
>  > > >
> https://github.com/gluster/glusterfs/blob/master/libglusterfs/src/gidcache.c
>  > > > 
>  > > > The default time for the gidcache to expire is 2 seconds. Users
>  > > > should be able to configure this to 30 seconds (or anything else)
>  > > > with:
>  > > > 
>  > > >     # gluster volume set <VOLUME> server.gid-timeout 30
>  > > > 
>  > > > 
>  > > > I think this should explain the use-case sufficiently, but let me
>  > > > know if there are any remaining questions. It might well be
>  > > > possible to make this code more sssd friendly. I'm sure that we
>  > > > as Gluster developers are open to any suggestions.
>  > > 
>  > > 
>  > > TBH this looks a little bit strange, other filesystems (as well as
>  > > the kernel) create a credentials token when a user first
>  > > authenticate and keep these credentials attached to the user
>  > > session for the duration. Why does GlusterFS keeps hammering the
>  > > system requesting the same information again and again ?
>  > 
>  > The GlusterFS protocol itself is very much stateless, similar to
>  > NFSv3. We need all the groups of the user on the server-side (brick)
>  > to allow the backing filesystem (mostly XFS) perform the permission
>  > checking. In the current GlusterFS protocol, there is no user
>  > authentication. (Well, there has been work done on adding support for
>  > SSL, maybe that could be used for tracking sessions on a per-client,
>  > not user, basis.)
>  > 
>  > Just for clarity, a GlusterFS client (like a fuse-mount, or the
>  > samba/vfs_glusterfs module) is used by many different users. The
>  > client builds the connection to the volume. After that, all users
>  > with access to the fuse-mount or samba-share are using the same
>  > client connection.
>  > 
>  > By default the client sends a list of groups in each RPC request, and
>  > the server-side trusts the list the client provides. However, for
>  > environments where these lists are too small to hold all the groups,
>  > there is an option to do the group resolving on the server side. This
>  > is the "server.manage-gids" volume option, which acts very much like
>  > the "rpc.mountd --manage-gids" functionality for NFS.
>  
>  Instead of sending a list of groups every time ... wouldn't it be
>  better to send a "session token" (a random 128bit uuid) and let the
>  bricks use this value to associate their cached lists ?
>  
>  This way you can control how caching is done from the client side. 
Yes, I was hoping RPCSEC_GSS could help with that. But that is a major
change and it'll take a while for it to get stable and used in
deployments.

Looking at it, there is a AUTH_SHORT option that we probably can use. We
do not use AUTH_SYS, but some variation called AUTH_GLUSTERFS. In the
end, they function pretty much the same. More on AUTH_SHORT:
- http://tools.ietf.org/html/rfc5531#page-25

One of the difficulties would be to have all the bricks be aware of the
token. There is no inter-brick communication...

>  > > Keep in mind that the use of getgrouplist() is an
> inherently costly
>  > > operation. Even adding caches, the system cannot cache for long
>  > > because it needs to return updated results eventually. Only the
>  > > application know when a user session terminates and/or the list
>  > > needs to be refreshed, so "caching" for this type of operation
>  > > should be done mostly on the application side.
>  > 
>  > I assume that your "application side" here is the brick process that
>  > runs on the same system as sssd. As mentioned above, the brick
>  > processes do cache the result of getgrouplist(). It may well be
>  > possible that the default expiry of 2 seconds is too short for many
>  > environments. But users can change that timeout easily with the
>  > "server.gid-timeout" volume option.
>  
>  Well the problem is that, unless you know you have some sort of user
>  session, longer caches have the only effect of upsetting users that
>  have had their credentials just changed.
>  
>  The way it *should* work, at least if you want posix compatibility[1],
>  is that once a user, on a client, start a session, is that his
>  credentials never change until the user logs out and logs back in.
>  Regardless of what happens in the identity management system (or the
>  passwd/group files). 
Well, we would like to improve the Gluster behaviour, making it "posix
complaint" at the same time works for me. I imagine that this would be
possible when we use RPCSEC_GSS, just like NFS does.

>  > From my understanding of this thread, we (the Gluster Community)
> have
>  > two things to do:
>  > 
>  > 1. Clearly document side-effects that can be caused by enabling the
>  >    "server.manage-gids" option, and suggest increasing the
>  >    "server.gid-timeout" value (maybe change the default?).
>  > 
>  > 2. Think about improving the GlusterFS protocol(s) and introduce some
>  >    kind of credentials token that is linked with the groups of a user.
>  >    Token expiry should invalidate the group-cache. One option would be
>  >    to use Kerberos like NFS (RPCSEC_GSS).
>  
>  Using RPCSEC_GSS is one good way to tie a user to its credentials, as
>  said credentials are tied to the GSS context and never changed until
>  the context is destroyed. Using, in general, a token created on
>  "session establishment"[2] and used until valid would resolve a host of
>  issues and make your filesystem more posix compliant and predictable
>  when it comes to access control decisions. 
The biggest advantage for the Gluster use-case seems to be that the
token is valid on all the systems hosting a brick for a particular
volume. At least, I hope that is the case. Because of the nature of the
scale-out, scale-up filesystem, systems and bricks can get added
whenever a sysadmin deems it necessary. I do not immediately see a
solution to prevent your [*] footnote, that would require Gluster to
pass credentials (and tokens?) around to all the bricks when they get
online. It is not impossible, but requires quite some more work.

>  > Does this all make sense to others too? I'm adding
> gluster-devel@ to
>  > CC so that others can chime in and this topic won't be forgotton.
>  
>  It does.
>  
>  Simo.
>  
>  
>  
>  [1] IIRC Posix requires that the credentials set in the kernel at login
>  time are used throughout the lifetime of the process unchanged. This is
>  particularly important as a process may *intentionally* drop auxiliary
>  groups or even change its credentials set entirely (like root switching
>  to a different uid and arbitrary set of gids).
>  You may decide this is not something you want to care about for network
>  access, and in fact NFS + RPC_GSSSEC do es *not* do this, as it always
>  computes the credentials set on the server side at (GSSAPI) context
>  establishment time. Up to you to decide what semantics you want to
>  follow, but they should be at least predictable if at all possible. 
If the NFS + RPC_GSSSEC semantics are well understood, they should work
for Gluster too. The main requirements would be that a userspace process
can get the token for a user, and pass that on through a library call
that then does the GlusterFS RPC stuff. Samba with vfs_glusterfs would
be one of these users, glusterfs-fuse an other.

>  [2] You will have to define what this means for GlusterFS, I can see
>  only a few constraints to make it useful.
>  - the session needs to be initiated by a specific client
>  - you need a way to either pass the information that a new session is
>    being established or pass the credential set to the bricks
>  - you need to cache this session on the bricks side and you cannot
>    discard it at will (yes, this means state needs to be kept)*
>  - if a client connects randomly to multiple bricks, it means this cache
>    needs to be distributed and accessible to any brick anywhere that
>    needs the information
>  - if state cannot be kept then you have no other option but to always
>    re-transmit the whole credential token, as big as it may be (maximum
>    size on a linux system would be 256K at the moment (1 32bit uid + 65k
>    32bit gids). 
Maybe we can ask for the whole credential token when a client connects
to the brick for the first time, and after that use the session token.
This would solve the issue I mentioned above about adding systems and
bricks.

>  * the reason you do not want to let each brick resolve the groups is
>  that you may end up with different bricks having a different list of
>  groups a uid is member of. This would lead to nasty, very hard to debug
>  access issues that admins would hate you for :) 
Yes, that is a very good point.

Thanks again,
Niels

[Roland Mainz]

----- Original Message -----
>  From: "Simo Sorce" <ssorce(a)redhat.com&gt;
>  To: "Niels de Vos" <ndevos(a)redhat.com&gt;
>  Cc: gluster-devel(a)gluster.org, sssd-devel(a)lists.fedorahosted.org, "Vijay
> Bellur" <vbellur(a)redhat.com&gt;
>  Sent: Friday, November 7, 2014 2:42:50 PM
>  Subject: Re: [SSSD] memory cache for initgroups
>  
>  On Fri, 7 Nov 2014 09:59:32 +0100
>  Niels de Vos <ndevos(a)redhat.com&gt; wrote: 
>  > On Thu, Nov 06, 2014 at 05:32:53PM -0500, Simo Sorce wrote:
>  > > On Thu, 6 Nov 2014 22:02:29 +0100
>  > > Niels de Vos <ndevos(a)redhat.com&gt; wrote:
>  > > > On Thu, Nov 06, 2014 at 11:45:18PM +0530, Vijay Bellur wrote:
>  > > > > On 11/03/2014 08:12 PM, Jakub Hrozek wrote:
>  > > > > >On Mon, Nov 03, 2014 at 03:41:43PM +0100, Jakub Hrozek wrote:
>  > > > > >>On Mon, Nov 03, 2014 at 08:53:06AM -0500, Simo Sorce wrote:
>  > > > > >>>On Mon, 3 Nov 2014 13:57:08 +0100
>  > > > > >>>Jakub Hrozek <jhrozek(a)redhat.com&gt; wrote:
[snip]
>  
>  [1] IIRC Posix requires that the credentials set in the kernel at login
>  time are used throughout the lifetime of the process unchanged. 
Right (except it's normally inherited from parent process, not login process directly)
... that's I was wondering whether the whole group information can be passed around
via shared memory (reducing the group lookup functions to
atomic_refcount+shared_mutex_lock+|memcpy()|+shared_mutex_unlock) so child
processes&co. can inherit it automagically without constant lookups.

>  This is
>  particularly important as a process may *intentionally* drop auxiliary
>  groups or even change its credentials set entirely (like root switching
>  to a different uid and arbitrary set of gids). 
... don't forget the case of newgrp(1) with groups with passwords, e.g. a random user
can obtain an extra group membership when the matching group has a password... ;-/

>  You may decide this is not something you want to care about for
> network
>  access, and in fact NFS + RPC_GSSSEC do es *not* do this, as it always
>  computes the credentials set on the server side at (GSSAPI) context
>  establishment time. Up to you to decide what semantics you want to
>  follow, but they should be at least predictable if at all possible. 
The problem with GlusterFS vs. RPC_GSSSEC might be that GlusterFS tries to be stateless
whereever possible...

----

Bye,
Roland

-- 
  __ .  . __
 (o.\ \/ /.o) rmainz(a)redhat.com
  \__\/\/__/  IPA/Kerberos5 team
  /O /==\ O\  
 (;O/ \/ \O;)

[Simo Sorce]

On Mon, 10 Nov 2014 16:56:22 -0500 (EST)
Roland Mainz <rmainz(a)redhat.com&gt; wrote:

>  
>  
>  ----- Original Message -----
>  > From: "Simo Sorce" <ssorce(a)redhat.com&gt;
>  > To: "Niels de Vos" <ndevos(a)redhat.com&gt;
>  > Cc: gluster-devel(a)gluster.org, sssd-devel(a)lists.fedorahosted.org,
>  > "Vijay Bellur" <vbellur(a)redhat.com&gt; Sent: Friday, November 7, 2014
>  > 2:42:50 PM Subject: Re: [SSSD] memory cache for initgroups
>  > 
>  > On Fri, 7 Nov 2014 09:59:32 +0100
>  > Niels de Vos <ndevos(a)redhat.com&gt; wrote: 
>  > > On Thu, Nov 06, 2014 at 05:32:53PM -0500, Simo Sorce wrote:
>  > > > On Thu, 6 Nov 2014 22:02:29 +0100
>  > > > Niels de Vos <ndevos(a)redhat.com&gt; wrote:
>  > > > > On Thu, Nov 06, 2014 at 11:45:18PM +0530, Vijay Bellur wrote:
>  > > > > > On 11/03/2014 08:12 PM, Jakub Hrozek wrote:
>  > > > > > >On Mon, Nov 03, 2014 at 03:41:43PM +0100, Jakub Hrozek
>  > > > > > >wrote:
>  > > > > > >>On Mon, Nov 03, 2014 at 08:53:06AM -0500, Simo Sorce
>  > > > > > >>wrote:
>  > > > > > >>>On Mon, 3 Nov 2014 13:57:08 +0100
>  > > > > > >>>Jakub Hrozek <jhrozek(a)redhat.com&gt; wrote:
>  [snip]
>  > 
>  > [1] IIRC Posix requires that the credentials set in the kernel at
>  > login time are used throughout the lifetime of the process
>  > unchanged.
>  
>  Right (except it's normally inherited from parent process, not login
>  process directly) ... that's I was wondering whether the whole group
>  information can be passed around via shared memory (reducing the
>  group lookup functions to
>  atomic_refcount+shared_mutex_lock+|memcpy()|+shared_mutex_unlock) so
>  child processes&co. can inherit it automagically without constant
>  lookups. 
The problem here is that the bricks run as a different user, there
isn't a user logged into the machine running the bricks, and the client
is generally on a completely separate machine.

>  > This is
>  > particularly important as a process may *intentionally* drop
>  > auxiliary groups or even change its credentials set entirely (like
>  > root switching to a different uid and arbitrary set of gids).
>  
>  ... don't forget the case of newgrp(1) with groups with passwords,
>  e.g. a random user can obtain an extra group membership when the
>  matching group has a password... ;-/ 
Indeed, in theory the GlusterFS client should check the group list for
each new process that talks to it, and have a cache of "sessions"
associated to each process that have a "unique credentials signature"

>  > You may decide this is not something you want to care about for
>  > network access, and in fact NFS + RPC_GSSSEC do es *not* do this,
>  > as it always computes the credentials set on the server side at
>  > (GSSAPI) context establishment time. Up to you to decide what
>  > semantics you want to follow, but they should be at least
>  > predictable if at all possible.
>  
>  The problem with GlusterFS vs. RPC_GSSSEC might be that GlusterFS
>  tries to be stateless whereever possible... 
It is indeed a problem, authentication/authorization and statelessness
often collide, a compromise needs to be found.

Simo.

-- 
Simo Sorce * Red Hat, Inc * New York

glibc |getgrouplist()|/|initgroups()| performace ... / was: Re: memory cache for initgroups

[Roland Mainz]

----- Original Message -----
>  From: "Simo Sorce" <ssorce(a)redhat.com&gt;
>  To: "Niels de Vos" <ndevos(a)redhat.com&gt;
>  Cc: sssd-devel(a)lists.fedorahosted.org, "Vijay Bellur"
> <vbellur(a)redhat.com&gt;
>  Sent: Thursday, November 6, 2014 11:32:53 PM
>  Subject: Re: [SSSD] memory cache for initgroups
>  
>  On Thu, 6 Nov 2014 22:02:29 +0100
>  Niels de Vos <ndevos(a)redhat.com&gt; wrote:
>  > On Thu, Nov 06, 2014 at 11:45:18PM +0530, Vijay Bellur wrote:
>  > > On 11/03/2014 08:12 PM, Jakub Hrozek wrote:
>  > > >On Mon, Nov 03, 2014 at 03:41:43PM +0100, Jakub Hrozek wrote:
>  > > >>On Mon, Nov 03, 2014 at 08:53:06AM -0500, Simo Sorce wrote:
>  > > >>>On Mon, 3 Nov 2014 13:57:08 +0100
>  > > >>>Jakub Hrozek <jhrozek(a)redhat.com&gt; wrote: 
[snip]
>  TBH this looks a little bit strange, other filesystems (as well as
> the
>  kernel) create a credentials token when a user first authenticate and
>  keep these credentials attached to the user session for the duration.
>  Why does GlusterFS keeps hammering the system requesting the same
>  information again and again ?
>  
>  Keep in mind that the use of getgrouplist() is an inherently costly
>  operation. 
Is there any detailed breakdown which parts consume which amount of CPU time ?

>  Even adding caches, the system cannot cache for long because
>  it needs to return updated results eventually. Only the application
>  know when a user session terminates and/or the list needs to be
>  refreshed, so "caching" for this type of operation should be done
>  mostly on the application side. 
BTW: What about using shared memory to transfer the groups data (basically reducing stuff
like |getgroups()| down to a atomic_refcount+shared_mutex_lock+|memcpy()|) ? It
doesn't bring much for small amounts of group entries but it scales much better when
the number of group entries becomes very large...

----

Bye,
Roland

-- 
  __ .  . __
 (o.\ \/ /.o) rmainz(a)redhat.com
  \__\/\/__/  IPA/Kerberos5 team
  /O /==\ O\  
 (;O/ \/ \O;)

[Simo Sorce]

On Mon, 3 Nov 2014 15:41:43 +0100
Jakub Hrozek <jhrozek(a)redhat.com&gt; wrote:

>  On Mon, Nov 03, 2014 at 08:53:06AM -0500, Simo Sorce wrote:
>  > On Mon, 3 Nov 2014 13:57:08 +0100
>  > Jakub Hrozek <jhrozek(a)redhat.com&gt; wrote:
>  > 
>  > > Hi,
>  > > 
>  > > we had short discussion on $SUBJECT with Simo on IRC already, but
>  > > there are multiple people involved from multiple timezones, so I
>  > > think a mailing list thread would be better trackable.
>  > > 
>  > > Can we add another memory cache file to SSSD, that would track
>  > > initgroups/getgrouplist results for the NSS responder? I realize
>  > > initgroups is a bit different operation than getpw{uid,nam} and
>  > > getgr{gid,nam} but what if the new memcache was only used by the
>  > > NSS responder and at the same time invalidated when initgroups is
>  > > initiated by the PAM responder to ensure the memcache is
>  > > up-to-date?
>  > 
>  > Can you describe the use case before jumping into a proposed
>  > solution ?
>  
>  Many getgrouplist() or initgroups() calls in a quick succession. One
>  user is GlusterFS -- I'm not quite sure what the reason is there,
>  maybe Vijay can elaborate.
>  
>  The other use-case is slapi-nis that calls getgrouplist() -- Alexander
>  would explain better here, I only know that slapi-nis uses
>  getgrouplist() during memberUID processing, but I'm fuzzy on the
>  details.
>  
>  > 
>  > > I already know about two projects that would benefit from faster
>  > > initgroups operation - GlusterFS uses getgrouplist() quite a lot
>  > > with some setups and also in the IPA server mode, the SSSD
>  > > running on the server becomes a bit of a bottlenec for some
>  > > operations.
>  > 
>  > Is the problem related to accessing the main ldb cache, or the fact
>  > we also go out and contact the IdM server ?
>  
>  Mostly the cache. In particular, with the GlusterFS case, there was a
>  huge filesystem directory that triggered 4000+ getgrouplist() calls,
>  one per a file. Even though the getgrouplist() calls were for fewer
>  than 10 users, we would hit LDB every time and that produced quite a
>  load on the SSSD.
>  
>  > 
>  > > Are there any technical reasons against a new memcache file that
>  > > would cache initgroups?
>  > 
>  > The main reason would be potential inconsistency between this cache
>  > and the users and groups caches, but I do not think that would be
>  > really a big deal if there is a pretty good reason for the cache.
>  
>  Hm, I see, then we have the same information duplicated in two caches,
>  just from a different angle. However, for cases where precision really
>  matters (access control), we don't use these caches anyway.
>  
>  Could we mitigate this problem by invalidating the memcache when group
>  membership changes at all during a backend operation? I think changes
>  to LDAP object are so rare, that invalidating the memcache (which is
>  quite shortlived anyway) would be worth it. 
Probably sufficient, what is the expected lifetime of the initgroups
memcache ?

>  > Do we need to insure all groups are resolvable in the groups
> cache ?
>  
>  I don't think so, if the group is not in the groups cache, then we
>  just hit the responder. 
ok

>  > Or is it ok if gids returned by the getgrouplist are not
> immediately
>  > available in the groups cache ?
>  > What about the user ?
>  
>  I *think* the user could also be left out. If the application also
>  needs user data, it simply calls getpwnam -- like id(1) does.
>  getgrouplist is a different interface with different outputs. 
fine then.

Note that one option could be to extend the user's cache, to have a
list of gids on the tail of the user's data, I am too stuffed of
medications though to think clearly if that is a good idea or a
colossal idiocy :-)

Simo.

-- 
Simo Sorce * Red Hat, Inc * New York

[Jakub Hrozek]

On Mon, Nov 03, 2014 at 09:50:04AM -0500, Simo Sorce wrote:
>  Note that one option could be to extend the user's cache, to have
> a
>  list of gids on the tail of the user's data, I am too stuffed of
>  medications though to think clearly if that is a good idea or a
>  colossal idiocy :-)
>  
>  Simo. 
Michal/Lukas, do you have an opinion here? I must confess I don't know
the memcache code, really, so I really don't know if extending the user
cache is a good idea or not.