Re: initgroups is taking longer with sssd-1.16.5-10.el7_9.5

[Sanjay Agrawal]

resending
Sanjay Agrawal

On Wednesday, December 2, 2020, 03:41:36 PM EST, Sanjay Agrawal  wrote:

Hi,

we are seeing an issue with newer version of sssd with centos 7.9 sssd version 1.16.5-10.el7_9.5.x86_64, where Initgroups is taking much longer compared to
previous version. Can you please look into it. Following are details, including a sample program to reproduce the issue with old and new version.

Sample Program: get_user_groups.py
    - it just call getgrouplist of a user supplied using libc

Old env -
OS Linux 3.10.0-1127.19.1.el7.x86_64 #1 SMP Wed Aug 26 11:48:49 BST 2020 x86_64 x86_64 x86_64 GNU/Linux
SSSD Version    - 1.16.4-37.el7_8.4.x86_64
Sample log file - sssd_nss-old.log
Flamegraph      - flamgraph_sssd_nss-old.svg
ID  879 'Initgroups by name' testuser1@example
    starttime Tue Dec  1 18:36:42:092541 2020
    endtime Tue Dec  1 18:36:42:124463 2020
    lookup_time 0.031922 sec

New env
OS Linux 3.10.0-1160.6.1.el7.x86_64 #1 SMP Wed Nov 18 22:40:48 GMT 2020 x86_64 x86_64 x86_64 GNU/Linux
SSSD Version    - 1.16.5-10.el7_9.5.x86_64
Sample log file - sssd_nss-new.log
Flamegraph      - flamgraph_sssd_nss-new.svg
ID  776 'Initgroups by name' testuser1@example
    starttime 2020-12-01 17:31:24:328419
    endtime 2020-12-01 17:31:24:451778
    lookup_time 0.123359 sec

It seem to be due to addtional .1 second taken during following two trace (from sssd_nss-new.log)
(2020-12-01 17:31:24:355458): [nss] [cache_req_done] (0x0400): CR #776: Finished: Success
(2020-12-01 17:31:24:451727): [nss] [sysdb_search_group_by_id] (0x0400): No such entry
(2020-12-01 17:31:24:451757): [nss] [nss_protocol_fill_initgr] (0x0080): Unable to find primary gid [2]: No such file or directory

It may be related to following change, which seems to ref to sysdb_search_group_by_id
nss: use real primary gid if the value is overriden · SSSD/sssd@80e6f71

| 
| 
| 
|  |  |

|

|
| 
|  | 
nss: use real primary gid if the value is overriden · SSSD/sssd@80e6f71

SYSDB_PRIMARY_GROUP_GIDNUM contains original primary group id from AD because any possible override may not be k...
 |

|

|

Sanjay Agrawal

initgroups is taking longer with sssd-1.16.5-10.el7_9.5

[Sumit Bose]

On Thu, Dec 03, 2020 at 05:50:31PM +0000, Sanjay Agrawal wrote:
> resending
> Sanjay Agrawal
> On Wednesday, December 2, 2020, 03:41:36 PM EST, Sanjay Agrawal <[e-mail]> wrote:  
>
> Hi,Â
>
> we are seeing an issue with newer version of sssd with centos 7.9 sssd version 1.16.5-10.el7_9.5.x86_64, where Initgroups is taking much longer compared to
> previous version. Can you please look into it. Following are details, including a sample program to reproduce the issue with old and new version.
>
> Sample Program: get_user_groups.py
> Â  Â  - it just call getgrouplist of a user supplied using libc
>
> Old env -
> OS Linux 3.10.0-1127.19.1.el7.x86_64 #1 SMP Wed Aug 26 11:48:49 BST 2020 x86_64 x86_64 x86_64 GNU/Linux
> SSSD Version    - 1.16.4-37.el7_8.4.x86_64
> Sample log file - sssd_nss-old.log
> Flamegraph      - flamgraph_sssd_nss-old.svg
> ID Â 879 'Initgroups by name' testuser1@example
>     starttime Tue Dec  1 18:36:42:092541 2020
>     endtime Tue Dec  1 18:36:42:124463 2020
> Â  Â  lookup_time 0.031922 sec
>
> New env
> OS Linux 3.10.0-1160.6.1.el7.x86_64 #1 SMP Wed Nov 18 22:40:48 GMT 2020 x86_64 x86_64 x86_64 GNU/Linux
> SSSD Version    - 1.16.5-10.el7_9.5.x86_64
> Sample log file - sssd_nss-new.log
> Flamegraph      - flamgraph_sssd_nss-new.svg
> ID Â 776 'Initgroups by name' testuser1@example
> Â  Â  starttime 2020-12-01 17:31:24:328419
> Â  Â  endtime 2020-12-01 17:31:24:451778
> Â  Â  lookup_time 0.123359 sec
>
> It seem to be due to addtional .1 second taken during following two trace (from sssd_nss-new.log)
> (2020-12-01 17:31:24:355458): [nss] [cache_req_done] (0x0400): CR #776: Finished: Success
> (2020-12-01 17:31:24:451727): [nss] [sysdb_search_group_by_id] (0x0400): No such entry
> (2020-12-01 17:31:24:451757): [nss] [nss_protocol_fill_initgr] (0x0080): Unable to find primary gid [2]: No such file or directory
>
> It may be related to following change, which seems to ref to sysdb_search_group_by_id
> nss: use real primary gid if the value is overriden · SSSD/sssd@80e6f71

Hi,

thanks for the analysis and I guess you might be right. The patch adds a
search where one attribute in the filter does not have an index. As a
result the search has to run over the whole cache.

I wonder if you would like to try to add such index? To do this please
stop SSSD and call:

ldbedit -H /var/lib/sss/db/cache_YOUR.DOMAIN.NAME.ldb -s base -b '@INDEXLIST'

which should open an editor showing something similar to

# editing 1 records
# record 1
dn: @INDEXLIST
@IDXATTR: cn
@IDXATTR: objectclass
@IDXATTR: member
@IDXATTR: memberof
@IDXATTR: name
@IDXATTR: uidNumber
@IDXATTR: gidNumber
@IDXATTR: lastUpdate
@IDXATTR: dataExpireTimestamp
@IDXATTR: originalDN
@IDXATTR: nameAlias
@IDXATTR: servicePort
@IDXATTR: serviceProtocol
@IDXATTR: sudoUser
@IDXATTR: sshKnownHostsExpire
@IDXATTR: objectSIDString
@IDXATTR: ghost
@IDXATTR: userPrincipalName
@IDXATTR: canonicalUserPrincipalName
@IDXATTR: uniqueID
@IDXATTR: mail
@IDXATTR: userMappedCertificate
@IDXATTR: ccacheFile
@IDXATTR: ipHostNumber
@IDXATTR: ipNetworkNumber
distinguishedName: @INDEXLIST

Please add in the @IDXATTR list a line

@IDXATTR: originalADgidNumber

and exit the editor, the databased will be reindexed at this point.
Finally start SSSD again and rerun the test.

HTH

bye,
Sumit
> | 
> | 
> | 
> |  |  |
>
> |
>
> |
> | 
> |  | 
> nss: use real primary gid if the value is overriden · SSSD/sssd@80e6f71
>
> SYSDB_PRIMARY_GROUP_GIDNUM contains original primary group id from AD because any possible override may not be k...
>  |
>
> |
>
> |
>
> Sanjay Agrawal
> sssd-users mailing list -- [e-mail]
> To unsubscribe send an email to [e-mail]
> Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives: https://lists.fedorahosted.org/archives/list/[e-mail].o...

initgroups is taking longer with sssd-1.16.5-10.el7_9.5

[Sanjay Agrawal]

Thanks for response. I re-ran test after adding index as suggested. Which seems to fix the underlying issue and results are inline with older version. So will this index would be added in newer version of sssd ? Is there an easier way to add this index w/o opening an editor ?

older version 
sssd-20201203163147/analysis.out:'Initgroups by name' count: 102 avg: 0.596225137254902 median 0.035176 min: 0.001218 max: 22.037332
sssd-20201203164328/analysis.out:'Initgroups by name' count: 201 avg: 0.313984855721393 median 0.039057 min: 0.000394 max: 23.138423
sssd-20201203165234/analysis.out:'Initgroups by name' count: 100 avg: 0.59339562 median 0.0362155 min: 0.000412 max: 21.559991

With new version + adding index as suggested above.
sssd-20201203145950/analysis.out:'Initgroups by name' count: 103 avg: 0.5026502815533981 median 0.03847 min: 0.000435 max: 21.532709
sssd-20201203152524/analysis.out:'Initgroups by name' count: 90 avg: 1.1182167555555556 median 0.0349585 min: 0.000536 max: 33.604103
sssd-20201203160837/analysis.out:'Initgroups by name' count: 102 avg: 0.4324193431372549 median 0.0338985 min: 0.000479 max: 19.493188

Above output is by parsing the sssd_nss.log with debug_level 9. As you can see from both older/newer versions, we are seeing a very high max value. All these are when request is being sent from nss->be for resolution. Is is possible similar issue is happening at BE side as well when updating ldb ? Please let me know, if you need additional data.

Thanks

Sanjay Agrawal

On Thursday, December 3, 2020, 01:32:32 PM EST, Sumit Bose  wrote:

On Thu, Dec 03, 2020 at 05:50:31PM +0000, Sanjay Agrawal wrote:
> resending
> Sanjay Agrawal
>
> On Wednesday, December 2, 2020, 03:41:36 PM EST, Sanjay Agrawal  wrote:  
>   
>   Hi,Â
>
> we are seeing an issue with newer version of sssd with centos 7.9 sssd version 1.16.5-10.el7_9.5.x86_64, where Initgroups is taking much longer compared to
> previous version. Can you please look into it. Following are details, including a sample program to reproduce the issue with old and new version.
>
> Sample Program: get_user_groups.py
> Â  Â  - it just call getgrouplist of a user supplied using libc
>
> Old env -
> OS Linux 3.10.0-1127.19.1.el7.x86_64 #1 SMP Wed Aug 26 11:48:49 BST 2020 x86_64 x86_64 x86_64 GNU/Linux
> SSSD Version    - 1.16.4-37.el7_8.4.x86_64
> Sample log file - sssd_nss-old.log
> Flamegraph      - flamgraph_sssd_nss-old.svg
> ID Â 879 'Initgroups by name' testuser1@example
>     starttime Tue Dec  1 18:36:42:092541 2020
>     endtime Tue Dec  1 18:36:42:124463 2020
> Â  Â  lookup_time 0.031922 sec
>
> New env
> OS Linux 3.10.0-1160.6.1.el7.x86_64 #1 SMP Wed Nov 18 22:40:48 GMT 2020 x86_64 x86_64 x86_64 GNU/Linux
> SSSD Version    - 1.16.5-10.el7_9.5.x86_64
> Sample log file - sssd_nss-new.log
> Flamegraph      - flamgraph_sssd_nss-new.svg
> ID Â 776 'Initgroups by name' testuser1@example
> Â  Â  starttime 2020-12-01 17:31:24:328419
> Â  Â  endtime 2020-12-01 17:31:24:451778
> Â  Â  lookup_time 0.123359 sec
>
> It seem to be due to addtional .1 second taken during following two trace (from sssd_nss-new.log)
> (2020-12-01 17:31:24:355458): [nss] [cache_req_done] (0x0400): CR #776: Finished: Success
> (2020-12-01 17:31:24:451727): [nss] [sysdb_search_group_by_id] (0x0400): No such entry
> (2020-12-01 17:31:24:451757): [nss] [nss_protocol_fill_initgr] (0x0080): Unable to find primary gid [2]: No such file or directory
>
> It may be related to following change, which seems to ref to sysdb_search_group_by_id
> nss: use real primary gid if the value is overriden · SSSD/sssd@80e6f71

Hi,

thanks for the analysis and I guess you might be right. The patch adds a
search where one attribute in the filter does not have an index. As a
result the search has to run over the whole cache.

I wonder if you would like to try to add such index? To do this please
stop SSSD and call:

ldbedit -H /var/lib/sss/db/cache_YOUR.DOMAIN.NAME.ldb -s base -b '@INDEXLIST'

which should open an editor showing something similar to

# editing 1 records
# record 1
dn: @INDEXLIST
@IDXATTR: cn
@IDXATTR: objectclass
@IDXATTR: member
@IDXATTR: memberof
@IDXATTR: name
@IDXATTR: uidNumber
@IDXATTR: gidNumber
@IDXATTR: lastUpdate
@IDXATTR: dataExpireTimestamp
@IDXATTR: originalDN
@IDXATTR: nameAlias
@IDXATTR: servicePort
@IDXATTR: serviceProtocol
@IDXATTR: sudoUser
@IDXATTR: sshKnownHostsExpire
@IDXATTR: objectSIDString
@IDXATTR: ghost
@IDXATTR: userPrincipalName
@IDXATTR: canonicalUserPrincipalName
@IDXATTR: uniqueID
@IDXATTR: mail
@IDXATTR: userMappedCertificate
@IDXATTR: ccacheFile
@IDXATTR: ipHostNumber
@IDXATTR: ipNetworkNumber
distinguishedName: @INDEXLIST

Please add in the @IDXATTR list a line

@IDXATTR: originalADgidNumber

and exit the editor, the databased will be reindexed at this point.
Finally start SSSD again and rerun the test.

HTH

bye,
Sumit
> | 
> | 
> | 
> |  |  |
>
> |
>
> |
> | 
> |  | 
> nss: use real primary gid if the value is overriden · SSSD/sssd@80e6f71
>
> SYSDB_PRIMARY_GROUP_GIDNUM contains original primary group id from AD because any possible override may not be k...
>   |
>
> |
>
> |
>
> Sanjay Agrawal
> sssd-users mailing list -- [e-mail]
> To unsubscribe send an email to [e-mail]
> Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives: https://lists.fedorahosted.org/archives/list/[e-mail].o...

_______________________________________________
sssd-users mailing list -- [e-mail]
To unsubscribe send an email to [e-mail]
Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: https://lists.fedorahosted.org/archives/list/[e-mail].o...

initgroups is taking longer with sssd-1.16.5-10.el7_9.5

[Alexey Tikhonov]

On Thu, Dec 3, 2020 at 11:46 PM Sanjay Agrawal 
wrote:
> Thanks for response. I re-ran test after adding index as suggested. Which
> seems to fix the underlying issue and results are inline with older
> version. So will this index would be added in newer version of sssd ?

I opened an upstream issue to track this:
https://github.com/SSSD/sssd/issues/5430

Looking at the package name in the subject, I guess you are a RHEL user.
If you need this fix in a RHEL, please open a customer case or bugzilla
ticket.
> Is there an easier way to add this index w/o opening an editor ?
>
> older version
> sssd-20201203163147/analysis.out:'Initgroups by name' count: 102 avg:
> 0.596225137254902 median 0.035176 min: 0.001218 max: 22.037332
> sssd-20201203164328/analysis.out:'Initgroups by name' count: 201 avg:
> 0.313984855721393 median 0.039057 min: 0.000394 max: 23.138423
> sssd-20201203165234/analysis.out:'Initgroups by name' count: 100 avg:
> 0.59339562 median 0.0362155 min: 0.000412 max: 21.559991
>
> With new version + adding index as suggested above.
> sssd-20201203145950/analysis.out:'Initgroups by name' count: 103 avg:
> 0.5026502815533981 median 0.03847 min: 0.000435 max: 21.532709
> sssd-20201203152524/analysis.out:'Initgroups by name' count: 90 avg:
> 1.1182167555555556 median 0.0349585 min: 0.000536 max: 33.604103
> sssd-20201203160837/analysis.out:'Initgroups by name' count: 102 avg:
> 0.4324193431372549 median 0.0338985 min: 0.000479 max: 19.493188
>
> Above output is by parsing the sssd_nss.log with debug_level 9. As you can
> see from both older/newer versions, we are seeing a very high max value.
> All these are when request is being sent from nss->be for resolution. Is is
> possible similar issue is happening at BE side as well when updating ldb ?
> Please let me know, if you need additional data.
>
> Thanks
>
> Sanjay Agrawal
>
> On Thursday, December 3, 2020, 01:32:32 PM EST, Sumit Bose <
> [e-mail]> wrote:
>
> On Thu, Dec 03, 2020 at 05:50:31PM +0000, Sanjay Agrawal wrote:
> > resending
> > Sanjay Agrawal
> >
> > On Wednesday, December 2, 2020, 03:41:36 PM EST, Sanjay Agrawal <
>
> [e-mail]> wrote:
> > Hi,Â
> >
> > we are seeing an issue with newer version of sssd with centos 7.9 sssd
>
> version 1.16.5-10.el7_9.5.x86_64, where Initgroups is taking much longer
> compared to
> > previous version. Can you please look into it. Following are details,
>
> including a sample program to reproduce the issue with old and new version.
> > Sample Program: get_user_groups.py
> > Â  Â  - it just call getgrouplist of a user supplied using libc
> >
> > Old env -
> > OS Linux 3.10.0-1127.19.1.el7.x86_64 #1 SMP Wed Aug 26 11:48:49 BST 2020
>
> x86_64 x86_64 x86_64 GNU/Linux
> > SSSD Version    - 1.16.4-37.el7_8.4.x86_64
> > Sample log file - sssd_nss-old.log
> > Flamegraph      - flamgraph_sssd_nss-old.svg
> > ID Â 879 'Initgroups by name' testuser1@example
> >     starttime Tue Dec  1 18:36:42:092541 2020
> >     endtime Tue Dec  1 18:36:42:124463 2020
> > Â  Â  lookup_time 0.031922 sec
> >
> > New env
> > OS Linux 3.10.0-1160.6.1.el7.x86_64 #1 SMP Wed Nov 18 22:40:48 GMT 2020
>
> x86_64 x86_64 x86_64 GNU/Linux
> > SSSD Version    - 1.16.5-10.el7_9.5.x86_64
> > Sample log file - sssd_nss-new.log
> > Flamegraph      - flamgraph_sssd_nss-new.svg
> > ID Â 776 'Initgroups by name' testuser1@example
> > Â  Â  starttime 2020-12-01 17:31:24:328419
> > Â  Â  endtime 2020-12-01 17:31:24:451778
> > Â  Â  lookup_time 0.123359 sec
> >
> > It seem to be due to addtional .1 second taken during following two
>
> trace (from sssd_nss-new.log)
> > (2020-12-01 17:31:24:355458): [nss] [cache_req_done] (0x0400): CR #776:
>
> Finished: Success
> > (2020-12-01 17:31:24:451727): [nss] [sysdb_search_group_by_id] (0x0400):
>
> No such entry
> > (2020-12-01 17:31:24:451757): [nss] [nss_protocol_fill_initgr] (0x0080):
>
> Unable to find primary gid [2]: No such file or directory
> > It may be related to following change, which seems to ref to
>
> sysdb_search_group_by_id
> > nss: use real primary gid if the value is overriden · SSSD/sssd@80e6f71
>
> Hi,
>
> thanks for the analysis and I guess you might be right. The patch adds a
> search where one attribute in the filter does not have an index. As a
> result the search has to run over the whole cache.
>
> I wonder if you would like to try to add such index? To do this please
> stop SSSD and call:
> ldbedit -H /var/lib/sss/db/cache_YOUR.DOMAIN.NAME.ldb -s base -b
>
> '@INDEXLIST'
>
> which should open an editor showing something similar to
>
> # editing 1 records
> # record 1
> dn: @INDEXLIST
> @IDXATTR: cn
> @IDXATTR: objectclass
> @IDXATTR: member
> @IDXATTR: memberof
> @IDXATTR: name
> @IDXATTR: uidNumber
> @IDXATTR: gidNumber
> @IDXATTR: lastUpdate
> @IDXATTR: dataExpireTimestamp
> @IDXATTR: originalDN
> @IDXATTR: nameAlias
> @IDXATTR: servicePort
> @IDXATTR: serviceProtocol
> @IDXATTR: sudoUser
> @IDXATTR: sshKnownHostsExpire
> @IDXATTR: objectSIDString
> @IDXATTR: ghost
> @IDXATTR: userPrincipalName
> @IDXATTR: canonicalUserPrincipalName
> @IDXATTR: uniqueID
> @IDXATTR: mail
> @IDXATTR: userMappedCertificate
> @IDXATTR: ccacheFile
> @IDXATTR: ipHostNumber
> @IDXATTR: ipNetworkNumber
> distinguishedName: @INDEXLIST
>
> Please add in the @IDXATTR list a line
> @IDXATTR: originalADgidNumber
>
> and exit the editor, the databased will be reindexed at this point.
> Finally start SSSD again and rerun the test.
>
> HTH
>
> bye,
> Sumit
> > |
> > |
> > |
> > |  |  |
> >
> > |
> >
> > |
> > |
> > |  |
> > nss: use real primary gid if the value is overriden · SSSD/sssd@80e6f71
> > SYSDB_PRIMARY_GROUP_GIDNUM contains original primary group id from AD
>
> because any possible override may not be k...
> > |
> >
> > |
> >
> > |
> >
> > Sanjay Agrawal
> > sssd-users mailing list -- [e-mail]
> > To unsubscribe send an email to [e-mail]
> > Fedora Code of Conduct:
>
> https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> > List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> > List Archives:
>
> https://lists.fedorahosted.org/archives/list/[e-mail].o...
> _______________________________________________
> sssd-users mailing list -- [e-mail]
> To unsubscribe send an email to [e-mail]
> Fedora Code of Conduct:
> https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives:
> https://lists.fedorahosted.org/archives/list/[e-mail].o...
>
> sssd-users mailing list -- [e-mail]
> To unsubscribe send an email to [e-mail]
> Fedora Code of Conduct:
> https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives:
> https://lists.fedorahosted.org/archives/list/[e-mail].o...

initgroups is taking longer with sssd-1.16.5-10.el7_9.5

[Sumit Bose]

On Thu, Dec 03, 2020 at 10:44:09PM +0000, Sanjay Agrawal wrote:
> Thanks for response. I re-ran test after adding index as suggested. Which seems to fix the underlying issue and results are inline with older version. So will this index would be added in newer version of sssd ? Is there an easier way to add this index w/o opening an editor ?

Hi,

something like:

ldbmodify -H /var/lib/sss/db/cache_YOUR.DOMAIN.NAME.ldb -s base -b '@INDEXLIST' << END_END
dn: @INDEXLIST
changetype: modify
add: @IDXATTR
@IDXATTR: originalADgidNumber

END_END

should work as well or you put ldif data into a file and add the
filename to the command line.
> older version 
> sssd-20201203163147/analysis.out:'Initgroups by name' count: 102 avg: 0.596225137254902 median 0.035176 min: 0.001218 max: 22.037332
> sssd-20201203164328/analysis.out:'Initgroups by name' count: 201 avg: 0.313984855721393 median 0.039057 min: 0.000394 max: 23.138423
> sssd-20201203165234/analysis.out:'Initgroups by name' count: 100 avg: 0.59339562 median 0.0362155 min: 0.000412 max: 21.559991
>
> With new version + adding index as suggested above.
> sssd-20201203145950/analysis.out:'Initgroups by name' count: 103 avg: 0.5026502815533981 median 0.03847 min: 0.000435 max: 21.532709
> sssd-20201203152524/analysis.out:'Initgroups by name' count: 90 avg: 1.1182167555555556 median 0.0349585 min: 0.000536 max: 33.604103
> sssd-20201203160837/analysis.out:'Initgroups by name' count: 102 avg: 0.4324193431372549 median 0.0338985 min: 0.000479 max: 19.493188
>
> Above output is by parsing the sssd_nss.log with debug_level 9. As you can see from both older/newer versions, we are seeing a very high max value. All these are when request is being sent from nss->be for resolution. Is is possible similar issue is happening at BE side as well when updating ldb ? Please let me know, if you need additional data.Â

Yes, the max times are most likely related to looking up fresh data from
the server via the backend. It is not unusual the Initgroups request
will take some time, e.g. if you user belongs to many groups and the
cache groups entries are expired as well and must be refreshed as well.
But to better understand what is the reason for the long delay in your
case we would need the backend logs with debug_level=9. There are some
code paths known to be slow, Alexey is e.g. currently working on one of
them. But there might be other reasons for delay as well, e.g. slowly
responding servers, failover to a different server etc.

bye,
Sumit
> Thanks
>
> Sanjay Agrawal
> On Thursday, December 3, 2020, 01:32:32 PM EST, Sumit Bose <[e-mail]> wrote:  
>
> On Thu, Dec 03, 2020 at 05:50:31PM +0000, Sanjay Agrawal wrote:
> > resending
> > Sanjay Agrawal
> >
> > Â  Â  On Wednesday, December 2, 2020, 03:41:36 PM EST, Sanjay Agrawal  wrote:Â  
> > Â  
> >   Hi,ÂÂ
> >
> > we are seeing an issue with newer version of sssd with centos 7.9 sssd version 1.16.5-10.el7_9.5.x86_64, where Initgroups is taking much longer compared to
> > previous version. Can you please look into it. Following are details, including a sample program to reproduce the issue with old and new version.
> >
> > Sample Program: get_user_groups.py
> >     - it just call getgrouplist of a user supplied using libc
> >
> > Old env -
> > OS Linux 3.10.0-1127.19.1.el7.x86_64 #1 SMP Wed Aug 26 11:48:49 BST 2020 x86_64 x86_64 x86_64 GNU/Linux
> > SSSD Version    - 1.16.4-37.el7_8.4.x86_64
> > Sample log file - sssd_nss-old.log
> > Flamegraph      - flamgraph_sssd_nss-old.svg
> > ID  879 'Initgroups by name' testuser1@example
> >     starttime Tue Dec  1 18:36:42:092541 2020
> >     endtime Tue Dec  1 18:36:42:124463 2020
> >     lookup_time 0.031922 sec
> >
> > New env
> > OS Linux 3.10.0-1160.6.1.el7.x86_64 #1 SMP Wed Nov 18 22:40:48 GMT 2020 x86_64 x86_64 x86_64 GNU/Linux
> > SSSD Version    - 1.16.5-10.el7_9.5.x86_64
> > Sample log file - sssd_nss-new.log
> > Flamegraph      - flamgraph_sssd_nss-new.svg
> > ID  776 'Initgroups by name' testuser1@example
> >     starttime 2020-12-01 17:31:24:328419
> >     endtime 2020-12-01 17:31:24:451778
> >     lookup_time 0.123359 sec
> >
> > It seem to be due to addtional .1 second taken during following two trace (from sssd_nss-new.log)
> > (2020-12-01 17:31:24:355458): [nss] [cache_req_done] (0x0400): CR #776: Finished: Success
> > (2020-12-01 17:31:24:451727): [nss] [sysdb_search_group_by_id] (0x0400): No such entry
> > (2020-12-01 17:31:24:451757): [nss] [nss_protocol_fill_initgr] (0x0080): Unable to find primary gid [2]: No such file or directory
> >
> > It may be related to following change, which seems to ref to sysdb_search_group_by_id
> > nss: use real primary gid if the value is overriden · SSSD/sssd@80e6f71
>
> Hi,
>
> thanks for the analysis and I guess you might be right. The patch adds a
> search where one attribute in the filter does not have an index. As a
> result the search has to run over the whole cache.
>
> I wonder if you would like to try to add such index? To do this please
> stop SSSD and call:
>
> Â  Â  ldbedit -H /var/lib/sss/db/cache_YOUR.DOMAIN.NAME.ldb -s base -b '@INDEXLIST'
>
> which should open an editor showing something similar to
>
> # editing 1 records
> # record 1
> dn: @INDEXLIST
> @IDXATTR: cn
> @IDXATTR: objectclass
> @IDXATTR: member
> @IDXATTR: memberof
> @IDXATTR: name
> @IDXATTR: uidNumber
> @IDXATTR: gidNumber
> @IDXATTR: lastUpdate
> @IDXATTR: dataExpireTimestamp
> @IDXATTR: originalDN
> @IDXATTR: nameAlias
> @IDXATTR: servicePort
> @IDXATTR: serviceProtocol
> @IDXATTR: sudoUser
> @IDXATTR: sshKnownHostsExpire
> @IDXATTR: objectSIDString
> @IDXATTR: ghost
> @IDXATTR: userPrincipalName
> @IDXATTR: canonicalUserPrincipalName
> @IDXATTR: uniqueID
> @IDXATTR: mail
> @IDXATTR: userMappedCertificate
> @IDXATTR: ccacheFile
> @IDXATTR: ipHostNumber
> @IDXATTR: ipNetworkNumber
> distinguishedName: @INDEXLIST
>
> Please add in the @IDXATTR list a line
>
> Â  Â  @IDXATTR: originalADgidNumber
>
> and exit the editor, the databased will be reindexed at this point.
> Finally start SSSD again and rerun the test.
>
> HTH
>
> bye,
> Sumit
> > | 
> > | 
> > | 
> > |Â  |Â  |
> >
> > Â  |
> >
> > Â  |
> > | 
> > |Â  | 
> > nss: use real primary gid if the value is overriden · SSSD/sssd@80e6f71
> >
> > SYSDB_PRIMARY_GROUP_GIDNUM contains original primary group id from AD because any possible override may not be k...
> > Â  |
> >
> > Â  |
> >
> > Â  |
> >
> > Sanjay AgrawalÂ
> > sssd-users mailing list -- [e-mail]
> > To unsubscribe send an email to [e-mail]
> > Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> > List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> > List Archives: https://lists.fedorahosted.org/archives/list/[e-mail].o...
>
> sssd-users mailing list -- [e-mail]
> To unsubscribe send an email to [e-mail]
> Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives: https://lists.fedorahosted.org/archives/list/[e-mail].o...
> sssd-users mailing list -- [e-mail]
> To unsubscribe send an email to [e-mail]
> Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives: https://lists.fedorahosted.org/archives/list/[e-mail].o...

initgroups is taking longer with sssd-1.16.5-10.el7_9.5

[Sanjay Agrawal]

Thanks Sumit, it worked for us. For some if queries still taking too long. I have collected all the logs. They are pretty large. Is there w site I can upload to ?

Thanks,Sanjay Agrawal

On Friday, December 4, 2020, 07:24:31 AM EST, Sumit Bose  wrote:

On Thu, Dec 03, 2020 at 10:44:09PM +0000, Sanjay Agrawal wrote:
> Thanks for response. I re-ran test after adding index as suggested. Which seems to fix the underlying issue and results are inline with older version. So will this index would be added in newer version of sssd ? Is there an easier way to add this index w/o opening an editor ?

Hi,

something like:

ldbmodify -H /var/lib/sss/db/cache_YOUR.DOMAIN.NAME.ldb -s base -b '@INDEXLIST' << END_END
dn: @INDEXLIST
changetype: modify
add: @IDXATTR
@IDXATTR: originalADgidNumber

END_END

should work as well or you put ldif data into a file and add the
filename to the command line.
> older version 
> sssd-20201203163147/analysis.out:'Initgroups by name' count: 102 avg: 0.596225137254902 median 0.035176 min: 0.001218 max: 22.037332
> sssd-20201203164328/analysis.out:'Initgroups by name' count: 201 avg: 0.313984855721393 median 0.039057 min: 0.000394 max: 23.138423
> sssd-20201203165234/analysis.out:'Initgroups by name' count: 100 avg: 0.59339562 median 0.0362155 min: 0.000412 max: 21.559991
>
> With new version + adding index as suggested above.
> sssd-20201203145950/analysis.out:'Initgroups by name' count: 103 avg: 0.5026502815533981 median 0.03847 min: 0.000435 max: 21.532709
> sssd-20201203152524/analysis.out:'Initgroups by name' count: 90 avg: 1.1182167555555556 median 0.0349585 min: 0.000536 max: 33.604103
> sssd-20201203160837/analysis.out:'Initgroups by name' count: 102 avg: 0.4324193431372549 median 0.0338985 min: 0.000479 max: 19.493188
>
> Above output is by parsing the sssd_nss.log with debug_level 9. As you can see from both older/newer versions, we are seeing a very high max value. All these are when request is being sent from nss->be for resolution. Is is possible similar issue is happening at BE side as well when updating ldb ? Please let me know, if you need additional data.Â

Yes, the max times are most likely related to looking up fresh data from
the server via the backend. It is not unusual the Initgroups request
will take some time, e.g. if you user belongs to many groups and the
cache groups entries are expired as well and must be refreshed as well.
But to better understand what is the reason for the long delay in your
case we would need the backend logs with debug_level=9. There are some
code paths known to be slow, Alexey is e.g. currently working on one of
them. But there might be other reasons for delay as well, e.g. slowly
responding servers, failover to a different server etc.

bye,
Sumit
> Thanks
>
> Sanjay Agrawal
>
> On Thursday, December 3, 2020, 01:32:32 PM EST, Sumit Bose  wrote:  
>   
>   On Thu, Dec 03, 2020 at 05:50:31PM +0000, Sanjay Agrawal wrote:
> > resending
> > Sanjay Agrawal
> >
> > Â  Â  On Wednesday, December 2, 2020, 03:41:36 PM EST, Sanjay Agrawal  wrote:Â  
> > Â  
> >   Hi,ÂÂ
> >
> > we are seeing an issue with newer version of sssd with centos 7.9 sssd version 1.16.5-10.el7_9.5.x86_64, where Initgroups is taking much longer compared to
> > previous version. Can you please look into it. Following are details, including a sample program to reproduce the issue with old and new version.
> >
> > Sample Program: get_user_groups.py
> >     - it just call getgrouplist of a user supplied using libc
> >
> > Old env -
> > OS Linux 3.10.0-1127.19.1.el7.x86_64 #1 SMP Wed Aug 26 11:48:49 BST 2020 x86_64 x86_64 x86_64 GNU/Linux
> > SSSD Version    - 1.16.4-37.el7_8.4.x86_64
> > Sample log file - sssd_nss-old.log
> > Flamegraph      - flamgraph_sssd_nss-old.svg
> > ID  879 'Initgroups by name' testuser1@example
> >     starttime Tue Dec  1 18:36:42:092541 2020
> >     endtime Tue Dec  1 18:36:42:124463 2020
> >     lookup_time 0.031922 sec
> >
> > New env
> > OS Linux 3.10.0-1160.6.1.el7.x86_64 #1 SMP Wed Nov 18 22:40:48 GMT 2020 x86_64 x86_64 x86_64 GNU/Linux
> > SSSD Version    - 1.16.5-10.el7_9.5.x86_64
> > Sample log file - sssd_nss-new.log
> > Flamegraph      - flamgraph_sssd_nss-new.svg
> > ID  776 'Initgroups by name' testuser1@example
> >     starttime 2020-12-01 17:31:24:328419
> >     endtime 2020-12-01 17:31:24:451778
> >     lookup_time 0.123359 sec
> >
> > It seem to be due to addtional .1 second taken during following two trace (from sssd_nss-new.log)
> > (2020-12-01 17:31:24:355458): [nss] [cache_req_done] (0x0400): CR #776: Finished: Success
> > (2020-12-01 17:31:24:451727): [nss] [sysdb_search_group_by_id] (0x0400): No such entry
> > (2020-12-01 17:31:24:451757): [nss] [nss_protocol_fill_initgr] (0x0080): Unable to find primary gid [2]: No such file or directory
> >
> > It may be related to following change, which seems to ref to sysdb_search_group_by_id
> > nss: use real primary gid if the value is overriden · SSSD/sssd@80e6f71
>
> Hi,
>
> thanks for the analysis and I guess you might be right. The patch adds a
> search where one attribute in the filter does not have an index. As a
> result the search has to run over the whole cache.
>
> I wonder if you would like to try to add such index? To do this please
> stop SSSD and call:
>
> Â  Â  ldbedit -H /var/lib/sss/db/cache_YOUR.DOMAIN.NAME.ldb -s base -b '@INDEXLIST'
>
> which should open an editor showing something similar to
>
> # editing 1 records
> # record 1
> dn: @INDEXLIST
> @IDXATTR: cn
> @IDXATTR: objectclass
> @IDXATTR: member
> @IDXATTR: memberof
> @IDXATTR: name
> @IDXATTR: uidNumber
> @IDXATTR: gidNumber
> @IDXATTR: lastUpdate
> @IDXATTR: dataExpireTimestamp
> @IDXATTR: originalDN
> @IDXATTR: nameAlias
> @IDXATTR: servicePort
> @IDXATTR: serviceProtocol
> @IDXATTR: sudoUser
> @IDXATTR: sshKnownHostsExpire
> @IDXATTR: objectSIDString
> @IDXATTR: ghost
> @IDXATTR: userPrincipalName
> @IDXATTR: canonicalUserPrincipalName
> @IDXATTR: uniqueID
> @IDXATTR: mail
> @IDXATTR: userMappedCertificate
> @IDXATTR: ccacheFile
> @IDXATTR: ipHostNumber
> @IDXATTR: ipNetworkNumber
> distinguishedName: @INDEXLIST
>
> Please add in the @IDXATTR list a line
>
> Â  Â  @IDXATTR: originalADgidNumber
>
> and exit the editor, the databased will be reindexed at this point.
> Finally start SSSD again and rerun the test.
>
> HTH
>
> bye,
> Sumit
> > | 
> > | 
> > | 
> > |Â  |Â  |
> >
> > Â  |
> >
> > Â  |
> > | 
> > |Â  | 
> > nss: use real primary gid if the value is overriden · SSSD/sssd@80e6f71
> >
> > SYSDB_PRIMARY_GROUP_GIDNUM contains original primary group id from AD because any possible override may not be k...
> > Â  |
> >
> > Â  |
> >
> > Â  |
> >
> > Sanjay AgrawalÂ
> > sssd-users mailing list -- [e-mail]
> > To unsubscribe send an email to [e-mail]
> > Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> > List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> > List Archives: https://lists.fedorahosted.org/archives/list/[e-mail].o...
>
> sssd-users mailing list -- [e-mail]
> To unsubscribe send an email to [e-mail]
> Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives: https://lists.fedorahosted.org/archives/list/[e-mail].o...
> sssd-users mailing list -- [e-mail]
> To unsubscribe send an email to [e-mail]
> Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives: https://lists.fedorahosted.org/archives/list/[e-mail].o...

_______________________________________________
sssd-users mailing list -- [e-mail]
To unsubscribe send an email to [e-mail]
Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: https://lists.fedorahosted.org/archives/list/[e-mail].o...

initgroups is taking longer with sssd-1.16.5-10.el7_9.5

[Sumit Bose]

On Tue, Dec 15, 2020 at 01:56:57AM +0000, Sanjay Agrawal wrote:
> Thanks Sumit, it worked for us. For some if queries still taking too long. I have collected all the logs. They are pretty large. Is there w site I can upload to ?

Hi,

if you have a place where you can upload them you can send me the link
directly. Otherwise you can try to open a ticket a bugzilla.redhat.com
and try to upload the logs there but I do not know which limit applies.

bye,
Sumit
> Thanks,Sanjay Agrawal
> On Friday, December 4, 2020, 07:24:31 AM EST, Sumit Bose <[e-mail]> wrote:  
>
> On Thu, Dec 03, 2020 at 10:44:09PM +0000, Sanjay Agrawal wrote:
> > Thanks for response. I re-ran test after adding index as suggested. Which seems to fix the underlying issue and results are inline with older version. So will this index would be added in newer version of sssd ? Is there an easier way to add this index w/o opening an editor ?
>
> Hi,
>
> something like:
>
> ldbmodify -H /var/lib/sss/db/cache_YOUR.DOMAIN.NAME.ldb -s base -b '@INDEXLIST' << END_END
> dn: @INDEXLIST
> changetype: modify
> add: @IDXATTR
> @IDXATTR: originalADgidNumber
>
> END_END
>
> should work as well or you put ldif data into a file and add the
> filename to the command line.
> > older version 
> > sssd-20201203163147/analysis.out:'Initgroups by name' count: 102 avg: 0.596225137254902 median 0.035176 min: 0.001218 max: 22.037332
> > sssd-20201203164328/analysis.out:'Initgroups by name' count: 201 avg: 0.313984855721393 median 0.039057 min: 0.000394 max: 23.138423
> > sssd-20201203165234/analysis.out:'Initgroups by name' count: 100 avg: 0.59339562 median 0.0362155 min: 0.000412 max: 21.559991
> >
> > With new version + adding index as suggested above.
> > sssd-20201203145950/analysis.out:'Initgroups by name' count: 103 avg: 0.5026502815533981 median 0.03847 min: 0.000435 max: 21.532709
> > sssd-20201203152524/analysis.out:'Initgroups by name' count: 90 avg: 1.1182167555555556 median 0.0349585 min: 0.000536 max: 33.604103
> > sssd-20201203160837/analysis.out:'Initgroups by name' count: 102 avg: 0.4324193431372549 median 0.0338985 min: 0.000479 max: 19.493188
> >
> > Above output is by parsing the sssd_nss.log with debug_level 9. As you can see from both older/newer versions, we are seeing a very high max value. All these are when request is being sent from nss->be for resolution. Is is possible similar issue is happening at BE side as well when updating ldb ? Please let me know, if you need additional data.Â
>
> Yes, the max times are most likely related to looking up fresh data from
> the server via the backend. It is not unusual the Initgroups request
> will take some time, e.g. if you user belongs to many groups and the
> cache groups entries are expired as well and must be refreshed as well.
> But to better understand what is the reason for the long delay in your
> case we would need the backend logs with debug_level=9. There are some
> code paths known to be slow, Alexey is e.g. currently working on one of
> them. But there might be other reasons for delay as well, e.g. slowly
> responding servers, failover to a different server etc.
>
> bye,
> Sumit
> > Thanks
> >
> > Sanjay Agrawal
> >
> > On Thursday, December 3, 2020, 01:32:32 PM EST, Sumit Bose  wrote:  
> >   
> >   On Thu, Dec 03, 2020 at 05:50:31PM +0000, Sanjay Agrawal wrote:
> > > resending
> > > Sanjay Agrawal
> > >
> > > Â  Â  On Wednesday, December 2, 2020, 03:41:36 PM EST, Sanjay Agrawal  wrote:Â  
> > > Â  
> > >   Hi,ÂÂ
> > >
> > > we are seeing an issue with newer version of sssd with centos 7.9 sssd version 1.16.5-10.el7_9.5.x86_64, where Initgroups is taking much longer compared to
> > > previous version. Can you please look into it. Following are details, including a sample program to reproduce the issue with old and new version.
> > >
> > > Sample Program: get_user_groups.py
> > >     - it just call getgrouplist of a user supplied using libc
> > >
> > > Old env -
> > > OS Linux 3.10.0-1127.19.1.el7.x86_64 #1 SMP Wed Aug 26 11:48:49 BST 2020 x86_64 x86_64 x86_64 GNU/Linux
> > > SSSD Version    - 1.16.4-37.el7_8.4.x86_64
> > > Sample log file - sssd_nss-old.log
> > > Flamegraph      - flamgraph_sssd_nss-old.svg
> > > ID  879 'Initgroups by name' testuser1@example
> > >     starttime Tue Dec  1 18:36:42:092541 2020
> > >     endtime Tue Dec  1 18:36:42:124463 2020
> > >     lookup_time 0.031922 sec
> > >
> > > New env
> > > OS Linux 3.10.0-1160.6.1.el7.x86_64 #1 SMP Wed Nov 18 22:40:48 GMT 2020 x86_64 x86_64 x86_64 GNU/Linux
> > > SSSD Version    - 1.16.5-10.el7_9.5.x86_64
> > > Sample log file - sssd_nss-new.log
> > > Flamegraph      - flamgraph_sssd_nss-new.svg
> > > ID  776 'Initgroups by name' testuser1@example
> > >     starttime 2020-12-01 17:31:24:328419
> > >     endtime 2020-12-01 17:31:24:451778
> > >     lookup_time 0.123359 sec
> > >
> > > It seem to be due to addtional .1 second taken during following two trace (from sssd_nss-new.log)
> > > (2020-12-01 17:31:24:355458): [nss] [cache_req_done] (0x0400): CR #776: Finished: Success
> > > (2020-12-01 17:31:24:451727): [nss] [sysdb_search_group_by_id] (0x0400): No such entry
> > > (2020-12-01 17:31:24:451757): [nss] [nss_protocol_fill_initgr] (0x0080): Unable to find primary gid [2]: No such file or directory
> > >
> > > It may be related to following change, which seems to ref to sysdb_search_group_by_id
> > > nss: use real primary gid if the value is overriden · SSSD/sssd@80e6f71
> >
> > Hi,
> >
> > thanks for the analysis and I guess you might be right. The patch adds a
> > search where one attribute in the filter does not have an index. As a
> > result the search has to run over the whole cache.
> >
> > I wonder if you would like to try to add such index? To do this please
> > stop SSSD and call:
> >
> > Â  Â  ldbedit -H /var/lib/sss/db/cache_YOUR.DOMAIN.NAME.ldb -s base -b '@INDEXLIST'
> >
> > which should open an editor showing something similar to
> >
> > # editing 1 records
> > # record 1
> > dn: @INDEXLIST
> > @IDXATTR: cn
> > @IDXATTR: objectclass
> > @IDXATTR: member
> > @IDXATTR: memberof
> > @IDXATTR: name
> > @IDXATTR: uidNumber
> > @IDXATTR: gidNumber
> > @IDXATTR: lastUpdate
> > @IDXATTR: dataExpireTimestamp
> > @IDXATTR: originalDN
> > @IDXATTR: nameAlias
> > @IDXATTR: servicePort
> > @IDXATTR: serviceProtocol
> > @IDXATTR: sudoUser
> > @IDXATTR: sshKnownHostsExpire
> > @IDXATTR: objectSIDString
> > @IDXATTR: ghost
> > @IDXATTR: userPrincipalName
> > @IDXATTR: canonicalUserPrincipalName
> > @IDXATTR: uniqueID
> > @IDXATTR: mail
> > @IDXATTR: userMappedCertificate
> > @IDXATTR: ccacheFile
> > @IDXATTR: ipHostNumber
> > @IDXATTR: ipNetworkNumber
> > distinguishedName: @INDEXLIST
> >
> > Please add in the @IDXATTR list a line
> >
> > Â  Â  @IDXATTR: originalADgidNumber
> >
> > and exit the editor, the databased will be reindexed at this point.
> > Finally start SSSD again and rerun the test.
> >
> > HTH
> >
> > bye,
> > Sumit
> > > | 
> > > | 
> > > | 
> > > |Â  |Â  |
> > >
> > > Â  |
> > >
> > > Â  |
> > > | 
> > > |Â  | 
> > > nss: use real primary gid if the value is overriden · SSSD/sssd@80e6f71
> > >
> > > SYSDB_PRIMARY_GROUP_GIDNUM contains original primary group id from AD because any possible override may not be k...
> > > Â  |
> > >
> > > Â  |
> > >
> > > Â  |
> > >
> > > Sanjay AgrawalÂ
> > > sssd-users mailing list -- [e-mail]
> > > To unsubscribe send an email to [e-mail]
> > > Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> > > List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> > > List Archives: https://lists.fedorahosted.org/archives/list/[e-mail].o...
> >
> > sssd-users mailing list -- [e-mail]
> > To unsubscribe send an email to [e-mail]
> > Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> > List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> > List Archives: https://lists.fedorahosted.org/archives/list/[e-mail].o...
> > sssd-users mailing list -- [e-mail]
> > To unsubscribe send an email to [e-mail]
> > Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> > List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> > List Archives: https://lists.fedorahosted.org/archives/list/[e-mail].o...
>
> sssd-users mailing list -- [e-mail]
> To unsubscribe send an email to [e-mail]
> Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives: https://lists.fedorahosted.org/archives/list/[e-mail].o...
> sssd-users mailing list -- [e-mail]
> To unsubscribe send an email to [e-mail]
> Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives: https://lists.fedorahosted.org/archives/list/[e-mail].o...