https://bugzilla.redhat.com/show_bug.cgi?id=1984591 not understanding the nature of the sssd bug introduced recently…
All (but particularly Sumit since he wrote the comments on https://bugzilla.redhat.com/show_bug.cgi?id=1984591),
There are at least two problems created by this recently-introduced sssd bug. One problem is solvable by the suggested work-around, the other is not. The work-around suggested is:
[domain/name.of.joined.domain]
ad_enabled_domains = dom1.example.com, dom2.example.com, dom3.example.com
In order to query only the desired AD domains.
What is the bug?
the sssd-ad man page says "The AD provider can be used to get user information and authenticate users from trusted domains. Currently only trusted domains in the same forest are recognized.".
What is happening is that untrusted AD domains are being discovered. A very specific type of untrusted domains. When the joined domain has no trust with that other domain, but that other domain trusts the original domain – that is a one-way trust (the wrong way). To the joined domain, this is an untrusted domain and should not be discovered.
This is actually very common in corporate environments.
You may have a main AD domain, call it CORP.COMPANY.COM. Then for testing and new production evaluation, you might have a test AD domain called LAB-TEST.COMPANY.COM. CORP.COMPANY.COM is tightly controlled, with full audits and corporate security. LAB-TEST.COMPANY.COM is a test AD domain – it’s the wild, wild west!
So LAB-TEST.COMPANY.COM trusts the main AD domain (in order that users can log into this test domain with their CORP accounts). But CORP.COMPANY.COM does not trust LAB-TEST.COMPANY.COM – nor should it!! (That’s the wild, wild west, doing so would compromise corporate security.)
Thus, a server joined to domain CORP.COMPANY.COM should discover CORP.COMPANY.COM and any domains trusted by CORP.COMPANY.COM. It should *NOT* discover LAB-TEST.COMPANY.COM, as CORP.COMPANY.COM does not trust this domain.
A server joined to LAB-TEST.COMPANY.COM should discover LAB-TEST.COMPANY.COM and all domains trusted by LAB-TEST.COMPANY.COM. Including CORP.COMPANY.COM, as LAB-TEST.COMPANY.COM trusts CORP.COMPANY.COM.
The bug is that a server joined to CORP.COMPANY.COM discovers LAB-TEST.COMPANY.COM, which it shouldn’t.
What problems does this cause?
Two problems.
1. Many of these untrusted discovered “lab” domains are accessible only to specific network locations. That is, they’re firewalled off to a particular lab. So sssd attempts to query these inaccessible AD domains and takes a long time to time out. This problem can be worked around by the suggested work-around in the Bugzilla:
[domain/corp.company.com]
ad_enabled_domains = corp.company.com
So then, while LAB-TEST.COMPANY.COM is still erroneously discovered, it is no longer searched. Sssd is again fast.
2. Bogus messages in /var/log/sssd_nss.log file. Even with no debug level set in the [nss] stanza, these error messages appear multiple times a second. It quickly fills up the /var/log filesystem.
[root@auspdfdlobv01 sssd]# cat sssd_nss.log |grep "The Data Provider returned an error"
(2021-10-07 9:50:02): [nss] [sss_dp_get_reply] (0x0010): The Data Provider returned an error [org.freedesktop.DBus.Error.Failed]
(2021-10-07 9:50:02): [nss] [sss_dp_get_reply] (0x0010): The Data Provider returned an error [org.freedesktop.DBus.Error.Failed]
(2021-10-07 9:50:02): [nss] [sss_dp_get_reply] (0x0010): The Data Provider returned an error [org.freedesktop.DBus.Error.Failed]
(2021-10-07 9:50:02): [nss] [sss_dp_get_reply] (0x0010): The Data Provider returned an error [org.freedesktop.DBus.Error.Failed]
(2021-10-07 9:50:02): [nss] [sss_dp_get_reply] (0x0010): The Data Provider returned an error [org.freedesktop.DBus.Error.Failed]
(2021-10-07 9:50:02): [nss] [sss_dp_get_reply] (0x0010): The Data Provider returned an error [org.freedesktop.DBus.Error.Failed]
(2021-10-07 9:50:02): [nss] [sss_dp_get_reply] (0x0010): The Data Provider returned an error [org.freedesktop.DBus.Error.Failed]
(2021-10-07 9:50:02): [nss] [sss_dp_get_reply] (0x0010): The Data Provider returned an error [org.freedesktop.DBus.Error.Failed]
(2021-10-07 9:50:02): [nss] [sss_dp_get_reply] (0x0010): The Data Provider returned an error [org.freedesktop.DBus.Error.Failed]
(2021-10-07 9:50:02): [nss] [sss_dp_get_reply] (0x0010): The Data Provider returned an error [org.freedesktop.DBus.Error.Failed]
(2021-10-07 9:50:02): [nss] [sss_dp_get_reply] (0x0010): The Data Provider returned an error [org.freedesktop.DBus.Error.Failed]
(2021-10-07 9:50:02): [nss] [sss_dp_get_reply] (0x0010): The Data Provider returned an error [org.freedesktop.DBus.Error.Failed]
(2021-10-07 9:50:02): [nss] [sss_dp_get_reply] (0x0010): The Data Provider returned an error [org.freedesktop.DBus.Error.Failed]
(2021-10-07 9:50:02): [nss] [sss_dp_get_reply] (0x0010): The Data Provider returned an error [org.freedesktop.DBus.Error.Failed]
(2021-10-07 9:50:02): [nss] [sss_dp_get_reply] (0x0010): The Data Provider returned an error [org.freedesktop.DBus.Error.Failed]
(2021-10-07 9:50:02): [nss] [sss_dp_get_reply] (0x0010): The Data Provider returned an error [org.freedesktop.DBus.Error.Failed]
(2021-10-07 9:50:02): [nss] [sss_dp_get_reply] (0x0010): The Data Provider returned an error [org.freedesktop.DBus.Error.Failed]
(2021-10-07 9:50:02): [nss] [sss_dp_get_reply] (0x0010): The Data Provider returned an error [org.freedesktop.DBus.Error.Failed]
(2021-10-07 9:50:02): [nss] [sss_dp_get_reply] (0x0010): The Data Provider returned an error [org.freedesktop.DBus.Error.Failed]
(2021-10-07 9:50:02): [nss] [sss_dp_get_reply] (0x0010): The Data Provider returned an error [org.freedesktop.DBus.Error.Failed]
(2021-10-07 9:50:02): [nss] [sss_dp_get_reply] (0x0010): The Data Provider returned an error [org.freedesktop.DBus.Error.Failed]
(2021-10-07 9:50:02): [nss] [sss_dp_get_reply] (0x0010): The Data Provider returned an error [org.freedesktop.DBus.Error.Failed]
(2021-10-07 9:50:02): [nss] [sss_dp_get_reply] (0x0010): The Data Provider returned an error [org.freedesktop.DBus.Error.Failed]
(2021-10-07 9:50:02): [nss] [sss_dp_get_reply] (0x0010): The Data Provider returned an error [org.freedesktop.DBus.Error.Failed]
(2021-10-07 9:50:02): [nss] [sss_dp_get_reply] (0x0010): The Data Provider returned an error [org.freedesktop.DBus.Error.Failed]
From debug level 9, it is clear that this is arising from a query of these erroneously-discovered untrusted domains. Here’s an example of one instance of above with debug level 9 turned on. So emeaicmd.geodll.company.com is one of these erroneously-discovered untrusted lab domains, that happens to be firewalled off from this particular AD client:
(2021-10-07 9:50:02): [nss] [sss_dp_get_reply] (0x1000): Got reply from Data Provider - DP error code: 0 errno: 0 error message: Success
(2021-10-07 9:50:02): [nss] [cache_req_search_cache] (0x0400): CR #9: Looking up [oracle@company.com] in cache
(2021-10-07 9:50:02): [nss] [cache_req_search_cache] (0x0400): CR #9: Object [oracle@company.com] was not found in cache
(2021-10-07 9:50:02): [nss] [cache_req_search_ncache_add_to_domain] (0x0400): CR #9: Adding [oracle@company.com] to negative cache
(2021-10-07 9:50:02): [nss] [is_user_local_by_name] (0x0400): User oracle@company.com is a local user
(2021-10-07 9:50:02): [nss] [sss_ncache_set_str] (0x0400): Adding [NCE/USER/company.com/oracle@company.com] to negative cache
(2021-10-07 9:50:02): [nss] [cache_req_validate_domain_type] (0x2000): Request type POSIX-only for domain EMEAICMD.geodll.company.com type POSIX is valid
(2021-10-07 9:50:02): [nss] [cache_req_set_domain] (0x0400): CR #9: Using domain [EMEAICMD.geodll.company.com]
(2021-10-07 9:50:02): [nss] [cache_req_prepare_domain_data] (0x0400): CR #9: Preparing input data for domain [EMEAICMD.geodll.company.com] rules
(2021-10-07 9:50:02): [nss] [cache_req_search_send] (0x0400): CR #9: Looking up oracle@emeaicmd.geodll.company.com
(2021-10-07 9:50:02): [nss] [cache_req_search_ncache] (0x0400): CR #9: Checking negative cache for [oracle@emeaicmd.geodll.company.com]
(2021-10-07 9:50:02): [nss] [sss_ncache_check_str] (0x2000): Checking negative cache for [NCE/USER/ EMEAICMD.geodll.company.com/oracle@emeaicmd.geodll.company.com]
(2021-10-07 9:50:02): [nss] [cache_req_search_ncache] (0x0400): CR #9: [ oracle@emeaicmd.geodll.company.com] is not present in negative cache
(2021-10-07 9:50:02): [nss] [cache_req_search_cache] (0x0400): CR #9: Looking up [oracle@emeaicmd.geodll.company.com] in cache
(2021-10-07 9:50:02): [nss] [cache_req_search_cache] (0x0400): CR #9: Object [oracle@emeaicmd.geodll.company.com] was not found in cache
(2021-10-07 9:50:02): [nss] [cache_req_search_dp] (0x0400): CR #9: Looking up [oracle@emeaicmd.geodll.company.com] in data provider
(2021-10-07 9:50:02): [nss] [sss_dp_issue_request] (0x0400): Issuing request for [0x564d6be36a70:3:oracle@emeaicmd.geodll.company.com@ EMEAICMD.geodll.company.com]
(2021-10-07 9:50:02): [nss] [sss_dp_get_account_msg] (0x0400): Creating request for [EMEAICMD.geodll.company.com ][0x3][BE_REQ_INITGROUPS][name=oracle@emeaicmd.geodll.company.com:-]
(2021-10-07 9:50:02): [nss] [sbus_add_timeout] (0x2000): 0x564d6ccd6670
(2021-10-07 9:50:02): [nss] [sss_dp_internal_get_send] (0x0400): Entering request [0x564d6be36a70:3:oracle@emeaicmd.geodll.company.com@ EMEAICMD.geodll.company.com]
(2021-10-07 9:50:02): [nss] [cache_req_search_cache] (0x0400): CR #12: Looking up [oracle@company.com] in cache
(2021-10-07 9:50:02): [nss] [cache_req_search_cache] (0x0400): CR #12: Object [oracle@company.com] was not found in cache
(2021-10-07 9:50:02): [nss] [cache_req_search_ncache_add_to_domain] (0x0400): CR #12: Adding [oracle@company.com] to negative cache
(2021-10-07 9:50:02): [nss] [is_user_local_by_name] (0x0400): User oracle@company.com is a local user
(2021-10-07 9:50:02): [nss] [sss_ncache_set_str] (0x0400): Adding [NCE/USER/company.com/oracle@company.com] to negative cache
(2021-10-07 9:50:02): [nss] [cache_req_validate_domain_type] (0x2000): Request type POSIX-only for domain EMEAICMD.geodll.company.com type POSIX is valid
(2021-10-07 9:50:02): [nss] [cache_req_set_domain] (0x0400): CR #12: Using domain [EMEAICMD.geodll.company.com]
(2021-10-07 9:50:02): [nss] [cache_req_prepare_domain_data] (0x0400): CR #12: Preparing input data for domain [EMEAICMD.geodll.company.com] rules
(2021-10-07 9:50:02): [nss] [cache_req_search_send] (0x0400): CR #12: Looking up oracle@emeaicmd.geodll.company.com
(2021-10-07 9:50:02): [nss] [cache_req_search_ncache] (0x0400): CR #12: Checking negative cache for [oracle@emeaicmd.geodll.company.com]
(2021-10-07 9:50:02): [nss] [sss_ncache_check_str] (0x2000): Checking negative cache for [NCE/USER/ EMEAICMD.geodll.company.com/oracle@emeaicmd.geodll.company.com]
(2021-10-07 9:50:02): [nss] [cache_req_search_ncache] (0x0400): CR #12: [ oracle@emeaicmd.geodll.company.com] is not present in negative cache
(2021-10-07 9:50:02): [nss] [cache_req_search_cache] (0x0400): CR #12: Looking up [oracle@emeaicmd.geodll.company.com] in cache
(2021-10-07 9:50:02): [nss] [cache_req_search_cache] (0x0400): CR #12: Object [oracle@emeaicmd.geodll.company.com] was not found in cache
(2021-10-07 9:50:02): [nss] [cache_req_search_dp] (0x0400): CR #12: Looking up [oracle@emeaicmd.geodll.company.com] in data provider
(2021-10-07 9:50:02): [nss] [sss_dp_issue_request] (0x0400): Issuing request for [0x564d6be36a70:3:oracle@emeaicmd.geodll.company.com@ EMEAICMD.geodll.company.com]
(2021-10-07 9:50:02): [nss] [sss_dp_issue_request] (0x0400): Identical request in progress: [0x564d6be36a70:3:oracle@emeaicmd.geodll.company.com@ EMEAICMD.geodll.company.com]
(2021-10-07 9:50:02): [nss] [sss_dp_req_destructor] (0x0400): Deleting request: [0x564d6be36a70:3:oracle@company.com@company.com]
(2021-10-07 9:50:02): [nss] [sbus_remove_timeout] (0x2000): 0x564d6ccd6670
(2021-10-07 9:50:02): [nss] [sbus_dispatch] (0x4000): dbus conn: 0x564d6ccc9300
(2021-10-07 9:50:02): [nss] [sbus_dispatch] (0x4000): Dispatching.
(2021-10-07 9:50:02): [nss] [sss_dp_get_reply] (0x0010): The Data Provider returned an error [org.freedesktop.DBus.Error.Failed]
The suggested work-around does not resolve problem #2.
BTW, here is a listing of the domains discovered on that sssd client:
[root@auspdfdlobv01 ~]# sssctl domain-list
amer.company.com
company.com
japn.company.com
emea.company.com
apac.company.com
EMEAICMD.geodll.company.com
geodll.company.com
EMEAICM.GEODLL.COMPANY.COM
alienware.com
corp.svcs
perotsystems.net
companyservices.dmz
Beer.Town
production.online.company.com
jp-poclab.companypoc.com
emea-poclab.companypoc.com
oldev.preol.company.com
olqa.preol.company.com
ap-poclab.companypoc.com
[root@auspdfdlobv01 ~]#
This sssd client is joined to amer.company.com, so the only trusted domains are the first 5. The parent domain and the 4 regional domains. All those other domains below that are untrusted domains. More specifically, they trust company.com, but company.com does not trust them. (one way trust – the wrong way.) Some look like the real wild wild west (Beer.Town ?).
Spike
Am Thu, Oct 07, 2021 at 11:38:54AM -0500 schrieb Spike White:
All (but particularly Sumit since he wrote the comments on https://bugzilla.redhat.com/show_bug.cgi?id=1984591),
Hi,
jfyi, I'm currently working on a fix for this to filter out domains from other forests and untrusted domains. My WIP branch is at https://github.com/sumit-bose/sssd/tree/ad_filter_domains. Can you do a test build of SSSD based on this or shall I try to create a test build for you? For the latter, please tell me for which platform.
bye, Sumit
There are at least two problems created by this recently-introduced sssd bug. One problem is solvable by the suggested work-around, the other is not. The work-around suggested is:
[domain/name.of.joined.domain]
ad_enabled_domains = dom1.example.com, dom2.example.com,dom3.example.com
In order to query only the desired AD domains.
What is the bug?
the sssd-ad man page says "The AD provider can be used to get user information and authenticate users from trusted domains. Currently only trusted domains in the same forest are recognized.".
What is happening is that untrusted AD domains are being discovered. A very specific type of untrusted domains. When the joined domain has no trust with that other domain, but that other domain trusts the original domain – that is a one-way trust (the wrong way). To the joined domain, this is an untrusted domain and should not be discovered.
This is actually very common in corporate environments.
You may have a main AD domain, call it CORP.COMPANY.COM. Then for testing and new production evaluation, you might have a test AD domain called LAB-TEST.COMPANY.COM. CORP.COMPANY.COM is tightly controlled, with full audits and corporate security. LAB-TEST.COMPANY.COM is a test AD domain – it’s the wild, wild west!
So LAB-TEST.COMPANY.COM trusts the main AD domain (in order that users can log into this test domain with their CORP accounts). But CORP.COMPANY.COM does not trust LAB-TEST.COMPANY.COM – nor should it!! (That’s the wild, wild west, doing so would compromise corporate security.)
Thus, a server joined to domain CORP.COMPANY.COM should discover CORP.COMPANY.COM and any domains trusted by CORP.COMPANY.COM. It should *NOT* discover LAB-TEST.COMPANY.COM, as CORP.COMPANY.COM does not trust this domain.
A server joined to LAB-TEST.COMPANY.COM should discover LAB-TEST.COMPANY.COM and all domains trusted by LAB-TEST.COMPANY.COM. Including CORP.COMPANY.COM, as LAB-TEST.COMPANY.COM trusts CORP.COMPANY.COM.
The bug is that a server joined to CORP.COMPANY.COM discovers LAB-TEST.COMPANY.COM, which it shouldn’t.
What problems does this cause?
Two problems.
Many of these untrusted discovered “lab” domains are accessibleonly to specific network locations. That is, they’re firewalled off to a particular lab. So sssd attempts to query these inaccessible AD domains and takes a long time to time out. This problem can be worked around by the suggested work-around in the Bugzilla:
[domain/corp.company.com]
ad_enabled_domains = corp.company.comSo then, while LAB-TEST.COMPANY.COM is still erroneously discovered, it is no longer searched. Sssd is again fast.
Bogus messages in /var/log/sssd_nss.log file. Even with no debuglevel set in the [nss] stanza, these error messages appear multiple times a second. It quickly fills up the /var/log filesystem.
[root@auspdfdlobv01 sssd]# cat sssd_nss.log |grep "The Data Provider returned an error"
(2021-10-07 9:50:02): [nss] [sss_dp_get_reply] (0x0010): The Data Provider returned an error [org.freedesktop.DBus.Error.Failed]
(2021-10-07 9:50:02): [nss] [sss_dp_get_reply] (0x0010): The Data Provider returned an error [org.freedesktop.DBus.Error.Failed]
(2021-10-07 9:50:02): [nss] [sss_dp_get_reply] (0x0010): The Data Provider returned an error [org.freedesktop.DBus.Error.Failed]
(2021-10-07 9:50:02): [nss] [sss_dp_get_reply] (0x0010): The Data Provider returned an error [org.freedesktop.DBus.Error.Failed]
(2021-10-07 9:50:02): [nss] [sss_dp_get_reply] (0x0010): The Data Provider returned an error [org.freedesktop.DBus.Error.Failed]
(2021-10-07 9:50:02): [nss] [sss_dp_get_reply] (0x0010): The Data Provider returned an error [org.freedesktop.DBus.Error.Failed]
(2021-10-07 9:50:02): [nss] [sss_dp_get_reply] (0x0010): The Data Provider returned an error [org.freedesktop.DBus.Error.Failed]
(2021-10-07 9:50:02): [nss] [sss_dp_get_reply] (0x0010): The Data Provider returned an error [org.freedesktop.DBus.Error.Failed]
(2021-10-07 9:50:02): [nss] [sss_dp_get_reply] (0x0010): The Data Provider returned an error [org.freedesktop.DBus.Error.Failed]
(2021-10-07 9:50:02): [nss] [sss_dp_get_reply] (0x0010): The Data Provider returned an error [org.freedesktop.DBus.Error.Failed]
(2021-10-07 9:50:02): [nss] [sss_dp_get_reply] (0x0010): The Data Provider returned an error [org.freedesktop.DBus.Error.Failed]
(2021-10-07 9:50:02): [nss] [sss_dp_get_reply] (0x0010): The Data Provider returned an error [org.freedesktop.DBus.Error.Failed]
(2021-10-07 9:50:02): [nss] [sss_dp_get_reply] (0x0010): The Data Provider returned an error [org.freedesktop.DBus.Error.Failed]
(2021-10-07 9:50:02): [nss] [sss_dp_get_reply] (0x0010): The Data Provider returned an error [org.freedesktop.DBus.Error.Failed]
(2021-10-07 9:50:02): [nss] [sss_dp_get_reply] (0x0010): The Data Provider returned an error [org.freedesktop.DBus.Error.Failed]
(2021-10-07 9:50:02): [nss] [sss_dp_get_reply] (0x0010): The Data Provider returned an error [org.freedesktop.DBus.Error.Failed]
(2021-10-07 9:50:02): [nss] [sss_dp_get_reply] (0x0010): The Data Provider returned an error [org.freedesktop.DBus.Error.Failed]
(2021-10-07 9:50:02): [nss] [sss_dp_get_reply] (0x0010): The Data Provider returned an error [org.freedesktop.DBus.Error.Failed]
(2021-10-07 9:50:02): [nss] [sss_dp_get_reply] (0x0010): The Data Provider returned an error [org.freedesktop.DBus.Error.Failed]
(2021-10-07 9:50:02): [nss] [sss_dp_get_reply] (0x0010): The Data Provider returned an error [org.freedesktop.DBus.Error.Failed]
(2021-10-07 9:50:02): [nss] [sss_dp_get_reply] (0x0010): The Data Provider returned an error [org.freedesktop.DBus.Error.Failed]
(2021-10-07 9:50:02): [nss] [sss_dp_get_reply] (0x0010): The Data Provider returned an error [org.freedesktop.DBus.Error.Failed]
(2021-10-07 9:50:02): [nss] [sss_dp_get_reply] (0x0010): The Data Provider returned an error [org.freedesktop.DBus.Error.Failed]
(2021-10-07 9:50:02): [nss] [sss_dp_get_reply] (0x0010): The Data Provider returned an error [org.freedesktop.DBus.Error.Failed]
(2021-10-07 9:50:02): [nss] [sss_dp_get_reply] (0x0010): The Data Provider returned an error [org.freedesktop.DBus.Error.Failed]
From debug level 9, it is clear that this is arising from a query of these erroneously-discovered untrusted domains. Here’s an example of one instance of above with debug level 9 turned on. So emeaicmd.geodll.company.com is one of these erroneously-discovered untrusted lab domains, that happens to be firewalled off from this particular AD client:
(2021-10-07 9:50:02): [nss] [sss_dp_get_reply] (0x1000): Got reply from Data Provider - DP error code: 0 errno: 0 error message: Success
(2021-10-07 9:50:02): [nss] [cache_req_search_cache] (0x0400): CR #9: Looking up [oracle@company.com] in cache
(2021-10-07 9:50:02): [nss] [cache_req_search_cache] (0x0400): CR #9: Object [oracle@company.com] was not found in cache
(2021-10-07 9:50:02): [nss] [cache_req_search_ncache_add_to_domain] (0x0400): CR #9: Adding [oracle@company.com] to negative cache
(2021-10-07 9:50:02): [nss] [is_user_local_by_name] (0x0400): User oracle@company.com is a local user
(2021-10-07 9:50:02): [nss] [sss_ncache_set_str] (0x0400): Adding [NCE/USER/company.com/oracle@company.com] to negative cache
(2021-10-07 9:50:02): [nss] [cache_req_validate_domain_type] (0x2000): Request type POSIX-only for domain EMEAICMD.geodll.company.com type POSIX is valid
(2021-10-07 9:50:02): [nss] [cache_req_set_domain] (0x0400): CR #9: Using domain [EMEAICMD.geodll.company.com]
(2021-10-07 9:50:02): [nss] [cache_req_prepare_domain_data] (0x0400): CR #9: Preparing input data for domain [EMEAICMD.geodll.company.com] rules
(2021-10-07 9:50:02): [nss] [cache_req_search_send] (0x0400): CR #9: Looking up oracle@emeaicmd.geodll.company.com
(2021-10-07 9:50:02): [nss] [cache_req_search_ncache] (0x0400): CR #9: Checking negative cache for [oracle@emeaicmd.geodll.company.com]
(2021-10-07 9:50:02): [nss] [sss_ncache_check_str] (0x2000): Checking negative cache for [NCE/USER/ EMEAICMD.geodll.company.com/oracle@emeaicmd.geodll.company.com]
(2021-10-07 9:50:02): [nss] [cache_req_search_ncache] (0x0400): CR #9: [ oracle@emeaicmd.geodll.company.com] is not present in negative cache
(2021-10-07 9:50:02): [nss] [cache_req_search_cache] (0x0400): CR #9: Looking up [oracle@emeaicmd.geodll.company.com] in cache
(2021-10-07 9:50:02): [nss] [cache_req_search_cache] (0x0400): CR #9: Object [oracle@emeaicmd.geodll.company.com] was not found in cache
(2021-10-07 9:50:02): [nss] [cache_req_search_dp] (0x0400): CR #9: Looking up [oracle@emeaicmd.geodll.company.com] in data provider
(2021-10-07 9:50:02): [nss] [sss_dp_issue_request] (0x0400): Issuing request for [0x564d6be36a70:3:oracle@emeaicmd.geodll.company.com@ EMEAICMD.geodll.company.com]
(2021-10-07 9:50:02): [nss] [sss_dp_get_account_msg] (0x0400): Creating request for [EMEAICMD.geodll.company.com ][0x3][BE_REQ_INITGROUPS][name=oracle@emeaicmd.geodll.company.com:-]
(2021-10-07 9:50:02): [nss] [sbus_add_timeout] (0x2000): 0x564d6ccd6670
(2021-10-07 9:50:02): [nss] [sss_dp_internal_get_send] (0x0400): Entering request [0x564d6be36a70:3:oracle@emeaicmd.geodll.company.com@ EMEAICMD.geodll.company.com]
(2021-10-07 9:50:02): [nss] [cache_req_search_cache] (0x0400): CR #12: Looking up [oracle@company.com] in cache
(2021-10-07 9:50:02): [nss] [cache_req_search_cache] (0x0400): CR #12: Object [oracle@company.com] was not found in cache
(2021-10-07 9:50:02): [nss] [cache_req_search_ncache_add_to_domain] (0x0400): CR #12: Adding [oracle@company.com] to negative cache
(2021-10-07 9:50:02): [nss] [is_user_local_by_name] (0x0400): User oracle@company.com is a local user
(2021-10-07 9:50:02): [nss] [sss_ncache_set_str] (0x0400): Adding [NCE/USER/company.com/oracle@company.com] to negative cache
(2021-10-07 9:50:02): [nss] [cache_req_validate_domain_type] (0x2000): Request type POSIX-only for domain EMEAICMD.geodll.company.com type POSIX is valid
(2021-10-07 9:50:02): [nss] [cache_req_set_domain] (0x0400): CR #12: Using domain [EMEAICMD.geodll.company.com]
(2021-10-07 9:50:02): [nss] [cache_req_prepare_domain_data] (0x0400): CR #12: Preparing input data for domain [EMEAICMD.geodll.company.com] rules
(2021-10-07 9:50:02): [nss] [cache_req_search_send] (0x0400): CR #12: Looking up oracle@emeaicmd.geodll.company.com
(2021-10-07 9:50:02): [nss] [cache_req_search_ncache] (0x0400): CR #12: Checking negative cache for [oracle@emeaicmd.geodll.company.com]
(2021-10-07 9:50:02): [nss] [sss_ncache_check_str] (0x2000): Checking negative cache for [NCE/USER/ EMEAICMD.geodll.company.com/oracle@emeaicmd.geodll.company.com]
(2021-10-07 9:50:02): [nss] [cache_req_search_ncache] (0x0400): CR #12: [ oracle@emeaicmd.geodll.company.com] is not present in negative cache
(2021-10-07 9:50:02): [nss] [cache_req_search_cache] (0x0400): CR #12: Looking up [oracle@emeaicmd.geodll.company.com] in cache
(2021-10-07 9:50:02): [nss] [cache_req_search_cache] (0x0400): CR #12: Object [oracle@emeaicmd.geodll.company.com] was not found in cache
(2021-10-07 9:50:02): [nss] [cache_req_search_dp] (0x0400): CR #12: Looking up [oracle@emeaicmd.geodll.company.com] in data provider
(2021-10-07 9:50:02): [nss] [sss_dp_issue_request] (0x0400): Issuing request for [0x564d6be36a70:3:oracle@emeaicmd.geodll.company.com@ EMEAICMD.geodll.company.com]
(2021-10-07 9:50:02): [nss] [sss_dp_issue_request] (0x0400): Identical request in progress: [0x564d6be36a70:3:oracle@emeaicmd.geodll.company.com@ EMEAICMD.geodll.company.com]
(2021-10-07 9:50:02): [nss] [sss_dp_req_destructor] (0x0400): Deleting request: [0x564d6be36a70:3:oracle@company.com@company.com]
(2021-10-07 9:50:02): [nss] [sbus_remove_timeout] (0x2000): 0x564d6ccd6670
(2021-10-07 9:50:02): [nss] [sbus_dispatch] (0x4000): dbus conn: 0x564d6ccc9300
(2021-10-07 9:50:02): [nss] [sbus_dispatch] (0x4000): Dispatching.
(2021-10-07 9:50:02): [nss] [sss_dp_get_reply] (0x0010): The Data Provider returned an error [org.freedesktop.DBus.Error.Failed]
The suggested work-around does not resolve problem #2.
BTW, here is a listing of the domains discovered on that sssd client:
[root@auspdfdlobv01 ~]# sssctl domain-list
amer.company.com
company.com
japn.company.com
emea.company.com
apac.company.com
EMEAICMD.geodll.company.com
geodll.company.com
EMEAICM.GEODLL.COMPANY.COM
alienware.com
corp.svcs
perotsystems.net
companyservices.dmz
Beer.Town
production.online.company.com
jp-poclab.companypoc.com
emea-poclab.companypoc.com
oldev.preol.company.com
olqa.preol.company.com
ap-poclab.companypoc.com
[root@auspdfdlobv01 ~]#
This sssd client is joined to amer.company.com, so the only trusted domains are the first 5. The parent domain and the 4 regional domains. All those other domains below that are untrusted domains. More specifically, they trust company.com, but company.com does not trust them. (one way trust – the wrong way.) Some look like the real wild wild west (Beer.Town ?).
Spike
sssd-users mailing list -- sssd-users@lists.fedorahosted.org To unsubscribe send an email to sssd-users-leave@lists.fedorahosted.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedorahosted.org/archives/list/sssd-users@lists.fedorahosted.o... Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Sumit,
It would probably be faster for you to do a test build. I'd have to fumble through pulling the source RPM, rpmbuild invocation, rpm install. You probably know those commands at your fingertips.
We have ~20K servers with RHEL7, RHEL8, OL7 (RHCK) and OL8 (RHCK) exhibiting this behavior. All x86_64. We have test servers of each of those flavors on which we can test. Your call.
We have beaucoup RHEL8/OL8 test boxes, so if that's convenient for you, it'll work for us.
It is super-easy for us to determine.if it's fixed or not. Previously 'sssctl domain-list' only showed the 5 trusted domains. Now with this new sssd version (~July), 'sssctl domain-list' shows the expected 5 trusted domains and the 14 untrusted domains.
Spike
On Fri, Oct 8, 2021 at 1:01 AM Sumit Bose sbose@redhat.com wrote:
Am Thu, Oct 07, 2021 at 11:38:54AM -0500 schrieb Spike White:
All (but particularly Sumit since he wrote the comments on https://bugzilla.redhat.com/show_bug.cgi?id=1984591),
Hi,
jfyi, I'm currently working on a fix for this to filter out domains from other forests and untrusted domains. My WIP branch is at https://github.com/sumit-bose/sssd/tree/ad_filter_domains. Can you do a test build of SSSD based on this or shall I try to create a test build for you? For the latter, please tell me for which platform.
bye, Sumit
There are at least two problems created by this recently-introduced sssd bug. One problem is solvable by the suggested work-around, the other is not. The work-around suggested is:
[domain/name.of.joined.domain]
ad_enabled_domains = dom1.example.com, dom2.example.com,dom3.example.com
In order to query only the desired AD domains.
What is the bug?
the sssd-ad man page says "The AD provider can be used to get user information and authenticate users from trusted domains. Currently only trusted domains in the same forest are recognized.".
What is happening is that untrusted AD domains are being discovered. A very specific type of untrusted domains. When the joined domain has no trust with that other domain, but that other domain trusts the original domain – that is a one-way trust (the wrong way). To the joined domain, this is an untrusted domain and should not be discovered.
This is actually very common in corporate environments.
You may have a main AD domain, call it CORP.COMPANY.COM. Then for
testing
and new production evaluation, you might have a test AD domain called LAB-TEST.COMPANY.COM. CORP.COMPANY.COM is tightly controlled, with full audits and corporate security. LAB-TEST.COMPANY.COM is a test AD
domain –
it’s the wild, wild west!
So LAB-TEST.COMPANY.COM trusts the main AD domain (in order that users
can
log into this test domain with their CORP accounts). But
CORP.COMPANY.COM
does not trust LAB-TEST.COMPANY.COM – nor should it!! (That’s the wild, wild west, doing so would compromise corporate security.)
Thus, a server joined to domain CORP.COMPANY.COM should discover CORP.COMPANY.COM and any domains trusted by CORP.COMPANY.COM. It
should
*NOT* discover LAB-TEST.COMPANY.COM, as CORP.COMPANY.COM does not trust this domain.
A server joined to LAB-TEST.COMPANY.COM should discover
LAB-TEST.COMPANY.COM
and all domains trusted by LAB-TEST.COMPANY.COM. Including
CORP.COMPANY.COM,
as LAB-TEST.COMPANY.COM trusts CORP.COMPANY.COM.
The bug is that a server joined to CORP.COMPANY.COM discovers LAB-TEST.COMPANY.COM, which it shouldn’t.
What problems does this cause?
Two problems.
Many of these untrusted discovered “lab” domains are accessibleonly to specific network locations. That is, they’re firewalled off to a particular lab. So sssd attempts to query these inaccessible AD domains and takes a long time to time out. This problem can be worked around by the suggested work-around in the Bugzilla:
[domain/corp.company.com]
ad_enabled_domains = corp.company.comSo then, while LAB-TEST.COMPANY.COM is still erroneously discovered,
it is
no longer searched. Sssd is again fast.
Bogus messages in /var/log/sssd_nss.log file. Even with nodebug
level set in the [nss] stanza, these error messages appear multiple
times a
second. It quickly fills up the /var/log filesystem.
[root@auspdfdlobv01 sssd]# cat sssd_nss.log |grep "The Data Provider returned an error"
(2021-10-07 9:50:02): [nss] [sss_dp_get_reply] (0x0010): The Data
Provider
returned an error [org.freedesktop.DBus.Error.Failed]
(2021-10-07 9:50:02): [nss] [sss_dp_get_reply] (0x0010): The Data
Provider
returned an error [org.freedesktop.DBus.Error.Failed]
(2021-10-07 9:50:02): [nss] [sss_dp_get_reply] (0x0010): The Data
Provider
returned an error [org.freedesktop.DBus.Error.Failed]
(2021-10-07 9:50:02): [nss] [sss_dp_get_reply] (0x0010): The Data
Provider
returned an error [org.freedesktop.DBus.Error.Failed]
(2021-10-07 9:50:02): [nss] [sss_dp_get_reply] (0x0010): The Data
Provider
returned an error [org.freedesktop.DBus.Error.Failed]
(2021-10-07 9:50:02): [nss] [sss_dp_get_reply] (0x0010): The Data
Provider
returned an error [org.freedesktop.DBus.Error.Failed]
(2021-10-07 9:50:02): [nss] [sss_dp_get_reply] (0x0010): The Data
Provider
returned an error [org.freedesktop.DBus.Error.Failed]
(2021-10-07 9:50:02): [nss] [sss_dp_get_reply] (0x0010): The Data
Provider
returned an error [org.freedesktop.DBus.Error.Failed]
(2021-10-07 9:50:02): [nss] [sss_dp_get_reply] (0x0010): The Data
Provider
returned an error [org.freedesktop.DBus.Error.Failed]
(2021-10-07 9:50:02): [nss] [sss_dp_get_reply] (0x0010): The Data
Provider
returned an error [org.freedesktop.DBus.Error.Failed]
(2021-10-07 9:50:02): [nss] [sss_dp_get_reply] (0x0010): The Data
Provider
returned an error [org.freedesktop.DBus.Error.Failed]
(2021-10-07 9:50:02): [nss] [sss_dp_get_reply] (0x0010): The Data
Provider
returned an error [org.freedesktop.DBus.Error.Failed]
(2021-10-07 9:50:02): [nss] [sss_dp_get_reply] (0x0010): The Data
Provider
returned an error [org.freedesktop.DBus.Error.Failed]
(2021-10-07 9:50:02): [nss] [sss_dp_get_reply] (0x0010): The Data
Provider
returned an error [org.freedesktop.DBus.Error.Failed]
(2021-10-07 9:50:02): [nss] [sss_dp_get_reply] (0x0010): The Data
Provider
returned an error [org.freedesktop.DBus.Error.Failed]
(2021-10-07 9:50:02): [nss] [sss_dp_get_reply] (0x0010): The Data
Provider
returned an error [org.freedesktop.DBus.Error.Failed]
(2021-10-07 9:50:02): [nss] [sss_dp_get_reply] (0x0010): The Data
Provider
returned an error [org.freedesktop.DBus.Error.Failed]
(2021-10-07 9:50:02): [nss] [sss_dp_get_reply] (0x0010): The Data
Provider
returned an error [org.freedesktop.DBus.Error.Failed]
(2021-10-07 9:50:02): [nss] [sss_dp_get_reply] (0x0010): The Data
Provider
returned an error [org.freedesktop.DBus.Error.Failed]
(2021-10-07 9:50:02): [nss] [sss_dp_get_reply] (0x0010): The Data
Provider
returned an error [org.freedesktop.DBus.Error.Failed]
(2021-10-07 9:50:02): [nss] [sss_dp_get_reply] (0x0010): The Data
Provider
returned an error [org.freedesktop.DBus.Error.Failed]
(2021-10-07 9:50:02): [nss] [sss_dp_get_reply] (0x0010): The Data
Provider
returned an error [org.freedesktop.DBus.Error.Failed]
(2021-10-07 9:50:02): [nss] [sss_dp_get_reply] (0x0010): The Data
Provider
returned an error [org.freedesktop.DBus.Error.Failed]
(2021-10-07 9:50:02): [nss] [sss_dp_get_reply] (0x0010): The Data
Provider
returned an error [org.freedesktop.DBus.Error.Failed]
(2021-10-07 9:50:02): [nss] [sss_dp_get_reply] (0x0010): The Data
Provider
returned an error [org.freedesktop.DBus.Error.Failed]
From debug level 9, it is clear that this is arising from a query of
these
erroneously-discovered untrusted domains. Here’s an example of one instance of above with debug level 9 turned on. So emeaicmd.geodll.company.com is one of these erroneously-discovered untrusted lab domains, that happens to be firewalled off from this particular AD client:
(2021-10-07 9:50:02): [nss] [sss_dp_get_reply] (0x1000): Got reply from Data Provider - DP error code: 0 errno: 0 error message: Success
(2021-10-07 9:50:02): [nss] [cache_req_search_cache] (0x0400): CR #9: Looking up [oracle@company.com] in cache
(2021-10-07 9:50:02): [nss] [cache_req_search_cache] (0x0400): CR #9: Object [oracle@company.com] was not found in cache
(2021-10-07 9:50:02): [nss] [cache_req_search_ncache_add_to_domain] (0x0400): CR #9: Adding [oracle@company.com] to negative cache
(2021-10-07 9:50:02): [nss] [is_user_local_by_name] (0x0400): User oracle@company.com is a local user
(2021-10-07 9:50:02): [nss] [sss_ncache_set_str] (0x0400): Adding [NCE/USER/company.com/oracle@company.com] to negative cache
(2021-10-07 9:50:02): [nss] [cache_req_validate_domain_type] (0x2000): Request type POSIX-only for domain EMEAICMD.geodll.company.com type
POSIX
is valid
(2021-10-07 9:50:02): [nss] [cache_req_set_domain] (0x0400): CR #9:
Using
domain [EMEAICMD.geodll.company.com]
(2021-10-07 9:50:02): [nss] [cache_req_prepare_domain_data] (0x0400): CR #9: Preparing input data for domain [EMEAICMD.geodll.company.com] rules
(2021-10-07 9:50:02): [nss] [cache_req_search_send] (0x0400): CR #9: Looking up oracle@emeaicmd.geodll.company.com
(2021-10-07 9:50:02): [nss] [cache_req_search_ncache] (0x0400): CR #9: Checking negative cache for [oracle@emeaicmd.geodll.company.com]
(2021-10-07 9:50:02): [nss] [sss_ncache_check_str] (0x2000): Checking negative cache for [NCE/USER/ EMEAICMD.geodll.company.com/oracle@emeaicmd.geodll.company.com]
(2021-10-07 9:50:02): [nss] [cache_req_search_ncache] (0x0400): CR #9: [ oracle@emeaicmd.geodll.company.com] is not present in negative cache
(2021-10-07 9:50:02): [nss] [cache_req_search_cache] (0x0400): CR #9: Looking up [oracle@emeaicmd.geodll.company.com] in cache
(2021-10-07 9:50:02): [nss] [cache_req_search_cache] (0x0400): CR #9: Object [oracle@emeaicmd.geodll.company.com] was not found in cache
(2021-10-07 9:50:02): [nss] [cache_req_search_dp] (0x0400): CR #9:
Looking
up [oracle@emeaicmd.geodll.company.com] in data provider
(2021-10-07 9:50:02): [nss] [sss_dp_issue_request] (0x0400): Issuing request for [0x564d6be36a70:3:oracle@emeaicmd.geodll.company.com@ EMEAICMD.geodll.company.com]
(2021-10-07 9:50:02): [nss] [sss_dp_get_account_msg] (0x0400): Creating request for [EMEAICMD.geodll.company.com ][0x3][BE_REQ_INITGROUPS][name=oracle@emeaicmd.geodll.company.com:-]
(2021-10-07 9:50:02): [nss] [sbus_add_timeout] (0x2000): 0x564d6ccd6670
(2021-10-07 9:50:02): [nss] [sss_dp_internal_get_send] (0x0400):
Entering
request [0x564d6be36a70:3:oracle@emeaicmd.geodll.company.com@ EMEAICMD.geodll.company.com]
(2021-10-07 9:50:02): [nss] [cache_req_search_cache] (0x0400): CR #12: Looking up [oracle@company.com] in cache
(2021-10-07 9:50:02): [nss] [cache_req_search_cache] (0x0400): CR #12: Object [oracle@company.com] was not found in cache
(2021-10-07 9:50:02): [nss] [cache_req_search_ncache_add_to_domain] (0x0400): CR #12: Adding [oracle@company.com] to negative cache
(2021-10-07 9:50:02): [nss] [is_user_local_by_name] (0x0400): User oracle@company.com is a local user
(2021-10-07 9:50:02): [nss] [sss_ncache_set_str] (0x0400): Adding [NCE/USER/company.com/oracle@company.com] to negative cache
(2021-10-07 9:50:02): [nss] [cache_req_validate_domain_type] (0x2000): Request type POSIX-only for domain EMEAICMD.geodll.company.com type
POSIX
is valid
(2021-10-07 9:50:02): [nss] [cache_req_set_domain] (0x0400): CR #12:
Using
domain [EMEAICMD.geodll.company.com]
(2021-10-07 9:50:02): [nss] [cache_req_prepare_domain_data] (0x0400): CR #12: Preparing input data for domain [EMEAICMD.geodll.company.com] rules
(2021-10-07 9:50:02): [nss] [cache_req_search_send] (0x0400): CR #12: Looking up oracle@emeaicmd.geodll.company.com
(2021-10-07 9:50:02): [nss] [cache_req_search_ncache] (0x0400): CR #12: Checking negative cache for [oracle@emeaicmd.geodll.company.com]
(2021-10-07 9:50:02): [nss] [sss_ncache_check_str] (0x2000): Checking negative cache for [NCE/USER/ EMEAICMD.geodll.company.com/oracle@emeaicmd.geodll.company.com]
(2021-10-07 9:50:02): [nss] [cache_req_search_ncache] (0x0400): CR #12:
[
oracle@emeaicmd.geodll.company.com] is not present in negative cache
(2021-10-07 9:50:02): [nss] [cache_req_search_cache] (0x0400): CR #12: Looking up [oracle@emeaicmd.geodll.company.com] in cache
(2021-10-07 9:50:02): [nss] [cache_req_search_cache] (0x0400): CR #12: Object [oracle@emeaicmd.geodll.company.com] was not found in cache
(2021-10-07 9:50:02): [nss] [cache_req_search_dp] (0x0400): CR #12: Looking up [oracle@emeaicmd.geodll.company.com] in data provider
(2021-10-07 9:50:02): [nss] [sss_dp_issue_request] (0x0400): Issuing request for [0x564d6be36a70:3:oracle@emeaicmd.geodll.company.com@ EMEAICMD.geodll.company.com]
(2021-10-07 9:50:02): [nss] [sss_dp_issue_request] (0x0400): Identical request in progress: [0x564d6be36a70:3:oracle@emeaicmd.geodll.company.com
@
EMEAICMD.geodll.company.com]
(2021-10-07 9:50:02): [nss] [sss_dp_req_destructor] (0x0400): Deleting request: [0x564d6be36a70:3:oracle@company.com@company.com]
(2021-10-07 9:50:02): [nss] [sbus_remove_timeout] (0x2000):
0x564d6ccd6670
(2021-10-07 9:50:02): [nss] [sbus_dispatch] (0x4000): dbus conn: 0x564d6ccc9300
(2021-10-07 9:50:02): [nss] [sbus_dispatch] (0x4000): Dispatching.
(2021-10-07 9:50:02): [nss] [sss_dp_get_reply] (0x0010): The Data
Provider
returned an error [org.freedesktop.DBus.Error.Failed]
The suggested work-around does not resolve problem #2.
BTW, here is a listing of the domains discovered on that sssd client:
[root@auspdfdlobv01 ~]# sssctl domain-list
amer.company.com
company.com
japn.company.com
emea.company.com
apac.company.com
EMEAICMD.geodll.company.com
geodll.company.com
EMEAICM.GEODLL.COMPANY.COM
alienware.com
corp.svcs
perotsystems.net
companyservices.dmz
Beer.Town
production.online.company.com
jp-poclab.companypoc.com
emea-poclab.companypoc.com
oldev.preol.company.com
olqa.preol.company.com
ap-poclab.companypoc.com
[root@auspdfdlobv01 ~]#
This sssd client is joined to amer.company.com, so the only trusted
domains
are the first 5. The parent domain and the 4 regional domains. All those other domains below that are untrusted domains. More specifically, they trust company.com, but company.com does not trust them. (one way trust – the wrong way.) Some look like the real wild wild west
(Beer.Town
?).
Spike
sssd-users mailing list -- sssd-users@lists.fedorahosted.org To unsubscribe send an email to sssd-users-leave@lists.fedorahosted.org Fedora Code of Conduct:
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives:
https://lists.fedorahosted.org/archives/list/sssd-users@lists.fedorahosted.o...
Do not reply to spam on the list, report it:
https://pagure.io/fedora-infrastructure _______________________________________________ sssd-users mailing list -- sssd-users@lists.fedorahosted.org To unsubscribe send an email to sssd-users-leave@lists.fedorahosted.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedorahosted.org/archives/list/sssd-users@lists.fedorahosted.o... Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Am Fri, Oct 08, 2021 at 08:51:55AM -0500 schrieb Spike White:
Sumit,
It would probably be faster for you to do a test build. I'd have to fumble through pulling the source RPM, rpmbuild invocation, rpm install. You probably know those commands at your fingertips.
We have ~20K servers with RHEL7, RHEL8, OL7 (RHCK) and OL8 (RHCK) exhibiting this behavior. All x86_64. We have test servers of each of those flavors on which we can test. Your call.
We have beaucoup RHEL8/OL8 test boxes, so if that's convenient for you, it'll work for us.
Hi,
please have a look at https://sbose.fedorapeople.org/sssd-2.4.0-9.el8_4.2sb1.tar.gz, a test build for RHEL-8.4. You do not have to install all packages, calling 'yum update *' in the new directory should just update the installed packages.
bye, Sumit
It is super-easy for us to determine.if it's fixed or not. Previously 'sssctl domain-list' only showed the 5 trusted domains. Now with this new sssd version (~July), 'sssctl domain-list' shows the expected 5 trusted domains and the 14 untrusted domains.
Spike
On Fri, Oct 8, 2021 at 1:01 AM Sumit Bose sbose@redhat.com wrote:
Am Thu, Oct 07, 2021 at 11:38:54AM -0500 schrieb Spike White:
All (but particularly Sumit since he wrote the comments on https://bugzilla.redhat.com/show_bug.cgi?id=1984591),
Hi,
jfyi, I'm currently working on a fix for this to filter out domains from other forests and untrusted domains. My WIP branch is at https://github.com/sumit-bose/sssd/tree/ad_filter_domains. Can you do a test build of SSSD based on this or shall I try to create a test build for you? For the latter, please tell me for which platform.
bye, Sumit
There are at least two problems created by this recently-introduced sssd bug. One problem is solvable by the suggested work-around, the other is not. The work-around suggested is:
[domain/name.of.joined.domain]
ad_enabled_domains = dom1.example.com, dom2.example.com,dom3.example.com
In order to query only the desired AD domains.
What is the bug?
the sssd-ad man page says "The AD provider can be used to get user information and authenticate users from trusted domains. Currently only trusted domains in the same forest are recognized.".
What is happening is that untrusted AD domains are being discovered. A very specific type of untrusted domains. When the joined domain has no trust with that other domain, but that other domain trusts the original domain – that is a one-way trust (the wrong way). To the joined domain, this is an untrusted domain and should not be discovered.
This is actually very common in corporate environments.
You may have a main AD domain, call it CORP.COMPANY.COM. Then for
testing
and new production evaluation, you might have a test AD domain called LAB-TEST.COMPANY.COM. CORP.COMPANY.COM is tightly controlled, with full audits and corporate security. LAB-TEST.COMPANY.COM is a test AD
domain –
it’s the wild, wild west!
So LAB-TEST.COMPANY.COM trusts the main AD domain (in order that users
can
log into this test domain with their CORP accounts). But
CORP.COMPANY.COM
does not trust LAB-TEST.COMPANY.COM – nor should it!! (That’s the wild, wild west, doing so would compromise corporate security.)
Thus, a server joined to domain CORP.COMPANY.COM should discover CORP.COMPANY.COM and any domains trusted by CORP.COMPANY.COM. It
should
*NOT* discover LAB-TEST.COMPANY.COM, as CORP.COMPANY.COM does not trust this domain.
A server joined to LAB-TEST.COMPANY.COM should discover
LAB-TEST.COMPANY.COM
and all domains trusted by LAB-TEST.COMPANY.COM. Including
CORP.COMPANY.COM,
as LAB-TEST.COMPANY.COM trusts CORP.COMPANY.COM.
The bug is that a server joined to CORP.COMPANY.COM discovers LAB-TEST.COMPANY.COM, which it shouldn’t.
What problems does this cause?
Two problems.
Many of these untrusted discovered “lab” domains are accessibleonly to specific network locations. That is, they’re firewalled off to a particular lab. So sssd attempts to query these inaccessible AD domains and takes a long time to time out. This problem can be worked around by the suggested work-around in the Bugzilla:
[domain/corp.company.com]
ad_enabled_domains = corp.company.comSo then, while LAB-TEST.COMPANY.COM is still erroneously discovered,
it is
no longer searched. Sssd is again fast.
Bogus messages in /var/log/sssd_nss.log file. Even with nodebug
level set in the [nss] stanza, these error messages appear multiple
times a
second. It quickly fills up the /var/log filesystem.
[root@auspdfdlobv01 sssd]# cat sssd_nss.log |grep "The Data Provider returned an error"
(2021-10-07 9:50:02): [nss] [sss_dp_get_reply] (0x0010): The Data
Provider
returned an error [org.freedesktop.DBus.Error.Failed]
(2021-10-07 9:50:02): [nss] [sss_dp_get_reply] (0x0010): The Data
Provider
returned an error [org.freedesktop.DBus.Error.Failed]
(2021-10-07 9:50:02): [nss] [sss_dp_get_reply] (0x0010): The Data
Provider
returned an error [org.freedesktop.DBus.Error.Failed]
(2021-10-07 9:50:02): [nss] [sss_dp_get_reply] (0x0010): The Data
Provider
returned an error [org.freedesktop.DBus.Error.Failed]
(2021-10-07 9:50:02): [nss] [sss_dp_get_reply] (0x0010): The Data
Provider
returned an error [org.freedesktop.DBus.Error.Failed]
(2021-10-07 9:50:02): [nss] [sss_dp_get_reply] (0x0010): The Data
Provider
returned an error [org.freedesktop.DBus.Error.Failed]
(2021-10-07 9:50:02): [nss] [sss_dp_get_reply] (0x0010): The Data
Provider
returned an error [org.freedesktop.DBus.Error.Failed]
(2021-10-07 9:50:02): [nss] [sss_dp_get_reply] (0x0010): The Data
Provider
returned an error [org.freedesktop.DBus.Error.Failed]
(2021-10-07 9:50:02): [nss] [sss_dp_get_reply] (0x0010): The Data
Provider
returned an error [org.freedesktop.DBus.Error.Failed]
(2021-10-07 9:50:02): [nss] [sss_dp_get_reply] (0x0010): The Data
Provider
returned an error [org.freedesktop.DBus.Error.Failed]
(2021-10-07 9:50:02): [nss] [sss_dp_get_reply] (0x0010): The Data
Provider
returned an error [org.freedesktop.DBus.Error.Failed]
(2021-10-07 9:50:02): [nss] [sss_dp_get_reply] (0x0010): The Data
Provider
returned an error [org.freedesktop.DBus.Error.Failed]
(2021-10-07 9:50:02): [nss] [sss_dp_get_reply] (0x0010): The Data
Provider
returned an error [org.freedesktop.DBus.Error.Failed]
(2021-10-07 9:50:02): [nss] [sss_dp_get_reply] (0x0010): The Data
Provider
returned an error [org.freedesktop.DBus.Error.Failed]
(2021-10-07 9:50:02): [nss] [sss_dp_get_reply] (0x0010): The Data
Provider
returned an error [org.freedesktop.DBus.Error.Failed]
(2021-10-07 9:50:02): [nss] [sss_dp_get_reply] (0x0010): The Data
Provider
returned an error [org.freedesktop.DBus.Error.Failed]
(2021-10-07 9:50:02): [nss] [sss_dp_get_reply] (0x0010): The Data
Provider
returned an error [org.freedesktop.DBus.Error.Failed]
(2021-10-07 9:50:02): [nss] [sss_dp_get_reply] (0x0010): The Data
Provider
returned an error [org.freedesktop.DBus.Error.Failed]
(2021-10-07 9:50:02): [nss] [sss_dp_get_reply] (0x0010): The Data
Provider
returned an error [org.freedesktop.DBus.Error.Failed]
(2021-10-07 9:50:02): [nss] [sss_dp_get_reply] (0x0010): The Data
Provider
returned an error [org.freedesktop.DBus.Error.Failed]
(2021-10-07 9:50:02): [nss] [sss_dp_get_reply] (0x0010): The Data
Provider
returned an error [org.freedesktop.DBus.Error.Failed]
(2021-10-07 9:50:02): [nss] [sss_dp_get_reply] (0x0010): The Data
Provider
returned an error [org.freedesktop.DBus.Error.Failed]
(2021-10-07 9:50:02): [nss] [sss_dp_get_reply] (0x0010): The Data
Provider
returned an error [org.freedesktop.DBus.Error.Failed]
(2021-10-07 9:50:02): [nss] [sss_dp_get_reply] (0x0010): The Data
Provider
returned an error [org.freedesktop.DBus.Error.Failed]
(2021-10-07 9:50:02): [nss] [sss_dp_get_reply] (0x0010): The Data
Provider
returned an error [org.freedesktop.DBus.Error.Failed]
From debug level 9, it is clear that this is arising from a query of
these
erroneously-discovered untrusted domains. Here’s an example of one instance of above with debug level 9 turned on. So emeaicmd.geodll.company.com is one of these erroneously-discovered untrusted lab domains, that happens to be firewalled off from this particular AD client:
(2021-10-07 9:50:02): [nss] [sss_dp_get_reply] (0x1000): Got reply from Data Provider - DP error code: 0 errno: 0 error message: Success
(2021-10-07 9:50:02): [nss] [cache_req_search_cache] (0x0400): CR #9: Looking up [oracle@company.com] in cache
(2021-10-07 9:50:02): [nss] [cache_req_search_cache] (0x0400): CR #9: Object [oracle@company.com] was not found in cache
(2021-10-07 9:50:02): [nss] [cache_req_search_ncache_add_to_domain] (0x0400): CR #9: Adding [oracle@company.com] to negative cache
(2021-10-07 9:50:02): [nss] [is_user_local_by_name] (0x0400): User oracle@company.com is a local user
(2021-10-07 9:50:02): [nss] [sss_ncache_set_str] (0x0400): Adding [NCE/USER/company.com/oracle@company.com] to negative cache
(2021-10-07 9:50:02): [nss] [cache_req_validate_domain_type] (0x2000): Request type POSIX-only for domain EMEAICMD.geodll.company.com type
POSIX
is valid
(2021-10-07 9:50:02): [nss] [cache_req_set_domain] (0x0400): CR #9:
Using
domain [EMEAICMD.geodll.company.com]
(2021-10-07 9:50:02): [nss] [cache_req_prepare_domain_data] (0x0400): CR #9: Preparing input data for domain [EMEAICMD.geodll.company.com] rules
(2021-10-07 9:50:02): [nss] [cache_req_search_send] (0x0400): CR #9: Looking up oracle@emeaicmd.geodll.company.com
(2021-10-07 9:50:02): [nss] [cache_req_search_ncache] (0x0400): CR #9: Checking negative cache for [oracle@emeaicmd.geodll.company.com]
(2021-10-07 9:50:02): [nss] [sss_ncache_check_str] (0x2000): Checking negative cache for [NCE/USER/ EMEAICMD.geodll.company.com/oracle@emeaicmd.geodll.company.com]
(2021-10-07 9:50:02): [nss] [cache_req_search_ncache] (0x0400): CR #9: [ oracle@emeaicmd.geodll.company.com] is not present in negative cache
(2021-10-07 9:50:02): [nss] [cache_req_search_cache] (0x0400): CR #9: Looking up [oracle@emeaicmd.geodll.company.com] in cache
(2021-10-07 9:50:02): [nss] [cache_req_search_cache] (0x0400): CR #9: Object [oracle@emeaicmd.geodll.company.com] was not found in cache
(2021-10-07 9:50:02): [nss] [cache_req_search_dp] (0x0400): CR #9:
Looking
up [oracle@emeaicmd.geodll.company.com] in data provider
(2021-10-07 9:50:02): [nss] [sss_dp_issue_request] (0x0400): Issuing request for [0x564d6be36a70:3:oracle@emeaicmd.geodll.company.com@ EMEAICMD.geodll.company.com]
(2021-10-07 9:50:02): [nss] [sss_dp_get_account_msg] (0x0400): Creating request for [EMEAICMD.geodll.company.com ][0x3][BE_REQ_INITGROUPS][name=oracle@emeaicmd.geodll.company.com:-]
(2021-10-07 9:50:02): [nss] [sbus_add_timeout] (0x2000): 0x564d6ccd6670
(2021-10-07 9:50:02): [nss] [sss_dp_internal_get_send] (0x0400):
Entering
request [0x564d6be36a70:3:oracle@emeaicmd.geodll.company.com@ EMEAICMD.geodll.company.com]
(2021-10-07 9:50:02): [nss] [cache_req_search_cache] (0x0400): CR #12: Looking up [oracle@company.com] in cache
(2021-10-07 9:50:02): [nss] [cache_req_search_cache] (0x0400): CR #12: Object [oracle@company.com] was not found in cache
(2021-10-07 9:50:02): [nss] [cache_req_search_ncache_add_to_domain] (0x0400): CR #12: Adding [oracle@company.com] to negative cache
(2021-10-07 9:50:02): [nss] [is_user_local_by_name] (0x0400): User oracle@company.com is a local user
(2021-10-07 9:50:02): [nss] [sss_ncache_set_str] (0x0400): Adding [NCE/USER/company.com/oracle@company.com] to negative cache
(2021-10-07 9:50:02): [nss] [cache_req_validate_domain_type] (0x2000): Request type POSIX-only for domain EMEAICMD.geodll.company.com type
POSIX
is valid
(2021-10-07 9:50:02): [nss] [cache_req_set_domain] (0x0400): CR #12:
Using
domain [EMEAICMD.geodll.company.com]
(2021-10-07 9:50:02): [nss] [cache_req_prepare_domain_data] (0x0400): CR #12: Preparing input data for domain [EMEAICMD.geodll.company.com] rules
(2021-10-07 9:50:02): [nss] [cache_req_search_send] (0x0400): CR #12: Looking up oracle@emeaicmd.geodll.company.com
(2021-10-07 9:50:02): [nss] [cache_req_search_ncache] (0x0400): CR #12: Checking negative cache for [oracle@emeaicmd.geodll.company.com]
(2021-10-07 9:50:02): [nss] [sss_ncache_check_str] (0x2000): Checking negative cache for [NCE/USER/ EMEAICMD.geodll.company.com/oracle@emeaicmd.geodll.company.com]
(2021-10-07 9:50:02): [nss] [cache_req_search_ncache] (0x0400): CR #12:
[
oracle@emeaicmd.geodll.company.com] is not present in negative cache
(2021-10-07 9:50:02): [nss] [cache_req_search_cache] (0x0400): CR #12: Looking up [oracle@emeaicmd.geodll.company.com] in cache
(2021-10-07 9:50:02): [nss] [cache_req_search_cache] (0x0400): CR #12: Object [oracle@emeaicmd.geodll.company.com] was not found in cache
(2021-10-07 9:50:02): [nss] [cache_req_search_dp] (0x0400): CR #12: Looking up [oracle@emeaicmd.geodll.company.com] in data provider
(2021-10-07 9:50:02): [nss] [sss_dp_issue_request] (0x0400): Issuing request for [0x564d6be36a70:3:oracle@emeaicmd.geodll.company.com@ EMEAICMD.geodll.company.com]
(2021-10-07 9:50:02): [nss] [sss_dp_issue_request] (0x0400): Identical request in progress: [0x564d6be36a70:3:oracle@emeaicmd.geodll.company.com
@
EMEAICMD.geodll.company.com]
(2021-10-07 9:50:02): [nss] [sss_dp_req_destructor] (0x0400): Deleting request: [0x564d6be36a70:3:oracle@company.com@company.com]
(2021-10-07 9:50:02): [nss] [sbus_remove_timeout] (0x2000):
0x564d6ccd6670
(2021-10-07 9:50:02): [nss] [sbus_dispatch] (0x4000): dbus conn: 0x564d6ccc9300
(2021-10-07 9:50:02): [nss] [sbus_dispatch] (0x4000): Dispatching.
(2021-10-07 9:50:02): [nss] [sss_dp_get_reply] (0x0010): The Data
Provider
returned an error [org.freedesktop.DBus.Error.Failed]
The suggested work-around does not resolve problem #2.
BTW, here is a listing of the domains discovered on that sssd client:
[root@auspdfdlobv01 ~]# sssctl domain-list
amer.company.com
company.com
japn.company.com
emea.company.com
apac.company.com
EMEAICMD.geodll.company.com
geodll.company.com
EMEAICM.GEODLL.COMPANY.COM
alienware.com
corp.svcs
perotsystems.net
companyservices.dmz
Beer.Town
production.online.company.com
jp-poclab.companypoc.com
emea-poclab.companypoc.com
oldev.preol.company.com
olqa.preol.company.com
ap-poclab.companypoc.com
[root@auspdfdlobv01 ~]#
This sssd client is joined to amer.company.com, so the only trusted
domains
are the first 5. The parent domain and the 4 regional domains. All those other domains below that are untrusted domains. More specifically, they trust company.com, but company.com does not trust them. (one way trust – the wrong way.) Some look like the real wild wild west
(Beer.Town
?).
Spike
sssd-users mailing list -- sssd-users@lists.fedorahosted.org To unsubscribe send an email to sssd-users-leave@lists.fedorahosted.org Fedora Code of Conduct:
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives:
https://lists.fedorahosted.org/archives/list/sssd-users@lists.fedorahosted.o...
Do not reply to spam on the list, report it:
https://pagure.io/fedora-infrastructure _______________________________________________ sssd-users mailing list -- sssd-users@lists.fedorahosted.org To unsubscribe send an email to sssd-users-leave@lists.fedorahosted.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedorahosted.org/archives/list/sssd-users@lists.fedorahosted.o... Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
sssd-users mailing list -- sssd-users@lists.fedorahosted.org To unsubscribe send an email to sssd-users-leave@lists.fedorahosted.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedorahosted.org/archives/list/sssd-users@lists.fedorahosted.o... Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Sumit,
It took all day, but I finally got these RPMs on a test box:
libsss_simpleifp-2.4.0-9.el8_4.2sb1.x86_64 sssd-ipa-2.4.0-9.el8_4.2sb1.x86_64 sssd-client-2.4.0-9.el8_4.2sb1.x86_64 sssd-krb5-common-2.4.0-9.el8_4.2sb1.x86_64 libsss_idmap-2.4.0-9.el8_4.2sb1.x86_64 sssd-common-2.4.0-9.el8_4.2sb1.x86_64 python3-sssdconfig-2.4.0-9.el8_4.2sb1.noarch sssd-common-pac-2.4.0-9.el8_4.2sb1.x86_64 sssd-ad-2.4.0-9.el8_4.2sb1.x86_64 libsss_certmap-2.4.0-9.el8_4.2sb1.x86_64 sssd-dbus-2.4.0-9.el8_4.2sb1.x86_64 sssd-tools-2.4.0-9.el8_4.2sb1.x86_64 libipa_hbac-2.4.0-9.el8_4.2sb1.x86_64 libsss_nss_idmap-2.4.0-9.el8_4.2sb1.x86_64 sssd-kcm-2.4.0-9.el8_4.2sb1.x86_64 python3-sss-2.4.0-9.el8_4.2sb1.x86_64
samba-client-libs-4.13.3-5.el8_4 must be *brand new*; the latest we have in our mirrored repos is samba-client-libs-4.13.3-4.el8_4. Which caused no end of manual work and installing 2-3 of the above rpms with 'rpm -Uvh --nodeps'.
Originally, all the bogus domains still appeared in sssctl domain-list and I was sad! But I realized they could have been cached from previously. Sure enough, when I shut down sssd and removed *all* cache:
rm -rf /var/lib/sss/db/*
rm -rf /var/lib/sss/mc/*
Then when it started it found only the expected 5 AD domains. (1 parent + 4 regional child sub-domains).
I'll let my fellow engineers poke at it in the next few days -- but I think it looks good. I think you've fixed it.
Spike PS I git cloned your github.com source tree, checked out your ad_domain_filter branch and attempted to build according to build instructions. Using a RHEL7 test box since it's what a fellow engineer had handy. Using the build instructions. https://sssd.io/contrib/building-sssd.html
It was an epic fail, even when I installed and enabled the devtoolset-9 SCL. I eventually generated binaries, but they installed into /usr/local/bin. I started to generate RPMs, but you had your RPMs ready for me by then and I switched over to a 8.4 test box, using your pre-build RPMs. Will the Fedora sssd build instructions on https://sssd.io/contrib/building-sssd.html ( dnf builddep sssd and following) work on RHEL8.4?
On Fri, Oct 8, 2021 at 12:42 PM Sumit Bose sbose@redhat.com wrote:
Am Fri, Oct 08, 2021 at 08:51:55AM -0500 schrieb Spike White:
Sumit,
It would probably be faster for you to do a test build. I'd have to
fumble
through pulling the source RPM, rpmbuild invocation, rpm install. You probably know those commands at your fingertips.
We have ~20K servers with RHEL7, RHEL8, OL7 (RHCK) and OL8 (RHCK) exhibiting this behavior. All x86_64. We have test servers of each of those flavors on which we can test. Your call.
We have beaucoup RHEL8/OL8 test boxes, so if that's convenient for you, it'll work for us.
Hi,
please have a look at https://sbose.fedorapeople.org/sssd-2.4.0-9.el8_4.2sb1.tar.gz, a test build for RHEL-8.4. You do not have to install all packages, calling 'yum update *' in the new directory should just update the installed packages.
bye, Sumit
It is super-easy for us to determine.if it's fixed or not. Previously 'sssctl domain-list' only showed the 5 trusted domains. Now with this
new
sssd version (~July), 'sssctl domain-list' shows the expected 5 trusted domains and the 14 untrusted domains.
Spike
On Fri, Oct 8, 2021 at 1:01 AM Sumit Bose sbose@redhat.com wrote:
Am Thu, Oct 07, 2021 at 11:38:54AM -0500 schrieb Spike White:
All (but particularly Sumit since he wrote the comments on https://bugzilla.redhat.com/show_bug.cgi?id=1984591),
Hi,
jfyi, I'm currently working on a fix for this to filter out domains
from
other forests and untrusted domains. My WIP branch is at https://github.com/sumit-bose/sssd/tree/ad_filter_domains. Can you do
a
test build of SSSD based on this or shall I try to create a test build for you? For the latter, please tell me for which platform.
bye, Sumit
There are at least two problems created by this recently-introduced
sssd
bug. One problem is solvable by the suggested work-around, the
other is
not. The work-around suggested is:
[domain/name.of.joined.domain]
ad_enabled_domains = dom1.example.com, dom2.example.com,dom3.example.com
In order to query only the desired AD domains.
What is the bug?
the sssd-ad man page says "The AD provider can be used to get user information and authenticate users from trusted domains. Currently only trusted domains in the same forest are recognized.".
What is happening is that untrusted AD domains are being
discovered. A
very specific type of untrusted domains. When the joined domain
has no
trust with that other domain, but that other domain trusts the
original
domain – that is a one-way trust (the wrong way). To the joined
domain,
this is an untrusted domain and should not be discovered.
This is actually very common in corporate environments.
You may have a main AD domain, call it CORP.COMPANY.COM. Then for
testing
and new production evaluation, you might have a test AD domain called LAB-TEST.COMPANY.COM. CORP.COMPANY.COM is tightly controlled, with
full
audits and corporate security. LAB-TEST.COMPANY.COM is a test AD
domain –
it’s the wild, wild west!
So LAB-TEST.COMPANY.COM trusts the main AD domain (in order that
users
can
log into this test domain with their CORP accounts). But
CORP.COMPANY.COM
does not trust LAB-TEST.COMPANY.COM – nor should it!! (That’s the
wild,
wild west, doing so would compromise corporate security.)
Thus, a server joined to domain CORP.COMPANY.COM should discover CORP.COMPANY.COM and any domains trusted by CORP.COMPANY.COM. It
should
*NOT* discover LAB-TEST.COMPANY.COM, as CORP.COMPANY.COM does not
trust
this domain.
A server joined to LAB-TEST.COMPANY.COM should discover
LAB-TEST.COMPANY.COM
and all domains trusted by LAB-TEST.COMPANY.COM. Including
CORP.COMPANY.COM,
as LAB-TEST.COMPANY.COM trusts CORP.COMPANY.COM.
The bug is that a server joined to CORP.COMPANY.COM discovers LAB-TEST.COMPANY.COM, which it shouldn’t.
What problems does this cause?
Two problems.
Many of these untrusted discovered “lab” domains areaccessible
only to specific network locations. That is, they’re firewalled off
to a
particular lab. So sssd attempts to query these inaccessible AD
domains
and takes a long time to time out. This problem can be worked
around by
the suggested work-around in the Bugzilla:
[domain/corp.company.com]
ad_enabled_domains = corp.company.comSo then, while LAB-TEST.COMPANY.COM is still erroneously discovered,
it is
no longer searched. Sssd is again fast.
Bogus messages in /var/log/sssd_nss.log file. Even with nodebug
level set in the [nss] stanza, these error messages appear multiple
times a
second. It quickly fills up the /var/log filesystem.
[root@auspdfdlobv01 sssd]# cat sssd_nss.log |grep "The Data Provider returned an error"
(2021-10-07 9:50:02): [nss] [sss_dp_get_reply] (0x0010): The Data
Provider
returned an error [org.freedesktop.DBus.Error.Failed]
(2021-10-07 9:50:02): [nss] [sss_dp_get_reply] (0x0010): The Data
Provider
returned an error [org.freedesktop.DBus.Error.Failed]
(2021-10-07 9:50:02): [nss] [sss_dp_get_reply] (0x0010): The Data
Provider
returned an error [org.freedesktop.DBus.Error.Failed]
(2021-10-07 9:50:02): [nss] [sss_dp_get_reply] (0x0010): The Data
Provider
returned an error [org.freedesktop.DBus.Error.Failed]
(2021-10-07 9:50:02): [nss] [sss_dp_get_reply] (0x0010): The Data
Provider
returned an error [org.freedesktop.DBus.Error.Failed]
(2021-10-07 9:50:02): [nss] [sss_dp_get_reply] (0x0010): The Data
Provider
returned an error [org.freedesktop.DBus.Error.Failed]
(2021-10-07 9:50:02): [nss] [sss_dp_get_reply] (0x0010): The Data
Provider
returned an error [org.freedesktop.DBus.Error.Failed]
(2021-10-07 9:50:02): [nss] [sss_dp_get_reply] (0x0010): The Data
Provider
returned an error [org.freedesktop.DBus.Error.Failed]
(2021-10-07 9:50:02): [nss] [sss_dp_get_reply] (0x0010): The Data
Provider
returned an error [org.freedesktop.DBus.Error.Failed]
(2021-10-07 9:50:02): [nss] [sss_dp_get_reply] (0x0010): The Data
Provider
returned an error [org.freedesktop.DBus.Error.Failed]
(2021-10-07 9:50:02): [nss] [sss_dp_get_reply] (0x0010): The Data
Provider
returned an error [org.freedesktop.DBus.Error.Failed]
(2021-10-07 9:50:02): [nss] [sss_dp_get_reply] (0x0010): The Data
Provider
returned an error [org.freedesktop.DBus.Error.Failed]
(2021-10-07 9:50:02): [nss] [sss_dp_get_reply] (0x0010): The Data
Provider
returned an error [org.freedesktop.DBus.Error.Failed]
(2021-10-07 9:50:02): [nss] [sss_dp_get_reply] (0x0010): The Data
Provider
returned an error [org.freedesktop.DBus.Error.Failed]
(2021-10-07 9:50:02): [nss] [sss_dp_get_reply] (0x0010): The Data
Provider
returned an error [org.freedesktop.DBus.Error.Failed]
(2021-10-07 9:50:02): [nss] [sss_dp_get_reply] (0x0010): The Data
Provider
returned an error [org.freedesktop.DBus.Error.Failed]
(2021-10-07 9:50:02): [nss] [sss_dp_get_reply] (0x0010): The Data
Provider
returned an error [org.freedesktop.DBus.Error.Failed]
(2021-10-07 9:50:02): [nss] [sss_dp_get_reply] (0x0010): The Data
Provider
returned an error [org.freedesktop.DBus.Error.Failed]
(2021-10-07 9:50:02): [nss] [sss_dp_get_reply] (0x0010): The Data
Provider
returned an error [org.freedesktop.DBus.Error.Failed]
(2021-10-07 9:50:02): [nss] [sss_dp_get_reply] (0x0010): The Data
Provider
returned an error [org.freedesktop.DBus.Error.Failed]
(2021-10-07 9:50:02): [nss] [sss_dp_get_reply] (0x0010): The Data
Provider
returned an error [org.freedesktop.DBus.Error.Failed]
(2021-10-07 9:50:02): [nss] [sss_dp_get_reply] (0x0010): The Data
Provider
returned an error [org.freedesktop.DBus.Error.Failed]
(2021-10-07 9:50:02): [nss] [sss_dp_get_reply] (0x0010): The Data
Provider
returned an error [org.freedesktop.DBus.Error.Failed]
(2021-10-07 9:50:02): [nss] [sss_dp_get_reply] (0x0010): The Data
Provider
returned an error [org.freedesktop.DBus.Error.Failed]
(2021-10-07 9:50:02): [nss] [sss_dp_get_reply] (0x0010): The Data
Provider
returned an error [org.freedesktop.DBus.Error.Failed]
From debug level 9, it is clear that this is arising from a query of
these
erroneously-discovered untrusted domains. Here’s an example of one instance of above with debug level 9 turned on. So emeaicmd.geodll.company.com is one of these erroneously-discovered untrusted lab domains, that happens to be firewalled off from this particular AD client:
(2021-10-07 9:50:02): [nss] [sss_dp_get_reply] (0x1000): Got reply
from
Data Provider - DP error code: 0 errno: 0 error message: Success
(2021-10-07 9:50:02): [nss] [cache_req_search_cache] (0x0400): CR
#9:
Looking up [oracle@company.com] in cache
(2021-10-07 9:50:02): [nss] [cache_req_search_cache] (0x0400): CR
#9:
Object [oracle@company.com] was not found in cache
(2021-10-07 9:50:02): [nss] [cache_req_search_ncache_add_to_domain] (0x0400): CR #9: Adding [oracle@company.com] to negative cache
(2021-10-07 9:50:02): [nss] [is_user_local_by_name] (0x0400): User oracle@company.com is a local user
(2021-10-07 9:50:02): [nss] [sss_ncache_set_str] (0x0400): Adding [NCE/USER/company.com/oracle@company.com] to negative cache
(2021-10-07 9:50:02): [nss] [cache_req_validate_domain_type]
(0x2000):
Request type POSIX-only for domain EMEAICMD.geodll.company.com type
POSIX
is valid
(2021-10-07 9:50:02): [nss] [cache_req_set_domain] (0x0400): CR #9:
Using
domain [EMEAICMD.geodll.company.com]
(2021-10-07 9:50:02): [nss] [cache_req_prepare_domain_data]
(0x0400): CR
#9: Preparing input data for domain [EMEAICMD.geodll.company.com]
rules
(2021-10-07 9:50:02): [nss] [cache_req_search_send] (0x0400): CR #9: Looking up oracle@emeaicmd.geodll.company.com
(2021-10-07 9:50:02): [nss] [cache_req_search_ncache] (0x0400): CR
#9:
Checking negative cache for [oracle@emeaicmd.geodll.company.com]
(2021-10-07 9:50:02): [nss] [sss_ncache_check_str] (0x2000):
Checking
negative cache for [NCE/USER/ EMEAICMD.geodll.company.com/oracle@emeaicmd.geodll.company.com]
(2021-10-07 9:50:02): [nss] [cache_req_search_ncache] (0x0400): CR
#9: [
oracle@emeaicmd.geodll.company.com] is not present in negative cache
(2021-10-07 9:50:02): [nss] [cache_req_search_cache] (0x0400): CR
#9:
Looking up [oracle@emeaicmd.geodll.company.com] in cache
(2021-10-07 9:50:02): [nss] [cache_req_search_cache] (0x0400): CR
#9:
Object [oracle@emeaicmd.geodll.company.com] was not found in cache
(2021-10-07 9:50:02): [nss] [cache_req_search_dp] (0x0400): CR #9:
Looking
up [oracle@emeaicmd.geodll.company.com] in data provider
(2021-10-07 9:50:02): [nss] [sss_dp_issue_request] (0x0400): Issuing request for [0x564d6be36a70:3:oracle@emeaicmd.geodll.company.com@ EMEAICMD.geodll.company.com]
(2021-10-07 9:50:02): [nss] [sss_dp_get_account_msg] (0x0400):
Creating
request for [EMEAICMD.geodll.company.com ][0x3][BE_REQ_INITGROUPS][name=oracle@emeaicmd.geodll.company.com:-]
(2021-10-07 9:50:02): [nss] [sbus_add_timeout] (0x2000):
0x564d6ccd6670
(2021-10-07 9:50:02): [nss] [sss_dp_internal_get_send] (0x0400):
Entering
request [0x564d6be36a70:3:oracle@emeaicmd.geodll.company.com@ EMEAICMD.geodll.company.com]
(2021-10-07 9:50:02): [nss] [cache_req_search_cache] (0x0400): CR
#12:
Looking up [oracle@company.com] in cache
(2021-10-07 9:50:02): [nss] [cache_req_search_cache] (0x0400): CR
#12:
Object [oracle@company.com] was not found in cache
(2021-10-07 9:50:02): [nss] [cache_req_search_ncache_add_to_domain] (0x0400): CR #12: Adding [oracle@company.com] to negative cache
(2021-10-07 9:50:02): [nss] [is_user_local_by_name] (0x0400): User oracle@company.com is a local user
(2021-10-07 9:50:02): [nss] [sss_ncache_set_str] (0x0400): Adding [NCE/USER/company.com/oracle@company.com] to negative cache
(2021-10-07 9:50:02): [nss] [cache_req_validate_domain_type]
(0x2000):
Request type POSIX-only for domain EMEAICMD.geodll.company.com type
POSIX
is valid
(2021-10-07 9:50:02): [nss] [cache_req_set_domain] (0x0400): CR #12:
Using
domain [EMEAICMD.geodll.company.com]
(2021-10-07 9:50:02): [nss] [cache_req_prepare_domain_data]
(0x0400): CR
#12: Preparing input data for domain [EMEAICMD.geodll.company.com]
rules
(2021-10-07 9:50:02): [nss] [cache_req_search_send] (0x0400): CR
#12:
Looking up oracle@emeaicmd.geodll.company.com
(2021-10-07 9:50:02): [nss] [cache_req_search_ncache] (0x0400): CR
#12:
Checking negative cache for [oracle@emeaicmd.geodll.company.com]
(2021-10-07 9:50:02): [nss] [sss_ncache_check_str] (0x2000):
Checking
negative cache for [NCE/USER/ EMEAICMD.geodll.company.com/oracle@emeaicmd.geodll.company.com]
(2021-10-07 9:50:02): [nss] [cache_req_search_ncache] (0x0400): CR
#12:
[
oracle@emeaicmd.geodll.company.com] is not present in negative cache
(2021-10-07 9:50:02): [nss] [cache_req_search_cache] (0x0400): CR
#12:
Looking up [oracle@emeaicmd.geodll.company.com] in cache
(2021-10-07 9:50:02): [nss] [cache_req_search_cache] (0x0400): CR
#12:
Object [oracle@emeaicmd.geodll.company.com] was not found in cache
(2021-10-07 9:50:02): [nss] [cache_req_search_dp] (0x0400): CR #12: Looking up [oracle@emeaicmd.geodll.company.com] in data provider
(2021-10-07 9:50:02): [nss] [sss_dp_issue_request] (0x0400): Issuing request for [0x564d6be36a70:3:oracle@emeaicmd.geodll.company.com@ EMEAICMD.geodll.company.com]
(2021-10-07 9:50:02): [nss] [sss_dp_issue_request] (0x0400):
Identical
request in progress: [
0x564d6be36a70:3:oracle@emeaicmd.geodll.company.com
@
EMEAICMD.geodll.company.com]
(2021-10-07 9:50:02): [nss] [sss_dp_req_destructor] (0x0400):
Deleting
request: [0x564d6be36a70:3:oracle@company.com@company.com]
(2021-10-07 9:50:02): [nss] [sbus_remove_timeout] (0x2000):
0x564d6ccd6670
(2021-10-07 9:50:02): [nss] [sbus_dispatch] (0x4000): dbus conn: 0x564d6ccc9300
(2021-10-07 9:50:02): [nss] [sbus_dispatch] (0x4000): Dispatching.
(2021-10-07 9:50:02): [nss] [sss_dp_get_reply] (0x0010): The Data
Provider
returned an error [org.freedesktop.DBus.Error.Failed]
The suggested work-around does not resolve problem #2.
BTW, here is a listing of the domains discovered on that sssd client:
[root@auspdfdlobv01 ~]# sssctl domain-list
amer.company.com
company.com
japn.company.com
emea.company.com
apac.company.com
EMEAICMD.geodll.company.com
geodll.company.com
EMEAICM.GEODLL.COMPANY.COM
alienware.com
corp.svcs
perotsystems.net
companyservices.dmz
Beer.Town
production.online.company.com
jp-poclab.companypoc.com
emea-poclab.companypoc.com
oldev.preol.company.com
olqa.preol.company.com
ap-poclab.companypoc.com
[root@auspdfdlobv01 ~]#
This sssd client is joined to amer.company.com, so the only trusted
domains
are the first 5. The parent domain and the 4 regional domains.
All
those other domains below that are untrusted domains. More
specifically,
they trust company.com, but company.com does not trust them. (one
way
trust – the wrong way.) Some look like the real wild wild west
(Beer.Town
?).
Spike
sssd-users mailing list -- sssd-users@lists.fedorahosted.org To unsubscribe send an email to
sssd-users-leave@lists.fedorahosted.org
Fedora Code of Conduct:
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines:
https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives:
https://lists.fedorahosted.org/archives/list/sssd-users@lists.fedorahosted.o...
Do not reply to spam on the list, report it:
https://pagure.io/fedora-infrastructure _______________________________________________ sssd-users mailing list -- sssd-users@lists.fedorahosted.org To unsubscribe send an email to
sssd-users-leave@lists.fedorahosted.org
Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines:
https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives:
https://lists.fedorahosted.org/archives/list/sssd-users@lists.fedorahosted.o...
Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
sssd-users mailing list -- sssd-users@lists.fedorahosted.org To unsubscribe send an email to sssd-users-leave@lists.fedorahosted.org Fedora Code of Conduct:
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives:
https://lists.fedorahosted.org/archives/list/sssd-users@lists.fedorahosted.o...
Do not reply to spam on the list, report it:
https://pagure.io/fedora-infrastructure _______________________________________________ sssd-users mailing list -- sssd-users@lists.fedorahosted.org To unsubscribe send an email to sssd-users-leave@lists.fedorahosted.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedorahosted.org/archives/list/sssd-users@lists.fedorahosted.o... Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Sumit,
It appears I spoke too soon. Yes, properly it does not discover those untrusted AD domains (i.e., one way trust -- the wrong way). But it also does not discover a couple of the expected trusted domains.
I see this in the sssd_amer.company.com.log when turning on debug level 9:
(2021-10-08 18:31:10): [be[amer.company.com]] [ad_create_2way_trust_options] (0x0400): 2way trust is defined to domain ' company.com' (2021-10-08 18:31:10): [be[amer.company.com]] [ad_create_2way_trust_options] (0x0400): 2way trust is defined to domain ' emea.company.com' ... (2021-10-08 18:31:11): [be[amer.company.com]] [ad_subdomains_process] (0x0400): Enabling subdomain japn.company.com (2021-10-08 18:31:11): [be[amer.company.com]] [ad_subdomains_process] (0x0400): Enabling subdomain amer.company.com (2021-10-08 18:31:11): [be[amer.company.com]] [ad_subdomains_process] (0x2000): Not including primary domain amer.company.com in the subdomain list (2021-10-08 18:31:11): [be[amer.company.com]] [ad_subdomains_process] (0x0400): Enabling subdomain emea.company.com (2021-10-08 18:31:11): [be[amer.company.com]] [ad_subdomains_process] (0x0400): Enabling subdomain apac.company.com (2021-10-08 18:31:11): [be[amer.company.com]] [ad_filter_domains] (0x0200): TRUST_ATTRIBUTE_WITHIN_FOREST not set for [japn.company.com]. (2021-10-08 18:31:11): [be[amer.company.com]] [ad_filter_domains] (0x0200): TRUST_ATTRIBUTE_WITHIN_FOREST not set for [emea.company.com]. (2021-10-08 18:31:11): [be[amer.company.com]] [ad_filter_domains] (0x0200): TRUST_ATTRIBUTE_WITHIN_FOREST not set for [apac.company.com]. ... (2021-10-08 18:31:11): [be[amer.company.com]] [ad_filter_domains] (0x0200): TRUST_ATTRIBUTE_WITHIN_FOREST not set for [EMEAICMD.geodll.company.com]. (2021-10-08 18:31:11): [be[amer.company.com]] [ad_filter_domains] (0x0200): TRUST_ATTRIBUTE_WITHIN_FOREST not set for [geocompany.company.com]. (2021-10-08 18:31:11): [be[amer.company.com]] [ad_filter_domains] (0x0200): TRUST_ATTRIBUTE_WITHIN_FOREST not set for [EMEAICM.GEOCOMPANY.COMPANY.COM]. (2021-10-08 18:31:11): [be[amer.company.com]] [ad_filter_domains] (0x0200): TRUST_ATTRIBUTE_WITHIN_FOREST not set for [alienware.com]. (2021-10-08 18:31:11): [be[amer.company.com]] [ad_filter_domains] (0x0200): TRUST_ATTRIBUTE_WITHIN_FOREST not set for [corp.svcs]. (2021-10-08 18:31:11): [be[amer.company.com]] [ad_filter_domains] (0x0200): TRUST_ATTRIBUTE_WITHIN_FOREST not set for [perotsystems.net]. (2021-10-08 18:31:11): [be[amer.company.com]] [ad_filter_domains] (0x0200): TRUST_ATTRIBUTE_WITHIN_FOREST not set for [companyservices.dmz]. (2021-10-08 18:31:11): [be[amer.company.com]] [ad_filter_domains] (0x0200): TRUST_ATTRIBUTE_WITHIN_FOREST not set for [Beer.Town]. (2021-10-08 18:31:11): [be[amer.company.com]] [ad_filter_domains] (0x0200): TRUST_ATTRIBUTE_WITHIN_FOREST not set for [production.online.company.com]. (2021-10-08 18:31:11): [be[amer.company.com]] [ad_filter_domains] (0x0200): TRUST_ATTRIBUTE_WITHIN_FOREST not set for [jp-poclab.companypoc.com]. (2021-10-08 18:31:11): [be[amer.company.com]] [ad_filter_domains] (0x0200): TRUST_ATTRIBUTE_WITHIN_FOREST not set for [japn.company.com]. (2021-10-08 18:31:11): [be[amer.company.com]] [ad_filter_domains] (0x0200): TRUST_ATTRIBUTE_WITHIN_FOREST not set for [apac.company.com]. (2021-10-08 18:31:11): [be[amer.company.com]] [ad_filter_domains] (0x0200): TRUST_ATTRIBUTE_WITHIN_FOREST not set for [emea-poclab.companypoc.com]. (2021-10-08 18:31:11): [be[amer.company.com]] [ad_filter_domains] (0x0200): TRUST_ATTRIBUTE_WITHIN_FOREST not set for [oldev.preol.company.com]. (2021-10-08 18:31:11): [be[amer.company.com]] [ad_filter_domains] (0x0200): TRUST_ATTRIBUTE_WITHIN_FOREST not set for [olqa.preol.company.com]. (2021-10-08 18:31:11): [be[amer.company.com]] [ad_filter_domains] (0x0200): TRUST_ATTRIBUTE_WITHIN_FOREST not set for [ap-poclab.companypoc.com].
It looks like it's discovering any AD domain with a two-way trust. Which is good.
And it's not discovering any domain with a one-way trust. That's good when it's a one-way trust the wrong way. I.e., untrusted from this domain. That's bad when it's a one-way trust (or a transitive trust) the right way.
Spike
On Fri, Oct 8, 2021 at 5:54 PM Spike White spikewhitetx@gmail.com wrote:
Sumit,
It took all day, but I finally got these RPMs on a test box:
libsss_simpleifp-2.4.0-9.el8_4.2sb1.x86_64 sssd-ipa-2.4.0-9.el8_4.2sb1.x86_64 sssd-client-2.4.0-9.el8_4.2sb1.x86_64 sssd-krb5-common-2.4.0-9.el8_4.2sb1.x86_64 libsss_idmap-2.4.0-9.el8_4.2sb1.x86_64 sssd-common-2.4.0-9.el8_4.2sb1.x86_64 python3-sssdconfig-2.4.0-9.el8_4.2sb1.noarch sssd-common-pac-2.4.0-9.el8_4.2sb1.x86_64 sssd-ad-2.4.0-9.el8_4.2sb1.x86_64 libsss_certmap-2.4.0-9.el8_4.2sb1.x86_64 sssd-dbus-2.4.0-9.el8_4.2sb1.x86_64 sssd-tools-2.4.0-9.el8_4.2sb1.x86_64 libipa_hbac-2.4.0-9.el8_4.2sb1.x86_64 libsss_nss_idmap-2.4.0-9.el8_4.2sb1.x86_64 sssd-kcm-2.4.0-9.el8_4.2sb1.x86_64 python3-sss-2.4.0-9.el8_4.2sb1.x86_64
samba-client-libs-4.13.3-5.el8_4 must be *brand new*; the latest we have in our mirrored repos is samba-client-libs-4.13.3-4.el8_4. Which caused no end of manual work and installing 2-3 of the above rpms with 'rpm -Uvh --nodeps'.
Originally, all the bogus domains still appeared in sssctl domain-list and I was sad! But I realized they could have been cached from previously. Sure enough, when I shut down sssd and removed *all* cache:
rm -rf /var/lib/sss/db/*
rm -rf /var/lib/sss/mc/*
Then when it started it found only the expected 5 AD domains. (1 parent + 4 regional child sub-domains).
I'll let my fellow engineers poke at it in the next few days -- but I think it looks good. I think you've fixed it.
Spike PS I git cloned your github.com source tree, checked out your ad_domain_filter branch and attempted to build according to build instructions. Using a RHEL7 test box since it's what a fellow engineer had handy. Using the build instructions. https://sssd.io/contrib/building-sssd.html
It was an epic fail, even when I installed and enabled the devtoolset-9 SCL. I eventually generated binaries, but they installed into /usr/local/bin. I started to generate RPMs, but you had your RPMs ready for me by then and I switched over to a 8.4 test box, using your pre-build RPMs. Will the Fedora sssd build instructions on https://sssd.io/contrib/building-sssd.html ( dnf builddep sssd and following) work on RHEL8.4?
On Fri, Oct 8, 2021 at 12:42 PM Sumit Bose sbose@redhat.com wrote:
Am Fri, Oct 08, 2021 at 08:51:55AM -0500 schrieb Spike White:
Sumit,
It would probably be faster for you to do a test build. I'd have to
fumble
through pulling the source RPM, rpmbuild invocation, rpm install. You probably know those commands at your fingertips.
We have ~20K servers with RHEL7, RHEL8, OL7 (RHCK) and OL8 (RHCK) exhibiting this behavior. All x86_64. We have test servers of each of those flavors on which we can test. Your call.
We have beaucoup RHEL8/OL8 test boxes, so if that's convenient for you, it'll work for us.
Hi,
please have a look at https://sbose.fedorapeople.org/sssd-2.4.0-9.el8_4.2sb1.tar.gz, a test build for RHEL-8.4. You do not have to install all packages, calling 'yum update *' in the new directory should just update the installed packages.
bye, Sumit
It is super-easy for us to determine.if it's fixed or not. Previously 'sssctl domain-list' only showed the 5 trusted domains. Now with this
new
sssd version (~July), 'sssctl domain-list' shows the expected 5 trusted domains and the 14 untrusted domains.
Spike
On Fri, Oct 8, 2021 at 1:01 AM Sumit Bose sbose@redhat.com wrote:
Am Thu, Oct 07, 2021 at 11:38:54AM -0500 schrieb Spike White:
All (but particularly Sumit since he wrote the comments on https://bugzilla.redhat.com/show_bug.cgi?id=1984591),
Hi,
jfyi, I'm currently working on a fix for this to filter out domains
from
other forests and untrusted domains. My WIP branch is at https://github.com/sumit-bose/sssd/tree/ad_filter_domains. Can you
do a
test build of SSSD based on this or shall I try to create a test build for you? For the latter, please tell me for which platform.
bye, Sumit
There are at least two problems created by this recently-introduced
sssd
bug. One problem is solvable by the suggested work-around, the
other is
not. The work-around suggested is:
[domain/name.of.joined.domain]
ad_enabled_domains = dom1.example.com, dom2.example.com,dom3.example.com
In order to query only the desired AD domains.
What is the bug?
the sssd-ad man page says "The AD provider can be used to get user information and authenticate users from trusted domains. Currently only trusted domains in the same forest are recognized.".
What is happening is that untrusted AD domains are being
discovered. A
very specific type of untrusted domains. When the joined domain
has no
trust with that other domain, but that other domain trusts the
original
domain – that is a one-way trust (the wrong way). To the joined
domain,
this is an untrusted domain and should not be discovered.
This is actually very common in corporate environments.
You may have a main AD domain, call it CORP.COMPANY.COM. Then for
testing
and new production evaluation, you might have a test AD domain
called
LAB-TEST.COMPANY.COM. CORP.COMPANY.COM is tightly controlled,
with full
audits and corporate security. LAB-TEST.COMPANY.COM is a test AD
domain –
it’s the wild, wild west!
So LAB-TEST.COMPANY.COM trusts the main AD domain (in order that
users
can
log into this test domain with their CORP accounts). But
CORP.COMPANY.COM
does not trust LAB-TEST.COMPANY.COM – nor should it!! (That’s the
wild,
wild west, doing so would compromise corporate security.)
Thus, a server joined to domain CORP.COMPANY.COM should discover CORP.COMPANY.COM and any domains trusted by CORP.COMPANY.COM. It
should
*NOT* discover LAB-TEST.COMPANY.COM, as CORP.COMPANY.COM does not
trust
this domain.
A server joined to LAB-TEST.COMPANY.COM should discover
LAB-TEST.COMPANY.COM
and all domains trusted by LAB-TEST.COMPANY.COM. Including
CORP.COMPANY.COM,
as LAB-TEST.COMPANY.COM trusts CORP.COMPANY.COM.
The bug is that a server joined to CORP.COMPANY.COM discovers LAB-TEST.COMPANY.COM, which it shouldn’t.
What problems does this cause?
Two problems.
Many of these untrusted discovered “lab” domains areaccessible
only to specific network locations. That is, they’re firewalled
off to a
particular lab. So sssd attempts to query these inaccessible AD
domains
and takes a long time to time out. This problem can be worked
around by
the suggested work-around in the Bugzilla:
[domain/corp.company.com]
ad_enabled_domains = corp.company.comSo then, while LAB-TEST.COMPANY.COM is still erroneously
discovered,
it is
no longer searched. Sssd is again fast.
Bogus messages in /var/log/sssd_nss.log file. Even with nodebug
level set in the [nss] stanza, these error messages appear multiple
times a
second. It quickly fills up the /var/log filesystem.
[root@auspdfdlobv01 sssd]# cat sssd_nss.log |grep "The Data
Provider
returned an error"
(2021-10-07 9:50:02): [nss] [sss_dp_get_reply] (0x0010): The Data
Provider
returned an error [org.freedesktop.DBus.Error.Failed]
(2021-10-07 9:50:02): [nss] [sss_dp_get_reply] (0x0010): The Data
Provider
returned an error [org.freedesktop.DBus.Error.Failed]
(2021-10-07 9:50:02): [nss] [sss_dp_get_reply] (0x0010): The Data
Provider
returned an error [org.freedesktop.DBus.Error.Failed]
(2021-10-07 9:50:02): [nss] [sss_dp_get_reply] (0x0010): The Data
Provider
returned an error [org.freedesktop.DBus.Error.Failed]
(2021-10-07 9:50:02): [nss] [sss_dp_get_reply] (0x0010): The Data
Provider
returned an error [org.freedesktop.DBus.Error.Failed]
(2021-10-07 9:50:02): [nss] [sss_dp_get_reply] (0x0010): The Data
Provider
returned an error [org.freedesktop.DBus.Error.Failed]
(2021-10-07 9:50:02): [nss] [sss_dp_get_reply] (0x0010): The Data
Provider
returned an error [org.freedesktop.DBus.Error.Failed]
(2021-10-07 9:50:02): [nss] [sss_dp_get_reply] (0x0010): The Data
Provider
returned an error [org.freedesktop.DBus.Error.Failed]
(2021-10-07 9:50:02): [nss] [sss_dp_get_reply] (0x0010): The Data
Provider
returned an error [org.freedesktop.DBus.Error.Failed]
(2021-10-07 9:50:02): [nss] [sss_dp_get_reply] (0x0010): The Data
Provider
returned an error [org.freedesktop.DBus.Error.Failed]
(2021-10-07 9:50:02): [nss] [sss_dp_get_reply] (0x0010): The Data
Provider
returned an error [org.freedesktop.DBus.Error.Failed]
(2021-10-07 9:50:02): [nss] [sss_dp_get_reply] (0x0010): The Data
Provider
returned an error [org.freedesktop.DBus.Error.Failed]
(2021-10-07 9:50:02): [nss] [sss_dp_get_reply] (0x0010): The Data
Provider
returned an error [org.freedesktop.DBus.Error.Failed]
(2021-10-07 9:50:02): [nss] [sss_dp_get_reply] (0x0010): The Data
Provider
returned an error [org.freedesktop.DBus.Error.Failed]
(2021-10-07 9:50:02): [nss] [sss_dp_get_reply] (0x0010): The Data
Provider
returned an error [org.freedesktop.DBus.Error.Failed]
(2021-10-07 9:50:02): [nss] [sss_dp_get_reply] (0x0010): The Data
Provider
returned an error [org.freedesktop.DBus.Error.Failed]
(2021-10-07 9:50:02): [nss] [sss_dp_get_reply] (0x0010): The Data
Provider
returned an error [org.freedesktop.DBus.Error.Failed]
(2021-10-07 9:50:02): [nss] [sss_dp_get_reply] (0x0010): The Data
Provider
returned an error [org.freedesktop.DBus.Error.Failed]
(2021-10-07 9:50:02): [nss] [sss_dp_get_reply] (0x0010): The Data
Provider
returned an error [org.freedesktop.DBus.Error.Failed]
(2021-10-07 9:50:02): [nss] [sss_dp_get_reply] (0x0010): The Data
Provider
returned an error [org.freedesktop.DBus.Error.Failed]
(2021-10-07 9:50:02): [nss] [sss_dp_get_reply] (0x0010): The Data
Provider
returned an error [org.freedesktop.DBus.Error.Failed]
(2021-10-07 9:50:02): [nss] [sss_dp_get_reply] (0x0010): The Data
Provider
returned an error [org.freedesktop.DBus.Error.Failed]
(2021-10-07 9:50:02): [nss] [sss_dp_get_reply] (0x0010): The Data
Provider
returned an error [org.freedesktop.DBus.Error.Failed]
(2021-10-07 9:50:02): [nss] [sss_dp_get_reply] (0x0010): The Data
Provider
returned an error [org.freedesktop.DBus.Error.Failed]
(2021-10-07 9:50:02): [nss] [sss_dp_get_reply] (0x0010): The Data
Provider
returned an error [org.freedesktop.DBus.Error.Failed]
From debug level 9, it is clear that this is arising from a query of
these
erroneously-discovered untrusted domains. Here’s an example of one instance of above with debug level 9 turned on. So emeaicmd.geodll.company.com is one of these erroneously-discovered untrusted lab domains, that happens to be firewalled off from this particular AD client:
(2021-10-07 9:50:02): [nss] [sss_dp_get_reply] (0x1000): Got reply
from
Data Provider - DP error code: 0 errno: 0 error message: Success
(2021-10-07 9:50:02): [nss] [cache_req_search_cache] (0x0400): CR
#9:
Looking up [oracle@company.com] in cache
(2021-10-07 9:50:02): [nss] [cache_req_search_cache] (0x0400): CR
#9:
Object [oracle@company.com] was not found in cache
(2021-10-07 9:50:02): [nss] [cache_req_search_ncache_add_to_domain] (0x0400): CR #9: Adding [oracle@company.com] to negative cache
(2021-10-07 9:50:02): [nss] [is_user_local_by_name] (0x0400): User oracle@company.com is a local user
(2021-10-07 9:50:02): [nss] [sss_ncache_set_str] (0x0400): Adding [NCE/USER/company.com/oracle@company.com] to negative cache
(2021-10-07 9:50:02): [nss] [cache_req_validate_domain_type]
(0x2000):
Request type POSIX-only for domain EMEAICMD.geodll.company.com type
POSIX
is valid
(2021-10-07 9:50:02): [nss] [cache_req_set_domain] (0x0400): CR #9:
Using
domain [EMEAICMD.geodll.company.com]
(2021-10-07 9:50:02): [nss] [cache_req_prepare_domain_data]
(0x0400): CR
#9: Preparing input data for domain [EMEAICMD.geodll.company.com]
rules
(2021-10-07 9:50:02): [nss] [cache_req_search_send] (0x0400): CR
#9:
Looking up oracle@emeaicmd.geodll.company.com
(2021-10-07 9:50:02): [nss] [cache_req_search_ncache] (0x0400): CR
#9:
Checking negative cache for [oracle@emeaicmd.geodll.company.com]
(2021-10-07 9:50:02): [nss] [sss_ncache_check_str] (0x2000):
Checking
negative cache for [NCE/USER/ EMEAICMD.geodll.company.com/oracle@emeaicmd.geodll.company.com]
(2021-10-07 9:50:02): [nss] [cache_req_search_ncache] (0x0400): CR
#9: [
oracle@emeaicmd.geodll.company.com] is not present in negative
cache
(2021-10-07 9:50:02): [nss] [cache_req_search_cache] (0x0400): CR
#9:
Looking up [oracle@emeaicmd.geodll.company.com] in cache
(2021-10-07 9:50:02): [nss] [cache_req_search_cache] (0x0400): CR
#9:
Object [oracle@emeaicmd.geodll.company.com] was not found in cache
(2021-10-07 9:50:02): [nss] [cache_req_search_dp] (0x0400): CR #9:
Looking
up [oracle@emeaicmd.geodll.company.com] in data provider
(2021-10-07 9:50:02): [nss] [sss_dp_issue_request] (0x0400):
Issuing
request for [0x564d6be36a70:3:oracle@emeaicmd.geodll.company.com@ EMEAICMD.geodll.company.com]
(2021-10-07 9:50:02): [nss] [sss_dp_get_account_msg] (0x0400):
Creating
request for [EMEAICMD.geodll.company.com ][0x3][BE_REQ_INITGROUPS][name=oracle@emeaicmd.geodll.company.com:
-]
(2021-10-07 9:50:02): [nss] [sbus_add_timeout] (0x2000):
0x564d6ccd6670
(2021-10-07 9:50:02): [nss] [sss_dp_internal_get_send] (0x0400):
Entering
request [0x564d6be36a70:3:oracle@emeaicmd.geodll.company.com@ EMEAICMD.geodll.company.com]
(2021-10-07 9:50:02): [nss] [cache_req_search_cache] (0x0400): CR
#12:
Looking up [oracle@company.com] in cache
(2021-10-07 9:50:02): [nss] [cache_req_search_cache] (0x0400): CR
#12:
Object [oracle@company.com] was not found in cache
(2021-10-07 9:50:02): [nss] [cache_req_search_ncache_add_to_domain] (0x0400): CR #12: Adding [oracle@company.com] to negative cache
(2021-10-07 9:50:02): [nss] [is_user_local_by_name] (0x0400): User oracle@company.com is a local user
(2021-10-07 9:50:02): [nss] [sss_ncache_set_str] (0x0400): Adding [NCE/USER/company.com/oracle@company.com] to negative cache
(2021-10-07 9:50:02): [nss] [cache_req_validate_domain_type]
(0x2000):
Request type POSIX-only for domain EMEAICMD.geodll.company.com type
POSIX
is valid
(2021-10-07 9:50:02): [nss] [cache_req_set_domain] (0x0400): CR
#12:
Using
domain [EMEAICMD.geodll.company.com]
(2021-10-07 9:50:02): [nss] [cache_req_prepare_domain_data]
(0x0400): CR
#12: Preparing input data for domain [EMEAICMD.geodll.company.com]
rules
(2021-10-07 9:50:02): [nss] [cache_req_search_send] (0x0400): CR
#12:
Looking up oracle@emeaicmd.geodll.company.com
(2021-10-07 9:50:02): [nss] [cache_req_search_ncache] (0x0400): CR
#12:
Checking negative cache for [oracle@emeaicmd.geodll.company.com]
(2021-10-07 9:50:02): [nss] [sss_ncache_check_str] (0x2000):
Checking
negative cache for [NCE/USER/ EMEAICMD.geodll.company.com/oracle@emeaicmd.geodll.company.com]
(2021-10-07 9:50:02): [nss] [cache_req_search_ncache] (0x0400): CR
#12:
[
oracle@emeaicmd.geodll.company.com] is not present in negative
cache
(2021-10-07 9:50:02): [nss] [cache_req_search_cache] (0x0400): CR
#12:
Looking up [oracle@emeaicmd.geodll.company.com] in cache
(2021-10-07 9:50:02): [nss] [cache_req_search_cache] (0x0400): CR
#12:
Object [oracle@emeaicmd.geodll.company.com] was not found in cache
(2021-10-07 9:50:02): [nss] [cache_req_search_dp] (0x0400): CR #12: Looking up [oracle@emeaicmd.geodll.company.com] in data provider
(2021-10-07 9:50:02): [nss] [sss_dp_issue_request] (0x0400):
Issuing
request for [0x564d6be36a70:3:oracle@emeaicmd.geodll.company.com@ EMEAICMD.geodll.company.com]
(2021-10-07 9:50:02): [nss] [sss_dp_issue_request] (0x0400):
Identical
request in progress: [
0x564d6be36a70:3:oracle@emeaicmd.geodll.company.com
@
EMEAICMD.geodll.company.com]
(2021-10-07 9:50:02): [nss] [sss_dp_req_destructor] (0x0400):
Deleting
request: [0x564d6be36a70:3:oracle@company.com@company.com]
(2021-10-07 9:50:02): [nss] [sbus_remove_timeout] (0x2000):
0x564d6ccd6670
(2021-10-07 9:50:02): [nss] [sbus_dispatch] (0x4000): dbus conn: 0x564d6ccc9300
(2021-10-07 9:50:02): [nss] [sbus_dispatch] (0x4000): Dispatching.
(2021-10-07 9:50:02): [nss] [sss_dp_get_reply] (0x0010): The Data
Provider
returned an error [org.freedesktop.DBus.Error.Failed]
The suggested work-around does not resolve problem #2.
BTW, here is a listing of the domains discovered on that sssd
client:
[root@auspdfdlobv01 ~]# sssctl domain-list
amer.company.com
company.com
japn.company.com
emea.company.com
apac.company.com
EMEAICMD.geodll.company.com
geodll.company.com
EMEAICM.GEODLL.COMPANY.COM
alienware.com
corp.svcs
perotsystems.net
companyservices.dmz
Beer.Town
production.online.company.com
jp-poclab.companypoc.com
emea-poclab.companypoc.com
oldev.preol.company.com
olqa.preol.company.com
ap-poclab.companypoc.com
[root@auspdfdlobv01 ~]#
This sssd client is joined to amer.company.com, so the only trusted
domains
are the first 5. The parent domain and the 4 regional domains.
All
those other domains below that are untrusted domains. More
specifically,
they trust company.com, but company.com does not trust them. (one
way
trust – the wrong way.) Some look like the real wild wild west
(Beer.Town
?).
Spike
sssd-users mailing list -- sssd-users@lists.fedorahosted.org To unsubscribe send an email to
sssd-users-leave@lists.fedorahosted.org
Fedora Code of Conduct:
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines:
https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives:
https://lists.fedorahosted.org/archives/list/sssd-users@lists.fedorahosted.o...
Do not reply to spam on the list, report it:
https://pagure.io/fedora-infrastructure _______________________________________________ sssd-users mailing list -- sssd-users@lists.fedorahosted.org To unsubscribe send an email to
sssd-users-leave@lists.fedorahosted.org
Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines:
https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives:
https://lists.fedorahosted.org/archives/list/sssd-users@lists.fedorahosted.o...
Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
sssd-users mailing list -- sssd-users@lists.fedorahosted.org To unsubscribe send an email to sssd-users-leave@lists.fedorahosted.org Fedora Code of Conduct:
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives:
https://lists.fedorahosted.org/archives/list/sssd-users@lists.fedorahosted.o...
Do not reply to spam on the list, report it:
https://pagure.io/fedora-infrastructure _______________________________________________ sssd-users mailing list -- sssd-users@lists.fedorahosted.org To unsubscribe send an email to sssd-users-leave@lists.fedorahosted.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedorahosted.org/archives/list/sssd-users@lists.fedorahosted.o... Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Am Fri, Oct 08, 2021 at 07:16:26PM -0500 schrieb Spike White:
Sumit,
It appears I spoke too soon. Yes, properly it does not discover those untrusted AD domains (i.e., one way trust -- the wrong way). But it also does not discover a couple of the expected trusted domains.
I see this in the sssd_amer.company.com.log when turning on debug level 9:
(2021-10-08 18:31:10): [be[amer.company.com]] [ad_create_2way_trust_options] (0x0400): 2way trust is defined to domain ' company.com' (2021-10-08 18:31:10): [be[amer.company.com]] [ad_create_2way_trust_options] (0x0400): 2way trust is defined to domain ' emea.company.com' ... (2021-10-08 18:31:11): [be[amer.company.com]] [ad_subdomains_process] (0x0400): Enabling subdomain japn.company.com (2021-10-08 18:31:11): [be[amer.company.com]] [ad_subdomains_process] (0x0400): Enabling subdomain amer.company.com (2021-10-08 18:31:11): [be[amer.company.com]] [ad_subdomains_process] (0x2000): Not including primary domain amer.company.com in the subdomain list (2021-10-08 18:31:11): [be[amer.company.com]] [ad_subdomains_process] (0x0400): Enabling subdomain emea.company.com (2021-10-08 18:31:11): [be[amer.company.com]] [ad_subdomains_process] (0x0400): Enabling subdomain apac.company.com (2021-10-08 18:31:11): [be[amer.company.com]] [ad_filter_domains] (0x0200): TRUST_ATTRIBUTE_WITHIN_FOREST not set for [japn.company.com]. (2021-10-08 18:31:11): [be[amer.company.com]] [ad_filter_domains] (0x0200): TRUST_ATTRIBUTE_WITHIN_FOREST not set for [emea.company.com]. (2021-10-08 18:31:11): [be[amer.company.com]] [ad_filter_domains] (0x0200): TRUST_ATTRIBUTE_WITHIN_FOREST not set for [apac.company.com].
Hi,
I guess the above domains should not have been filtered out? Can you send me the logs form the time before as well covering the ldap_search for the domains and the processing of the results? This would tell if the 'trustAttributes' LDAP attributes was not returned by the AD DC for those domains (this is what I would expect) or if the TRUST_ATTRIBUTE_WITHIN_FOREST is not set which would be surprising because I would expect that those domains are in the same forest as the client.
bye, Sumit
... (2021-10-08 18:31:11): [be[amer.company.com]] [ad_filter_domains] (0x0200): TRUST_ATTRIBUTE_WITHIN_FOREST not set for [EMEAICMD.geodll.company.com]. (2021-10-08 18:31:11): [be[amer.company.com]] [ad_filter_domains] (0x0200): TRUST_ATTRIBUTE_WITHIN_FOREST not set for [geocompany.company.com]. (2021-10-08 18:31:11): [be[amer.company.com]] [ad_filter_domains] (0x0200): TRUST_ATTRIBUTE_WITHIN_FOREST not set for [EMEAICM.GEOCOMPANY.COMPANY.COM]. (2021-10-08 18:31:11): [be[amer.company.com]] [ad_filter_domains] (0x0200): TRUST_ATTRIBUTE_WITHIN_FOREST not set for [alienware.com]. (2021-10-08 18:31:11): [be[amer.company.com]] [ad_filter_domains] (0x0200): TRUST_ATTRIBUTE_WITHIN_FOREST not set for [corp.svcs]. (2021-10-08 18:31:11): [be[amer.company.com]] [ad_filter_domains] (0x0200): TRUST_ATTRIBUTE_WITHIN_FOREST not set for [perotsystems.net]. (2021-10-08 18:31:11): [be[amer.company.com]] [ad_filter_domains] (0x0200): TRUST_ATTRIBUTE_WITHIN_FOREST not set for [companyservices.dmz]. (2021-10-08 18:31:11): [be[amer.company.com]] [ad_filter_domains] (0x0200): TRUST_ATTRIBUTE_WITHIN_FOREST not set for [Beer.Town]. (2021-10-08 18:31:11): [be[amer.company.com]] [ad_filter_domains] (0x0200): TRUST_ATTRIBUTE_WITHIN_FOREST not set for [production.online.company.com]. (2021-10-08 18:31:11): [be[amer.company.com]] [ad_filter_domains] (0x0200): TRUST_ATTRIBUTE_WITHIN_FOREST not set for [jp-poclab.companypoc.com]. (2021-10-08 18:31:11): [be[amer.company.com]] [ad_filter_domains] (0x0200): TRUST_ATTRIBUTE_WITHIN_FOREST not set for [japn.company.com]. (2021-10-08 18:31:11): [be[amer.company.com]] [ad_filter_domains] (0x0200): TRUST_ATTRIBUTE_WITHIN_FOREST not set for [apac.company.com]. (2021-10-08 18:31:11): [be[amer.company.com]] [ad_filter_domains] (0x0200): TRUST_ATTRIBUTE_WITHIN_FOREST not set for [emea-poclab.companypoc.com]. (2021-10-08 18:31:11): [be[amer.company.com]] [ad_filter_domains] (0x0200): TRUST_ATTRIBUTE_WITHIN_FOREST not set for [oldev.preol.company.com]. (2021-10-08 18:31:11): [be[amer.company.com]] [ad_filter_domains] (0x0200): TRUST_ATTRIBUTE_WITHIN_FOREST not set for [olqa.preol.company.com]. (2021-10-08 18:31:11): [be[amer.company.com]] [ad_filter_domains] (0x0200): TRUST_ATTRIBUTE_WITHIN_FOREST not set for [ap-poclab.companypoc.com].
It looks like it's discovering any AD domain with a two-way trust. Which is good.
And it's not discovering any domain with a one-way trust. That's good when it's a one-way trust the wrong way. I.e., untrusted from this domain. That's bad when it's a one-way trust (or a transitive trust) the right way.
Spike
On Fri, Oct 8, 2021 at 5:54 PM Spike White spikewhitetx@gmail.com wrote:
Sumit,
It took all day, but I finally got these RPMs on a test box:
libsss_simpleifp-2.4.0-9.el8_4.2sb1.x86_64 sssd-ipa-2.4.0-9.el8_4.2sb1.x86_64 sssd-client-2.4.0-9.el8_4.2sb1.x86_64 sssd-krb5-common-2.4.0-9.el8_4.2sb1.x86_64 libsss_idmap-2.4.0-9.el8_4.2sb1.x86_64 sssd-common-2.4.0-9.el8_4.2sb1.x86_64 python3-sssdconfig-2.4.0-9.el8_4.2sb1.noarch sssd-common-pac-2.4.0-9.el8_4.2sb1.x86_64 sssd-ad-2.4.0-9.el8_4.2sb1.x86_64 libsss_certmap-2.4.0-9.el8_4.2sb1.x86_64 sssd-dbus-2.4.0-9.el8_4.2sb1.x86_64 sssd-tools-2.4.0-9.el8_4.2sb1.x86_64 libipa_hbac-2.4.0-9.el8_4.2sb1.x86_64 libsss_nss_idmap-2.4.0-9.el8_4.2sb1.x86_64 sssd-kcm-2.4.0-9.el8_4.2sb1.x86_64 python3-sss-2.4.0-9.el8_4.2sb1.x86_64
samba-client-libs-4.13.3-5.el8_4 must be *brand new*; the latest we have in our mirrored repos is samba-client-libs-4.13.3-4.el8_4. Which caused no end of manual work and installing 2-3 of the above rpms with 'rpm -Uvh --nodeps'.
Originally, all the bogus domains still appeared in sssctl domain-list and I was sad! But I realized they could have been cached from previously. Sure enough, when I shut down sssd and removed *all* cache:
rm -rf /var/lib/sss/db/*
rm -rf /var/lib/sss/mc/*
Then when it started it found only the expected 5 AD domains. (1 parent + 4 regional child sub-domains).
I'll let my fellow engineers poke at it in the next few days -- but I think it looks good. I think you've fixed it.
Spike PS I git cloned your github.com source tree, checked out your ad_domain_filter branch and attempted to build according to build instructions. Using a RHEL7 test box since it's what a fellow engineer had handy. Using the build instructions. https://sssd.io/contrib/building-sssd.html
It was an epic fail, even when I installed and enabled the devtoolset-9 SCL. I eventually generated binaries, but they installed into /usr/local/bin. I started to generate RPMs, but you had your RPMs ready for me by then and I switched over to a 8.4 test box, using your pre-build RPMs. Will the Fedora sssd build instructions on https://sssd.io/contrib/building-sssd.html ( dnf builddep sssd and following) work on RHEL8.4?
On Fri, Oct 8, 2021 at 12:42 PM Sumit Bose sbose@redhat.com wrote:
Am Fri, Oct 08, 2021 at 08:51:55AM -0500 schrieb Spike White:
Sumit,
It would probably be faster for you to do a test build. I'd have to
fumble
through pulling the source RPM, rpmbuild invocation, rpm install. You probably know those commands at your fingertips.
We have ~20K servers with RHEL7, RHEL8, OL7 (RHCK) and OL8 (RHCK) exhibiting this behavior. All x86_64. We have test servers of each of those flavors on which we can test. Your call.
We have beaucoup RHEL8/OL8 test boxes, so if that's convenient for you, it'll work for us.
Hi,
please have a look at https://sbose.fedorapeople.org/sssd-2.4.0-9.el8_4.2sb1.tar.gz, a test build for RHEL-8.4. You do not have to install all packages, calling 'yum update *' in the new directory should just update the installed packages.
bye, Sumit
It is super-easy for us to determine.if it's fixed or not. Previously 'sssctl domain-list' only showed the 5 trusted domains. Now with this
new
sssd version (~July), 'sssctl domain-list' shows the expected 5 trusted domains and the 14 untrusted domains.
Spike
On Fri, Oct 8, 2021 at 1:01 AM Sumit Bose sbose@redhat.com wrote:
Am Thu, Oct 07, 2021 at 11:38:54AM -0500 schrieb Spike White:
All (but particularly Sumit since he wrote the comments on https://bugzilla.redhat.com/show_bug.cgi?id=1984591),
Hi,
jfyi, I'm currently working on a fix for this to filter out domains
from
other forests and untrusted domains. My WIP branch is at https://github.com/sumit-bose/sssd/tree/ad_filter_domains. Can you
do a
test build of SSSD based on this or shall I try to create a test build for you? For the latter, please tell me for which platform.
bye, Sumit
There are at least two problems created by this recently-introduced
sssd
bug. One problem is solvable by the suggested work-around, the
other is
not. The work-around suggested is:
[domain/name.of.joined.domain]
ad_enabled_domains = dom1.example.com, dom2.example.com,dom3.example.com
In order to query only the desired AD domains.
What is the bug?
the sssd-ad man page says "The AD provider can be used to get user information and authenticate users from trusted domains. Currently only trusted domains in the same forest are recognized.".
What is happening is that untrusted AD domains are being
discovered. A
very specific type of untrusted domains. When the joined domain
has no
trust with that other domain, but that other domain trusts the
original
domain – that is a one-way trust (the wrong way). To the joined
domain,
this is an untrusted domain and should not be discovered.
This is actually very common in corporate environments.
You may have a main AD domain, call it CORP.COMPANY.COM. Then for
testing
and new production evaluation, you might have a test AD domain
called
LAB-TEST.COMPANY.COM. CORP.COMPANY.COM is tightly controlled,
with full
audits and corporate security. LAB-TEST.COMPANY.COM is a test AD
domain –
it’s the wild, wild west!
So LAB-TEST.COMPANY.COM trusts the main AD domain (in order that
users
can
log into this test domain with their CORP accounts). But
CORP.COMPANY.COM
does not trust LAB-TEST.COMPANY.COM – nor should it!! (That’s the
wild,
wild west, doing so would compromise corporate security.)
Thus, a server joined to domain CORP.COMPANY.COM should discover CORP.COMPANY.COM and any domains trusted by CORP.COMPANY.COM. It
should
*NOT* discover LAB-TEST.COMPANY.COM, as CORP.COMPANY.COM does not
trust
this domain.
A server joined to LAB-TEST.COMPANY.COM should discover
LAB-TEST.COMPANY.COM
and all domains trusted by LAB-TEST.COMPANY.COM. Including
CORP.COMPANY.COM,
as LAB-TEST.COMPANY.COM trusts CORP.COMPANY.COM.
The bug is that a server joined to CORP.COMPANY.COM discovers LAB-TEST.COMPANY.COM, which it shouldn’t.
What problems does this cause?
Two problems.
Many of these untrusted discovered “lab” domains areaccessible
only to specific network locations. That is, they’re firewalled
off to a
particular lab. So sssd attempts to query these inaccessible AD
domains
and takes a long time to time out. This problem can be worked
around by
the suggested work-around in the Bugzilla:
[domain/corp.company.com]
ad_enabled_domains = corp.company.comSo then, while LAB-TEST.COMPANY.COM is still erroneously
discovered,
it is
no longer searched. Sssd is again fast.
Bogus messages in /var/log/sssd_nss.log file. Even with nodebug
level set in the [nss] stanza, these error messages appear multiple
times a
second. It quickly fills up the /var/log filesystem.
[root@auspdfdlobv01 sssd]# cat sssd_nss.log |grep "The Data
Provider
returned an error"
(2021-10-07 9:50:02): [nss] [sss_dp_get_reply] (0x0010): The Data
Provider
returned an error [org.freedesktop.DBus.Error.Failed]
(2021-10-07 9:50:02): [nss] [sss_dp_get_reply] (0x0010): The Data
Provider
returned an error [org.freedesktop.DBus.Error.Failed]
(2021-10-07 9:50:02): [nss] [sss_dp_get_reply] (0x0010): The Data
Provider
returned an error [org.freedesktop.DBus.Error.Failed]
(2021-10-07 9:50:02): [nss] [sss_dp_get_reply] (0x0010): The Data
Provider
returned an error [org.freedesktop.DBus.Error.Failed]
(2021-10-07 9:50:02): [nss] [sss_dp_get_reply] (0x0010): The Data
Provider
returned an error [org.freedesktop.DBus.Error.Failed]
(2021-10-07 9:50:02): [nss] [sss_dp_get_reply] (0x0010): The Data
Provider
returned an error [org.freedesktop.DBus.Error.Failed]
(2021-10-07 9:50:02): [nss] [sss_dp_get_reply] (0x0010): The Data
Provider
returned an error [org.freedesktop.DBus.Error.Failed]
(2021-10-07 9:50:02): [nss] [sss_dp_get_reply] (0x0010): The Data
Provider
returned an error [org.freedesktop.DBus.Error.Failed]
(2021-10-07 9:50:02): [nss] [sss_dp_get_reply] (0x0010): The Data
Provider
returned an error [org.freedesktop.DBus.Error.Failed]
(2021-10-07 9:50:02): [nss] [sss_dp_get_reply] (0x0010): The Data
Provider
returned an error [org.freedesktop.DBus.Error.Failed]
(2021-10-07 9:50:02): [nss] [sss_dp_get_reply] (0x0010): The Data
Provider
returned an error [org.freedesktop.DBus.Error.Failed]
(2021-10-07 9:50:02): [nss] [sss_dp_get_reply] (0x0010): The Data
Provider
returned an error [org.freedesktop.DBus.Error.Failed]
(2021-10-07 9:50:02): [nss] [sss_dp_get_reply] (0x0010): The Data
Provider
returned an error [org.freedesktop.DBus.Error.Failed]
(2021-10-07 9:50:02): [nss] [sss_dp_get_reply] (0x0010): The Data
Provider
returned an error [org.freedesktop.DBus.Error.Failed]
(2021-10-07 9:50:02): [nss] [sss_dp_get_reply] (0x0010): The Data
Provider
returned an error [org.freedesktop.DBus.Error.Failed]
(2021-10-07 9:50:02): [nss] [sss_dp_get_reply] (0x0010): The Data
Provider
returned an error [org.freedesktop.DBus.Error.Failed]
(2021-10-07 9:50:02): [nss] [sss_dp_get_reply] (0x0010): The Data
Provider
returned an error [org.freedesktop.DBus.Error.Failed]
(2021-10-07 9:50:02): [nss] [sss_dp_get_reply] (0x0010): The Data
Provider
returned an error [org.freedesktop.DBus.Error.Failed]
(2021-10-07 9:50:02): [nss] [sss_dp_get_reply] (0x0010): The Data
Provider
returned an error [org.freedesktop.DBus.Error.Failed]
(2021-10-07 9:50:02): [nss] [sss_dp_get_reply] (0x0010): The Data
Provider
returned an error [org.freedesktop.DBus.Error.Failed]
(2021-10-07 9:50:02): [nss] [sss_dp_get_reply] (0x0010): The Data
Provider
returned an error [org.freedesktop.DBus.Error.Failed]
(2021-10-07 9:50:02): [nss] [sss_dp_get_reply] (0x0010): The Data
Provider
returned an error [org.freedesktop.DBus.Error.Failed]
(2021-10-07 9:50:02): [nss] [sss_dp_get_reply] (0x0010): The Data
Provider
returned an error [org.freedesktop.DBus.Error.Failed]
(2021-10-07 9:50:02): [nss] [sss_dp_get_reply] (0x0010): The Data
Provider
returned an error [org.freedesktop.DBus.Error.Failed]
(2021-10-07 9:50:02): [nss] [sss_dp_get_reply] (0x0010): The Data
Provider
returned an error [org.freedesktop.DBus.Error.Failed]
From debug level 9, it is clear that this is arising from a query of
these
erroneously-discovered untrusted domains. Here’s an example of one instance of above with debug level 9 turned on. So emeaicmd.geodll.company.com is one of these erroneously-discovered untrusted lab domains, that happens to be firewalled off from this particular AD client:
(2021-10-07 9:50:02): [nss] [sss_dp_get_reply] (0x1000): Got reply
from
Data Provider - DP error code: 0 errno: 0 error message: Success
(2021-10-07 9:50:02): [nss] [cache_req_search_cache] (0x0400): CR
#9:
Looking up [oracle@company.com] in cache
(2021-10-07 9:50:02): [nss] [cache_req_search_cache] (0x0400): CR
#9:
Object [oracle@company.com] was not found in cache
(2021-10-07 9:50:02): [nss] [cache_req_search_ncache_add_to_domain] (0x0400): CR #9: Adding [oracle@company.com] to negative cache
(2021-10-07 9:50:02): [nss] [is_user_local_by_name] (0x0400): User oracle@company.com is a local user
(2021-10-07 9:50:02): [nss] [sss_ncache_set_str] (0x0400): Adding [NCE/USER/company.com/oracle@company.com] to negative cache
(2021-10-07 9:50:02): [nss] [cache_req_validate_domain_type]
(0x2000):
Request type POSIX-only for domain EMEAICMD.geodll.company.com type
POSIX
is valid
(2021-10-07 9:50:02): [nss] [cache_req_set_domain] (0x0400): CR #9:
Using
domain [EMEAICMD.geodll.company.com]
(2021-10-07 9:50:02): [nss] [cache_req_prepare_domain_data]
(0x0400): CR
#9: Preparing input data for domain [EMEAICMD.geodll.company.com]
rules
(2021-10-07 9:50:02): [nss] [cache_req_search_send] (0x0400): CR
#9:
Looking up oracle@emeaicmd.geodll.company.com
(2021-10-07 9:50:02): [nss] [cache_req_search_ncache] (0x0400): CR
#9:
Checking negative cache for [oracle@emeaicmd.geodll.company.com]
(2021-10-07 9:50:02): [nss] [sss_ncache_check_str] (0x2000):
Checking
negative cache for [NCE/USER/ EMEAICMD.geodll.company.com/oracle@emeaicmd.geodll.company.com]
(2021-10-07 9:50:02): [nss] [cache_req_search_ncache] (0x0400): CR
#9: [
oracle@emeaicmd.geodll.company.com] is not present in negative
cache
(2021-10-07 9:50:02): [nss] [cache_req_search_cache] (0x0400): CR
#9:
Looking up [oracle@emeaicmd.geodll.company.com] in cache
(2021-10-07 9:50:02): [nss] [cache_req_search_cache] (0x0400): CR
#9:
Object [oracle@emeaicmd.geodll.company.com] was not found in cache
(2021-10-07 9:50:02): [nss] [cache_req_search_dp] (0x0400): CR #9:
Looking
up [oracle@emeaicmd.geodll.company.com] in data provider
(2021-10-07 9:50:02): [nss] [sss_dp_issue_request] (0x0400):
Issuing
request for [0x564d6be36a70:3:oracle@emeaicmd.geodll.company.com@ EMEAICMD.geodll.company.com]
(2021-10-07 9:50:02): [nss] [sss_dp_get_account_msg] (0x0400):
Creating
request for [EMEAICMD.geodll.company.com ][0x3][BE_REQ_INITGROUPS][name=oracle@emeaicmd.geodll.company.com:
-]
(2021-10-07 9:50:02): [nss] [sbus_add_timeout] (0x2000):
0x564d6ccd6670
(2021-10-07 9:50:02): [nss] [sss_dp_internal_get_send] (0x0400):
Entering
request [0x564d6be36a70:3:oracle@emeaicmd.geodll.company.com@ EMEAICMD.geodll.company.com]
(2021-10-07 9:50:02): [nss] [cache_req_search_cache] (0x0400): CR
#12:
Looking up [oracle@company.com] in cache
(2021-10-07 9:50:02): [nss] [cache_req_search_cache] (0x0400): CR
#12:
Object [oracle@company.com] was not found in cache
(2021-10-07 9:50:02): [nss] [cache_req_search_ncache_add_to_domain] (0x0400): CR #12: Adding [oracle@company.com] to negative cache
(2021-10-07 9:50:02): [nss] [is_user_local_by_name] (0x0400): User oracle@company.com is a local user
(2021-10-07 9:50:02): [nss] [sss_ncache_set_str] (0x0400): Adding [NCE/USER/company.com/oracle@company.com] to negative cache
(2021-10-07 9:50:02): [nss] [cache_req_validate_domain_type]
(0x2000):
Request type POSIX-only for domain EMEAICMD.geodll.company.com type
POSIX
is valid
(2021-10-07 9:50:02): [nss] [cache_req_set_domain] (0x0400): CR
#12:
Using
domain [EMEAICMD.geodll.company.com]
(2021-10-07 9:50:02): [nss] [cache_req_prepare_domain_data]
(0x0400): CR
#12: Preparing input data for domain [EMEAICMD.geodll.company.com]
rules
(2021-10-07 9:50:02): [nss] [cache_req_search_send] (0x0400): CR
#12:
Looking up oracle@emeaicmd.geodll.company.com
(2021-10-07 9:50:02): [nss] [cache_req_search_ncache] (0x0400): CR
#12:
Checking negative cache for [oracle@emeaicmd.geodll.company.com]
(2021-10-07 9:50:02): [nss] [sss_ncache_check_str] (0x2000):
Checking
negative cache for [NCE/USER/ EMEAICMD.geodll.company.com/oracle@emeaicmd.geodll.company.com]
(2021-10-07 9:50:02): [nss] [cache_req_search_ncache] (0x0400): CR
#12:
[
oracle@emeaicmd.geodll.company.com] is not present in negative
cache
(2021-10-07 9:50:02): [nss] [cache_req_search_cache] (0x0400): CR
#12:
Looking up [oracle@emeaicmd.geodll.company.com] in cache
(2021-10-07 9:50:02): [nss] [cache_req_search_cache] (0x0400): CR
#12:
Object [oracle@emeaicmd.geodll.company.com] was not found in cache
(2021-10-07 9:50:02): [nss] [cache_req_search_dp] (0x0400): CR #12: Looking up [oracle@emeaicmd.geodll.company.com] in data provider
(2021-10-07 9:50:02): [nss] [sss_dp_issue_request] (0x0400):
Issuing
request for [0x564d6be36a70:3:oracle@emeaicmd.geodll.company.com@ EMEAICMD.geodll.company.com]
(2021-10-07 9:50:02): [nss] [sss_dp_issue_request] (0x0400):
Identical
request in progress: [
0x564d6be36a70:3:oracle@emeaicmd.geodll.company.com
@
EMEAICMD.geodll.company.com]
(2021-10-07 9:50:02): [nss] [sss_dp_req_destructor] (0x0400):
Deleting
request: [0x564d6be36a70:3:oracle@company.com@company.com]
(2021-10-07 9:50:02): [nss] [sbus_remove_timeout] (0x2000):
0x564d6ccd6670
(2021-10-07 9:50:02): [nss] [sbus_dispatch] (0x4000): dbus conn: 0x564d6ccc9300
(2021-10-07 9:50:02): [nss] [sbus_dispatch] (0x4000): Dispatching.
(2021-10-07 9:50:02): [nss] [sss_dp_get_reply] (0x0010): The Data
Provider
returned an error [org.freedesktop.DBus.Error.Failed]
The suggested work-around does not resolve problem #2.
BTW, here is a listing of the domains discovered on that sssd
client:
[root@auspdfdlobv01 ~]# sssctl domain-list
amer.company.com
company.com
japn.company.com
emea.company.com
apac.company.com
EMEAICMD.geodll.company.com
geodll.company.com
EMEAICM.GEODLL.COMPANY.COM
alienware.com
corp.svcs
perotsystems.net
companyservices.dmz
Beer.Town
production.online.company.com
jp-poclab.companypoc.com
emea-poclab.companypoc.com
oldev.preol.company.com
olqa.preol.company.com
ap-poclab.companypoc.com
[root@auspdfdlobv01 ~]#
This sssd client is joined to amer.company.com, so the only trusted
domains
are the first 5. The parent domain and the 4 regional domains.
All
those other domains below that are untrusted domains. More
specifically,
they trust company.com, but company.com does not trust them. (one
way
trust – the wrong way.) Some look like the real wild wild west
(Beer.Town
?).
Spike
sssd-users mailing list -- sssd-users@lists.fedorahosted.org To unsubscribe send an email to
sssd-users-leave@lists.fedorahosted.org
Fedora Code of Conduct:
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines:
https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives:
https://lists.fedorahosted.org/archives/list/sssd-users@lists.fedorahosted.o...
Do not reply to spam on the list, report it:
https://pagure.io/fedora-infrastructure _______________________________________________ sssd-users mailing list -- sssd-users@lists.fedorahosted.org To unsubscribe send an email to
sssd-users-leave@lists.fedorahosted.org
Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines:
https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives:
https://lists.fedorahosted.org/archives/list/sssd-users@lists.fedorahosted.o...
Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
sssd-users mailing list -- sssd-users@lists.fedorahosted.org To unsubscribe send an email to sssd-users-leave@lists.fedorahosted.org Fedora Code of Conduct:
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives:
https://lists.fedorahosted.org/archives/list/sssd-users@lists.fedorahosted.o...
Do not reply to spam on the list, report it:
https://pagure.io/fedora-infrastructure _______________________________________________ sssd-users mailing list -- sssd-users@lists.fedorahosted.org To unsubscribe send an email to sssd-users-leave@lists.fedorahosted.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedorahosted.org/archives/list/sssd-users@lists.fedorahosted.o... Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
sssd-users mailing list -- sssd-users@lists.fedorahosted.org To unsubscribe send an email to sssd-users-leave@lists.fedorahosted.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedorahosted.org/archives/list/sssd-users@lists.fedorahosted.o... Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
sssd-users@lists.fedorahosted.org
-
Spike White -
Sumit Bose