On Wed, Dec 02, 2020 at 03:04:14PM -0500, J. Adam Craig wrote:
> Thanks, all, for your assistance with this issue!
>
> Your notes have prompted some further investigation on my end.
>
> 1. In response to Alexey's question, I can confirm that, when lookups
> are successful, port 389 is also used.
>
> 2. After conversing with our Active Directory team, I've learned that,
> in at least one case of this issue occurring (the one cited above), the IP
> address used for the lookup was actually not a domain controller at all,
> but was rather one of our Active Directory DNS servers. This leads me to
> wonder why SSSD could be attempting to use an Active Directory DNS server
> to perform lookups in the first place.
>
> When I run an 'nslookup' against our Active Directory domain name, only
> IP addresses of domain controllers are returned, so the issue doesn't
> appear to be originating from there.
Hi,
what kind of lookup did you do with nslookup? Typically SSSD does a SRV
lookup for '_ldap._tcp.ad.domain.name' to find the domain controllers of
a given domain. But it might also try to find site specific DC with
'_ldap._tcp.sitename,_sites.ad.domain.name'.
If you search in the logs where the IP address or the related DNS name
is recorded first this should help to understand by which request this
host was found.
>
> What other mechanisms (besides DNS) might SSSD be using to determine the
> domain controllers it should use?
No, the other way is to specific the DCs in sssd.conf.
>
> In our environment, our AD DNS servers live in a root domain, with
> users, etc. residing in a child domain.
I guess the DNS servers are domain controllers for the root domain as
well? If the one wrongly used a global catalog server as well? If that's
the case you most probably hit the bug I mentioned earlier.
bye,
Sumit
>
> Thanks again for any insight and assistance.
>
> *J. Adam Craig*
> Lead Unix Operating Systems Analyst
> VCU Computer Center <https://www.ucc.vcu.edu/>
> 804.828.4886
> jacraig@vcu.edu
>
> <https://adminmicro2.questionpro.com/?t_340030260=J.%20Adam%20Craig&u_65977055=351791134>
> *Don't be a phishing victim -- VCU and other reputable organisations will
> never use email to request that you reply with your password, social
> security number or confidential personal information. For more details,
> visit **https://ts.vcu.edu/about-us/information-security/common-questions/what-is-phishing
> <https://ts.vcu.edu/about-us/information-security/common-questions/what-is-phishing>*
>
>
>
> On Wed, Dec 2, 2020 at 2:18 AM Sumit Bose <sbose@redhat.com> wrote:
>
> > On Tue, Dec 01, 2020 at 07:21:15PM +0100, Alexey Tikhonov wrote:
> > > Hi,
> > >
> > > according to the sssd.conf you do not have `ad_enable_gc` set so this
> > > should be `true` by default.
> > >
> > > But in the log it contacts LDAP port:
> > > [sdap_print_server] (0x2000): Searching 192.168.1.4:389 -- IIUC, GC
> > port
> > > should be 3269...
> > > -- what port is used when everything is working as expected?
> > >
> > > If I understand "Referral(10), 0000202B: RefErr: DSID-0310074A, data 0, 1
> > > access points" correctly, this specific DC "doesn't know" this user and
> > > refers to other DCs, but referrals chasing is disabled for ad provider.
> > >
> > > This doesn't explain what happens though... Perhaps clue can be found
> > > earlier in the logs.
> > >
> > >
> > > On Tue, Dec 1, 2020 at 4:43 PM J. Adam Craig <jacraig@vcu.edu> wrote:
> > >
> > > > Hello!
> > > >
> > > > I am currently troubleshooting a very mysterious and difficult to tack
> > > > down issue with SSSD running on RHEL/CentOS 7.x and 8.x (SSSD ver.
> > 1.16.5
> > > > and 2.2.3). The EL 7.x and 8.x systems are attached to a Windows
> > Active
> > > > Directory domain using 'adcli'. We used this guide (
> > > > https://access.redhat.com/solutions/2653771), with some minor tweaks
> > > > appropriate for our Active Directory environment and security
> > requirements
> > > > to set this up.
> > > >
> > > > The vast majority of the time, the configuration works as expected.
> > > > However, occasionally, we experience sporadic and temporary issues
> > where
> > > > users attempting to authenticate using valid Active Directory
> > credentials
> > > > (via SSH) are unable to login.
> >
> > Hi,
> >
> > you might hit https://github.com/SSSD/sssd/issues/5351, the fix is
> > currently not available in any released RHEL or CentOS version.
> >
> > bye,
> > Sumit
> >
> > > >
> > > > When the issue presents itself, if we leave an affected system alone
> > and
> > > > do nothing, the issue will eventually self-correct and users are able
> > to
> > > > authenticate as expected. We have also discovered that we can manually
> > > > "fix" the issue by either (1) restarting SSSD with 'sudo systemctl
> > restart
> > > > sssd' or (2) by running a 'kdestroy'/'kinit' sequence as the 'root'
> > user on
> > > > the affected system like so:
> > > >
> > > > # kdestroy -A ; kinit -k 'MYSERVER$@MYDOMAIN.EXAMPLE.COM'
> > > >
> > > >
> > > > At times when the issue is occurring, we observe the following
> > messages in
> > > > '/var/log/sssd/sssd_MYDOMAIN.example.com.log' (debug_level 9):
> > > >
> > > > (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]]
> > > > [dp_get_account_info_handler] (0x0200): Got request for
> > > > [0x1][BE_REQ_USER][name=someuser@mydomain.example.com]
> > > > (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] [dp_attach_req]
> > > > (0x0400): DP Request [Account #6104]: New request. Flags [0x0001].
> > > > (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] [dp_attach_req]
> > > > (0x0400): Number of active DP request: 1
> > > > (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]]
> > [sss_domain_get_state]
> > > > (0x1000): Domain MYDOMAIN.example.com is Active
> > > > (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]]
> > [sss_domain_get_state]
> > > > (0x1000): Domain MYDOMAIN.example.com is Active
> > > > (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]]
> > > > [sdap_id_op_connect_step] (0x4000): reusing cached connection
> > > > (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]]
> > > > [sdap_search_user_next_base] (0x0400): Searching for users with base
> > > > [DC=MYDOMAIN,DC=example,DC=com]
> > > > (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] [sdap_print_server]
> > > > (0x2000): Searching 192.168.1.4:389
> > > > (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]]
> > > > [sdap_get_generic_ext_step] (0x0400): calling ldap_search_ext with
> > > >
> > [(&(sAMAccountName=someuser)(objectclass=user)(sAMAccountName=*)(objectSID=*))][DC=MYDOMAIN,DC=example,DC=com].
> > > > (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]]
> > > > [sdap_get_generic_ext_step] (0x1000): Requesting attrs: [objectClass]
> > > > (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]]
> > > > [sdap_get_generic_ext_step] (0x1000): Requesting attrs:
> > [sAMAccountName]
> > > > (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]]
> > > > [sdap_get_generic_ext_step] (0x1000): Requesting attrs:
> > [unixUserPassword]
> > > > (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]]
> > > > [sdap_get_generic_ext_step] (0x1000): Requesting attrs: [uidNumber]
> > > > (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]]
> > > > [sdap_get_generic_ext_step] (0x1000): Requesting attrs: [gidNumber]
> > > > (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]]
> > > > [sdap_get_generic_ext_step] (0x1000): Requesting attrs: [gecos]
> > > > (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]]
> > > > [sdap_get_generic_ext_step] (0x1000): Requesting attrs:
> > [unixHomeDirectory]
> > > > (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]]
> > > > [sdap_get_generic_ext_step] (0x1000): Requesting attrs: [loginShell]
> > > > (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]]
> > > > [sdap_get_generic_ext_step] (0x1000): Requesting attrs:
> > [userPrincipalName]
> > > > (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]]
> > > > [sdap_get_generic_ext_step] (0x1000): Requesting attrs: [name]
> > > > (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]]
> > > > [sdap_get_generic_ext_step] (0x1000): Requesting attrs: [memberOf]
> > > > (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]]
> > > > [sdap_get_generic_ext_step] (0x1000): Requesting attrs: [objectGUID]
> > > > (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]]
> > > > [sdap_get_generic_ext_step] (0x1000): Requesting attrs: [objectSID]
> > > > (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]]
> > > > [sdap_get_generic_ext_step] (0x1000): Requesting attrs:
> > [primaryGroupID]
> > > > (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]]
> > > > [sdap_get_generic_ext_step] (0x1000): Requesting attrs: [whenChanged]
> > > > (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]]
> > > > [sdap_get_generic_ext_step] (0x1000): Requesting attrs: [uSNChanged]
> > > > (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]]
> > > > [sdap_get_generic_ext_step] (0x1000): Requesting attrs:
> > [accountExpires]
> > > > (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]]
> > > > [sdap_get_generic_ext_step] (0x1000): Requesting attrs:
> > [userAccountControl]
> > > > (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]]
> > > > [sdap_get_generic_ext_step] (0x1000): Requesting attrs:
> > > > [userCertificate;binary]
> > > > (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]]
> > > > [sdap_get_generic_ext_step] (0x1000): Requesting attrs: [mail]
> > > > (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]]
> > > > [sdap_get_generic_ext_step] (0x2000): ldap_search_ext called, msgid =
> > 26
> > > > (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] [sdap_op_add]
> > (0x2000):
> > > > New operation 26 timeout 6
> > > > (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]]
> > [sdap_process_result]
> > > > (0x2000): Trace: sh[0x55ca2802d840], connected[1], ops[0x55ca28028ef0],
> > > > ldap[0x55ca27fe0610]
> > > > (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]]
> > [sdap_process_message]
> > > > (0x4000): Message type: [LDAP_RES_SEARCH_RESULT]
> > > > (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]]
> > > > [sdap_get_generic_op_finished] (0x0400): Search result: Referral(10),
> > > > 0000202B: RefErr: DSID-0310074A, data 0, 1 access points
> > > > ref 1: 'MYDOMAIN.example.com'
> > > >
> > > > (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]]
> > > > [sdap_get_generic_ext_add_references] (0x1000): Additional References:
> > > > ldap://MYDOMAIN.example.com/DC=MYDOMAIN,DC=example,DC=com
> > > > (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] [sdap_op_destructor]
> > > > (0x2000): Operation 26 finished
> > > > (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]]
> > > > [generic_ext_search_handler] (0x4000): Request included referrals which
> > > > were ignored.
> > > > (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]]
> > > > [generic_ext_search_handler] (0x4000): Ref: ldap://
> > > > MYDOMAIN.example.com/DC=MYDOMAIN,DC=example,DC=com
> > > > (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]]
> > > > [sdap_search_user_process] (0x0400): Search for users, returned 0
> > results.
> > > > (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]]
> > > > [sdap_search_user_process] (0x2000): Retrieved total 0 users
> > > > (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] [sdap_id_op_done]
> > > > (0x4000): releasing operation connection
> > > > (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]]
> > [sysdb_search_by_name]
> > > > (0x0400): No such entry
> > > > (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]]
> > > > [sysdb_cache_search_groups] (0x2000): Search groups with filter:
> > > > (&(objectCategory=group)(ghost=someuser@mydomain.example.com))
> > > > (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]]
> > > > [sysdb_cache_search_groups] (0x2000): No such entry
> > > > (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] [sysdb_delete_user]
> > > > (0x0400): Error: 2 (No such file or directory)
> > > > (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] [dp_req_done]
> > (0x0400):
> > > > DP Request [Account #6104]: Request handler finished [0]: Success
> > > > (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] [_dp_req_recv]
> > > > (0x0400): DP Request [Account #6104]: Receiving request data.
> > > > (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]]
> > > > [dp_req_reply_list_success] (0x0400): DP Request [Account #6104]:
> > Finished.
> > > > Success.
> > > > (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] [dp_req_reply_std]
> > > > (0x1000): DP Request [Account #6104]: Returning [Success]: 0,0,Success
> > > > (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]]
> > > > [dp_table_value_destructor] (0x0400): Removing
> > > > [0:1:0x0001:1::MYDOMAIN.example.com:name=someuser@mydomain.example.com
> > ]
> > > > from reply table
> > > > (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] [dp_req_destructor]
> > > > (0x0400): DP Request [Account #6104]: Request removed.
> > > > (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] [dp_req_destructor]
> > > > (0x0400): Number of active DP request: 0
> > > > (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]]
> > [sdap_process_result]
> > > > (0x2000): Trace: sh[0x55ca2802d840], connected[1], ops[(nil)],
> > > > ldap[0x55ca27fe0610]
> > > > (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]]
> > [sdap_process_result]
> > > > (0x2000): Trace: end of ldap_result list
> > > >
> > > >
> > > > And the following corresponding messages in '/var/log/secure':
> > > >
> > > > Nov 10 07:40:57 myserver sshd[755]: pam_unix(sshd:auth): check pass;
> > user
> > > > unknown
> > > > Nov 10 07:40:57 myserver sshd[755]: pam_unix(sshd:auth): authentication
> > > > failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=192.168.1.123
> > > > Nov 10 07:40:57 myserver sshd[755]: pam_sss(sshd:auth): authentication
> > > > failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=192.168.1.123
> > > > user=someuser
> > > > Nov 10 07:40:57 myserver sshd[755]: pam_sss(sshd:auth): received for
> > user
> > > > someuser: 10 (User not known to the underlying authentication module)
> > > > Nov 10 07:40:59 myserver sshd[735]: error: PAM: User not known to the
> > > > underlying authentication module for illegal user someuser from
> > > > 192.168.1.123
> > > > Nov 10 07:40:59 myserver sshd[735]: Failed keyboard-interactive/pam for
> > > > invalid user someuser from 192.168.1.123 port 53300 ssh2
> > > > Nov 10 07:40:59 myserver sshd[735]: Connection closed by 192.168.1.123
> > > > port 53300 [preauth]
> > > >
> > > >
> > > > The effective '/etc/sssd/sssd.conf' file is as follows:
> > > >
> > > > [sssd]
> > > > domains = MYDOMAIN.example.com
> > > > config_file_version = 2
> > > > services = nss, pam
> > > > debug_level = 9
> > > >
> > > > [domain/MYDOMAIN.example.com]
> > > > ad_domain = MYDOMAIN.example.com
> > > > krb5_realm = MYDOMAIN.EXAMPLE.COM
> > > > krb5_lifetime = 10h
> > > > subdomain_inherit = ignore_group_members, ldap_purge_cache_timeout
> > > > ignore_group_members = true
> > > > ldap_purge_cache_timeout = 0
> > > > realmd_tags = joined-with-adcli, manages-system
> > > > cache_credentials = false
> > > > id_provider = ad
> > > > krb5_store_password_if_offline = true
> > > > default_shell = /bin/bash
> > > > ldap_id_mapping = true
> > > > ldap_sasl_authid = MYSERVER$@MYDOMAIN.EXAMPLE.COM
> > > > ldap_use_tokengroups = true
> > > > use_fully_qualified_names = false
> > > > fallback_homedir = /home/%d/%u
> > > > access_provider = simple
> > > > simple_allow_groups = linux_admins
> > > > simple_allow_users = someuser, someuser2, someuser3
> > > > debug_level = 9
> > > >
> > > >
> > > > Running either of the following commands appears to correct the issue
> > > > (until it presents again at an unpredictable time):
> > > >
> > > > # systemctl restart sssd
> > > >
> > > >
> > > > or
> > > >
> > > > # kdestroy -A ; kinit -k 'MYSERVER$@MYDOMAIN.EXAMPLE.COM'
> > > >
> > > >
> > > > Any assistance or insight you can offer would be greatly appreciated.
> > We
> > > > have run countless internet searches over recent weeks, as well as
> > > > consulted with Red Hat Support without breakthroughs, so I decided to
> > take
> > > > this straight to the experts!
> > > >
> > > > Best,
> > > >
> > > > *J. Adam Craig*
> > > > Lead Unix Operating Systems Analyst
> > > > VCU Computer Center <https://www.ucc.vcu.edu/>
> > > > 804.828.4886
> > > > jacraig@vcu.edu
> > > >
> > > >
> > > > <
> > https://adminmicro2.questionpro.com/?t_340030260=J.%20Adam%20Craig&u_65977055=351791134
> > >
> > > > *Don't be a phishing victim -- VCU and other reputable organisations
> > will
> > > > never use email to request that you reply with your password, social
> > > > security number or confidential personal information. For more
> > details,
> > > > visit **
> > https://ts.vcu.edu/about-us/information-security/common-questions/what-is-phishing
> > > > <
> > https://ts.vcu.edu/about-us/information-security/common-questions/what-is-phishing
> > >*
> > > >
> > > > _______________________________________________
> > > > sssd-users mailing list -- sssd-users@lists.fedorahosted.org
> > > > To unsubscribe send an email to
> > sssd-users-leave@lists.fedorahosted.org
> > > > Fedora Code of Conduct:
> > > > https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> > > > List Guidelines:
> > https://fedoraproject.org/wiki/Mailing_list_guidelines
> > > > List Archives:
> > > >
> > https://lists.fedorahosted.org/archives/list/sssd-users@lists.fedorahosted.org
> > > >
> >
> > > _______________________________________________
> > > sssd-users mailing list -- sssd-users@lists.fedorahosted.org
> > > To unsubscribe send an email to sssd-users-leave@lists.fedorahosted.org
> > > Fedora Code of Conduct:
> > https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> > > List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> > > List Archives:
> > https://lists.fedorahosted.org/archives/list/sssd-users@lists.fedorahosted.org
> > _______________________________________________
> > sssd-users mailing list -- sssd-users@lists.fedorahosted.org
> > To unsubscribe send an email to sssd-users-leave@lists.fedorahosted.org
> > Fedora Code of Conduct:
> > https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> > List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> > List Archives:
> > https://lists.fedorahosted.org/archives/list/sssd-users@lists.fedorahosted.org
> >
> _______________________________________________
> sssd-users mailing list -- sssd-users@lists.fedorahosted.org
> To unsubscribe send an email to sssd-users-leave@lists.fedorahosted.org
> Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives: https://lists.fedorahosted.org/archives/list/sssd-users@lists.fedorahosted.org
_______________________________________________
sssd-users mailing list -- sssd-users@lists.fedorahosted.org
To unsubscribe send an email to sssd-users-leave@lists.fedorahosted.org
Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: https://lists.fedorahosted.org/archives/list/sssd-users@lists.fedorahosted.org