Sporadic Issue When Searching Active Directory for a Valid User, "returned 0 results", Restarting SSSD Resolves
Hello!
I am currently troubleshooting a very mysterious and difficult to tack down issue with SSSD running on RHEL/CentOS 7.x and 8.x (SSSD ver. 1.16.5 and 2.2.3). The EL 7.x and 8.x systems are attached to a Windows Active Directory domain using 'adcli'. We used this guide ( https://access.redhat.com/solutions/2653771), with some minor tweaks appropriate for our Active Directory environment and security requirements to set this up.
The vast majority of the time, the configuration works as expected. However, occasionally, we experience sporadic and temporary issues where users attempting to authenticate using valid Active Directory credentials (via SSH) are unable to login.
When the issue presents itself, if we leave an affected system alone and do nothing, the issue will eventually self-correct and users are able to authenticate as expected. We have also discovered that we can manually "fix" the issue by either (1) restarting SSSD with 'sudo systemctl restart sssd' or (2) by running a 'kdestroy'/'kinit' sequence as the 'root' user on the affected system like so:
# kdestroy -A ; kinit -k 'MYSERVER$@MYDOMAIN.EXAMPLE.COM'
At times when the issue is occurring, we observe the following messages in '/var/log/sssd/sssd_MYDOMAIN.example.com.log' (debug_level 9):
(2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] [dp_get_account_info_handler] (0x0200): Got request for [0x1][BE_REQ_USER][name=someuser@mydomain.example.com] (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] [dp_attach_req] (0x0400): DP Request [Account #6104]: New request. Flags [0x0001]. (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] [dp_attach_req] (0x0400): Number of active DP request: 1 (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] [sss_domain_get_state] (0x1000): Domain MYDOMAIN.example.com is Active (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] [sss_domain_get_state] (0x1000): Domain MYDOMAIN.example.com is Active (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] [sdap_id_op_connect_step] (0x4000): reusing cached connection (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] [sdap_search_user_next_base] (0x0400): Searching for users with base [DC=MYDOMAIN,DC=example,DC=com] (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] [sdap_print_server] (0x2000): Searching 192.168.1.4:389 (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] [sdap_get_generic_ext_step] (0x0400): calling ldap_search_ext with [(&(sAMAccountName=someuser)(objectclass=user)(sAMAccountName=*)(objectSID=*))][DC=MYDOMAIN,DC=example,DC=com]. (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] [sdap_get_generic_ext_step] (0x1000): Requesting attrs: [objectClass] (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] [sdap_get_generic_ext_step] (0x1000): Requesting attrs: [sAMAccountName] (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] [sdap_get_generic_ext_step] (0x1000): Requesting attrs: [unixUserPassword] (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] [sdap_get_generic_ext_step] (0x1000): Requesting attrs: [uidNumber] (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] [sdap_get_generic_ext_step] (0x1000): Requesting attrs: [gidNumber] (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] [sdap_get_generic_ext_step] (0x1000): Requesting attrs: [gecos] (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] [sdap_get_generic_ext_step] (0x1000): Requesting attrs: [unixHomeDirectory] (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] [sdap_get_generic_ext_step] (0x1000): Requesting attrs: [loginShell] (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] [sdap_get_generic_ext_step] (0x1000): Requesting attrs: [userPrincipalName] (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] [sdap_get_generic_ext_step] (0x1000): Requesting attrs: [name] (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] [sdap_get_generic_ext_step] (0x1000): Requesting attrs: [memberOf] (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] [sdap_get_generic_ext_step] (0x1000): Requesting attrs: [objectGUID] (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] [sdap_get_generic_ext_step] (0x1000): Requesting attrs: [objectSID] (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] [sdap_get_generic_ext_step] (0x1000): Requesting attrs: [primaryGroupID] (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] [sdap_get_generic_ext_step] (0x1000): Requesting attrs: [whenChanged] (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] [sdap_get_generic_ext_step] (0x1000): Requesting attrs: [uSNChanged] (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] [sdap_get_generic_ext_step] (0x1000): Requesting attrs: [accountExpires] (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] [sdap_get_generic_ext_step] (0x1000): Requesting attrs: [userAccountControl] (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] [sdap_get_generic_ext_step] (0x1000): Requesting attrs: [userCertificate;binary] (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] [sdap_get_generic_ext_step] (0x1000): Requesting attrs: [mail] (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] [sdap_get_generic_ext_step] (0x2000): ldap_search_ext called, msgid = 26 (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] [sdap_op_add] (0x2000): New operation 26 timeout 6 (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] [sdap_process_result] (0x2000): Trace: sh[0x55ca2802d840], connected[1], ops[0x55ca28028ef0], ldap[0x55ca27fe0610] (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] [sdap_process_message] (0x4000): Message type: [LDAP_RES_SEARCH_RESULT] (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] [sdap_get_generic_op_finished] (0x0400): Search result: Referral(10), 0000202B: RefErr: DSID-0310074A, data 0, 1 access points ref 1: 'MYDOMAIN.example.com'
(2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] [sdap_get_generic_ext_add_references] (0x1000): Additional References: ldap://MYDOMAIN.example.com/DC=MYDOMAIN,DC=example,DC=com (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] [sdap_op_destructor] (0x2000): Operation 26 finished (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] [generic_ext_search_handler] (0x4000): Request included referrals which were ignored. (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] [generic_ext_search_handler] (0x4000): Ref: ldap:// MYDOMAIN.example.com/DC=MYDOMAIN,DC=example,DC=com (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] [sdap_search_user_process] (0x0400): Search for users, returned 0 results. (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] [sdap_search_user_process] (0x2000): Retrieved total 0 users (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] [sdap_id_op_done] (0x4000): releasing operation connection (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] [sysdb_search_by_name] (0x0400): No such entry (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] [sysdb_cache_search_groups] (0x2000): Search groups with filter: (&(objectCategory=group)(ghost=someuser@mydomain.example.com)) (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] [sysdb_cache_search_groups] (0x2000): No such entry (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] [sysdb_delete_user] (0x0400): Error: 2 (No such file or directory) (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] [dp_req_done] (0x0400): DP Request [Account #6104]: Request handler finished [0]: Success (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] [_dp_req_recv] (0x0400): DP Request [Account #6104]: Receiving request data. (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] [dp_req_reply_list_success] (0x0400): DP Request [Account #6104]: Finished. Success. (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] [dp_req_reply_std] (0x1000): DP Request [Account #6104]: Returning [Success]: 0,0,Success (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] [dp_table_value_destructor] (0x0400): Removing [0:1:0x0001:1::MYDOMAIN.example.com:name=someuser@mydomain.example.com] from reply table (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] [dp_req_destructor] (0x0400): DP Request [Account #6104]: Request removed. (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] [dp_req_destructor] (0x0400): Number of active DP request: 0 (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] [sdap_process_result] (0x2000): Trace: sh[0x55ca2802d840], connected[1], ops[(nil)], ldap[0x55ca27fe0610] (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] [sdap_process_result] (0x2000): Trace: end of ldap_result list
And the following corresponding messages in '/var/log/secure':
Nov 10 07:40:57 myserver sshd[755]: pam_unix(sshd:auth): check pass; user unknown Nov 10 07:40:57 myserver sshd[755]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=192.168.1.123 Nov 10 07:40:57 myserver sshd[755]: pam_sss(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=192.168.1.123 user=someuser Nov 10 07:40:57 myserver sshd[755]: pam_sss(sshd:auth): received for user someuser: 10 (User not known to the underlying authentication module) Nov 10 07:40:59 myserver sshd[735]: error: PAM: User not known to the underlying authentication module for illegal user someuser from 192.168.1.123 Nov 10 07:40:59 myserver sshd[735]: Failed keyboard-interactive/pam for invalid user someuser from 192.168.1.123 port 53300 ssh2 Nov 10 07:40:59 myserver sshd[735]: Connection closed by 192.168.1.123 port 53300 [preauth]
The effective '/etc/sssd/sssd.conf' file is as follows:
[sssd] domains = MYDOMAIN.example.com config_file_version = 2 services = nss, pam debug_level = 9
[domain/MYDOMAIN.example.com] ad_domain = MYDOMAIN.example.com krb5_realm = MYDOMAIN.EXAMPLE.COM krb5_lifetime = 10h subdomain_inherit = ignore_group_members, ldap_purge_cache_timeout ignore_group_members = true ldap_purge_cache_timeout = 0 realmd_tags = joined-with-adcli, manages-system cache_credentials = false id_provider = ad krb5_store_password_if_offline = true default_shell = /bin/bash ldap_id_mapping = true ldap_sasl_authid = MYSERVER$@MYDOMAIN.EXAMPLE.COM ldap_use_tokengroups = true use_fully_qualified_names = false fallback_homedir = /home/%d/%u access_provider = simple simple_allow_groups = linux_admins simple_allow_users = someuser, someuser2, someuser3 debug_level = 9
Running either of the following commands appears to correct the issue (until it presents again at an unpredictable time):
# systemctl restart sssd
or
# kdestroy -A ; kinit -k 'MYSERVER$@MYDOMAIN.EXAMPLE.COM'
Any assistance or insight you can offer would be greatly appreciated. We have run countless internet searches over recent weeks, as well as consulted with Red Hat Support without breakthroughs, so I decided to take this straight to the experts!
Best,
*J. Adam Craig* Lead Unix Operating Systems Analyst VCU Computer Center https://www.ucc.vcu.edu/ 804.828.4886 jacraig@vcu.edu
https://adminmicro2.questionpro.com/?t_340030260=J.%20Adam%20Craig&u_65977055=351791134 *Don't be a phishing victim -- VCU and other reputable organisations will never use email to request that you reply with your password, social security number or confidential personal information. For more details, visit **https://ts.vcu.edu/about-us/information-security/common-questions/what-is-ph... https://ts.vcu.edu/about-us/information-security/common-questions/what-is-phishing*
Hi,
according to the sssd.conf you do not have `ad_enable_gc` set so this should be `true` by default.
But in the log it contacts LDAP port: [sdap_print_server] (0x2000): Searching 192.168.1.4:389 -- IIUC, GC port should be 3269... -- what port is used when everything is working as expected?
If I understand "Referral(10), 0000202B: RefErr: DSID-0310074A, data 0, 1 access points" correctly, this specific DC "doesn't know" this user and refers to other DCs, but referrals chasing is disabled for ad provider.
This doesn't explain what happens though... Perhaps clue can be found earlier in the logs.
On Tue, Dec 1, 2020 at 4:43 PM J. Adam Craig jacraig@vcu.edu wrote:
Hello!
I am currently troubleshooting a very mysterious and difficult to tack down issue with SSSD running on RHEL/CentOS 7.x and 8.x (SSSD ver. 1.16.5 and 2.2.3). The EL 7.x and 8.x systems are attached to a Windows Active Directory domain using 'adcli'. We used this guide ( https://access.redhat.com/solutions/2653771), with some minor tweaks appropriate for our Active Directory environment and security requirements to set this up.
The vast majority of the time, the configuration works as expected. However, occasionally, we experience sporadic and temporary issues where users attempting to authenticate using valid Active Directory credentials (via SSH) are unable to login.
When the issue presents itself, if we leave an affected system alone and do nothing, the issue will eventually self-correct and users are able to authenticate as expected. We have also discovered that we can manually "fix" the issue by either (1) restarting SSSD with 'sudo systemctl restart sssd' or (2) by running a 'kdestroy'/'kinit' sequence as the 'root' user on the affected system like so:
# kdestroy -A ; kinit -k 'MYSERVER$@MYDOMAIN.EXAMPLE.COM'
At times when the issue is occurring, we observe the following messages in '/var/log/sssd/sssd_MYDOMAIN.example.com.log' (debug_level 9):
(2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] [dp_get_account_info_handler] (0x0200): Got request for [0x1][BE_REQ_USER][name=someuser@mydomain.example.com] (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] [dp_attach_req] (0x0400): DP Request [Account #6104]: New request. Flags [0x0001]. (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] [dp_attach_req] (0x0400): Number of active DP request: 1 (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] [sss_domain_get_state] (0x1000): Domain MYDOMAIN.example.com is Active (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] [sss_domain_get_state] (0x1000): Domain MYDOMAIN.example.com is Active (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] [sdap_id_op_connect_step] (0x4000): reusing cached connection (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] [sdap_search_user_next_base] (0x0400): Searching for users with base [DC=MYDOMAIN,DC=example,DC=com] (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] [sdap_print_server] (0x2000): Searching 192.168.1.4:389 (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] [sdap_get_generic_ext_step] (0x0400): calling ldap_search_ext with [(&(sAMAccountName=someuser)(objectclass=user)(sAMAccountName=*)(objectSID=*))][DC=MYDOMAIN,DC=example,DC=com]. (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] [sdap_get_generic_ext_step] (0x1000): Requesting attrs: [objectClass] (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] [sdap_get_generic_ext_step] (0x1000): Requesting attrs: [sAMAccountName] (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] [sdap_get_generic_ext_step] (0x1000): Requesting attrs: [unixUserPassword] (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] [sdap_get_generic_ext_step] (0x1000): Requesting attrs: [uidNumber] (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] [sdap_get_generic_ext_step] (0x1000): Requesting attrs: [gidNumber] (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] [sdap_get_generic_ext_step] (0x1000): Requesting attrs: [gecos] (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] [sdap_get_generic_ext_step] (0x1000): Requesting attrs: [unixHomeDirectory] (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] [sdap_get_generic_ext_step] (0x1000): Requesting attrs: [loginShell] (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] [sdap_get_generic_ext_step] (0x1000): Requesting attrs: [userPrincipalName] (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] [sdap_get_generic_ext_step] (0x1000): Requesting attrs: [name] (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] [sdap_get_generic_ext_step] (0x1000): Requesting attrs: [memberOf] (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] [sdap_get_generic_ext_step] (0x1000): Requesting attrs: [objectGUID] (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] [sdap_get_generic_ext_step] (0x1000): Requesting attrs: [objectSID] (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] [sdap_get_generic_ext_step] (0x1000): Requesting attrs: [primaryGroupID] (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] [sdap_get_generic_ext_step] (0x1000): Requesting attrs: [whenChanged] (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] [sdap_get_generic_ext_step] (0x1000): Requesting attrs: [uSNChanged] (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] [sdap_get_generic_ext_step] (0x1000): Requesting attrs: [accountExpires] (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] [sdap_get_generic_ext_step] (0x1000): Requesting attrs: [userAccountControl] (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] [sdap_get_generic_ext_step] (0x1000): Requesting attrs: [userCertificate;binary] (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] [sdap_get_generic_ext_step] (0x1000): Requesting attrs: [mail] (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] [sdap_get_generic_ext_step] (0x2000): ldap_search_ext called, msgid = 26 (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] [sdap_op_add] (0x2000): New operation 26 timeout 6 (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] [sdap_process_result] (0x2000): Trace: sh[0x55ca2802d840], connected[1], ops[0x55ca28028ef0], ldap[0x55ca27fe0610] (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] [sdap_process_message] (0x4000): Message type: [LDAP_RES_SEARCH_RESULT] (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] [sdap_get_generic_op_finished] (0x0400): Search result: Referral(10), 0000202B: RefErr: DSID-0310074A, data 0, 1 access points ref 1: 'MYDOMAIN.example.com'
(2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] [sdap_get_generic_ext_add_references] (0x1000): Additional References: ldap://MYDOMAIN.example.com/DC=MYDOMAIN,DC=example,DC=com (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] [sdap_op_destructor] (0x2000): Operation 26 finished (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] [generic_ext_search_handler] (0x4000): Request included referrals which were ignored. (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] [generic_ext_search_handler] (0x4000): Ref: ldap:// MYDOMAIN.example.com/DC=MYDOMAIN,DC=example,DC=com (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] [sdap_search_user_process] (0x0400): Search for users, returned 0 results. (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] [sdap_search_user_process] (0x2000): Retrieved total 0 users (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] [sdap_id_op_done] (0x4000): releasing operation connection (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] [sysdb_search_by_name] (0x0400): No such entry (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] [sysdb_cache_search_groups] (0x2000): Search groups with filter: (&(objectCategory=group)(ghost=someuser@mydomain.example.com)) (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] [sysdb_cache_search_groups] (0x2000): No such entry (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] [sysdb_delete_user] (0x0400): Error: 2 (No such file or directory) (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] [dp_req_done] (0x0400): DP Request [Account #6104]: Request handler finished [0]: Success (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] [_dp_req_recv] (0x0400): DP Request [Account #6104]: Receiving request data. (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] [dp_req_reply_list_success] (0x0400): DP Request [Account #6104]: Finished. Success. (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] [dp_req_reply_std] (0x1000): DP Request [Account #6104]: Returning [Success]: 0,0,Success (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] [dp_table_value_destructor] (0x0400): Removing [0:1:0x0001:1::MYDOMAIN.example.com:name=someuser@mydomain.example.com] from reply table (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] [dp_req_destructor] (0x0400): DP Request [Account #6104]: Request removed. (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] [dp_req_destructor] (0x0400): Number of active DP request: 0 (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] [sdap_process_result] (0x2000): Trace: sh[0x55ca2802d840], connected[1], ops[(nil)], ldap[0x55ca27fe0610] (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] [sdap_process_result] (0x2000): Trace: end of ldap_result list
And the following corresponding messages in '/var/log/secure':
Nov 10 07:40:57 myserver sshd[755]: pam_unix(sshd:auth): check pass; user unknown Nov 10 07:40:57 myserver sshd[755]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=192.168.1.123 Nov 10 07:40:57 myserver sshd[755]: pam_sss(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=192.168.1.123 user=someuser Nov 10 07:40:57 myserver sshd[755]: pam_sss(sshd:auth): received for user someuser: 10 (User not known to the underlying authentication module) Nov 10 07:40:59 myserver sshd[735]: error: PAM: User not known to the underlying authentication module for illegal user someuser from 192.168.1.123 Nov 10 07:40:59 myserver sshd[735]: Failed keyboard-interactive/pam for invalid user someuser from 192.168.1.123 port 53300 ssh2 Nov 10 07:40:59 myserver sshd[735]: Connection closed by 192.168.1.123 port 53300 [preauth]
The effective '/etc/sssd/sssd.conf' file is as follows:
[sssd] domains = MYDOMAIN.example.com config_file_version = 2 services = nss, pam debug_level = 9
[domain/MYDOMAIN.example.com] ad_domain = MYDOMAIN.example.com krb5_realm = MYDOMAIN.EXAMPLE.COM krb5_lifetime = 10h subdomain_inherit = ignore_group_members, ldap_purge_cache_timeout ignore_group_members = true ldap_purge_cache_timeout = 0 realmd_tags = joined-with-adcli, manages-system cache_credentials = false id_provider = ad krb5_store_password_if_offline = true default_shell = /bin/bash ldap_id_mapping = true ldap_sasl_authid = MYSERVER$@MYDOMAIN.EXAMPLE.COM ldap_use_tokengroups = true use_fully_qualified_names = false fallback_homedir = /home/%d/%u access_provider = simple simple_allow_groups = linux_admins simple_allow_users = someuser, someuser2, someuser3 debug_level = 9
Running either of the following commands appears to correct the issue (until it presents again at an unpredictable time):
# systemctl restart sssd
or
# kdestroy -A ; kinit -k 'MYSERVER$@MYDOMAIN.EXAMPLE.COM'
Any assistance or insight you can offer would be greatly appreciated. We have run countless internet searches over recent weeks, as well as consulted with Red Hat Support without breakthroughs, so I decided to take this straight to the experts!
Best,
*J. Adam Craig* Lead Unix Operating Systems Analyst VCU Computer Center https://www.ucc.vcu.edu/ 804.828.4886 jacraig@vcu.edu
https://adminmicro2.questionpro.com/?t_340030260=J.%20Adam%20Craig&u_65977055=351791134 *Don't be a phishing victim -- VCU and other reputable organisations will never use email to request that you reply with your password, social security number or confidential personal information. For more details, visit **https://ts.vcu.edu/about-us/information-security/common-questions/what-is-ph... https://ts.vcu.edu/about-us/information-security/common-questions/what-is-phishing*
sssd-users mailing list -- sssd-users@lists.fedorahosted.org To unsubscribe send an email to sssd-users-leave@lists.fedorahosted.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedorahosted.org/archives/list/sssd-users@lists.fedorahosted.o...
On Tue, Dec 01, 2020 at 07:21:15PM +0100, Alexey Tikhonov wrote:
Hi,
according to the sssd.conf you do not have `ad_enable_gc` set so this should be `true` by default.
But in the log it contacts LDAP port: [sdap_print_server] (0x2000): Searching 192.168.1.4:389 -- IIUC, GC port should be 3269... -- what port is used when everything is working as expected?
If I understand "Referral(10), 0000202B: RefErr: DSID-0310074A, data 0, 1 access points" correctly, this specific DC "doesn't know" this user and refers to other DCs, but referrals chasing is disabled for ad provider.
This doesn't explain what happens though... Perhaps clue can be found earlier in the logs.
On Tue, Dec 1, 2020 at 4:43 PM J. Adam Craig jacraig@vcu.edu wrote:
Hello!
I am currently troubleshooting a very mysterious and difficult to tack down issue with SSSD running on RHEL/CentOS 7.x and 8.x (SSSD ver. 1.16.5 and 2.2.3). The EL 7.x and 8.x systems are attached to a Windows Active Directory domain using 'adcli'. We used this guide ( https://access.redhat.com/solutions/2653771), with some minor tweaks appropriate for our Active Directory environment and security requirements to set this up.
The vast majority of the time, the configuration works as expected. However, occasionally, we experience sporadic and temporary issues where users attempting to authenticate using valid Active Directory credentials (via SSH) are unable to login.
Hi,
you might hit https://github.com/SSSD/sssd/issues/5351, the fix is currently not available in any released RHEL or CentOS version.
bye, Sumit
When the issue presents itself, if we leave an affected system alone and do nothing, the issue will eventually self-correct and users are able to authenticate as expected. We have also discovered that we can manually "fix" the issue by either (1) restarting SSSD with 'sudo systemctl restart sssd' or (2) by running a 'kdestroy'/'kinit' sequence as the 'root' user on the affected system like so:
# kdestroy -A ; kinit -k 'MYSERVER$@MYDOMAIN.EXAMPLE.COM'
At times when the issue is occurring, we observe the following messages in '/var/log/sssd/sssd_MYDOMAIN.example.com.log' (debug_level 9):
(2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] [dp_get_account_info_handler] (0x0200): Got request for [0x1][BE_REQ_USER][name=someuser@mydomain.example.com] (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] [dp_attach_req] (0x0400): DP Request [Account #6104]: New request. Flags [0x0001]. (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] [dp_attach_req] (0x0400): Number of active DP request: 1 (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] [sss_domain_get_state] (0x1000): Domain MYDOMAIN.example.com is Active (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] [sss_domain_get_state] (0x1000): Domain MYDOMAIN.example.com is Active (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] [sdap_id_op_connect_step] (0x4000): reusing cached connection (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] [sdap_search_user_next_base] (0x0400): Searching for users with base [DC=MYDOMAIN,DC=example,DC=com] (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] [sdap_print_server] (0x2000): Searching 192.168.1.4:389 (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] [sdap_get_generic_ext_step] (0x0400): calling ldap_search_ext with [(&(sAMAccountName=someuser)(objectclass=user)(sAMAccountName=*)(objectSID=*))][DC=MYDOMAIN,DC=example,DC=com]. (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] [sdap_get_generic_ext_step] (0x1000): Requesting attrs: [objectClass] (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] [sdap_get_generic_ext_step] (0x1000): Requesting attrs: [sAMAccountName] (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] [sdap_get_generic_ext_step] (0x1000): Requesting attrs: [unixUserPassword] (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] [sdap_get_generic_ext_step] (0x1000): Requesting attrs: [uidNumber] (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] [sdap_get_generic_ext_step] (0x1000): Requesting attrs: [gidNumber] (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] [sdap_get_generic_ext_step] (0x1000): Requesting attrs: [gecos] (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] [sdap_get_generic_ext_step] (0x1000): Requesting attrs: [unixHomeDirectory] (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] [sdap_get_generic_ext_step] (0x1000): Requesting attrs: [loginShell] (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] [sdap_get_generic_ext_step] (0x1000): Requesting attrs: [userPrincipalName] (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] [sdap_get_generic_ext_step] (0x1000): Requesting attrs: [name] (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] [sdap_get_generic_ext_step] (0x1000): Requesting attrs: [memberOf] (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] [sdap_get_generic_ext_step] (0x1000): Requesting attrs: [objectGUID] (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] [sdap_get_generic_ext_step] (0x1000): Requesting attrs: [objectSID] (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] [sdap_get_generic_ext_step] (0x1000): Requesting attrs: [primaryGroupID] (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] [sdap_get_generic_ext_step] (0x1000): Requesting attrs: [whenChanged] (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] [sdap_get_generic_ext_step] (0x1000): Requesting attrs: [uSNChanged] (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] [sdap_get_generic_ext_step] (0x1000): Requesting attrs: [accountExpires] (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] [sdap_get_generic_ext_step] (0x1000): Requesting attrs: [userAccountControl] (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] [sdap_get_generic_ext_step] (0x1000): Requesting attrs: [userCertificate;binary] (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] [sdap_get_generic_ext_step] (0x1000): Requesting attrs: [mail] (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] [sdap_get_generic_ext_step] (0x2000): ldap_search_ext called, msgid = 26 (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] [sdap_op_add] (0x2000): New operation 26 timeout 6 (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] [sdap_process_result] (0x2000): Trace: sh[0x55ca2802d840], connected[1], ops[0x55ca28028ef0], ldap[0x55ca27fe0610] (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] [sdap_process_message] (0x4000): Message type: [LDAP_RES_SEARCH_RESULT] (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] [sdap_get_generic_op_finished] (0x0400): Search result: Referral(10), 0000202B: RefErr: DSID-0310074A, data 0, 1 access points ref 1: 'MYDOMAIN.example.com'
(2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] [sdap_get_generic_ext_add_references] (0x1000): Additional References: ldap://MYDOMAIN.example.com/DC=MYDOMAIN,DC=example,DC=com (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] [sdap_op_destructor] (0x2000): Operation 26 finished (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] [generic_ext_search_handler] (0x4000): Request included referrals which were ignored. (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] [generic_ext_search_handler] (0x4000): Ref: ldap:// MYDOMAIN.example.com/DC=MYDOMAIN,DC=example,DC=com (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] [sdap_search_user_process] (0x0400): Search for users, returned 0 results. (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] [sdap_search_user_process] (0x2000): Retrieved total 0 users (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] [sdap_id_op_done] (0x4000): releasing operation connection (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] [sysdb_search_by_name] (0x0400): No such entry (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] [sysdb_cache_search_groups] (0x2000): Search groups with filter: (&(objectCategory=group)(ghost=someuser@mydomain.example.com)) (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] [sysdb_cache_search_groups] (0x2000): No such entry (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] [sysdb_delete_user] (0x0400): Error: 2 (No such file or directory) (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] [dp_req_done] (0x0400): DP Request [Account #6104]: Request handler finished [0]: Success (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] [_dp_req_recv] (0x0400): DP Request [Account #6104]: Receiving request data. (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] [dp_req_reply_list_success] (0x0400): DP Request [Account #6104]: Finished. Success. (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] [dp_req_reply_std] (0x1000): DP Request [Account #6104]: Returning [Success]: 0,0,Success (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] [dp_table_value_destructor] (0x0400): Removing [0:1:0x0001:1::MYDOMAIN.example.com:name=someuser@mydomain.example.com] from reply table (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] [dp_req_destructor] (0x0400): DP Request [Account #6104]: Request removed. (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] [dp_req_destructor] (0x0400): Number of active DP request: 0 (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] [sdap_process_result] (0x2000): Trace: sh[0x55ca2802d840], connected[1], ops[(nil)], ldap[0x55ca27fe0610] (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] [sdap_process_result] (0x2000): Trace: end of ldap_result list
And the following corresponding messages in '/var/log/secure':
Nov 10 07:40:57 myserver sshd[755]: pam_unix(sshd:auth): check pass; user unknown Nov 10 07:40:57 myserver sshd[755]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=192.168.1.123 Nov 10 07:40:57 myserver sshd[755]: pam_sss(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=192.168.1.123 user=someuser Nov 10 07:40:57 myserver sshd[755]: pam_sss(sshd:auth): received for user someuser: 10 (User not known to the underlying authentication module) Nov 10 07:40:59 myserver sshd[735]: error: PAM: User not known to the underlying authentication module for illegal user someuser from 192.168.1.123 Nov 10 07:40:59 myserver sshd[735]: Failed keyboard-interactive/pam for invalid user someuser from 192.168.1.123 port 53300 ssh2 Nov 10 07:40:59 myserver sshd[735]: Connection closed by 192.168.1.123 port 53300 [preauth]
The effective '/etc/sssd/sssd.conf' file is as follows:
[sssd] domains = MYDOMAIN.example.com config_file_version = 2 services = nss, pam debug_level = 9
[domain/MYDOMAIN.example.com] ad_domain = MYDOMAIN.example.com krb5_realm = MYDOMAIN.EXAMPLE.COM krb5_lifetime = 10h subdomain_inherit = ignore_group_members, ldap_purge_cache_timeout ignore_group_members = true ldap_purge_cache_timeout = 0 realmd_tags = joined-with-adcli, manages-system cache_credentials = false id_provider = ad krb5_store_password_if_offline = true default_shell = /bin/bash ldap_id_mapping = true ldap_sasl_authid = MYSERVER$@MYDOMAIN.EXAMPLE.COM ldap_use_tokengroups = true use_fully_qualified_names = false fallback_homedir = /home/%d/%u access_provider = simple simple_allow_groups = linux_admins simple_allow_users = someuser, someuser2, someuser3 debug_level = 9
Running either of the following commands appears to correct the issue (until it presents again at an unpredictable time):
# systemctl restart sssd
or
# kdestroy -A ; kinit -k 'MYSERVER$@MYDOMAIN.EXAMPLE.COM'
Any assistance or insight you can offer would be greatly appreciated. We have run countless internet searches over recent weeks, as well as consulted with Red Hat Support without breakthroughs, so I decided to take this straight to the experts!
Best,
*J. Adam Craig* Lead Unix Operating Systems Analyst VCU Computer Center https://www.ucc.vcu.edu/ 804.828.4886 jacraig@vcu.edu
https://adminmicro2.questionpro.com/?t_340030260=J.%20Adam%20Craig&u_65977055=351791134 *Don't be a phishing victim -- VCU and other reputable organisations will never use email to request that you reply with your password, social security number or confidential personal information. For more details, visit **https://ts.vcu.edu/about-us/information-security/common-questions/what-is-ph... https://ts.vcu.edu/about-us/information-security/common-questions/what-is-phishing*
sssd-users mailing list -- sssd-users@lists.fedorahosted.org To unsubscribe send an email to sssd-users-leave@lists.fedorahosted.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedorahosted.org/archives/list/sssd-users@lists.fedorahosted.o...
sssd-users mailing list -- sssd-users@lists.fedorahosted.org To unsubscribe send an email to sssd-users-leave@lists.fedorahosted.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedorahosted.org/archives/list/sssd-users@lists.fedorahosted.o...
I am currently out of the office, but plan to return to my desk on Wednesday at 7am.
If you require assistance with a server that is hosted by the VCU Computer Center, please submit a support ticket https://itsupport.vcu.edu/CherwellPortal.
If this is an emergency, please contact the Network Operations Center at (804) 828-1802.
I apologise for any inconvenience.
Make it be a great day,
*J. Adam Craig* Lead Unix Operating Systems Analyst VCU Computer Center https://www.ucc.vcu.edu/ 804.828.4886 jacraig@vcu.edu
https://adminmicro2.questionpro.com/?t_340030260=J.%20Adam%20Craig&u_65977055=351791134 *Don't be a phishing victim -- VCU and other reputable organisations will never use email to request that you reply with your password, social security number or confidential personal information. For more details, visit https://ts.vcu.edu/about-us/information-security/common-questions/what-is-ph... https://ts.vcu.edu/about-us/information-security/common-questions/what-is-phishing*
Thanks, all, for your assistance with this issue!
Your notes have prompted some further investigation on my end.
1. In response to Alexey's question, I can confirm that, when lookups are successful, port 389 is also used.
2. After conversing with our Active Directory team, I've learned that, in at least one case of this issue occurring (the one cited above), the IP address used for the lookup was actually not a domain controller at all, but was rather one of our Active Directory DNS servers. This leads me to wonder why SSSD could be attempting to use an Active Directory DNS server to perform lookups in the first place.
When I run an 'nslookup' against our Active Directory domain name, only IP addresses of domain controllers are returned, so the issue doesn't appear to be originating from there.
What other mechanisms (besides DNS) might SSSD be using to determine the domain controllers it should use?
In our environment, our AD DNS servers live in a root domain, with users, etc. residing in a child domain.
Thanks again for any insight and assistance.
*J. Adam Craig* Lead Unix Operating Systems Analyst VCU Computer Center https://www.ucc.vcu.edu/ 804.828.4886 jacraig@vcu.edu
https://adminmicro2.questionpro.com/?t_340030260=J.%20Adam%20Craig&u_65977055=351791134 *Don't be a phishing victim -- VCU and other reputable organisations will never use email to request that you reply with your password, social security number or confidential personal information. For more details, visit **https://ts.vcu.edu/about-us/information-security/common-questions/what-is-ph... https://ts.vcu.edu/about-us/information-security/common-questions/what-is-phishing*
On Wed, Dec 2, 2020 at 2:18 AM Sumit Bose sbose@redhat.com wrote:
On Tue, Dec 01, 2020 at 07:21:15PM +0100, Alexey Tikhonov wrote:
Hi,
according to the sssd.conf you do not have `ad_enable_gc` set so this should be `true` by default.
But in the log it contacts LDAP port: [sdap_print_server] (0x2000): Searching 192.168.1.4:389 -- IIUC, GC
port
should be 3269... -- what port is used when everything is working as expected?
If I understand "Referral(10), 0000202B: RefErr: DSID-0310074A, data 0, 1 access points" correctly, this specific DC "doesn't know" this user and refers to other DCs, but referrals chasing is disabled for ad provider.
This doesn't explain what happens though... Perhaps clue can be found earlier in the logs.
On Tue, Dec 1, 2020 at 4:43 PM J. Adam Craig jacraig@vcu.edu wrote:
Hello!
I am currently troubleshooting a very mysterious and difficult to tack down issue with SSSD running on RHEL/CentOS 7.x and 8.x (SSSD ver.
1.16.5
and 2.2.3). The EL 7.x and 8.x systems are attached to a Windows
Active
Directory domain using 'adcli'. We used this guide ( https://access.redhat.com/solutions/2653771), with some minor tweaks appropriate for our Active Directory environment and security
requirements
to set this up.
The vast majority of the time, the configuration works as expected. However, occasionally, we experience sporadic and temporary issues
where
users attempting to authenticate using valid Active Directory
credentials
(via SSH) are unable to login.
Hi,
you might hit https://github.com/SSSD/sssd/issues/5351, the fix is currently not available in any released RHEL or CentOS version.
bye, Sumit
When the issue presents itself, if we leave an affected system alone
and
do nothing, the issue will eventually self-correct and users are able
to
authenticate as expected. We have also discovered that we can manually "fix" the issue by either (1) restarting SSSD with 'sudo systemctl
restart
sssd' or (2) by running a 'kdestroy'/'kinit' sequence as the 'root'
user on
the affected system like so:
# kdestroy -A ; kinit -k 'MYSERVER$@MYDOMAIN.EXAMPLE.COM'
At times when the issue is occurring, we observe the following
messages in
'/var/log/sssd/sssd_MYDOMAIN.example.com.log' (debug_level 9):
(2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] [dp_get_account_info_handler] (0x0200): Got request for [0x1][BE_REQ_USER][name=someuser@mydomain.example.com] (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] [dp_attach_req] (0x0400): DP Request [Account #6104]: New request. Flags [0x0001]. (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] [dp_attach_req] (0x0400): Number of active DP request: 1 (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]]
[sss_domain_get_state]
(0x1000): Domain MYDOMAIN.example.com is Active (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]]
[sss_domain_get_state]
(0x1000): Domain MYDOMAIN.example.com is Active (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] [sdap_id_op_connect_step] (0x4000): reusing cached connection (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] [sdap_search_user_next_base] (0x0400): Searching for users with base [DC=MYDOMAIN,DC=example,DC=com] (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] [sdap_print_server] (0x2000): Searching 192.168.1.4:389 (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] [sdap_get_generic_ext_step] (0x0400): calling ldap_search_ext with
[(&(sAMAccountName=someuser)(objectclass=user)(sAMAccountName=*)(objectSID=*))][DC=MYDOMAIN,DC=example,DC=com].
(2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] [sdap_get_generic_ext_step] (0x1000): Requesting attrs: [objectClass] (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] [sdap_get_generic_ext_step] (0x1000): Requesting attrs:
[sAMAccountName]
(2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] [sdap_get_generic_ext_step] (0x1000): Requesting attrs:
[unixUserPassword]
(2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] [sdap_get_generic_ext_step] (0x1000): Requesting attrs: [uidNumber] (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] [sdap_get_generic_ext_step] (0x1000): Requesting attrs: [gidNumber] (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] [sdap_get_generic_ext_step] (0x1000): Requesting attrs: [gecos] (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] [sdap_get_generic_ext_step] (0x1000): Requesting attrs:
[unixHomeDirectory]
(2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] [sdap_get_generic_ext_step] (0x1000): Requesting attrs: [loginShell] (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] [sdap_get_generic_ext_step] (0x1000): Requesting attrs:
[userPrincipalName]
(2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] [sdap_get_generic_ext_step] (0x1000): Requesting attrs: [name] (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] [sdap_get_generic_ext_step] (0x1000): Requesting attrs: [memberOf] (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] [sdap_get_generic_ext_step] (0x1000): Requesting attrs: [objectGUID] (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] [sdap_get_generic_ext_step] (0x1000): Requesting attrs: [objectSID] (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] [sdap_get_generic_ext_step] (0x1000): Requesting attrs:
[primaryGroupID]
(2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] [sdap_get_generic_ext_step] (0x1000): Requesting attrs: [whenChanged] (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] [sdap_get_generic_ext_step] (0x1000): Requesting attrs: [uSNChanged] (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] [sdap_get_generic_ext_step] (0x1000): Requesting attrs:
[accountExpires]
(2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] [sdap_get_generic_ext_step] (0x1000): Requesting attrs:
[userAccountControl]
(2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] [sdap_get_generic_ext_step] (0x1000): Requesting attrs: [userCertificate;binary] (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] [sdap_get_generic_ext_step] (0x1000): Requesting attrs: [mail] (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] [sdap_get_generic_ext_step] (0x2000): ldap_search_ext called, msgid =
26
(2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] [sdap_op_add]
(0x2000):
New operation 26 timeout 6 (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]]
[sdap_process_result]
(0x2000): Trace: sh[0x55ca2802d840], connected[1], ops[0x55ca28028ef0], ldap[0x55ca27fe0610] (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]]
[sdap_process_message]
(0x4000): Message type: [LDAP_RES_SEARCH_RESULT] (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] [sdap_get_generic_op_finished] (0x0400): Search result: Referral(10), 0000202B: RefErr: DSID-0310074A, data 0, 1 access points ref 1: 'MYDOMAIN.example.com'
(2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] [sdap_get_generic_ext_add_references] (0x1000): Additional References: ldap://MYDOMAIN.example.com/DC=MYDOMAIN,DC=example,DC=com (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] [sdap_op_destructor] (0x2000): Operation 26 finished (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] [generic_ext_search_handler] (0x4000): Request included referrals which were ignored. (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] [generic_ext_search_handler] (0x4000): Ref: ldap:// MYDOMAIN.example.com/DC=MYDOMAIN,DC=example,DC=com (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] [sdap_search_user_process] (0x0400): Search for users, returned 0
results.
(2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] [sdap_search_user_process] (0x2000): Retrieved total 0 users (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] [sdap_id_op_done] (0x4000): releasing operation connection (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]]
[sysdb_search_by_name]
(0x0400): No such entry (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] [sysdb_cache_search_groups] (0x2000): Search groups with filter: (&(objectCategory=group)(ghost=someuser@mydomain.example.com)) (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] [sysdb_cache_search_groups] (0x2000): No such entry (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] [sysdb_delete_user] (0x0400): Error: 2 (No such file or directory) (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] [dp_req_done]
(0x0400):
DP Request [Account #6104]: Request handler finished [0]: Success (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] [_dp_req_recv] (0x0400): DP Request [Account #6104]: Receiving request data. (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] [dp_req_reply_list_success] (0x0400): DP Request [Account #6104]:
Finished.
Success. (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] [dp_req_reply_std] (0x1000): DP Request [Account #6104]: Returning [Success]: 0,0,Success (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] [dp_table_value_destructor] (0x0400): Removing [0:1:0x0001:1::MYDOMAIN.example.com:name=someuser@mydomain.example.com
]
from reply table (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] [dp_req_destructor] (0x0400): DP Request [Account #6104]: Request removed. (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] [dp_req_destructor] (0x0400): Number of active DP request: 0 (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]]
[sdap_process_result]
(0x2000): Trace: sh[0x55ca2802d840], connected[1], ops[(nil)], ldap[0x55ca27fe0610] (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]]
[sdap_process_result]
(0x2000): Trace: end of ldap_result list
And the following corresponding messages in '/var/log/secure':
Nov 10 07:40:57 myserver sshd[755]: pam_unix(sshd:auth): check pass;
user
unknown Nov 10 07:40:57 myserver sshd[755]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=192.168.1.123 Nov 10 07:40:57 myserver sshd[755]: pam_sss(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=192.168.1.123 user=someuser Nov 10 07:40:57 myserver sshd[755]: pam_sss(sshd:auth): received for
user
someuser: 10 (User not known to the underlying authentication module) Nov 10 07:40:59 myserver sshd[735]: error: PAM: User not known to the underlying authentication module for illegal user someuser from 192.168.1.123 Nov 10 07:40:59 myserver sshd[735]: Failed keyboard-interactive/pam for invalid user someuser from 192.168.1.123 port 53300 ssh2 Nov 10 07:40:59 myserver sshd[735]: Connection closed by 192.168.1.123 port 53300 [preauth]
The effective '/etc/sssd/sssd.conf' file is as follows:
[sssd] domains = MYDOMAIN.example.com config_file_version = 2 services = nss, pam debug_level = 9
[domain/MYDOMAIN.example.com] ad_domain = MYDOMAIN.example.com krb5_realm = MYDOMAIN.EXAMPLE.COM krb5_lifetime = 10h subdomain_inherit = ignore_group_members, ldap_purge_cache_timeout ignore_group_members = true ldap_purge_cache_timeout = 0 realmd_tags = joined-with-adcli, manages-system cache_credentials = false id_provider = ad krb5_store_password_if_offline = true default_shell = /bin/bash ldap_id_mapping = true ldap_sasl_authid = MYSERVER$@MYDOMAIN.EXAMPLE.COM ldap_use_tokengroups = true use_fully_qualified_names = false fallback_homedir = /home/%d/%u access_provider = simple simple_allow_groups = linux_admins simple_allow_users = someuser, someuser2, someuser3 debug_level = 9
Running either of the following commands appears to correct the issue (until it presents again at an unpredictable time):
# systemctl restart sssd
or
# kdestroy -A ; kinit -k 'MYSERVER$@MYDOMAIN.EXAMPLE.COM'
Any assistance or insight you can offer would be greatly appreciated.
We
have run countless internet searches over recent weeks, as well as consulted with Red Hat Support without breakthroughs, so I decided to
take
this straight to the experts!
Best,
*J. Adam Craig* Lead Unix Operating Systems Analyst VCU Computer Center https://www.ucc.vcu.edu/ 804.828.4886 jacraig@vcu.edu
<
https://adminmicro2.questionpro.com/?t_340030260=J.%20Adam%20Craig&u_659...
*Don't be a phishing victim -- VCU and other reputable organisations
will
never use email to request that you reply with your password, social security number or confidential personal information. For more
details,
visit **
https://ts.vcu.edu/about-us/information-security/common-questions/what-is-ph...
<
https://ts.vcu.edu/about-us/information-security/common-questions/what-is-ph...
sssd-users mailing list -- sssd-users@lists.fedorahosted.org To unsubscribe send an email to
sssd-users-leave@lists.fedorahosted.org
Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines:
https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives:
https://lists.fedorahosted.org/archives/list/sssd-users@lists.fedorahosted.o...
sssd-users mailing list -- sssd-users@lists.fedorahosted.org To unsubscribe send an email to sssd-users-leave@lists.fedorahosted.org Fedora Code of Conduct:
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives:
https://lists.fedorahosted.org/archives/list/sssd-users@lists.fedorahosted.o... _______________________________________________ sssd-users mailing list -- sssd-users@lists.fedorahosted.org To unsubscribe send an email to sssd-users-leave@lists.fedorahosted.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedorahosted.org/archives/list/sssd-users@lists.fedorahosted.o...
On Wed, Dec 02, 2020 at 03:04:14PM -0500, J. Adam Craig wrote:
Thanks, all, for your assistance with this issue!
Your notes have prompted some further investigation on my end.
- In response to Alexey's question, I can confirm that, when lookups
are successful, port 389 is also used.
- After conversing with our Active Directory team, I've learned that,
in at least one case of this issue occurring (the one cited above), the IP address used for the lookup was actually not a domain controller at all, but was rather one of our Active Directory DNS servers. This leads me to wonder why SSSD could be attempting to use an Active Directory DNS server to perform lookups in the first place.
When I run an 'nslookup' against our Active Directory domain name, only IP addresses of domain controllers are returned, so the issue doesn't appear to be originating from there.
Hi,
what kind of lookup did you do with nslookup? Typically SSSD does a SRV lookup for '_ldap._tcp.ad.domain.name' to find the domain controllers of a given domain. But it might also try to find site specific DC with '_ldap._tcp.sitename,_sites.ad.domain.name'.
If you search in the logs where the IP address or the related DNS name is recorded first this should help to understand by which request this host was found.
What other mechanisms (besides DNS) might SSSD be using to determine the domain controllers it should use?
No, the other way is to specific the DCs in sssd.conf.
In our environment, our AD DNS servers live in a root domain, with users, etc. residing in a child domain.
I guess the DNS servers are domain controllers for the root domain as well? If the one wrongly used a global catalog server as well? If that's the case you most probably hit the bug I mentioned earlier.
bye, Sumit
Thanks again for any insight and assistance.
*J. Adam Craig* Lead Unix Operating Systems Analyst VCU Computer Center https://www.ucc.vcu.edu/ 804.828.4886 jacraig@vcu.edu
https://adminmicro2.questionpro.com/?t_340030260=J.%20Adam%20Craig&u_65977055=351791134 *Don't be a phishing victim -- VCU and other reputable organisations will never use email to request that you reply with your password, social security number or confidential personal information. For more details, visit **https://ts.vcu.edu/about-us/information-security/common-questions/what-is-ph... https://ts.vcu.edu/about-us/information-security/common-questions/what-is-phishing*
On Wed, Dec 2, 2020 at 2:18 AM Sumit Bose sbose@redhat.com wrote:
On Tue, Dec 01, 2020 at 07:21:15PM +0100, Alexey Tikhonov wrote:
Hi,
according to the sssd.conf you do not have `ad_enable_gc` set so this should be `true` by default.
But in the log it contacts LDAP port: [sdap_print_server] (0x2000): Searching 192.168.1.4:389 -- IIUC, GC
port
should be 3269... -- what port is used when everything is working as expected?
If I understand "Referral(10), 0000202B: RefErr: DSID-0310074A, data 0, 1 access points" correctly, this specific DC "doesn't know" this user and refers to other DCs, but referrals chasing is disabled for ad provider.
This doesn't explain what happens though... Perhaps clue can be found earlier in the logs.
On Tue, Dec 1, 2020 at 4:43 PM J. Adam Craig jacraig@vcu.edu wrote:
Hello!
I am currently troubleshooting a very mysterious and difficult to tack down issue with SSSD running on RHEL/CentOS 7.x and 8.x (SSSD ver.
1.16.5
and 2.2.3). The EL 7.x and 8.x systems are attached to a Windows
Active
Directory domain using 'adcli'. We used this guide ( https://access.redhat.com/solutions/2653771), with some minor tweaks appropriate for our Active Directory environment and security
requirements
to set this up.
The vast majority of the time, the configuration works as expected. However, occasionally, we experience sporadic and temporary issues
where
users attempting to authenticate using valid Active Directory
credentials
(via SSH) are unable to login.
Hi,
you might hit https://github.com/SSSD/sssd/issues/5351, the fix is currently not available in any released RHEL or CentOS version.
bye, Sumit
When the issue presents itself, if we leave an affected system alone
and
do nothing, the issue will eventually self-correct and users are able
to
authenticate as expected. We have also discovered that we can manually "fix" the issue by either (1) restarting SSSD with 'sudo systemctl
restart
sssd' or (2) by running a 'kdestroy'/'kinit' sequence as the 'root'
user on
the affected system like so:
# kdestroy -A ; kinit -k 'MYSERVER$@MYDOMAIN.EXAMPLE.COM'
At times when the issue is occurring, we observe the following
messages in
'/var/log/sssd/sssd_MYDOMAIN.example.com.log' (debug_level 9):
(2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] [dp_get_account_info_handler] (0x0200): Got request for [0x1][BE_REQ_USER][name=someuser@mydomain.example.com] (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] [dp_attach_req] (0x0400): DP Request [Account #6104]: New request. Flags [0x0001]. (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] [dp_attach_req] (0x0400): Number of active DP request: 1 (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]]
[sss_domain_get_state]
(0x1000): Domain MYDOMAIN.example.com is Active (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]]
[sss_domain_get_state]
(0x1000): Domain MYDOMAIN.example.com is Active (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] [sdap_id_op_connect_step] (0x4000): reusing cached connection (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] [sdap_search_user_next_base] (0x0400): Searching for users with base [DC=MYDOMAIN,DC=example,DC=com] (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] [sdap_print_server] (0x2000): Searching 192.168.1.4:389 (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] [sdap_get_generic_ext_step] (0x0400): calling ldap_search_ext with
[(&(sAMAccountName=someuser)(objectclass=user)(sAMAccountName=*)(objectSID=*))][DC=MYDOMAIN,DC=example,DC=com].
(2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] [sdap_get_generic_ext_step] (0x1000): Requesting attrs: [objectClass] (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] [sdap_get_generic_ext_step] (0x1000): Requesting attrs:
[sAMAccountName]
(2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] [sdap_get_generic_ext_step] (0x1000): Requesting attrs:
[unixUserPassword]
(2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] [sdap_get_generic_ext_step] (0x1000): Requesting attrs: [uidNumber] (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] [sdap_get_generic_ext_step] (0x1000): Requesting attrs: [gidNumber] (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] [sdap_get_generic_ext_step] (0x1000): Requesting attrs: [gecos] (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] [sdap_get_generic_ext_step] (0x1000): Requesting attrs:
[unixHomeDirectory]
(2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] [sdap_get_generic_ext_step] (0x1000): Requesting attrs: [loginShell] (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] [sdap_get_generic_ext_step] (0x1000): Requesting attrs:
[userPrincipalName]
(2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] [sdap_get_generic_ext_step] (0x1000): Requesting attrs: [name] (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] [sdap_get_generic_ext_step] (0x1000): Requesting attrs: [memberOf] (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] [sdap_get_generic_ext_step] (0x1000): Requesting attrs: [objectGUID] (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] [sdap_get_generic_ext_step] (0x1000): Requesting attrs: [objectSID] (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] [sdap_get_generic_ext_step] (0x1000): Requesting attrs:
[primaryGroupID]
(2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] [sdap_get_generic_ext_step] (0x1000): Requesting attrs: [whenChanged] (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] [sdap_get_generic_ext_step] (0x1000): Requesting attrs: [uSNChanged] (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] [sdap_get_generic_ext_step] (0x1000): Requesting attrs:
[accountExpires]
(2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] [sdap_get_generic_ext_step] (0x1000): Requesting attrs:
[userAccountControl]
(2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] [sdap_get_generic_ext_step] (0x1000): Requesting attrs: [userCertificate;binary] (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] [sdap_get_generic_ext_step] (0x1000): Requesting attrs: [mail] (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] [sdap_get_generic_ext_step] (0x2000): ldap_search_ext called, msgid =
26
(2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] [sdap_op_add]
(0x2000):
New operation 26 timeout 6 (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]]
[sdap_process_result]
(0x2000): Trace: sh[0x55ca2802d840], connected[1], ops[0x55ca28028ef0], ldap[0x55ca27fe0610] (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]]
[sdap_process_message]
(0x4000): Message type: [LDAP_RES_SEARCH_RESULT] (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] [sdap_get_generic_op_finished] (0x0400): Search result: Referral(10), 0000202B: RefErr: DSID-0310074A, data 0, 1 access points ref 1: 'MYDOMAIN.example.com'
(2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] [sdap_get_generic_ext_add_references] (0x1000): Additional References: ldap://MYDOMAIN.example.com/DC=MYDOMAIN,DC=example,DC=com (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] [sdap_op_destructor] (0x2000): Operation 26 finished (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] [generic_ext_search_handler] (0x4000): Request included referrals which were ignored. (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] [generic_ext_search_handler] (0x4000): Ref: ldap:// MYDOMAIN.example.com/DC=MYDOMAIN,DC=example,DC=com (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] [sdap_search_user_process] (0x0400): Search for users, returned 0
results.
(2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] [sdap_search_user_process] (0x2000): Retrieved total 0 users (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] [sdap_id_op_done] (0x4000): releasing operation connection (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]]
[sysdb_search_by_name]
(0x0400): No such entry (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] [sysdb_cache_search_groups] (0x2000): Search groups with filter: (&(objectCategory=group)(ghost=someuser@mydomain.example.com)) (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] [sysdb_cache_search_groups] (0x2000): No such entry (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] [sysdb_delete_user] (0x0400): Error: 2 (No such file or directory) (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] [dp_req_done]
(0x0400):
DP Request [Account #6104]: Request handler finished [0]: Success (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] [_dp_req_recv] (0x0400): DP Request [Account #6104]: Receiving request data. (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] [dp_req_reply_list_success] (0x0400): DP Request [Account #6104]:
Finished.
Success. (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] [dp_req_reply_std] (0x1000): DP Request [Account #6104]: Returning [Success]: 0,0,Success (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] [dp_table_value_destructor] (0x0400): Removing [0:1:0x0001:1::MYDOMAIN.example.com:name=someuser@mydomain.example.com
]
from reply table (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] [dp_req_destructor] (0x0400): DP Request [Account #6104]: Request removed. (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] [dp_req_destructor] (0x0400): Number of active DP request: 0 (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]]
[sdap_process_result]
(0x2000): Trace: sh[0x55ca2802d840], connected[1], ops[(nil)], ldap[0x55ca27fe0610] (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]]
[sdap_process_result]
(0x2000): Trace: end of ldap_result list
And the following corresponding messages in '/var/log/secure':
Nov 10 07:40:57 myserver sshd[755]: pam_unix(sshd:auth): check pass;
user
unknown Nov 10 07:40:57 myserver sshd[755]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=192.168.1.123 Nov 10 07:40:57 myserver sshd[755]: pam_sss(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=192.168.1.123 user=someuser Nov 10 07:40:57 myserver sshd[755]: pam_sss(sshd:auth): received for
user
someuser: 10 (User not known to the underlying authentication module) Nov 10 07:40:59 myserver sshd[735]: error: PAM: User not known to the underlying authentication module for illegal user someuser from 192.168.1.123 Nov 10 07:40:59 myserver sshd[735]: Failed keyboard-interactive/pam for invalid user someuser from 192.168.1.123 port 53300 ssh2 Nov 10 07:40:59 myserver sshd[735]: Connection closed by 192.168.1.123 port 53300 [preauth]
The effective '/etc/sssd/sssd.conf' file is as follows:
[sssd] domains = MYDOMAIN.example.com config_file_version = 2 services = nss, pam debug_level = 9
[domain/MYDOMAIN.example.com] ad_domain = MYDOMAIN.example.com krb5_realm = MYDOMAIN.EXAMPLE.COM krb5_lifetime = 10h subdomain_inherit = ignore_group_members, ldap_purge_cache_timeout ignore_group_members = true ldap_purge_cache_timeout = 0 realmd_tags = joined-with-adcli, manages-system cache_credentials = false id_provider = ad krb5_store_password_if_offline = true default_shell = /bin/bash ldap_id_mapping = true ldap_sasl_authid = MYSERVER$@MYDOMAIN.EXAMPLE.COM ldap_use_tokengroups = true use_fully_qualified_names = false fallback_homedir = /home/%d/%u access_provider = simple simple_allow_groups = linux_admins simple_allow_users = someuser, someuser2, someuser3 debug_level = 9
Running either of the following commands appears to correct the issue (until it presents again at an unpredictable time):
# systemctl restart sssd
or
# kdestroy -A ; kinit -k 'MYSERVER$@MYDOMAIN.EXAMPLE.COM'
Any assistance or insight you can offer would be greatly appreciated.
We
have run countless internet searches over recent weeks, as well as consulted with Red Hat Support without breakthroughs, so I decided to
take
this straight to the experts!
Best,
*J. Adam Craig* Lead Unix Operating Systems Analyst VCU Computer Center https://www.ucc.vcu.edu/ 804.828.4886 jacraig@vcu.edu
<
https://adminmicro2.questionpro.com/?t_340030260=J.%20Adam%20Craig&u_659...
*Don't be a phishing victim -- VCU and other reputable organisations
will
never use email to request that you reply with your password, social security number or confidential personal information. For more
details,
visit **
https://ts.vcu.edu/about-us/information-security/common-questions/what-is-ph...
<
https://ts.vcu.edu/about-us/information-security/common-questions/what-is-ph...
sssd-users mailing list -- sssd-users@lists.fedorahosted.org To unsubscribe send an email to
sssd-users-leave@lists.fedorahosted.org
Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines:
https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives:
https://lists.fedorahosted.org/archives/list/sssd-users@lists.fedorahosted.o...
sssd-users mailing list -- sssd-users@lists.fedorahosted.org To unsubscribe send an email to sssd-users-leave@lists.fedorahosted.org Fedora Code of Conduct:
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives:
https://lists.fedorahosted.org/archives/list/sssd-users@lists.fedorahosted.o... _______________________________________________ sssd-users mailing list -- sssd-users@lists.fedorahosted.org To unsubscribe send an email to sssd-users-leave@lists.fedorahosted.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedorahosted.org/archives/list/sssd-users@lists.fedorahosted.o...
sssd-users mailing list -- sssd-users@lists.fedorahosted.org To unsubscribe send an email to sssd-users-leave@lists.fedorahosted.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedorahosted.org/archives/list/sssd-users@lists.fedorahosted.o...
Thanks! This is, once again, very helpful!
I did some searching through the logs, as per your suggestion, and came up with the SRV queries that include the (problematic) AD DNS servers in the parent domain (which, as you suspected, are also Global Catalog servers).
So, now I'm guessing that the next course of action, in view of the fact that the bug you cited is not yet resolved in EL SSSD packages, is to try to find a workaround that will result in queries only being made against the child DCs.
Currently, I'm considering the following options, but would be eager to learn if there is a preferred/better approach:
1. One thought is to set 'ad_enable_gc = false' in '/etc/sssd/sssd.conf'., but I am not sure if this would resolve the bug or expose us to other problems that I'm not aware of. I currently have this setting in place on a test system, and am monitoring the SSSD logs for any of the SRV queries that include the problematic servers in the parent domain. I'm currently not seeing any of those entries, but want to continue watching this for a little while longer before I draw any conclusions.
2. Explicitly listing the child DCs in '/etc/sssd/sssd.conf'. The biggest drawback I see to this approach is that we'd have to manually adjust this list every time we add or remove a DC from the child domain. We use automation tools, naturally, so this wouldn't be terribly difficult to accomplish, but it would be tedious.
Best,
*J. Adam Craig* Lead Unix Operating Systems Analyst VCU Computer Center https://www.ucc.vcu.edu/ 804.828.4886 jacraig@vcu.edu
https://adminmicro2.questionpro.com/?t_340030260=J.%20Adam%20Craig&u_65977055=351791134 *Don't be a phishing victim -- VCU and other reputable organisations will never use email to request that you reply with your password, social security number or confidential personal information. For more details, visit **https://ts.vcu.edu/about-us/information-security/common-questions/what-is-ph... https://ts.vcu.edu/about-us/information-security/common-questions/what-is-phishing*
On Thu, Dec 3, 2020 at 5:19 AM Sumit Bose sbose@redhat.com wrote:
On Wed, Dec 02, 2020 at 03:04:14PM -0500, J. Adam Craig wrote:
Thanks, all, for your assistance with this issue!
Your notes have prompted some further investigation on my end.
- In response to Alexey's question, I can confirm that, when lookups
are successful, port 389 is also used.
- After conversing with our Active Directory team, I've learned that,
in at least one case of this issue occurring (the one cited above),
the IP
address used for the lookup was actually not a domain controller at
all,
but was rather one of our Active Directory DNS servers. This leads
me to
wonder why SSSD could be attempting to use an Active Directory DNS
server
to perform lookups in the first place.
When I run an 'nslookup' against our Active Directory domain name,
only
IP addresses of domain controllers are returned, so the issue doesn't appear to be originating from there.
Hi,
what kind of lookup did you do with nslookup? Typically SSSD does a SRV lookup for '_ldap._tcp.ad.domain.name' to find the domain controllers of a given domain. But it might also try to find site specific DC with '_ldap._tcp.sitename,_sites.ad.domain.name'.
If you search in the logs where the IP address or the related DNS name is recorded first this should help to understand by which request this host was found.
What other mechanisms (besides DNS) might SSSD be using to determine
the
domain controllers it should use?
No, the other way is to specific the DCs in sssd.conf.
In our environment, our AD DNS servers live in a root domain, with users, etc. residing in a child domain.
I guess the DNS servers are domain controllers for the root domain as well? If the one wrongly used a global catalog server as well? If that's the case you most probably hit the bug I mentioned earlier.
bye, Sumit
Thanks again for any insight and assistance.
*J. Adam Craig* Lead Unix Operating Systems Analyst VCU Computer Center https://www.ucc.vcu.edu/ 804.828.4886 jacraig@vcu.edu
<
https://adminmicro2.questionpro.com/?t_340030260=J.%20Adam%20Craig&u_659...
*Don't be a phishing victim -- VCU and other reputable organisations will never use email to request that you reply with your password, social security number or confidential personal information. For more details, visit **
https://ts.vcu.edu/about-us/information-security/common-questions/what-is-ph...
<
https://ts.vcu.edu/about-us/information-security/common-questions/what-is-ph...
On Wed, Dec 2, 2020 at 2:18 AM Sumit Bose sbose@redhat.com wrote:
On Tue, Dec 01, 2020 at 07:21:15PM +0100, Alexey Tikhonov wrote:
Hi,
according to the sssd.conf you do not have `ad_enable_gc` set so this should be `true` by default.
But in the log it contacts LDAP port: [sdap_print_server] (0x2000): Searching 192.168.1.4:389 -- IIUC,
GC
port
should be 3269... -- what port is used when everything is working as expected?
If I understand "Referral(10), 0000202B: RefErr: DSID-0310074A, data
0, 1
access points" correctly, this specific DC "doesn't know" this user
and
refers to other DCs, but referrals chasing is disabled for ad
provider.
This doesn't explain what happens though... Perhaps clue can be found earlier in the logs.
On Tue, Dec 1, 2020 at 4:43 PM J. Adam Craig jacraig@vcu.edu
wrote:
Hello!
I am currently troubleshooting a very mysterious and difficult to
tack
down issue with SSSD running on RHEL/CentOS 7.x and 8.x (SSSD ver.
1.16.5
and 2.2.3). The EL 7.x and 8.x systems are attached to a Windows
Active
Directory domain using 'adcli'. We used this guide ( https://access.redhat.com/solutions/2653771), with some minor
tweaks
appropriate for our Active Directory environment and security
requirements
to set this up.
The vast majority of the time, the configuration works as expected. However, occasionally, we experience sporadic and temporary issues
where
users attempting to authenticate using valid Active Directory
credentials
(via SSH) are unable to login.
Hi,
you might hit https://github.com/SSSD/sssd/issues/5351, the fix is currently not available in any released RHEL or CentOS version.
bye, Sumit
When the issue presents itself, if we leave an affected system
alone
and
do nothing, the issue will eventually self-correct and users are
able
to
authenticate as expected. We have also discovered that we can
manually
"fix" the issue by either (1) restarting SSSD with 'sudo systemctl
restart
sssd' or (2) by running a 'kdestroy'/'kinit' sequence as the 'root'
user on
the affected system like so:
# kdestroy -A ; kinit -k 'MYSERVER$@MYDOMAIN.EXAMPLE.COM'
At times when the issue is occurring, we observe the following
messages in
'/var/log/sssd/sssd_MYDOMAIN.example.com.log' (debug_level 9):
(2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] [dp_get_account_info_handler] (0x0200): Got request for [0x1][BE_REQ_USER][name=someuser@mydomain.example.com] (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] [dp_attach_req] (0x0400): DP Request [Account #6104]: New request. Flags [0x0001]. (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] [dp_attach_req] (0x0400): Number of active DP request: 1 (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]]
[sss_domain_get_state]
(0x1000): Domain MYDOMAIN.example.com is Active (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]]
[sss_domain_get_state]
(0x1000): Domain MYDOMAIN.example.com is Active (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] [sdap_id_op_connect_step] (0x4000): reusing cached connection (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] [sdap_search_user_next_base] (0x0400): Searching for users with
base
[DC=MYDOMAIN,DC=example,DC=com] (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]]
[sdap_print_server]
(0x2000): Searching 192.168.1.4:389 (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] [sdap_get_generic_ext_step] (0x0400): calling ldap_search_ext with
[(&(sAMAccountName=someuser)(objectclass=user)(sAMAccountName=*)(objectSID=*))][DC=MYDOMAIN,DC=example,DC=com].
(2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] [sdap_get_generic_ext_step] (0x1000): Requesting attrs:
[objectClass]
(2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] [sdap_get_generic_ext_step] (0x1000): Requesting attrs:
[sAMAccountName]
(2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] [sdap_get_generic_ext_step] (0x1000): Requesting attrs:
[unixUserPassword]
(2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] [sdap_get_generic_ext_step] (0x1000): Requesting attrs: [uidNumber] (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] [sdap_get_generic_ext_step] (0x1000): Requesting attrs: [gidNumber] (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] [sdap_get_generic_ext_step] (0x1000): Requesting attrs: [gecos] (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] [sdap_get_generic_ext_step] (0x1000): Requesting attrs:
[unixHomeDirectory]
(2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] [sdap_get_generic_ext_step] (0x1000): Requesting attrs:
[loginShell]
(2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] [sdap_get_generic_ext_step] (0x1000): Requesting attrs:
[userPrincipalName]
(2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] [sdap_get_generic_ext_step] (0x1000): Requesting attrs: [name] (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] [sdap_get_generic_ext_step] (0x1000): Requesting attrs: [memberOf] (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] [sdap_get_generic_ext_step] (0x1000): Requesting attrs:
[objectGUID]
(2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] [sdap_get_generic_ext_step] (0x1000): Requesting attrs: [objectSID] (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] [sdap_get_generic_ext_step] (0x1000): Requesting attrs:
[primaryGroupID]
(2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] [sdap_get_generic_ext_step] (0x1000): Requesting attrs:
[whenChanged]
(2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] [sdap_get_generic_ext_step] (0x1000): Requesting attrs:
[uSNChanged]
(2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] [sdap_get_generic_ext_step] (0x1000): Requesting attrs:
[accountExpires]
(2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] [sdap_get_generic_ext_step] (0x1000): Requesting attrs:
[userAccountControl]
(2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] [sdap_get_generic_ext_step] (0x1000): Requesting attrs: [userCertificate;binary] (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] [sdap_get_generic_ext_step] (0x1000): Requesting attrs: [mail] (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] [sdap_get_generic_ext_step] (0x2000): ldap_search_ext called,
msgid =
26
(2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] [sdap_op_add]
(0x2000):
New operation 26 timeout 6 (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]]
[sdap_process_result]
(0x2000): Trace: sh[0x55ca2802d840], connected[1],
ops[0x55ca28028ef0],
ldap[0x55ca27fe0610] (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]]
[sdap_process_message]
(0x4000): Message type: [LDAP_RES_SEARCH_RESULT] (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] [sdap_get_generic_op_finished] (0x0400): Search result:
Referral(10),
0000202B: RefErr: DSID-0310074A, data 0, 1 access points ref 1: 'MYDOMAIN.example.com'
(2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] [sdap_get_generic_ext_add_references] (0x1000): Additional
References:
ldap://MYDOMAIN.example.com/DC=MYDOMAIN,DC=example,DC=com (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]]
[sdap_op_destructor]
(0x2000): Operation 26 finished (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] [generic_ext_search_handler] (0x4000): Request included referrals
which
were ignored. (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] [generic_ext_search_handler] (0x4000): Ref: ldap:// MYDOMAIN.example.com/DC=MYDOMAIN,DC=example,DC=com (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] [sdap_search_user_process] (0x0400): Search for users, returned 0
results.
(2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] [sdap_search_user_process] (0x2000): Retrieved total 0 users (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]]
[sdap_id_op_done]
(0x4000): releasing operation connection (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]]
[sysdb_search_by_name]
(0x0400): No such entry (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] [sysdb_cache_search_groups] (0x2000): Search groups with filter: (&(objectCategory=group)(ghost=someuser@mydomain.example.com)) (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] [sysdb_cache_search_groups] (0x2000): No such entry (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]]
[sysdb_delete_user]
(0x0400): Error: 2 (No such file or directory) (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] [dp_req_done]
(0x0400):
DP Request [Account #6104]: Request handler finished [0]: Success (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] [_dp_req_recv] (0x0400): DP Request [Account #6104]: Receiving request data. (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] [dp_req_reply_list_success] (0x0400): DP Request [Account #6104]:
Finished.
Success. (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]]
[dp_req_reply_std]
(0x1000): DP Request [Account #6104]: Returning [Success]:
0,0,Success
(2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] [dp_table_value_destructor] (0x0400): Removing [0:1:0x0001:1::MYDOMAIN.example.com:name=
someuser@mydomain.example.com
]
from reply table (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]]
[dp_req_destructor]
(0x0400): DP Request [Account #6104]: Request removed. (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]]
[dp_req_destructor]
(0x0400): Number of active DP request: 0 (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]]
[sdap_process_result]
(0x2000): Trace: sh[0x55ca2802d840], connected[1], ops[(nil)], ldap[0x55ca27fe0610] (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]]
[sdap_process_result]
(0x2000): Trace: end of ldap_result list
And the following corresponding messages in '/var/log/secure':
Nov 10 07:40:57 myserver sshd[755]: pam_unix(sshd:auth): check
pass;
user
unknown Nov 10 07:40:57 myserver sshd[755]: pam_unix(sshd:auth):
authentication
failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=192.168.1.123 Nov 10 07:40:57 myserver sshd[755]: pam_sss(sshd:auth):
authentication
failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=192.168.1.123 user=someuser Nov 10 07:40:57 myserver sshd[755]: pam_sss(sshd:auth): received
for
user
someuser: 10 (User not known to the underlying authentication
module)
Nov 10 07:40:59 myserver sshd[735]: error: PAM: User not known to
the
underlying authentication module for illegal user someuser from 192.168.1.123 Nov 10 07:40:59 myserver sshd[735]: Failed
keyboard-interactive/pam for
invalid user someuser from 192.168.1.123 port 53300 ssh2 Nov 10 07:40:59 myserver sshd[735]: Connection closed by
192.168.1.123
port 53300 [preauth]
The effective '/etc/sssd/sssd.conf' file is as follows:
[sssd] domains = MYDOMAIN.example.com config_file_version = 2 services = nss, pam debug_level = 9
[domain/MYDOMAIN.example.com] ad_domain = MYDOMAIN.example.com krb5_realm = MYDOMAIN.EXAMPLE.COM krb5_lifetime = 10h subdomain_inherit = ignore_group_members, ldap_purge_cache_timeout ignore_group_members = true ldap_purge_cache_timeout = 0 realmd_tags = joined-with-adcli, manages-system cache_credentials = false id_provider = ad krb5_store_password_if_offline = true default_shell = /bin/bash ldap_id_mapping = true ldap_sasl_authid = MYSERVER$@MYDOMAIN.EXAMPLE.COM ldap_use_tokengroups = true use_fully_qualified_names = false fallback_homedir = /home/%d/%u access_provider = simple simple_allow_groups = linux_admins simple_allow_users = someuser, someuser2, someuser3 debug_level = 9
Running either of the following commands appears to correct the
issue
(until it presents again at an unpredictable time):
# systemctl restart sssd
or
# kdestroy -A ; kinit -k 'MYSERVER$@MYDOMAIN.EXAMPLE.COM'
Any assistance or insight you can offer would be greatly
appreciated.
We
have run countless internet searches over recent weeks, as well as consulted with Red Hat Support without breakthroughs, so I decided
to
take
this straight to the experts!
Best,
*J. Adam Craig* Lead Unix Operating Systems Analyst VCU Computer Center https://www.ucc.vcu.edu/ 804.828.4886 jacraig@vcu.edu
<
https://adminmicro2.questionpro.com/?t_340030260=J.%20Adam%20Craig&u_659...
*Don't be a phishing victim -- VCU and other reputable
organisations
will
never use email to request that you reply with your password,
social
security number or confidential personal information. For more
details,
visit **
https://ts.vcu.edu/about-us/information-security/common-questions/what-is-ph...
<
https://ts.vcu.edu/about-us/information-security/common-questions/what-is-ph...
sssd-users mailing list -- sssd-users@lists.fedorahosted.org To unsubscribe send an email to
sssd-users-leave@lists.fedorahosted.org
Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines:
https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives:
https://lists.fedorahosted.org/archives/list/sssd-users@lists.fedorahosted.o...
sssd-users mailing list -- sssd-users@lists.fedorahosted.org To unsubscribe send an email to
sssd-users-leave@lists.fedorahosted.org
Fedora Code of Conduct:
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines:
https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives:
https://lists.fedorahosted.org/archives/list/sssd-users@lists.fedorahosted.o...
sssd-users mailing list -- sssd-users@lists.fedorahosted.org To unsubscribe send an email to
sssd-users-leave@lists.fedorahosted.org
Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines:
https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives:
https://lists.fedorahosted.org/archives/list/sssd-users@lists.fedorahosted.o...
sssd-users mailing list -- sssd-users@lists.fedorahosted.org To unsubscribe send an email to sssd-users-leave@lists.fedorahosted.org Fedora Code of Conduct:
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives:
https://lists.fedorahosted.org/archives/list/sssd-users@lists.fedorahosted.o... _______________________________________________ sssd-users mailing list -- sssd-users@lists.fedorahosted.org To unsubscribe send an email to sssd-users-leave@lists.fedorahosted.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedorahosted.org/archives/list/sssd-users@lists.fedorahosted.o...
On Thu, Dec 3, 2020 at 4:23 PM J. Adam Craig jacraig@vcu.edu wrote:
Thanks! This is, once again, very helpful!
I did some searching through the logs, as per your suggestion, and came up with the SRV queries that include the (problematic) AD DNS servers in the parent domain (which, as you suspected, are also Global Catalog servers).
So, now I'm guessing that the next course of action, in view of the fact that the bug you cited is not yet resolved in EL SSSD packages, is to try to find a workaround that will result in queries only being made against the child DCs.
I can't comment on the workaround, but could you please open a case and/or bugzilla ticket to have this fix included into RHEL?
Currently, I'm considering the following options, but would be eager to learn if there is a preferred/better approach:
- One thought is to set 'ad_enable_gc = false' in
'/etc/sssd/sssd.conf'., but I am not sure if this would resolve the bug or expose us to other problems that I'm not aware of. I currently have this setting in place on a test system, and am monitoring the SSSD logs for any of the SRV queries that include the problematic servers in the parent domain. I'm currently not seeing any of those entries, but want to continue watching this for a little while longer before I draw any conclusions.
- Explicitly listing the child DCs in '/etc/sssd/sssd.conf'. The
biggest drawback I see to this approach is that we'd have to manually adjust this list every time we add or remove a DC from the child domain. We use automation tools, naturally, so this wouldn't be terribly difficult to accomplish, but it would be tedious.
Best,
*J. Adam Craig* Lead Unix Operating Systems Analyst VCU Computer Center https://www.ucc.vcu.edu/ 804.828.4886 jacraig@vcu.edu
https://adminmicro2.questionpro.com/?t_340030260=J.%20Adam%20Craig&u_65977055=351791134 *Don't be a phishing victim -- VCU and other reputable organisations will never use email to request that you reply with your password, social security number or confidential personal information. For more details, visit **https://ts.vcu.edu/about-us/information-security/common-questions/what-is-ph... https://ts.vcu.edu/about-us/information-security/common-questions/what-is-phishing*
On Thu, Dec 3, 2020 at 5:19 AM Sumit Bose sbose@redhat.com wrote:
On Wed, Dec 02, 2020 at 03:04:14PM -0500, J. Adam Craig wrote:
Thanks, all, for your assistance with this issue!
Your notes have prompted some further investigation on my end.
- In response to Alexey's question, I can confirm that, when lookups
are successful, port 389 is also used.
- After conversing with our Active Directory team, I've learned
that,
in at least one case of this issue occurring (the one cited above),
the IP
address used for the lookup was actually not a domain controller at
all,
but was rather one of our Active Directory DNS servers. This leads
me to
wonder why SSSD could be attempting to use an Active Directory DNS
server
to perform lookups in the first place.
When I run an 'nslookup' against our Active Directory domain name,
only
IP addresses of domain controllers are returned, so the issue doesn't appear to be originating from there.
Hi,
what kind of lookup did you do with nslookup? Typically SSSD does a SRV lookup for '_ldap._tcp.ad.domain.name' to find the domain controllers of a given domain. But it might also try to find site specific DC with '_ldap._tcp.sitename,_sites.ad.domain.name'.
If you search in the logs where the IP address or the related DNS name is recorded first this should help to understand by which request this host was found.
What other mechanisms (besides DNS) might SSSD be using to determine
the
domain controllers it should use?
No, the other way is to specific the DCs in sssd.conf.
In our environment, our AD DNS servers live in a root domain, with users, etc. residing in a child domain.
I guess the DNS servers are domain controllers for the root domain as well? If the one wrongly used a global catalog server as well? If that's the case you most probably hit the bug I mentioned earlier.
bye, Sumit
Thanks again for any insight and assistance.
*J. Adam Craig* Lead Unix Operating Systems Analyst VCU Computer Center https://www.ucc.vcu.edu/ 804.828.4886 jacraig@vcu.edu
<
https://adminmicro2.questionpro.com/?t_340030260=J.%20Adam%20Craig&u_659...
*Don't be a phishing victim -- VCU and other reputable organisations
will
never use email to request that you reply with your password, social security number or confidential personal information. For more details, visit **
https://ts.vcu.edu/about-us/information-security/common-questions/what-is-ph...
<
https://ts.vcu.edu/about-us/information-security/common-questions/what-is-ph...
On Wed, Dec 2, 2020 at 2:18 AM Sumit Bose sbose@redhat.com wrote:
On Tue, Dec 01, 2020 at 07:21:15PM +0100, Alexey Tikhonov wrote:
Hi,
according to the sssd.conf you do not have `ad_enable_gc` set so
this
should be `true` by default.
But in the log it contacts LDAP port: [sdap_print_server] (0x2000): Searching 192.168.1.4:389 -- IIUC,
GC
port
should be 3269... -- what port is used when everything is working as expected?
If I understand "Referral(10), 0000202B: RefErr: DSID-0310074A,
data 0, 1
access points" correctly, this specific DC "doesn't know" this user
and
refers to other DCs, but referrals chasing is disabled for ad
provider.
This doesn't explain what happens though... Perhaps clue can be
found
earlier in the logs.
On Tue, Dec 1, 2020 at 4:43 PM J. Adam Craig jacraig@vcu.edu
wrote:
Hello!
I am currently troubleshooting a very mysterious and difficult to
tack
down issue with SSSD running on RHEL/CentOS 7.x and 8.x (SSSD ver.
1.16.5
and 2.2.3). The EL 7.x and 8.x systems are attached to a Windows
Active
Directory domain using 'adcli'. We used this guide ( https://access.redhat.com/solutions/2653771), with some minor
tweaks
appropriate for our Active Directory environment and security
requirements
to set this up.
The vast majority of the time, the configuration works as
expected.
However, occasionally, we experience sporadic and temporary issues
where
users attempting to authenticate using valid Active Directory
credentials
(via SSH) are unable to login.
Hi,
you might hit https://github.com/SSSD/sssd/issues/5351, the fix is currently not available in any released RHEL or CentOS version.
bye, Sumit
When the issue presents itself, if we leave an affected system
alone
and
do nothing, the issue will eventually self-correct and users are
able
to
authenticate as expected. We have also discovered that we can
manually
"fix" the issue by either (1) restarting SSSD with 'sudo systemctl
restart
sssd' or (2) by running a 'kdestroy'/'kinit' sequence as the
'root'
user on
the affected system like so:
# kdestroy -A ; kinit -k 'MYSERVER$@MYDOMAIN.EXAMPLE.COM'
At times when the issue is occurring, we observe the following
messages in
'/var/log/sssd/sssd_MYDOMAIN.example.com.log' (debug_level 9):
(2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] [dp_get_account_info_handler] (0x0200): Got request for [0x1][BE_REQ_USER][name=someuser@mydomain.example.com] (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] [dp_attach_req] (0x0400): DP Request [Account #6104]: New request. Flags [0x0001]. (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] [dp_attach_req] (0x0400): Number of active DP request: 1 (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]]
[sss_domain_get_state]
(0x1000): Domain MYDOMAIN.example.com is Active (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]]
[sss_domain_get_state]
(0x1000): Domain MYDOMAIN.example.com is Active (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] [sdap_id_op_connect_step] (0x4000): reusing cached connection (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] [sdap_search_user_next_base] (0x0400): Searching for users with
base
[DC=MYDOMAIN,DC=example,DC=com] (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]]
[sdap_print_server]
(0x2000): Searching 192.168.1.4:389 (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] [sdap_get_generic_ext_step] (0x0400): calling ldap_search_ext with
[(&(sAMAccountName=someuser)(objectclass=user)(sAMAccountName=*)(objectSID=*))][DC=MYDOMAIN,DC=example,DC=com].
(2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] [sdap_get_generic_ext_step] (0x1000): Requesting attrs:
[objectClass]
(2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] [sdap_get_generic_ext_step] (0x1000): Requesting attrs:
[sAMAccountName]
(2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] [sdap_get_generic_ext_step] (0x1000): Requesting attrs:
[unixUserPassword]
(2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] [sdap_get_generic_ext_step] (0x1000): Requesting attrs:
[uidNumber]
(2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] [sdap_get_generic_ext_step] (0x1000): Requesting attrs:
[gidNumber]
(2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] [sdap_get_generic_ext_step] (0x1000): Requesting attrs: [gecos] (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] [sdap_get_generic_ext_step] (0x1000): Requesting attrs:
[unixHomeDirectory]
(2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] [sdap_get_generic_ext_step] (0x1000): Requesting attrs:
[loginShell]
(2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] [sdap_get_generic_ext_step] (0x1000): Requesting attrs:
[userPrincipalName]
(2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] [sdap_get_generic_ext_step] (0x1000): Requesting attrs: [name] (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] [sdap_get_generic_ext_step] (0x1000): Requesting attrs: [memberOf] (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] [sdap_get_generic_ext_step] (0x1000): Requesting attrs:
[objectGUID]
(2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] [sdap_get_generic_ext_step] (0x1000): Requesting attrs:
[objectSID]
(2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] [sdap_get_generic_ext_step] (0x1000): Requesting attrs:
[primaryGroupID]
(2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] [sdap_get_generic_ext_step] (0x1000): Requesting attrs:
[whenChanged]
(2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] [sdap_get_generic_ext_step] (0x1000): Requesting attrs:
[uSNChanged]
(2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] [sdap_get_generic_ext_step] (0x1000): Requesting attrs:
[accountExpires]
(2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] [sdap_get_generic_ext_step] (0x1000): Requesting attrs:
[userAccountControl]
(2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] [sdap_get_generic_ext_step] (0x1000): Requesting attrs: [userCertificate;binary] (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] [sdap_get_generic_ext_step] (0x1000): Requesting attrs: [mail] (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] [sdap_get_generic_ext_step] (0x2000): ldap_search_ext called,
msgid =
26
(2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] [sdap_op_add]
(0x2000):
New operation 26 timeout 6 (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]]
[sdap_process_result]
(0x2000): Trace: sh[0x55ca2802d840], connected[1],
ops[0x55ca28028ef0],
ldap[0x55ca27fe0610] (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]]
[sdap_process_message]
(0x4000): Message type: [LDAP_RES_SEARCH_RESULT] (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] [sdap_get_generic_op_finished] (0x0400): Search result:
Referral(10),
0000202B: RefErr: DSID-0310074A, data 0, 1 access points ref 1: 'MYDOMAIN.example.com'
(2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] [sdap_get_generic_ext_add_references] (0x1000): Additional
References:
ldap://MYDOMAIN.example.com/DC=MYDOMAIN,DC=example,DC=com (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]]
[sdap_op_destructor]
(0x2000): Operation 26 finished (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] [generic_ext_search_handler] (0x4000): Request included referrals
which
were ignored. (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] [generic_ext_search_handler] (0x4000): Ref: ldap:// MYDOMAIN.example.com/DC=MYDOMAIN,DC=example,DC=com (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] [sdap_search_user_process] (0x0400): Search for users, returned 0
results.
(2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] [sdap_search_user_process] (0x2000): Retrieved total 0 users (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]]
[sdap_id_op_done]
(0x4000): releasing operation connection (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]]
[sysdb_search_by_name]
(0x0400): No such entry (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] [sysdb_cache_search_groups] (0x2000): Search groups with filter: (&(objectCategory=group)(ghost=someuser@mydomain.example.com)) (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] [sysdb_cache_search_groups] (0x2000): No such entry (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]]
[sysdb_delete_user]
(0x0400): Error: 2 (No such file or directory) (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] [dp_req_done]
(0x0400):
DP Request [Account #6104]: Request handler finished [0]: Success (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] [_dp_req_recv] (0x0400): DP Request [Account #6104]: Receiving request data. (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] [dp_req_reply_list_success] (0x0400): DP Request [Account #6104]:
Finished.
Success. (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]]
[dp_req_reply_std]
(0x1000): DP Request [Account #6104]: Returning [Success]:
0,0,Success
(2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] [dp_table_value_destructor] (0x0400): Removing [0:1:0x0001:1::MYDOMAIN.example.com:name=
someuser@mydomain.example.com
]
from reply table (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]]
[dp_req_destructor]
(0x0400): DP Request [Account #6104]: Request removed. (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]]
[dp_req_destructor]
(0x0400): Number of active DP request: 0 (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]]
[sdap_process_result]
(0x2000): Trace: sh[0x55ca2802d840], connected[1], ops[(nil)], ldap[0x55ca27fe0610] (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]]
[sdap_process_result]
(0x2000): Trace: end of ldap_result list
And the following corresponding messages in '/var/log/secure':
Nov 10 07:40:57 myserver sshd[755]: pam_unix(sshd:auth): check
pass;
user
unknown Nov 10 07:40:57 myserver sshd[755]: pam_unix(sshd:auth):
authentication
failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=192.168.1.123 Nov 10 07:40:57 myserver sshd[755]: pam_sss(sshd:auth):
authentication
failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=192.168.1.123 user=someuser Nov 10 07:40:57 myserver sshd[755]: pam_sss(sshd:auth): received
for
user
someuser: 10 (User not known to the underlying authentication
module)
Nov 10 07:40:59 myserver sshd[735]: error: PAM: User not known to
the
underlying authentication module for illegal user someuser from 192.168.1.123 Nov 10 07:40:59 myserver sshd[735]: Failed
keyboard-interactive/pam for
invalid user someuser from 192.168.1.123 port 53300 ssh2 Nov 10 07:40:59 myserver sshd[735]: Connection closed by
192.168.1.123
port 53300 [preauth]
The effective '/etc/sssd/sssd.conf' file is as follows:
[sssd] domains = MYDOMAIN.example.com config_file_version = 2 services = nss, pam debug_level = 9
[domain/MYDOMAIN.example.com] ad_domain = MYDOMAIN.example.com krb5_realm = MYDOMAIN.EXAMPLE.COM krb5_lifetime = 10h subdomain_inherit = ignore_group_members, ldap_purge_cache_timeout ignore_group_members = true ldap_purge_cache_timeout = 0 realmd_tags = joined-with-adcli, manages-system cache_credentials = false id_provider = ad krb5_store_password_if_offline = true default_shell = /bin/bash ldap_id_mapping = true ldap_sasl_authid = MYSERVER$@MYDOMAIN.EXAMPLE.COM ldap_use_tokengroups = true use_fully_qualified_names = false fallback_homedir = /home/%d/%u access_provider = simple simple_allow_groups = linux_admins simple_allow_users = someuser, someuser2, someuser3 debug_level = 9
Running either of the following commands appears to correct the
issue
(until it presents again at an unpredictable time):
# systemctl restart sssd
or
# kdestroy -A ; kinit -k 'MYSERVER$@MYDOMAIN.EXAMPLE.COM'
Any assistance or insight you can offer would be greatly
appreciated.
We
have run countless internet searches over recent weeks, as well as consulted with Red Hat Support without breakthroughs, so I
decided to
take
this straight to the experts!
Best,
*J. Adam Craig* Lead Unix Operating Systems Analyst VCU Computer Center https://www.ucc.vcu.edu/ 804.828.4886 jacraig@vcu.edu
<
https://adminmicro2.questionpro.com/?t_340030260=J.%20Adam%20Craig&u_659...
*Don't be a phishing victim -- VCU and other reputable
organisations
will
never use email to request that you reply with your password,
social
security number or confidential personal information. For more
details,
visit **
https://ts.vcu.edu/about-us/information-security/common-questions/what-is-ph...
<
https://ts.vcu.edu/about-us/information-security/common-questions/what-is-ph...
sssd-users mailing list -- sssd-users@lists.fedorahosted.org To unsubscribe send an email to
sssd-users-leave@lists.fedorahosted.org
Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines:
https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives:
https://lists.fedorahosted.org/archives/list/sssd-users@lists.fedorahosted.o...
sssd-users mailing list -- sssd-users@lists.fedorahosted.org To unsubscribe send an email to
sssd-users-leave@lists.fedorahosted.org
Fedora Code of Conduct:
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines:
https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives:
https://lists.fedorahosted.org/archives/list/sssd-users@lists.fedorahosted.o...
sssd-users mailing list -- sssd-users@lists.fedorahosted.org To unsubscribe send an email to
sssd-users-leave@lists.fedorahosted.org
Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines:
https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives:
https://lists.fedorahosted.org/archives/list/sssd-users@lists.fedorahosted.o...
sssd-users mailing list -- sssd-users@lists.fedorahosted.org To unsubscribe send an email to sssd-users-leave@lists.fedorahosted.org Fedora Code of Conduct:
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives:
https://lists.fedorahosted.org/archives/list/sssd-users@lists.fedorahosted.o... _______________________________________________ sssd-users mailing list -- sssd-users@lists.fedorahosted.org To unsubscribe send an email to sssd-users-leave@lists.fedorahosted.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedorahosted.org/archives/list/sssd-users@lists.fedorahosted.o...
sssd-users mailing list -- sssd-users@lists.fedorahosted.org To unsubscribe send an email to sssd-users-leave@lists.fedorahosted.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedorahosted.org/archives/list/sssd-users@lists.fedorahosted.o...
Alexey --
Sure! I have referred a link to this thread to Red Hat Support, via the active case that I already have open regarding this issue.
From the Bugzilla https://bugzilla.redhat.com/show_bug.cgi?id=1695606 linked in Sumit's GitHub issue https://github.com/SSSD/sssd/issues/5351, I can see that there has been some discussion about setting `ad_enable_gc = false` in `/etc/sssd/sssd.conf`, but that in one case, this didn't resolve the issue. Still, I will leave `ad_enable_gc` set to `false` for now on the test server I'm working on to see if I have a similar result.
Best,
*J. Adam Craig* Lead Unix Operating Systems Analyst VCU Computer Center https://www.ucc.vcu.edu/ 804.828.4886 jacraig@vcu.edu
https://adminmicro2.questionpro.com/?t_340030260=J.%20Adam%20Craig&u_65977055=351791134 *Don't be a phishing victim -- VCU and other reputable organisations will never use email to request that you reply with your password, social security number or confidential personal information. For more details, visit **https://ts.vcu.edu/about-us/information-security/common-questions/what-is-ph... https://ts.vcu.edu/about-us/information-security/common-questions/what-is-phishing*
On Thu, Dec 3, 2020 at 10:33 AM Alexey Tikhonov atikhono@redhat.com wrote:
On Thu, Dec 3, 2020 at 4:23 PM J. Adam Craig jacraig@vcu.edu wrote:
Thanks! This is, once again, very helpful!
I did some searching through the logs, as per your suggestion, and came up with the SRV queries that include the (problematic) AD DNS servers in the parent domain (which, as you suspected, are also Global Catalog servers).
So, now I'm guessing that the next course of action, in view of the fact that the bug you cited is not yet resolved in EL SSSD packages, is to try to find a workaround that will result in queries only being made against the child DCs.
I can't comment on the workaround, but could you please open a case and/or bugzilla ticket to have this fix included into RHEL?
Currently, I'm considering the following options, but would be eager to learn if there is a preferred/better approach:
- One thought is to set 'ad_enable_gc = false' in
'/etc/sssd/sssd.conf'., but I am not sure if this would resolve the bug or expose us to other problems that I'm not aware of. I currently have this setting in place on a test system, and am monitoring the SSSD logs for any of the SRV queries that include the problematic servers in the parent domain. I'm currently not seeing any of those entries, but want to continue watching this for a little while longer before I draw any conclusions.
- Explicitly listing the child DCs in '/etc/sssd/sssd.conf'. The
biggest drawback I see to this approach is that we'd have to manually adjust this list every time we add or remove a DC from the child domain. We use automation tools, naturally, so this wouldn't be terribly difficult to accomplish, but it would be tedious.
Best,
*J. Adam Craig* Lead Unix Operating Systems Analyst VCU Computer Center https://www.ucc.vcu.edu/ 804.828.4886 jacraig@vcu.edu
https://adminmicro2.questionpro.com/?t_340030260=J.%20Adam%20Craig&u_65977055=351791134 *Don't be a phishing victim -- VCU and other reputable organisations will never use email to request that you reply with your password, social security number or confidential personal information. For more details, visit **https://ts.vcu.edu/about-us/information-security/common-questions/what-is-ph... https://ts.vcu.edu/about-us/information-security/common-questions/what-is-phishing*
On Thu, Dec 3, 2020 at 5:19 AM Sumit Bose sbose@redhat.com wrote:
On Wed, Dec 02, 2020 at 03:04:14PM -0500, J. Adam Craig wrote:
Thanks, all, for your assistance with this issue!
Your notes have prompted some further investigation on my end.
- In response to Alexey's question, I can confirm that, when
lookups
are successful, port 389 is also used.
- After conversing with our Active Directory team, I've learned
that,
in at least one case of this issue occurring (the one cited above),
the IP
address used for the lookup was actually not a domain controller at
all,
but was rather one of our Active Directory DNS servers. This leads
me to
wonder why SSSD could be attempting to use an Active Directory DNS
server
to perform lookups in the first place.
When I run an 'nslookup' against our Active Directory domain name,
only
IP addresses of domain controllers are returned, so the issue
doesn't
appear to be originating from there.
Hi,
what kind of lookup did you do with nslookup? Typically SSSD does a SRV lookup for '_ldap._tcp.ad.domain.name' to find the domain controllers of a given domain. But it might also try to find site specific DC with '_ldap._tcp.sitename,_sites.ad.domain.name'.
If you search in the logs where the IP address or the related DNS name is recorded first this should help to understand by which request this host was found.
What other mechanisms (besides DNS) might SSSD be using to
determine the
domain controllers it should use?
No, the other way is to specific the DCs in sssd.conf.
In our environment, our AD DNS servers live in a root domain, with users, etc. residing in a child domain.
I guess the DNS servers are domain controllers for the root domain as well? If the one wrongly used a global catalog server as well? If that's the case you most probably hit the bug I mentioned earlier.
bye, Sumit
Thanks again for any insight and assistance.
*J. Adam Craig* Lead Unix Operating Systems Analyst VCU Computer Center https://www.ucc.vcu.edu/ 804.828.4886 jacraig@vcu.edu
<
https://adminmicro2.questionpro.com/?t_340030260=J.%20Adam%20Craig&u_659...
*Don't be a phishing victim -- VCU and other reputable organisations
will
never use email to request that you reply with your password, social security number or confidential personal information. For more
details,
visit **
https://ts.vcu.edu/about-us/information-security/common-questions/what-is-ph...
<
https://ts.vcu.edu/about-us/information-security/common-questions/what-is-ph...
On Wed, Dec 2, 2020 at 2:18 AM Sumit Bose sbose@redhat.com wrote:
On Tue, Dec 01, 2020 at 07:21:15PM +0100, Alexey Tikhonov wrote:
Hi,
according to the sssd.conf you do not have `ad_enable_gc` set so
this
should be `true` by default.
But in the log it contacts LDAP port: [sdap_print_server] (0x2000): Searching 192.168.1.4:389 --
IIUC, GC
port
should be 3269... -- what port is used when everything is working as expected?
If I understand "Referral(10), 0000202B: RefErr: DSID-0310074A,
data 0, 1
access points" correctly, this specific DC "doesn't know" this
user and
refers to other DCs, but referrals chasing is disabled for ad
provider.
This doesn't explain what happens though... Perhaps clue can be
found
earlier in the logs.
On Tue, Dec 1, 2020 at 4:43 PM J. Adam Craig jacraig@vcu.edu
wrote:
> Hello! > > I am currently troubleshooting a very mysterious and difficult
to tack
> down issue with SSSD running on RHEL/CentOS 7.x and 8.x (SSSD
ver.
1.16.5
> and 2.2.3). The EL 7.x and 8.x systems are attached to a Windows
Active
> Directory domain using 'adcli'. We used this guide ( > https://access.redhat.com/solutions/2653771), with some minor
tweaks
> appropriate for our Active Directory environment and security
requirements
> to set this up. > > The vast majority of the time, the configuration works as
expected.
> However, occasionally, we experience sporadic and temporary
issues
where
> users attempting to authenticate using valid Active Directory
credentials
> (via SSH) are unable to login.
Hi,
you might hit https://github.com/SSSD/sssd/issues/5351, the fix is currently not available in any released RHEL or CentOS version.
bye, Sumit
> > When the issue presents itself, if we leave an affected system
alone
and
> do nothing, the issue will eventually self-correct and users are
able
to
> authenticate as expected. We have also discovered that we can
manually
> "fix" the issue by either (1) restarting SSSD with 'sudo
systemctl
restart
> sssd' or (2) by running a 'kdestroy'/'kinit' sequence as the
'root'
user on
> the affected system like so: > > # kdestroy -A ; kinit -k 'MYSERVER$@MYDOMAIN.EXAMPLE.COM' > > > At times when the issue is occurring, we observe the following
messages in
> '/var/log/sssd/sssd_MYDOMAIN.example.com.log' (debug_level 9): > > (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] > [dp_get_account_info_handler] (0x0200): Got request for > [0x1][BE_REQ_USER][name=someuser@mydomain.example.com] > (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]]
[dp_attach_req]
> (0x0400): DP Request [Account #6104]: New request. Flags
[0x0001].
> (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]]
[dp_attach_req]
> (0x0400): Number of active DP request: 1 > (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]]
[sss_domain_get_state]
> (0x1000): Domain MYDOMAIN.example.com is Active > (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]]
[sss_domain_get_state]
> (0x1000): Domain MYDOMAIN.example.com is Active > (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] > [sdap_id_op_connect_step] (0x4000): reusing cached connection > (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] > [sdap_search_user_next_base] (0x0400): Searching for users with
base
> [DC=MYDOMAIN,DC=example,DC=com] > (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]]
[sdap_print_server]
> (0x2000): Searching 192.168.1.4:389 > (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] > [sdap_get_generic_ext_step] (0x0400): calling ldap_search_ext
with
>
[(&(sAMAccountName=someuser)(objectclass=user)(sAMAccountName=*)(objectSID=*))][DC=MYDOMAIN,DC=example,DC=com].
> (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] > [sdap_get_generic_ext_step] (0x1000): Requesting attrs:
[objectClass]
> (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] > [sdap_get_generic_ext_step] (0x1000): Requesting attrs:
[sAMAccountName]
> (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] > [sdap_get_generic_ext_step] (0x1000): Requesting attrs:
[unixUserPassword]
> (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] > [sdap_get_generic_ext_step] (0x1000): Requesting attrs:
[uidNumber]
> (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] > [sdap_get_generic_ext_step] (0x1000): Requesting attrs:
[gidNumber]
> (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] > [sdap_get_generic_ext_step] (0x1000): Requesting attrs: [gecos] > (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] > [sdap_get_generic_ext_step] (0x1000): Requesting attrs:
[unixHomeDirectory]
> (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] > [sdap_get_generic_ext_step] (0x1000): Requesting attrs:
[loginShell]
> (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] > [sdap_get_generic_ext_step] (0x1000): Requesting attrs:
[userPrincipalName]
> (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] > [sdap_get_generic_ext_step] (0x1000): Requesting attrs: [name] > (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] > [sdap_get_generic_ext_step] (0x1000): Requesting attrs:
[memberOf]
> (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] > [sdap_get_generic_ext_step] (0x1000): Requesting attrs:
[objectGUID]
> (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] > [sdap_get_generic_ext_step] (0x1000): Requesting attrs:
[objectSID]
> (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] > [sdap_get_generic_ext_step] (0x1000): Requesting attrs:
[primaryGroupID]
> (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] > [sdap_get_generic_ext_step] (0x1000): Requesting attrs:
[whenChanged]
> (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] > [sdap_get_generic_ext_step] (0x1000): Requesting attrs:
[uSNChanged]
> (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] > [sdap_get_generic_ext_step] (0x1000): Requesting attrs:
[accountExpires]
> (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] > [sdap_get_generic_ext_step] (0x1000): Requesting attrs:
[userAccountControl]
> (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] > [sdap_get_generic_ext_step] (0x1000): Requesting attrs: > [userCertificate;binary] > (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] > [sdap_get_generic_ext_step] (0x1000): Requesting attrs: [mail] > (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] > [sdap_get_generic_ext_step] (0x2000): ldap_search_ext called,
msgid =
26
> (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] [sdap_op_add]
(0x2000):
> New operation 26 timeout 6 > (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]]
[sdap_process_result]
> (0x2000): Trace: sh[0x55ca2802d840], connected[1],
ops[0x55ca28028ef0],
> ldap[0x55ca27fe0610] > (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]]
[sdap_process_message]
> (0x4000): Message type: [LDAP_RES_SEARCH_RESULT] > (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] > [sdap_get_generic_op_finished] (0x0400): Search result:
Referral(10),
> 0000202B: RefErr: DSID-0310074A, data 0, 1 access points > ref 1: 'MYDOMAIN.example.com' > > (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] > [sdap_get_generic_ext_add_references] (0x1000): Additional
References:
> ldap://MYDOMAIN.example.com/DC=MYDOMAIN,DC=example,DC=com > (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]]
[sdap_op_destructor]
> (0x2000): Operation 26 finished > (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] > [generic_ext_search_handler] (0x4000): Request included
referrals which
> were ignored. > (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] > [generic_ext_search_handler] (0x4000): Ref: ldap:// > MYDOMAIN.example.com/DC=MYDOMAIN,DC=example,DC=com > (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] > [sdap_search_user_process] (0x0400): Search for users, returned 0
results.
> (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] > [sdap_search_user_process] (0x2000): Retrieved total 0 users > (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]]
[sdap_id_op_done]
> (0x4000): releasing operation connection > (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]]
[sysdb_search_by_name]
> (0x0400): No such entry > (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] > [sysdb_cache_search_groups] (0x2000): Search groups with filter: > (&(objectCategory=group)(ghost=someuser@mydomain.example.com)) > (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] > [sysdb_cache_search_groups] (0x2000): No such entry > (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]]
[sysdb_delete_user]
> (0x0400): Error: 2 (No such file or directory) > (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] [dp_req_done]
(0x0400):
> DP Request [Account #6104]: Request handler finished [0]: Success > (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] [_dp_req_recv] > (0x0400): DP Request [Account #6104]: Receiving request data. > (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] > [dp_req_reply_list_success] (0x0400): DP Request [Account #6104]:
Finished.
> Success. > (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]]
[dp_req_reply_std]
> (0x1000): DP Request [Account #6104]: Returning [Success]:
0,0,Success
> (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] > [dp_table_value_destructor] (0x0400): Removing > [0:1:0x0001:1::MYDOMAIN.example.com:name=
someuser@mydomain.example.com
]
> from reply table > (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]]
[dp_req_destructor]
> (0x0400): DP Request [Account #6104]: Request removed. > (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]]
[dp_req_destructor]
> (0x0400): Number of active DP request: 0 > (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]]
[sdap_process_result]
> (0x2000): Trace: sh[0x55ca2802d840], connected[1], ops[(nil)], > ldap[0x55ca27fe0610] > (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]]
[sdap_process_result]
> (0x2000): Trace: end of ldap_result list > > > And the following corresponding messages in '/var/log/secure': > > Nov 10 07:40:57 myserver sshd[755]: pam_unix(sshd:auth): check
pass;
user
> unknown > Nov 10 07:40:57 myserver sshd[755]: pam_unix(sshd:auth):
authentication
> failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=192.168.1.123 > Nov 10 07:40:57 myserver sshd[755]: pam_sss(sshd:auth):
authentication
> failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=192.168.1.123 > user=someuser > Nov 10 07:40:57 myserver sshd[755]: pam_sss(sshd:auth): received
for
user
> someuser: 10 (User not known to the underlying authentication
module)
> Nov 10 07:40:59 myserver sshd[735]: error: PAM: User not known
to the
> underlying authentication module for illegal user someuser from > 192.168.1.123 > Nov 10 07:40:59 myserver sshd[735]: Failed
keyboard-interactive/pam for
> invalid user someuser from 192.168.1.123 port 53300 ssh2 > Nov 10 07:40:59 myserver sshd[735]: Connection closed by
192.168.1.123
> port 53300 [preauth] > > > The effective '/etc/sssd/sssd.conf' file is as follows: > > [sssd] > domains = MYDOMAIN.example.com > config_file_version = 2 > services = nss, pam > debug_level = 9 > > [domain/MYDOMAIN.example.com] > ad_domain = MYDOMAIN.example.com > krb5_realm = MYDOMAIN.EXAMPLE.COM > krb5_lifetime = 10h > subdomain_inherit = ignore_group_members,
ldap_purge_cache_timeout
> ignore_group_members = true > ldap_purge_cache_timeout = 0 > realmd_tags = joined-with-adcli, manages-system > cache_credentials = false > id_provider = ad > krb5_store_password_if_offline = true > default_shell = /bin/bash > ldap_id_mapping = true > ldap_sasl_authid = MYSERVER$@MYDOMAIN.EXAMPLE.COM > ldap_use_tokengroups = true > use_fully_qualified_names = false > fallback_homedir = /home/%d/%u > access_provider = simple > simple_allow_groups = linux_admins > simple_allow_users = someuser, someuser2, someuser3 > debug_level = 9 > > > Running either of the following commands appears to correct the
issue
> (until it presents again at an unpredictable time): > > # systemctl restart sssd > > > or > > # kdestroy -A ; kinit -k 'MYSERVER$@MYDOMAIN.EXAMPLE.COM' > > > Any assistance or insight you can offer would be greatly
appreciated.
We
> have run countless internet searches over recent weeks, as well
as
> consulted with Red Hat Support without breakthroughs, so I
decided to
take
> this straight to the experts! > > Best, > > *J. Adam Craig* > Lead Unix Operating Systems Analyst > VCU Computer Center https://www.ucc.vcu.edu/ > 804.828.4886 > jacraig@vcu.edu > > > <
https://adminmicro2.questionpro.com/?t_340030260=J.%20Adam%20Craig&u_659...
> *Don't be a phishing victim -- VCU and other reputable
organisations
will
> never use email to request that you reply with your password,
social
> security number or confidential personal information. For more
details,
> visit **
https://ts.vcu.edu/about-us/information-security/common-questions/what-is-ph...
> <
https://ts.vcu.edu/about-us/information-security/common-questions/what-is-ph...
> > _______________________________________________ > sssd-users mailing list -- sssd-users@lists.fedorahosted.org > To unsubscribe send an email to
sssd-users-leave@lists.fedorahosted.org
> Fedora Code of Conduct: > https://docs.fedoraproject.org/en-US/project/code-of-conduct/ > List Guidelines:
https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives: >
https://lists.fedorahosted.org/archives/list/sssd-users@lists.fedorahosted.o...
>
sssd-users mailing list -- sssd-users@lists.fedorahosted.org To unsubscribe send an email to
sssd-users-leave@lists.fedorahosted.org
Fedora Code of Conduct:
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines:
https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives:
https://lists.fedorahosted.org/archives/list/sssd-users@lists.fedorahosted.o...
sssd-users mailing list -- sssd-users@lists.fedorahosted.org To unsubscribe send an email to
sssd-users-leave@lists.fedorahosted.org
Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines:
https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives:
https://lists.fedorahosted.org/archives/list/sssd-users@lists.fedorahosted.o...
sssd-users mailing list -- sssd-users@lists.fedorahosted.org To unsubscribe send an email to
sssd-users-leave@lists.fedorahosted.org
Fedora Code of Conduct:
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines:
https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives:
https://lists.fedorahosted.org/archives/list/sssd-users@lists.fedorahosted.o... _______________________________________________ sssd-users mailing list -- sssd-users@lists.fedorahosted.org To unsubscribe send an email to sssd-users-leave@lists.fedorahosted.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedorahosted.org/archives/list/sssd-users@lists.fedorahosted.o...
sssd-users mailing list -- sssd-users@lists.fedorahosted.org To unsubscribe send an email to sssd-users-leave@lists.fedorahosted.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedorahosted.org/archives/list/sssd-users@lists.fedorahosted.o...
sssd-users mailing list -- sssd-users@lists.fedorahosted.org To unsubscribe send an email to sssd-users-leave@lists.fedorahosted.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedorahosted.org/archives/list/sssd-users@lists.fedorahosted.o...
On Thu, Dec 03, 2020 at 10:19:12AM -0500, J. Adam Craig wrote:
Thanks! This is, once again, very helpful!
I did some searching through the logs, as per your suggestion, and came up with the SRV queries that include the (problematic) AD DNS servers in the parent domain (which, as you suspected, are also Global Catalog servers).
So, now I'm guessing that the next course of action, in view of the fact that the bug you cited is not yet resolved in EL SSSD packages, is to try to find a workaround that will result in queries only being made against the child DCs.
Currently, I'm considering the following options, but would be eager to learn if there is a preferred/better approach:
- One thought is to set 'ad_enable_gc = false' in
'/etc/sssd/sssd.conf'., but I am not sure if this would resolve the bug or expose us to other problems that I'm not aware of. I currently have this setting in place on a test system, and am monitoring the SSSD logs for any of the SRV queries that include the problematic servers in the parent domain. I'm currently not seeing any of those entries, but want to continue watching this for a little while longer before I draw any conclusions.
Hi,
I would recommend 'ad_enable_gc = false' until the issue is fixed. With this I'm pretty sure this issue will be avoided. I haven't checked the second option but it might work as well.
bye, Sumit
- Explicitly listing the child DCs in '/etc/sssd/sssd.conf'. The
biggest drawback I see to this approach is that we'd have to manually adjust this list every time we add or remove a DC from the child domain. We use automation tools, naturally, so this wouldn't be terribly difficult to accomplish, but it would be tedious.
Best,
*J. Adam Craig* Lead Unix Operating Systems Analyst VCU Computer Center https://www.ucc.vcu.edu/ 804.828.4886 jacraig@vcu.edu
https://adminmicro2.questionpro.com/?t_340030260=J.%20Adam%20Craig&u_65977055=351791134 *Don't be a phishing victim -- VCU and other reputable organisations will never use email to request that you reply with your password, social security number or confidential personal information. For more details, visit **https://ts.vcu.edu/about-us/information-security/common-questions/what-is-ph... https://ts.vcu.edu/about-us/information-security/common-questions/what-is-phishing*
On Thu, Dec 3, 2020 at 5:19 AM Sumit Bose sbose@redhat.com wrote:
On Wed, Dec 02, 2020 at 03:04:14PM -0500, J. Adam Craig wrote:
Thanks, all, for your assistance with this issue!
Your notes have prompted some further investigation on my end.
- In response to Alexey's question, I can confirm that, when lookups
are successful, port 389 is also used.
- After conversing with our Active Directory team, I've learned that,
in at least one case of this issue occurring (the one cited above),
the IP
address used for the lookup was actually not a domain controller at
all,
but was rather one of our Active Directory DNS servers. This leads
me to
wonder why SSSD could be attempting to use an Active Directory DNS
server
to perform lookups in the first place.
When I run an 'nslookup' against our Active Directory domain name,
only
IP addresses of domain controllers are returned, so the issue doesn't appear to be originating from there.
Hi,
what kind of lookup did you do with nslookup? Typically SSSD does a SRV lookup for '_ldap._tcp.ad.domain.name' to find the domain controllers of a given domain. But it might also try to find site specific DC with '_ldap._tcp.sitename,_sites.ad.domain.name'.
If you search in the logs where the IP address or the related DNS name is recorded first this should help to understand by which request this host was found.
What other mechanisms (besides DNS) might SSSD be using to determine
the
domain controllers it should use?
No, the other way is to specific the DCs in sssd.conf.
In our environment, our AD DNS servers live in a root domain, with users, etc. residing in a child domain.
I guess the DNS servers are domain controllers for the root domain as well? If the one wrongly used a global catalog server as well? If that's the case you most probably hit the bug I mentioned earlier.
bye, Sumit
Thanks again for any insight and assistance.
*J. Adam Craig* Lead Unix Operating Systems Analyst VCU Computer Center https://www.ucc.vcu.edu/ 804.828.4886 jacraig@vcu.edu
<
https://adminmicro2.questionpro.com/?t_340030260=J.%20Adam%20Craig&u_659...
*Don't be a phishing victim -- VCU and other reputable organisations will never use email to request that you reply with your password, social security number or confidential personal information. For more details, visit **
https://ts.vcu.edu/about-us/information-security/common-questions/what-is-ph...
<
https://ts.vcu.edu/about-us/information-security/common-questions/what-is-ph...
On Wed, Dec 2, 2020 at 2:18 AM Sumit Bose sbose@redhat.com wrote:
On Tue, Dec 01, 2020 at 07:21:15PM +0100, Alexey Tikhonov wrote:
Hi,
according to the sssd.conf you do not have `ad_enable_gc` set so this should be `true` by default.
But in the log it contacts LDAP port: [sdap_print_server] (0x2000): Searching 192.168.1.4:389 -- IIUC,
GC
port
should be 3269... -- what port is used when everything is working as expected?
If I understand "Referral(10), 0000202B: RefErr: DSID-0310074A, data
0, 1
access points" correctly, this specific DC "doesn't know" this user
and
refers to other DCs, but referrals chasing is disabled for ad
provider.
This doesn't explain what happens though... Perhaps clue can be found earlier in the logs.
On Tue, Dec 1, 2020 at 4:43 PM J. Adam Craig jacraig@vcu.edu
wrote:
Hello!
I am currently troubleshooting a very mysterious and difficult to
tack
down issue with SSSD running on RHEL/CentOS 7.x and 8.x (SSSD ver.
1.16.5
and 2.2.3). The EL 7.x and 8.x systems are attached to a Windows
Active
Directory domain using 'adcli'. We used this guide ( https://access.redhat.com/solutions/2653771), with some minor
tweaks
appropriate for our Active Directory environment and security
requirements
to set this up.
The vast majority of the time, the configuration works as expected. However, occasionally, we experience sporadic and temporary issues
where
users attempting to authenticate using valid Active Directory
credentials
(via SSH) are unable to login.
Hi,
you might hit https://github.com/SSSD/sssd/issues/5351, the fix is currently not available in any released RHEL or CentOS version.
bye, Sumit
When the issue presents itself, if we leave an affected system
alone
and
do nothing, the issue will eventually self-correct and users are
able
to
authenticate as expected. We have also discovered that we can
manually
"fix" the issue by either (1) restarting SSSD with 'sudo systemctl
restart
sssd' or (2) by running a 'kdestroy'/'kinit' sequence as the 'root'
user on
the affected system like so:
# kdestroy -A ; kinit -k 'MYSERVER$@MYDOMAIN.EXAMPLE.COM'
At times when the issue is occurring, we observe the following
messages in
'/var/log/sssd/sssd_MYDOMAIN.example.com.log' (debug_level 9):
(2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] [dp_get_account_info_handler] (0x0200): Got request for [0x1][BE_REQ_USER][name=someuser@mydomain.example.com] (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] [dp_attach_req] (0x0400): DP Request [Account #6104]: New request. Flags [0x0001]. (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] [dp_attach_req] (0x0400): Number of active DP request: 1 (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]]
[sss_domain_get_state]
(0x1000): Domain MYDOMAIN.example.com is Active (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]]
[sss_domain_get_state]
(0x1000): Domain MYDOMAIN.example.com is Active (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] [sdap_id_op_connect_step] (0x4000): reusing cached connection (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] [sdap_search_user_next_base] (0x0400): Searching for users with
base
[DC=MYDOMAIN,DC=example,DC=com] (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]]
[sdap_print_server]
(0x2000): Searching 192.168.1.4:389 (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] [sdap_get_generic_ext_step] (0x0400): calling ldap_search_ext with
[(&(sAMAccountName=someuser)(objectclass=user)(sAMAccountName=*)(objectSID=*))][DC=MYDOMAIN,DC=example,DC=com].
(2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] [sdap_get_generic_ext_step] (0x1000): Requesting attrs:
[objectClass]
(2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] [sdap_get_generic_ext_step] (0x1000): Requesting attrs:
[sAMAccountName]
(2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] [sdap_get_generic_ext_step] (0x1000): Requesting attrs:
[unixUserPassword]
(2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] [sdap_get_generic_ext_step] (0x1000): Requesting attrs: [uidNumber] (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] [sdap_get_generic_ext_step] (0x1000): Requesting attrs: [gidNumber] (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] [sdap_get_generic_ext_step] (0x1000): Requesting attrs: [gecos] (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] [sdap_get_generic_ext_step] (0x1000): Requesting attrs:
[unixHomeDirectory]
(2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] [sdap_get_generic_ext_step] (0x1000): Requesting attrs:
[loginShell]
(2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] [sdap_get_generic_ext_step] (0x1000): Requesting attrs:
[userPrincipalName]
(2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] [sdap_get_generic_ext_step] (0x1000): Requesting attrs: [name] (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] [sdap_get_generic_ext_step] (0x1000): Requesting attrs: [memberOf] (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] [sdap_get_generic_ext_step] (0x1000): Requesting attrs:
[objectGUID]
(2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] [sdap_get_generic_ext_step] (0x1000): Requesting attrs: [objectSID] (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] [sdap_get_generic_ext_step] (0x1000): Requesting attrs:
[primaryGroupID]
(2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] [sdap_get_generic_ext_step] (0x1000): Requesting attrs:
[whenChanged]
(2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] [sdap_get_generic_ext_step] (0x1000): Requesting attrs:
[uSNChanged]
(2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] [sdap_get_generic_ext_step] (0x1000): Requesting attrs:
[accountExpires]
(2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] [sdap_get_generic_ext_step] (0x1000): Requesting attrs:
[userAccountControl]
(2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] [sdap_get_generic_ext_step] (0x1000): Requesting attrs: [userCertificate;binary] (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] [sdap_get_generic_ext_step] (0x1000): Requesting attrs: [mail] (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] [sdap_get_generic_ext_step] (0x2000): ldap_search_ext called,
msgid =
26
(2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] [sdap_op_add]
(0x2000):
New operation 26 timeout 6 (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]]
[sdap_process_result]
(0x2000): Trace: sh[0x55ca2802d840], connected[1],
ops[0x55ca28028ef0],
ldap[0x55ca27fe0610] (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]]
[sdap_process_message]
(0x4000): Message type: [LDAP_RES_SEARCH_RESULT] (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] [sdap_get_generic_op_finished] (0x0400): Search result:
Referral(10),
0000202B: RefErr: DSID-0310074A, data 0, 1 access points ref 1: 'MYDOMAIN.example.com'
(2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] [sdap_get_generic_ext_add_references] (0x1000): Additional
References:
ldap://MYDOMAIN.example.com/DC=MYDOMAIN,DC=example,DC=com (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]]
[sdap_op_destructor]
(0x2000): Operation 26 finished (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] [generic_ext_search_handler] (0x4000): Request included referrals
which
were ignored. (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] [generic_ext_search_handler] (0x4000): Ref: ldap:// MYDOMAIN.example.com/DC=MYDOMAIN,DC=example,DC=com (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] [sdap_search_user_process] (0x0400): Search for users, returned 0
results.
(2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] [sdap_search_user_process] (0x2000): Retrieved total 0 users (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]]
[sdap_id_op_done]
(0x4000): releasing operation connection (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]]
[sysdb_search_by_name]
(0x0400): No such entry (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] [sysdb_cache_search_groups] (0x2000): Search groups with filter: (&(objectCategory=group)(ghost=someuser@mydomain.example.com)) (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] [sysdb_cache_search_groups] (0x2000): No such entry (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]]
[sysdb_delete_user]
(0x0400): Error: 2 (No such file or directory) (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] [dp_req_done]
(0x0400):
DP Request [Account #6104]: Request handler finished [0]: Success (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] [_dp_req_recv] (0x0400): DP Request [Account #6104]: Receiving request data. (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] [dp_req_reply_list_success] (0x0400): DP Request [Account #6104]:
Finished.
Success. (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]]
[dp_req_reply_std]
(0x1000): DP Request [Account #6104]: Returning [Success]:
0,0,Success
(2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] [dp_table_value_destructor] (0x0400): Removing [0:1:0x0001:1::MYDOMAIN.example.com:name=
someuser@mydomain.example.com
]
from reply table (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]]
[dp_req_destructor]
(0x0400): DP Request [Account #6104]: Request removed. (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]]
[dp_req_destructor]
(0x0400): Number of active DP request: 0 (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]]
[sdap_process_result]
(0x2000): Trace: sh[0x55ca2802d840], connected[1], ops[(nil)], ldap[0x55ca27fe0610] (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]]
[sdap_process_result]
(0x2000): Trace: end of ldap_result list
And the following corresponding messages in '/var/log/secure':
Nov 10 07:40:57 myserver sshd[755]: pam_unix(sshd:auth): check
pass;
user
unknown Nov 10 07:40:57 myserver sshd[755]: pam_unix(sshd:auth):
authentication
failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=192.168.1.123 Nov 10 07:40:57 myserver sshd[755]: pam_sss(sshd:auth):
authentication
failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=192.168.1.123 user=someuser Nov 10 07:40:57 myserver sshd[755]: pam_sss(sshd:auth): received
for
user
someuser: 10 (User not known to the underlying authentication
module)
Nov 10 07:40:59 myserver sshd[735]: error: PAM: User not known to
the
underlying authentication module for illegal user someuser from 192.168.1.123 Nov 10 07:40:59 myserver sshd[735]: Failed
keyboard-interactive/pam for
invalid user someuser from 192.168.1.123 port 53300 ssh2 Nov 10 07:40:59 myserver sshd[735]: Connection closed by
192.168.1.123
port 53300 [preauth]
The effective '/etc/sssd/sssd.conf' file is as follows:
[sssd] domains = MYDOMAIN.example.com config_file_version = 2 services = nss, pam debug_level = 9
[domain/MYDOMAIN.example.com] ad_domain = MYDOMAIN.example.com krb5_realm = MYDOMAIN.EXAMPLE.COM krb5_lifetime = 10h subdomain_inherit = ignore_group_members, ldap_purge_cache_timeout ignore_group_members = true ldap_purge_cache_timeout = 0 realmd_tags = joined-with-adcli, manages-system cache_credentials = false id_provider = ad krb5_store_password_if_offline = true default_shell = /bin/bash ldap_id_mapping = true ldap_sasl_authid = MYSERVER$@MYDOMAIN.EXAMPLE.COM ldap_use_tokengroups = true use_fully_qualified_names = false fallback_homedir = /home/%d/%u access_provider = simple simple_allow_groups = linux_admins simple_allow_users = someuser, someuser2, someuser3 debug_level = 9
Running either of the following commands appears to correct the
issue
(until it presents again at an unpredictable time):
# systemctl restart sssd
or
# kdestroy -A ; kinit -k 'MYSERVER$@MYDOMAIN.EXAMPLE.COM'
Any assistance or insight you can offer would be greatly
appreciated.
We
have run countless internet searches over recent weeks, as well as consulted with Red Hat Support without breakthroughs, so I decided
to
take
this straight to the experts!
Best,
*J. Adam Craig* Lead Unix Operating Systems Analyst VCU Computer Center https://www.ucc.vcu.edu/ 804.828.4886 jacraig@vcu.edu
<
https://adminmicro2.questionpro.com/?t_340030260=J.%20Adam%20Craig&u_659...
*Don't be a phishing victim -- VCU and other reputable
organisations
will
never use email to request that you reply with your password,
social
security number or confidential personal information. For more
details,
visit **
https://ts.vcu.edu/about-us/information-security/common-questions/what-is-ph...
<
https://ts.vcu.edu/about-us/information-security/common-questions/what-is-ph...
sssd-users mailing list -- sssd-users@lists.fedorahosted.org To unsubscribe send an email to
sssd-users-leave@lists.fedorahosted.org
Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines:
https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives:
https://lists.fedorahosted.org/archives/list/sssd-users@lists.fedorahosted.o...
sssd-users mailing list -- sssd-users@lists.fedorahosted.org To unsubscribe send an email to
sssd-users-leave@lists.fedorahosted.org
Fedora Code of Conduct:
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines:
https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives:
https://lists.fedorahosted.org/archives/list/sssd-users@lists.fedorahosted.o...
sssd-users mailing list -- sssd-users@lists.fedorahosted.org To unsubscribe send an email to
sssd-users-leave@lists.fedorahosted.org
Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines:
https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives:
https://lists.fedorahosted.org/archives/list/sssd-users@lists.fedorahosted.o...
sssd-users mailing list -- sssd-users@lists.fedorahosted.org To unsubscribe send an email to sssd-users-leave@lists.fedorahosted.org Fedora Code of Conduct:
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives:
https://lists.fedorahosted.org/archives/list/sssd-users@lists.fedorahosted.o... _______________________________________________ sssd-users mailing list -- sssd-users@lists.fedorahosted.org To unsubscribe send an email to sssd-users-leave@lists.fedorahosted.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedorahosted.org/archives/list/sssd-users@lists.fedorahosted.o...
sssd-users mailing list -- sssd-users@lists.fedorahosted.org To unsubscribe send an email to sssd-users-leave@lists.fedorahosted.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedorahosted.org/archives/list/sssd-users@lists.fedorahosted.o...
Thanks, Sumit.
I will keep the list apprised of anything I may learn from Red Hat with respect to backporting your patch to the versions of SSSD that are currently in use on supported versions of EL.
Best,
*J. Adam Craig* Lead Unix Operating Systems Analyst VCU Computer Center https://www.ucc.vcu.edu/ 804.828.4886 jacraig@vcu.edu
https://adminmicro2.questionpro.com/?t_340030260=J.%20Adam%20Craig&u_65977055=351791134 *Don't be a phishing victim -- VCU and other reputable organisations will never use email to request that you reply with your password, social security number or confidential personal information. For more details, visit **https://ts.vcu.edu/about-us/information-security/common-questions/what-is-ph... https://ts.vcu.edu/about-us/information-security/common-questions/what-is-phishing*
On Thu, Dec 3, 2020 at 12:49 PM Sumit Bose sbose@redhat.com wrote:
On Thu, Dec 03, 2020 at 10:19:12AM -0500, J. Adam Craig wrote:
Thanks! This is, once again, very helpful!
I did some searching through the logs, as per your suggestion, and came
up
with the SRV queries that include the (problematic) AD DNS servers in the parent domain (which, as you suspected, are also Global Catalog servers).
So, now I'm guessing that the next course of action, in view of the fact that the bug you cited is not yet resolved in EL SSSD packages, is to try to find a workaround that will result in queries only being made against the child DCs.
Currently, I'm considering the following options, but would be eager to learn if there is a preferred/better approach:
- One thought is to set 'ad_enable_gc = false' in
'/etc/sssd/sssd.conf'., but I am not sure if this would resolve the
bug or
expose us to other problems that I'm not aware of. I currently have
this
setting in place on a test system, and am monitoring the SSSD logs
for any
of the SRV queries that include the problematic servers in the parent domain. I'm currently not seeing any of those entries, but want to continue watching this for a little while longer before I draw any conclusions.
Hi,
I would recommend 'ad_enable_gc = false' until the issue is fixed. With this I'm pretty sure this issue will be avoided. I haven't checked the second option but it might work as well.
bye, Sumit
- Explicitly listing the child DCs in '/etc/sssd/sssd.conf'. The
biggest drawback I see to this approach is that we'd have to manually adjust this list every time we add or remove a DC from the child
domain.
We use automation tools, naturally, so this wouldn't be terribly
difficult
to accomplish, but it would be tedious.
Best,
*J. Adam Craig* Lead Unix Operating Systems Analyst VCU Computer Center https://www.ucc.vcu.edu/ 804.828.4886 jacraig@vcu.edu
<
https://adminmicro2.questionpro.com/?t_340030260=J.%20Adam%20Craig&u_659...
*Don't be a phishing victim -- VCU and other reputable organisations will never use email to request that you reply with your password, social security number or confidential personal information. For more details, visit **
https://ts.vcu.edu/about-us/information-security/common-questions/what-is-ph...
<
https://ts.vcu.edu/about-us/information-security/common-questions/what-is-ph...
On Thu, Dec 3, 2020 at 5:19 AM Sumit Bose sbose@redhat.com wrote:
On Wed, Dec 02, 2020 at 03:04:14PM -0500, J. Adam Craig wrote:
Thanks, all, for your assistance with this issue!
Your notes have prompted some further investigation on my end.
- In response to Alexey's question, I can confirm that, when
lookups
are successful, port 389 is also used.
- After conversing with our Active Directory team, I've learned
that,
in at least one case of this issue occurring (the one cited
above),
the IP
address used for the lookup was actually not a domain controller
at
all,
but was rather one of our Active Directory DNS servers. This
leads
me to
wonder why SSSD could be attempting to use an Active Directory DNS
server
to perform lookups in the first place.
When I run an 'nslookup' against our Active Directory domain name,
only
IP addresses of domain controllers are returned, so the issue
doesn't
appear to be originating from there.
Hi,
what kind of lookup did you do with nslookup? Typically SSSD does a SRV lookup for '_ldap._tcp.ad.domain.name' to find the domain controllers
of
a given domain. But it might also try to find site specific DC with '_ldap._tcp.sitename,_sites.ad.domain.name'.
If you search in the logs where the IP address or the related DNS name is recorded first this should help to understand by which request this host was found.
What other mechanisms (besides DNS) might SSSD be using to
determine
the
domain controllers it should use?
No, the other way is to specific the DCs in sssd.conf.
In our environment, our AD DNS servers live in a root domain, with users, etc. residing in a child domain.
I guess the DNS servers are domain controllers for the root domain as well? If the one wrongly used a global catalog server as well? If
that's
the case you most probably hit the bug I mentioned earlier.
bye, Sumit
Thanks again for any insight and assistance.
*J. Adam Craig* Lead Unix Operating Systems Analyst VCU Computer Center https://www.ucc.vcu.edu/ 804.828.4886 jacraig@vcu.edu
<
https://adminmicro2.questionpro.com/?t_340030260=J.%20Adam%20Craig&u_659...
*Don't be a phishing victim -- VCU and other reputable organisations
will
never use email to request that you reply with your password, social security number or confidential personal information. For more
details,
visit **
https://ts.vcu.edu/about-us/information-security/common-questions/what-is-ph...
<
https://ts.vcu.edu/about-us/information-security/common-questions/what-is-ph...
On Wed, Dec 2, 2020 at 2:18 AM Sumit Bose sbose@redhat.com wrote:
On Tue, Dec 01, 2020 at 07:21:15PM +0100, Alexey Tikhonov wrote:
Hi,
according to the sssd.conf you do not have `ad_enable_gc` set so
this
should be `true` by default.
But in the log it contacts LDAP port: [sdap_print_server] (0x2000): Searching 192.168.1.4:389 --
IIUC,
GC
port
should be 3269... -- what port is used when everything is working as expected?
If I understand "Referral(10), 0000202B: RefErr: DSID-0310074A,
data
0, 1
access points" correctly, this specific DC "doesn't know" this
user
and
refers to other DCs, but referrals chasing is disabled for ad
provider.
This doesn't explain what happens though... Perhaps clue can be
found
earlier in the logs.
On Tue, Dec 1, 2020 at 4:43 PM J. Adam Craig jacraig@vcu.edu
wrote:
> Hello! > > I am currently troubleshooting a very mysterious and difficult
to
tack
> down issue with SSSD running on RHEL/CentOS 7.x and 8.x (SSSD
ver.
1.16.5
> and 2.2.3). The EL 7.x and 8.x systems are attached to a
Windows
Active
> Directory domain using 'adcli'. We used this guide ( > https://access.redhat.com/solutions/2653771), with some minor
tweaks
> appropriate for our Active Directory environment and security
requirements
> to set this up. > > The vast majority of the time, the configuration works as
expected.
> However, occasionally, we experience sporadic and temporary
issues
where
> users attempting to authenticate using valid Active Directory
credentials
> (via SSH) are unable to login.
Hi,
you might hit https://github.com/SSSD/sssd/issues/5351, the fix is currently not available in any released RHEL or CentOS version.
bye, Sumit
> > When the issue presents itself, if we leave an affected system
alone
and
> do nothing, the issue will eventually self-correct and users
are
able
to
> authenticate as expected. We have also discovered that we can
manually
> "fix" the issue by either (1) restarting SSSD with 'sudo
systemctl
restart
> sssd' or (2) by running a 'kdestroy'/'kinit' sequence as the
'root'
user on
> the affected system like so: > > # kdestroy -A ; kinit -k 'MYSERVER$@MYDOMAIN.EXAMPLE.COM' > > > At times when the issue is occurring, we observe the following
messages in
> '/var/log/sssd/sssd_MYDOMAIN.example.com.log' (debug_level 9): > > (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] > [dp_get_account_info_handler] (0x0200): Got request for > [0x1][BE_REQ_USER][name=someuser@mydomain.example.com] > (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]]
[dp_attach_req]
> (0x0400): DP Request [Account #6104]: New request. Flags
[0x0001].
> (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]]
[dp_attach_req]
> (0x0400): Number of active DP request: 1 > (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]]
[sss_domain_get_state]
> (0x1000): Domain MYDOMAIN.example.com is Active > (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]]
[sss_domain_get_state]
> (0x1000): Domain MYDOMAIN.example.com is Active > (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] > [sdap_id_op_connect_step] (0x4000): reusing cached connection > (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] > [sdap_search_user_next_base] (0x0400): Searching for users with
base
> [DC=MYDOMAIN,DC=example,DC=com] > (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]]
[sdap_print_server]
> (0x2000): Searching 192.168.1.4:389 > (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] > [sdap_get_generic_ext_step] (0x0400): calling ldap_search_ext
with
>
[(&(sAMAccountName=someuser)(objectclass=user)(sAMAccountName=*)(objectSID=*))][DC=MYDOMAIN,DC=example,DC=com].
> (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] > [sdap_get_generic_ext_step] (0x1000): Requesting attrs:
[objectClass]
> (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] > [sdap_get_generic_ext_step] (0x1000): Requesting attrs:
[sAMAccountName]
> (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] > [sdap_get_generic_ext_step] (0x1000): Requesting attrs:
[unixUserPassword]
> (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] > [sdap_get_generic_ext_step] (0x1000): Requesting attrs:
[uidNumber]
> (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] > [sdap_get_generic_ext_step] (0x1000): Requesting attrs:
[gidNumber]
> (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] > [sdap_get_generic_ext_step] (0x1000): Requesting attrs: [gecos] > (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] > [sdap_get_generic_ext_step] (0x1000): Requesting attrs:
[unixHomeDirectory]
> (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] > [sdap_get_generic_ext_step] (0x1000): Requesting attrs:
[loginShell]
> (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] > [sdap_get_generic_ext_step] (0x1000): Requesting attrs:
[userPrincipalName]
> (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] > [sdap_get_generic_ext_step] (0x1000): Requesting attrs: [name] > (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] > [sdap_get_generic_ext_step] (0x1000): Requesting attrs:
[memberOf]
> (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] > [sdap_get_generic_ext_step] (0x1000): Requesting attrs:
[objectGUID]
> (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] > [sdap_get_generic_ext_step] (0x1000): Requesting attrs:
[objectSID]
> (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] > [sdap_get_generic_ext_step] (0x1000): Requesting attrs:
[primaryGroupID]
> (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] > [sdap_get_generic_ext_step] (0x1000): Requesting attrs:
[whenChanged]
> (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] > [sdap_get_generic_ext_step] (0x1000): Requesting attrs:
[uSNChanged]
> (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] > [sdap_get_generic_ext_step] (0x1000): Requesting attrs:
[accountExpires]
> (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] > [sdap_get_generic_ext_step] (0x1000): Requesting attrs:
[userAccountControl]
> (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] > [sdap_get_generic_ext_step] (0x1000): Requesting attrs: > [userCertificate;binary] > (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] > [sdap_get_generic_ext_step] (0x1000): Requesting attrs: [mail] > (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] > [sdap_get_generic_ext_step] (0x2000): ldap_search_ext called,
msgid =
26
> (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]]
[sdap_op_add]
(0x2000):
> New operation 26 timeout 6 > (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]]
[sdap_process_result]
> (0x2000): Trace: sh[0x55ca2802d840], connected[1],
ops[0x55ca28028ef0],
> ldap[0x55ca27fe0610] > (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]]
[sdap_process_message]
> (0x4000): Message type: [LDAP_RES_SEARCH_RESULT] > (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] > [sdap_get_generic_op_finished] (0x0400): Search result:
Referral(10),
> 0000202B: RefErr: DSID-0310074A, data 0, 1 access points > ref 1: 'MYDOMAIN.example.com' > > (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] > [sdap_get_generic_ext_add_references] (0x1000): Additional
References:
> ldap://MYDOMAIN.example.com/DC=MYDOMAIN,DC=example,DC=com > (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]]
[sdap_op_destructor]
> (0x2000): Operation 26 finished > (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] > [generic_ext_search_handler] (0x4000): Request included
referrals
which
> were ignored. > (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] > [generic_ext_search_handler] (0x4000): Ref: ldap:// > MYDOMAIN.example.com/DC=MYDOMAIN,DC=example,DC=com > (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] > [sdap_search_user_process] (0x0400): Search for users,
returned 0
results.
> (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] > [sdap_search_user_process] (0x2000): Retrieved total 0 users > (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]]
[sdap_id_op_done]
> (0x4000): releasing operation connection > (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]]
[sysdb_search_by_name]
> (0x0400): No such entry > (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] > [sysdb_cache_search_groups] (0x2000): Search groups with
filter:
> (&(objectCategory=group)(ghost=someuser@mydomain.example.com)) > (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] > [sysdb_cache_search_groups] (0x2000): No such entry > (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]]
[sysdb_delete_user]
> (0x0400): Error: 2 (No such file or directory) > (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]]
[dp_req_done]
(0x0400):
> DP Request [Account #6104]: Request handler finished [0]:
Success
> (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]]
[_dp_req_recv]
> (0x0400): DP Request [Account #6104]: Receiving request data. > (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] > [dp_req_reply_list_success] (0x0400): DP Request [Account
#6104]:
Finished.
> Success. > (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]]
[dp_req_reply_std]
> (0x1000): DP Request [Account #6104]: Returning [Success]:
0,0,Success
> (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]] > [dp_table_value_destructor] (0x0400): Removing > [0:1:0x0001:1::MYDOMAIN.example.com:name=
someuser@mydomain.example.com
]
> from reply table > (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]]
[dp_req_destructor]
> (0x0400): DP Request [Account #6104]: Request removed. > (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]]
[dp_req_destructor]
> (0x0400): Number of active DP request: 0 > (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]]
[sdap_process_result]
> (0x2000): Trace: sh[0x55ca2802d840], connected[1], ops[(nil)], > ldap[0x55ca27fe0610] > (2020-11-10 7:40:57): [be[MYDOMAIN.example.com]]
[sdap_process_result]
> (0x2000): Trace: end of ldap_result list > > > And the following corresponding messages in '/var/log/secure': > > Nov 10 07:40:57 myserver sshd[755]: pam_unix(sshd:auth): check
pass;
user
> unknown > Nov 10 07:40:57 myserver sshd[755]: pam_unix(sshd:auth):
authentication
> failure; logname= uid=0 euid=0 tty=ssh ruser=
rhost=192.168.1.123
> Nov 10 07:40:57 myserver sshd[755]: pam_sss(sshd:auth):
authentication
> failure; logname= uid=0 euid=0 tty=ssh ruser=
rhost=192.168.1.123
> user=someuser > Nov 10 07:40:57 myserver sshd[755]: pam_sss(sshd:auth):
received
for
user
> someuser: 10 (User not known to the underlying authentication
module)
> Nov 10 07:40:59 myserver sshd[735]: error: PAM: User not known
to
the
> underlying authentication module for illegal user someuser from > 192.168.1.123 > Nov 10 07:40:59 myserver sshd[735]: Failed
keyboard-interactive/pam for
> invalid user someuser from 192.168.1.123 port 53300 ssh2 > Nov 10 07:40:59 myserver sshd[735]: Connection closed by
192.168.1.123
> port 53300 [preauth] > > > The effective '/etc/sssd/sssd.conf' file is as follows: > > [sssd] > domains = MYDOMAIN.example.com > config_file_version = 2 > services = nss, pam > debug_level = 9 > > [domain/MYDOMAIN.example.com] > ad_domain = MYDOMAIN.example.com > krb5_realm = MYDOMAIN.EXAMPLE.COM > krb5_lifetime = 10h > subdomain_inherit = ignore_group_members,
ldap_purge_cache_timeout
> ignore_group_members = true > ldap_purge_cache_timeout = 0 > realmd_tags = joined-with-adcli, manages-system > cache_credentials = false > id_provider = ad > krb5_store_password_if_offline = true > default_shell = /bin/bash > ldap_id_mapping = true > ldap_sasl_authid = MYSERVER$@MYDOMAIN.EXAMPLE.COM > ldap_use_tokengroups = true > use_fully_qualified_names = false > fallback_homedir = /home/%d/%u > access_provider = simple > simple_allow_groups = linux_admins > simple_allow_users = someuser, someuser2, someuser3 > debug_level = 9 > > > Running either of the following commands appears to correct the
issue
> (until it presents again at an unpredictable time): > > # systemctl restart sssd > > > or > > # kdestroy -A ; kinit -k 'MYSERVER$@MYDOMAIN.EXAMPLE.COM' > > > Any assistance or insight you can offer would be greatly
appreciated.
We
> have run countless internet searches over recent weeks, as
well as
> consulted with Red Hat Support without breakthroughs, so I
decided
to
take
> this straight to the experts! > > Best, > > *J. Adam Craig* > Lead Unix Operating Systems Analyst > VCU Computer Center https://www.ucc.vcu.edu/ > 804.828.4886 > jacraig@vcu.edu > > > <
https://adminmicro2.questionpro.com/?t_340030260=J.%20Adam%20Craig&u_659...
> *Don't be a phishing victim -- VCU and other reputable
organisations
will
> never use email to request that you reply with your password,
social
> security number or confidential personal information. For more
details,
> visit **
https://ts.vcu.edu/about-us/information-security/common-questions/what-is-ph...
> <
https://ts.vcu.edu/about-us/information-security/common-questions/what-is-ph...
> > _______________________________________________ > sssd-users mailing list -- sssd-users@lists.fedorahosted.org > To unsubscribe send an email to
sssd-users-leave@lists.fedorahosted.org
> Fedora Code of Conduct: > https://docs.fedoraproject.org/en-US/project/code-of-conduct/ > List Guidelines:
https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives: >
https://lists.fedorahosted.org/archives/list/sssd-users@lists.fedorahosted.o...
>
sssd-users mailing list -- sssd-users@lists.fedorahosted.org To unsubscribe send an email to
sssd-users-leave@lists.fedorahosted.org
Fedora Code of Conduct:
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines:
https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives:
https://lists.fedorahosted.org/archives/list/sssd-users@lists.fedorahosted.o...
sssd-users mailing list -- sssd-users@lists.fedorahosted.org To unsubscribe send an email to
sssd-users-leave@lists.fedorahosted.org
Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines:
https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives:
https://lists.fedorahosted.org/archives/list/sssd-users@lists.fedorahosted.o...
sssd-users mailing list -- sssd-users@lists.fedorahosted.org To unsubscribe send an email to
sssd-users-leave@lists.fedorahosted.org
Fedora Code of Conduct:
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines:
https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives:
https://lists.fedorahosted.org/archives/list/sssd-users@lists.fedorahosted.o...
sssd-users mailing list -- sssd-users@lists.fedorahosted.org To unsubscribe send an email to
sssd-users-leave@lists.fedorahosted.org
Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines:
https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives:
https://lists.fedorahosted.org/archives/list/sssd-users@lists.fedorahosted.o...
sssd-users mailing list -- sssd-users@lists.fedorahosted.org To unsubscribe send an email to sssd-users-leave@lists.fedorahosted.org Fedora Code of Conduct:
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives:
https://lists.fedorahosted.org/archives/list/sssd-users@lists.fedorahosted.o... _______________________________________________ sssd-users mailing list -- sssd-users@lists.fedorahosted.org To unsubscribe send an email to sssd-users-leave@lists.fedorahosted.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedorahosted.org/archives/list/sssd-users@lists.fedorahosted.o...
sssd-users@lists.fedorahosted.org
-
Alexey Tikhonov -
J. Adam Craig -
jacraig＠vcu.edu
-
Sumit Bose