Hi,
On Thu, Jul 16, 2020 at 2:10 PM Jerry Morey <themorey(a)gmail.com> wrote:
Hi,
I have a weird situation when a user launches a Slurm interactive job on
a new
compute node with empty sssd cache. I'm not sure if its an issue
with Slurm or SSSD/NSS. Below is the output of the Slurm session
originated on node ip-0A0B0004 and starting interactive node ip-0A0B0006 by
user1 with uid 705601104:
I'm sorry I'm not familiar with Slurm.
What does it mean to "start a new compute node"? Does this new node have
isolated sssd process and /var/lib/sss/* ?
>[user1@ip-0A0B0004 ~]$ srun -p lowmem -w lowmem-2 --pty bash
/usr/bin/id: cannot find name for group ID 705600513
/usr/bin/id: cannot find name for user ID 705601104
[I have no name!@ip-0A0B0006 ~]$
From this node I can successfully id user1 or any other AD user (this
populates my
cache and subsequent Slurm sessions for any user will resolve):
[I have no name!@ip-0A0B0006 ~]$ id user1
uid=705601104(user1) gid=705600513 groups=705600513,705601103(group1)
[I have no name!@ip-0A0B0006 ~]$
Here is the relevant session from my sssd_nss.log:
(Fri Jul 10 22:47:58 2020) [sssd[nss]] [cache_req_send] (0x0400): CR #0:
New
request 'User by ID'
(Fri Jul 10 22:47:58 2020) [sssd[nss]] [cache_req_select_domains]
(0x0400): CR #0: Performing a multi-domain search
(Fri Jul 10 22:47:58 2020) [sssd[nss]] [cache_req_search_domains]
(0x0400): CR #0: Search will check the cache and check the data provider
(Fri Jul 10 22:47:58 2020) [sssd[nss]] [cache_req_set_domain]
(0x0400):
CR #0: Using domain [
jmorey.net]
(Fri Jul 10 22:47:58 2020) [sssd[nss]] [cache_req_search_send]
(0x0400):
CR #0: Looking up UID:705601104@jmorey.net
(Fri Jul 10 22:47:58 2020) [sssd[nss]] [cache_req_search_ncache]
(0x0400): CR #0: Checking negative cache for [UID:705601104@jmorey.net]
(Fri Jul 10 22:47:58 2020) [sssd[nss]] [cache_req_search_ncache]
(0x0400): CR #0: [UID:705601104@jmorey.net] is not present in negative cache
(Fri Jul 10 22:47:58 2020) [sssd[nss]] [cache_req_search_cache]
(0x0400):
CR #0: Looking up [UID:705601104@jmorey.net] in cache
(Fri Jul 10 22:47:58 2020) [sssd[nss]] [cache_req_search_cache]
(0x0400):
CR #0: Object [UID:705601104@jmorey.net] was not found in cache
(Fri Jul 10 22:47:58 2020) [sssd[nss]] [cache_req_search_dp]
(0x0400): CR
#0: Looking up [UID:705601104@jmorey.net] in data provider
(Fri Jul 10 22:47:58 2020) [sssd[nss]] [sss_dp_issue_request]
(0x0400):
Issuing request for [0x562848473820:1:705601104@jmorey.net]
(Fri Jul 10 22:47:58 2020) [sssd[nss]] [sss_dp_get_account_msg]
(0x0400):
Creating request for [jmorey.net][0x1][BE_REQ_USER][idnumber=705601104:-]
(Fri Jul 10 22:47:58 2020) [sssd[nss]] [sss_dp_internal_get_send]
(0x0400): Entering request [0x562848473820:1:705601104@jmorey.net]
(Fri Jul 10 22:47:58 2020)
(Fri Jul 10 22:47:58 2020) [sssd[nss]] [cache_req_common_dp_recv]
(0x0040): CR #0:
Data Provider Error: 3, 0, Success
(Fri Jul 10 22:47:58 2020) [sssd[nss]] [cache_req_common_dp_recv]
(0x0400): CR #0: Due to an error we will return cached data
I would advise to consult "sssd_$domain.log" to figure out what was a
reason of failed "[BE_REQ_USER][idnumber=705601104:-]"
But chances are, the backend process was still offline (if SSSD was just
started in this fresh virtual env)...
(Fri Jul 10 22:47:58 2020) [sssd[nss]] [cache_req_search_cache]
(0x0400):
CR #0: Looking up [UID:705601104@jmorey.net] in cache
(Fri Jul 10 22:47:58 2020) [sssd[nss]] [cache_req_search_cache]
(0x0400):
CR #0: Object [UID:705601104@jmorey.net] was not found in cache
(Fri Jul 10 22:47:58 2020) [sssd[nss]] [cache_req_process_result]
(0x0400): CR #0: Finished: Not found
To my inexperienced eye it looks like it is communicating with the
backend but
receives some kind of error so it reverts to the cache, which
is empty. The other oddity is the 'User by id' request that is trying to
resolve based on the uid not the name. While this is happening I can
successfully resolve user1 using SSH to the same node.
The SSH session:
(Mon Jul 13 20:30:27 2020) [sssd[nss]] [cache_req_send] (0x0400): CR #6:
New
request 'User by name'
(Mon Jul 13 20:30:27 2020) [sssd[nss]] [cache_req_process_input]
(0x0400): CR #6: Parsing input name [user1]
(Mon Jul 13 20:30:27 2020) [sssd[nss]] [sss_parse_name_for_domains]
(0x0200): name 'user1' matched without domain, user is user1
(Mon Jul 13 20:30:27 2020) [sssd[nss]] [cache_req_set_name] (0x0400):
CR
#6: Setting name [user1]
(Mon Jul 13 20:30:27 2020) [sssd[nss]] [cache_req_select_domains]
(0x0400): CR #6: Performing a multi-domain search
(Mon Jul 13 20:30:27 2020) [sssd[nss]] [cache_req_search_domains]
(0x0400): CR #6: Search will check the cache and check the data provider
(Mon Jul 13 20:30:27 2020) [sssd[nss]] [cache_req_set_domain]
(0x0400):
CR #6: Using domain [
jmorey.net] (Mon Jul 13 20:30:27 2020) [sssd[nss]]
[cache_req_prepare_domain_data] (0x0400): CR #6: Preparing input data for
domain [
jmorey.net] rules
(Mon Jul 13 20:30:27 2020) [sssd[nss]] [cache_req_search_send]
(0x0400):
CR #6: Looking up user1(a)jmorey.net
(Mon Jul 13 20:30:27 2020) [sssd[nss]] [cache_req_search_ncache]
(0x0400): CR #6: Checking negative cache for [user1(a)jmorey.net]
(Mon Jul 13 20:30:27 2020) [sssd[nss]] [cache_req_search_ncache]
(0x0400): CR #6: [user1(a)jmorey.net] is not present in negative cache
(Mon Jul 13 20:30:27 2020) [sssd[nss]] [cache_req_search_cache]
(0x0400):
CR #6: Looking up [user1(a)jmorey.net] in cache
(Mon Jul 13 20:30:27 2020) [sssd[nss]] [cache_req_search_cache]
(0x0400):
CR #6: Object [user1(a)jmorey.net] was not found in cache
(Mon Jul 13 20:30:27 2020) [sssd[nss]] [cache_req_search_dp]
(0x0400): CR
#6: Looking up [user1(a)jmorey.net] in data provider
(Mon Jul 13 20:30:27 2020) [sssd[nss]] [sss_dp_issue_request]
(0x0400):
Issuing request for [0x564629422820:1:user1@jmorey.net@jmorey.net]
(Mon Jul 13 20:30:27 2020) [sssd[nss]] [sss_dp_get_account_msg]
(0x0400):
Creating request for
[jmorey.net][0x1][BE_REQ_USER][name=user1@jmorey.net:-]
(Mon Jul 13 20:30:27 2020) [sssd[nss]] [sss_dp_internal_get_send]
(0x0400): Entering request [0x564629422820:1:user1@jmorey.net@jmorey.net]
(Mon Jul 13 20:30:27 2020) [sssd[nss]] [sss_dp_get_reply] (0x1000):
Got
reply from Data Provider - DP error code: 0 errno: 0 error message: Success
(Mon Jul 13 20:30:27 2020) [sssd[nss]] [cache_req_search_cache]
(0x0400):
CR #6: Looking up [user1(a)jmorey.net] in cache
(Mon Jul 13 20:30:27 2020) [sssd[nss]]
[cache_req_search_ncache_filter]
(0x0400): CR #6: This request type does not support
filtering result by
negative cache
(Mon Jul 13 20:30:27 2020) [sssd[nss]] [cache_req_search_done]
(0x0400):
CR #6: Returning updated object [user1(a)jmorey.net]
(Mon Jul 13 20:30:27 2020) [sssd[nss]]
[cache_req_create_and_add_result]
(0x0400): CR #6: Found 1 entries in domain
jmorey.net
(Mon Jul 13 20:30:27 2020) [sssd[nss]] [sss_dp_req_destructor]
(0x0400):
Deleting request: [0x564629422820:1:user1@jmorey.net@jmorey.net]
(Mon Jul 13 20:30:27 2020) [sssd[nss]] [cache_req_done] (0x0400): CR
#6:
Finished: Success
Here is my sssd.conf:
[sssd]
config_file_version = 2
services = nss,pam
domains =
jmorey.net
[nss]
filter_users = root
filter_groups = root
debug_level = 7
[pam]
[
domain/jmorey.net]
id_provider = ldap
auth_provider = ldap
access_provider = ldap
debug_level = 7
dns_discovery_domain =
jmorey.net
enumerate = False
cache_credentials = True
case_sensitive = false
ldap_schema = ad
ldap_uri =
ldaps://jmorey-novo-ad.jmorey.net
ldap_user_search_base = cn=Users,dc=jmorey,dc=net
ldap_group_search_base = cn=group1,cn=Users,dc=jmorey,dc=net
ldap_referrals = False
ldap_tls_reqcert = never
ldap_use_tokengroups = True
ldap_id_mapping = True
override_homedir = /mnt/exports/shared/home/%u
fallback_homedir = /shared/home/%u
default_shell = /bin/bash
ldap_access_order = filter, expire
ldap_account_expire_policy = ad
ldap_access_filter = (|(memberOf=cn=group1,cn=Users,dc=jmorey,dc=net))
ldap_default_bind_dn = cn=user 1,cn=Users,dc=jmorey,dc=net
ldap_default_authtok_type = password
ldap_default_authtok =
thanks,
Jerry
_______________________________________________
sssd-users mailing list -- sssd-users(a)lists.fedorahosted.org
To unsubscribe send an email to sssd-users-leave(a)lists.fedorahosted.org
Fedora Code of Conduct:
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
https://lists.fedorahosted.org/archives/list/sssd-users@lists.fedorahoste...