On Thu, Jul 16, 2020 at 4:42 PM Jerry Morey <themorey@gmail.com> wrote:
Thanks Alexey.

> I'm sorry I'm not familiar with Slurm.
> What does it mean to "start a new compute node"? Does this new node have
> isolated sssd process and /var/lib/sss/* ?

Slurm is a job scheduler that will submit tasks to other compute nodes to process.  The compute nodes are separate Linux VMs each with its own sssd process and /var/lib/sss/*.

> I would advise to consult "sssd_$domain.log" to figure out what was a
> reason of failed "[BE_REQ_USER][idnumber=705601104:-]"
> But chances are, the backend process was still offline (if SSSD was just
> started in this fresh virtual env)...

The sssd process appears to be online but I see a mapping issue in sssd_jmorey.net.log:

[dp_get_account_info_handler] (0x0200): Got request for [0x1][BE_REQ_USER][idnumber=705601104]
[sssd[be[jmorey.net]]] [dp_attach_req] (0x0400): DP Request [Account #4]: New request. Flags [0x0001].
[sssd[be[jmorey.net]]] [dp_attach_req] (0x0400): Number of active DP request: 1
[sssd[be[jmorey.net]]] [sss_domain_get_state] (0x1000): Domain jmorey.net is Active
[sssd[be[jmorey.net]]] [users_get_send] (0x0080): [705601104] did not match any configured ID mapping domain

"Domain SID <-> ID range (slice)" map isn't populated yet.

`man sssd-ldap`: "When a user or group entry for a particular domain is encountered for the first time, the SSSD allocates one of the available slices for that domain."

I'm not entirely sure, but it seems this slice (range) for a domain is only created when a first entry SID from this domain is read.
Since in this case the first request is "by UID", this didn't happen yet, and there is no data for SSSD to convert UID to a SID.

IMO, work-arounds could be:
1) trigger "by name" look up first (e.g. ssh)
2) using `ldap_idmap_default_domain_sid` option to "bind" domain to a fixed slice (0). IIUC, this should pre-populate id mapping.
But please be careful with it, as this result in a *new* UIDs generated for all objects in this domain (since currently this domain clearly maps to a non-zero slice)

[sysdb_search_user_by_uid] (0x0400): No such entry
[sssd[be[jmorey.net]]] [sysdb_delete_user] (0x0400): Error: 2 (No such file or directory)
[sssd[be[jmorey.net]]] [dp_req_done] (0x0400): DP Request [Account #4]: Request handler finished [0]: Success
[sssd[be[jmorey.net]]] [_dp_req_recv] (0x0400): DP Request [Account #4]: Receiving request data.
[sssd[be[jmorey.net]]] [dp_req_reply_list_success] (0x0400): DP Request [Account #4]: Finished. Success.
[sssd[be[jmorey.net]]] [dp_req_reply_std] (0x1000): DP Request [Account #4]: Returning [Internal Error]: 3,0,Success

What is this last line, "Returning [Internal Error]: 3,0,Success"?