Re: SSSD and Kerberos-related problems during joining a RHEL8.4-host to AD

[Spike White]

Aron,

Several things.    Some backgroun -- in our company, we have thousands of
OL8.x and hundreds of RHEL 8.x Linux servers directly AD integrated to our
corp AD domain.

I compared our sssd.conf with yours.  I think you want to add the 'ifp'
service for *L8.  It's the infopipe service.  Used by support utilities
such as sssctl domain-list, etc.

I thought you had to have a sssd.conf stanza for each service you enable.
For instance, we have this:

[nss]
debug_level = 0x0100
#debug_level = 9
filter_groups = root
filter_users = root

[pam]
pam_verbosity = 3
#debug_level = 9
offline_credentials_expiration = 3

[ifp]

Because we have
[sssd]
...
services = nss,pam,ifp

From your ldap_child.log, it looks like your SASL bind and then the LDAP
query is working.  Which is surprising to me.  We set up

ldap_sasl_authid = host/<fqdn>@<ad domain>

in our sssd.conf file.  But I'm guessing if that's not explicitly set, it
uses HOSTNAME$@REALM.  At least that's what it appears from your
ldap_child.log. It appears to use:  TEST0003$@ourlab.se

Several things to try to narrow down where the failure is occuring.
1. try 'kinit -k'  on the command line.   That uses the first entry in
/etc/krb5.keytab file to attempt to authenticate as this machine account.
It's not a perfect test, since it's acquiring a TGT ticket instead of a
service ticket.
2. If that succeeds, try kvno -S host TEST0003
     That should report the KVNO of your machine account credentials, as
stored in AD.  This is a better test, as you're acquiring a service ticket
here.  (Like sssd does).
3. Try 'adcli testjoin -D ourlab.se -v'.  This tests the sssd connectivity,
similar to how sssd does it.

if all that works, your problem is not with LDAP or your SASL binding.  So
then,

4. try 'getent passwd [e-mail]'.  That should return the
entry for this user.  If this fails, possibly your /etc/nsswitch.conf file
isn't set up right or maybe your attribute mapping in AD isn't agreeing
with your sssd.conf setting.  (We use the MS-supported RFC2307bis AD schema
extension)..

5. If that's good, then probably the problem is something in your PAM
stack.  Specifically, the auth phase.   In our /etc/pam.d/sshd file, we
have:

#%PAM-1.0
auth       substack     password-auth
auth       include      postlogin
...

and in password-auth, we have:

auth        required                                     pam_env.so
# OL7/8 version. Per I/T's stated policy for service & process accounts,
lock-out time = 30 mins
auth        required                                     pam_faillock.so
preauth silent deny=5 unlock_time=1800
auth        sufficient                                   pam_sss.so
forward_pass
auth        sufficient                                   pam_unix.so nullok
try_first_pass
auth        requisite                                    pam_succeed_if.so
uid >= 1000 quiet_success
auth        required                                     pam_faillock.so
authfail deny=5 unlock_time=1800
auth        required                                     pam_deny.so

Spike

On Thu, Nov 18, 2021 at 7:16 AM Aron Kelemen Szabo <
[e-mail]> wrote:
> Hello!
>
> We are trying to join a RHEL8.4-server to our Active Directory with the
> realm name ourlab.se.
>
> Our first attempt was to follow the RedHat-guide (
> https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/8/htm...)
> to join RHEL8.4 to an AD, also covering the installation and configuration
> of SSSD and the machine seem to have got joined to the AD (the computer
> account appears and both adcli info ourlab.se as well as the id
> [e-mail] commands return valid lookup-results from the Active
> Directory)… The TGT also seems to be fetched successfully.
>
> However, we cannot log on to the system with any AD-account and
> /var/log/sssd/krb5_child.log contains the errors below (please view the
> attached log-files for a complete log-listing), hitting the critical
> failure (SSSDBG_CRIT_FAILURE), emitting the internal error
> “[111][Connection refused]” already at the call to the function
> krb5_cc_cache_match(), called from the function create_ccache():
>
> (2021-11-18 11:02:16): [krb5_child[3585]] [sss_send_pac] (0x0080): failed
> to contact PAC responder
>
> (2021-11-18 11:02:16): [krb5_child[3585]] [validate_tgt] (0x0040):
> sss_send_pac failed, group membership for user with principal
> [[e-mail]@OURLAB.SE] might not be correct.
>
> (2021-11-18 11:02:16): [krb5_child[3585]] [sss_child_krb5_trace_cb]
> (0x4000): [3585] 1637229736.192904: Destroying ccache MEMORY:rd_req2
>
> (2021-11-18 11:02:16): [krb5_child[3585]] [get_and_save_tgt] (0x2000):
> Running as [285279201][285200513].
>
> (2021-11-18 11:02:16): [krb5_child[3585]]
> [sss_get_ccache_name_for_principal] (0x4000): Location: [KCM:]
>
> (2021-11-18 11:02:16): [krb5_child[3585]]
> [sss_get_ccache_name_for_principal] (0x2000): krb5_cc_cache_match failed:
> [111][Connection refused]
>
> (2021-11-18 11:02:16): [krb5_child[3585]] [create_ccache] (0x0020): 1000:
> [111][Connection refused]
>
> (2021-11-18 11:02:16): [krb5_child[3585]] [map_krb5_error] (0x0020): 1853:
> [111][Connection refused]
>
> (2021-11-18 11:02:16): [krb5_child[3585]] [k5c_send_data] (0x0200):
> Received error code 1432158209
>
> (2021-11-18 11:02:16): [krb5_child[3585]] [pack_response_packet] (0x2000):
> response packet size: [20]
>
> (2021-11-18 11:02:16): [krb5_child[3585]] [k5c_send_data] (0x4000):
> Response sent.
>
> (2021-11-18 11:02:16): [krb5_child[3585]] [main] (0x0400): krb5_child
> completed successfully
>
> Interestingly, the following lines appeared in /var/log/secure when
> performing the logon-attempt above (with the AD-user kelaro):
>
> Nov 18 13:12:50 test0003 sshd[4183]: pam_sss(sshd:auth): authentication
> failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=172.17.111.92
> user=kelaro
>
> Nov 18 13:12:50 test0003 sshd[4183]: pam_sss(sshd:auth): received for user
> kelaro: 4 (System error)
>
> Nov 18 13:12:52 test0003 sshd[4183]: Failed password for kelaro from
> 172.17.111.92 port 54138 ssh2
>
> /etc/sssd/sssd.conf contains the following settings:
>
> [sssd]
>
> domains = ourlab.se
>
> config_file_version = 2
>
> services = nss, pam
>
> default_domain_suffix = ourlab.se
>
> [domain/ourlab.se]
>
> ad_domain = ourlab.se
>
> krb5_realm = OURLAB.SE
>
> debug_level = 10
>
> ad_server = STLBYDCVPLV003.ourlab.se
>
> realmd_tags = manages-system joined-with-samba
>
> cache_credentials = True
>
> id_provider = ad
>
> ad_hostname = test0003.ourlab.se
>
> krb5_store_password_if_offline = True
>
> default_shell = /bin/bash
>
> ldap_id_mapping = True
>
> use_fully_qualified_names = True
>
> fallback_homedir = /home/%u@%d
>
> access_provider = ad
>
> We have also tried to:
> Set krb5_validate to false as well as setting the default_ccache_name =
>
> FILE:/tmp/krb5cc_:%{uid} but none of these changes helped either.
> Set the system crypto-policy from DEFAULT to DEFAULT:AD-SUPPORT
>
> (update-crypto-policies --set DEFAULT:AD-SUPPORT) did not help either.
> Remove the host from the realm (realm leave), deleted it’s computer
>
> account from the Active Directory and tried to set it’s hostname as FDQN
> (hostnamectl set-hostname test0003.ourlab.se) and then re-joined it again
> (realm join), but still with the same results…
> Apply the realm permit --realm=ourlab.se --all command, but we could
>
> still not log in with our user ([e-mail]). User principal for
> [e-mail] is [e-mail]@OURLAB.SE.
> Set realmd_tags to “manages-system joined-with-adcli”, but the problem
>
> still remained
> Follow several other similar tutorials (for example
>
> https://www.redhat.com/sysadmin/linux-active-directory) to join the host
> to the AD, however all of them resulted in the same error described here.
>
> The following SSSD and KRB5 package-versions are installed on the host:
>
> sssd-client-2.5.2-2.el8_5.1.x86_64
>
> sssd-krb5-2.5.2-2.el8_5.1.x86_64
>
> sssd-2.5.2-2.el8_5.1.x86_64
>
> sssd-nfs-idmap-2.4.0-9.el8_4.2.x86_64
>
> sssd-common-2.5.2-2.el8_5.1.x86_64
>
> sssd-ldap-2.5.2-2.el8_5.1.x86_64
>
> sssd-proxy-2.5.2-2.el8_5.1.x86_64
>
> sssd-ipa-2.5.2-2.el8_5.1.x86_64
>
> sssd-kcm-2.5.2-2.el8_5.1.x86_64
>
> sssd-tools-2.5.2-2.el8_5.1.x86_64
>
> sssd-krb5-common-2.5.2-2.el8_5.1.x86_64
>
> sssd-common-pac-2.5.2-2.el8_5.1.x86_64
>
> sssd-ad-2.5.2-2.el8_5.1.x86_64
>
> sssd-dbus-2.5.2-2.el8_5.1.x86_64
>
> krb5-libs-1.18.2-8.3.el8_4.x86_64
>
> I have also attached the krb5_child.log and ldap_child.log (created with
> log-level 0x3ff0) after my latest logon-attempt (as the user kelaro).
>
> Any help/tips about:
> Why the “Connection refused”-error is being generated of the
>
> krb5_cc_cache_match-function
>    2. Why the UPN is getting appended to the user name (
>    [e-mail]@OURLAB.SE despite I have actively attempted to
>    disable this by setting use_fully_qualified_names = False), and whether
>    this can be declared as a fatal condition
>    3. Possible failure reasons for sss_send_pac, leading to the “failed
>    to contact PAC responder” error message…
>
> …would be greatly appreciated!
>
> Med Vänliga Hälsningar / Best Regards
>
> *Áron Kelemen Szabó*
> IT Data Center Administrator / Linux Engineer
> [e-mail]
>
> *PostNord Strålfors AB*
>
> sssd-users mailing list -- [e-mail]
> To unsubscribe send an email to [e-mail]
> Fedora Code of Conduct:
> https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives:
> https://lists.fedorahosted.org/archives/list/[e-mail].o...
> Do not reply to spam on the list, report it:
> https://pagure.io/fedora-infrastructure

SSSD and Kerberos-related problems during joining a RHEL8.4-host to AD

[Aron Kelemen Szabo]

Hi Spike!

Thanks for your answer! I have extended the /etc/sssd/sssd.conf with the missing stanzas like below:

[sssd]
domains = ourlab.se
config_file_version = 2
services = nss, pam, ifp
default_domain_suffix = ourlab.se
#default_ccache_name = KEYRING:persistent:%{uid}
#default_ccache_name = FILE:/tmp/krb5cc_:%{uid}

[nss]
debug_level = 0x0100

[pam]
pam_verbosity = 3

[ifp]
debug_level = 0x0100

[domain/ourlab.se]
ad_domain = ourlab.se
krb5_realm = OURLAB.SE
debug_level = 10
#krb5_validate = false
realmd_tags = manages-system joined-with-adcli
cache_credentials = True
id_provider = ad
krb5_store_password_if_offline = True
default_shell = /bin/bash
ldap_id_mapping = True
use_fully_qualified_names = True
fallback_homedir = /home/%u@%d
access_provider = ad

I have cleared the SSSD-cache and restarted SSSD and the logons still not work. However, kinit -k returns the following error:
kinit: Connection refused while getting default ccache

as well as kvno -S host TEST0003:
kvno: Connection refused while opening ccache

Exactly what connection is attempted to be established at this state? Does the error denote a failed file open-operation or is it a network connection?

Thanks again!

/Áron

From: Spike White 
Sent: den 18 november 2021 18:46
To: End-user discussions about the System Security Services Daemon 
Subject: [SSSD-users] Re: SSSD and Kerberos-related problems during joining a RHEL8.4-host to AD

Aron,

Several things.    Some backgroun -- in our company, we have thousands of OL8.x and hundreds of RHEL 8.x Linux servers directly AD integrated to our corp AD domain.

I compared our sssd.conf with yours.  I think you want to add the 'ifp' service for *L8.  It's the infopipe service.  Used by support utilities such as sssctl domain-list, etc.

I thought you had to have a sssd.conf stanza for each service you enable.  For instance, we have this:

[nss]
debug_level = 0x0100
#debug_level = 9
filter_groups = root
filter_users = root

[pam]
pam_verbosity = 3
#debug_level = 9
offline_credentials_expiration = 3

[ifp]

Because we have
[sssd]
...
services = nss,pam,ifp

From your ldap_child.log, it looks like your SASL bind and then the LDAP query is working.  Which is surprising to me.  We set up

ldap_sasl_authid = host/<fqdn>@<ad domain>

in our sssd.conf file.  But I'm guessing if that's not explicitly set, it uses HOSTNAME$@REALM.  At least that's what it appears from your ldap_child.log. It appears to use:  TEST0003$@ourlab.sehttps://eur03.safelinks.protection.outlook.com/?url=http%3A%2F%2Fourlab.se%2F&data=04%7C01%7Caron.kelemen%40stralfors.se%7Cee4423c7606c4b41204308d9aabb6635%7C91f09566a8504faebbe129ad3804a2f6%7C1%7C1%7C637728544138573813%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=aOUkFrUQ067nE%2BpYXJ5S2o%2BnFBsq81bT%2FohpsjkPX50%3D&reserved=0

Several things to try to narrow down where the failure is occuring.
1. try 'kinit -k'  on the command line.   That uses the first entry in /etc/krb5.keytab file to attempt to authenticate as this machine account.  It's not a perfect test, since it's acquiring a TGT ticket instead of a service ticket.
2. If that succeeds, try kvno -S host TEST0003
     That should report the KVNO of your machine account credentials, as stored in AD.  This is a better test, as you're acquiring a service ticket here.  (Like sssd does).
3. Try 'adcli testjoin -D ourlab.sehttps://eur03.safelinks.protection.outlook.com/?url=http%3A%2F%2Fourlab.se%2F&data=04%7C01%7Caron.kelemen%40stralfors.se%7Cee4423c7606c4b41204308d9aabb6635%7C91f09566a8504faebbe129ad3804a2f6%7C1%7C1%7C637728544138583805%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=qG%2BMWmyeGr6Z4%2Fw7dIY4F77u21DJvpOW4oTsSy5C0ds%3D&reserved=0 -v'.  This tests the sssd connectivity, similar to how sssd does it.

if all that works, your problem is not with LDAP or your SASL binding.  So then,

4. try 'getent passwd [e-mail]'.  That should return the entry for this user.  If this fails, possibly your /etc/nsswitch.conf file isn't set up right or maybe your attribute mapping in AD isn't agreeing with your sssd.conf setting.  (We use the MS-supported RFC2307bis AD schema extension)..

5. If that's good, then probably the problem is something in your PAM stack.  Specifically, the auth phase.   In our /etc/pam.d/sshd file, we have:

#%PAM-1.0
auth       substack     password-auth
auth       include      postlogin
...

and in password-auth, we have:

auth        required                                     pam_env.so
# OL7/8 version. Per I/T's stated policy for service & process accounts, lock-out time = 30 mins
auth        required                                     pam_faillock.so preauth silent deny=5 unlock_time=1800
auth        sufficient                                   pam_sss.so forward_pass
auth        sufficient                                   pam_unix.so nullok try_first_pass
auth        requisite                                    pam_succeed_if.so uid >= 1000 quiet_success
auth        required                                     pam_faillock.so authfail deny=5 unlock_time=1800
auth        required                                     pam_deny.so

Spike

SSSD and Kerberos-related problems during joining a RHEL8.4-host to AD

[Aron Kelemen Szabo]

Hi again!

I think I have found the problem! 😊

kinit -k returned the following error:
kinit: Connection refused while getting default ccache
…while kvno -S host TEST0003 returned the following error:
kvno: Connection refused while opening ccache

A bit of googling on these error messages revealed that the culprit might be the KCM-service not running... When googling a bit on the error messages thrown by kinit and kvno and issuing the following commands revealed that this was indeed the case.
systemctl status sssd-kcm.socket
systemctl status sssd-kcm.service

I came across this error report which among other things recommends upgrading the packages: https://bugzilla.redhat.com/show_bug.cgi?id=1716981

So I run a “yum update” and re-joined the host to the realm, and now the AD-logons seem to be working fine! Now I “only” need to find out the very happening that rendered KCM to fail. :-)

Thanks for all your input again!

Best regards,
Áron

From: Spike White 
Sent: den 18 november 2021 18:46
To: End-user discussions about the System Security Services Daemon 
Subject: [SSSD-users] Re: SSSD and Kerberos-related problems during joining a RHEL8.4-host to AD

Aron,

Several things.    Some backgroun -- in our company, we have thousands of OL8.x and hundreds of RHEL 8.x Linux servers directly AD integrated to our corp AD domain.

I compared our sssd.conf with yours.  I think you want to add the 'ifp' service for *L8.  It's the infopipe service.  Used by support utilities such as sssctl domain-list, etc.

I thought you had to have a sssd.conf stanza for each service you enable.  For instance, we have this:

[nss]
debug_level = 0x0100
#debug_level = 9
filter_groups = root
filter_users = root

[pam]
pam_verbosity = 3
#debug_level = 9
offline_credentials_expiration = 3

[ifp]

Because we have
[sssd]
...
services = nss,pam,ifp

From your ldap_child.log, it looks like your SASL bind and then the LDAP query is working.  Which is surprising to me.  We set up

ldap_sasl_authid = host/<fqdn>@<ad domain>

in our sssd.conf file.  But I'm guessing if that's not explicitly set, it uses HOSTNAME$@REALM.  At least that's what it appears from your ldap_child.log. It appears to use:  TEST0003$@ourlab.sehttps://eur03.safelinks.protection.outlook.com/?url=http%3A%2F%2Fourlab.se%2F&data=04%7C01%7Caron.kelemen%40stralfors.se%7Cee4423c7606c4b41204308d9aabb6635%7C91f09566a8504faebbe129ad3804a2f6%7C1%7C1%7C637728544138573813%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=aOUkFrUQ067nE%2BpYXJ5S2o%2BnFBsq81bT%2FohpsjkPX50%3D&reserved=0

Several things to try to narrow down where the failure is occuring.
1. try 'kinit -k'  on the command line.   That uses the first entry in /etc/krb5.keytab file to attempt to authenticate as this machine account.  It's not a perfect test, since it's acquiring a TGT ticket instead of a service ticket.
2. If that succeeds, try kvno -S host TEST0003
     That should report the KVNO of your machine account credentials, as stored in AD.  This is a better test, as you're acquiring a service ticket here.  (Like sssd does).
3. Try 'adcli testjoin -D ourlab.sehttps://eur03.safelinks.protection.outlook.com/?url=http%3A%2F%2Fourlab.se%2F&data=04%7C01%7Caron.kelemen%40stralfors.se%7Cee4423c7606c4b41204308d9aabb6635%7C91f09566a8504faebbe129ad3804a2f6%7C1%7C1%7C637728544138583805%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=qG%2BMWmyeGr6Z4%2Fw7dIY4F77u21DJvpOW4oTsSy5C0ds%3D&reserved=0 -v'.  This tests the sssd connectivity, similar to how sssd does it.

if all that works, your problem is not with LDAP or your SASL binding.  So then,

4. try 'getent passwd [e-mail]'.  That should return the entry for this user.  If this fails, possibly your /etc/nsswitch.conf file isn't set up right or maybe your attribute mapping in AD isn't agreeing with your sssd.conf setting.  (We use the MS-supported RFC2307bis AD schema extension)..

5. If that's good, then probably the problem is something in your PAM stack.  Specifically, the auth phase.   In our /etc/pam.d/sshd file, we have:

#%PAM-1.0
auth       substack     password-auth
auth       include      postlogin
...

and in password-auth, we have:

auth        required                                     pam_env.so
# OL7/8 version. Per I/T's stated policy for service & process accounts, lock-out time = 30 mins
auth        required                                     pam_faillock.so preauth silent deny=5 unlock_time=1800
auth        sufficient                                   pam_sss.so forward_pass
auth        sufficient                                   pam_unix.so nullok try_first_pass
auth        requisite                                    pam_succeed_if.so uid >= 1000 quiet_success
auth        required                                     pam_faillock.so authfail deny=5 unlock_time=1800
auth        required                                     pam_deny.so

Spike