I tried this on another test server, and configured NIS for the users, which are different. Same issue. All the verbose output adds a lot of log noise but I'm hoping it provides a clue.
OpenSSH_8.4p1, OpenSSL 1.1.1i FIPS 8 Dec 2020
debug1: Reading configuration data /etc/ssh/ssh_config
debug3: /etc/ssh/ssh_config line 55: Including file /etc/ssh/ssh_config.d/50-redhat.conf depth 0
debug1: Reading configuration data /etc/ssh/ssh_config.d/50-redhat.conf
debug2: checking match for 'final all' host ourserver originally ourserver
debug3: /etc/ssh/ssh_config.d/50-redhat.conf line 3: not matched 'final'
debug2: match not found
debug3: /etc/ssh/ssh_config.d/50-redhat.conf line 5: Including file /etc/crypto-policies/back-ends/openssh.config depth 1 (parse only)
debug1: Reading configuration data /etc/crypto-policies/back-ends/openssh.config
debug3: gss kex names ok: [gss-curve25519-sha256-,gss-nistp256-sha256-,gss-group14-sha256-,gss-group16-sha512-]
debug3: kex names ok: [curve25519-sha256,
curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group14-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512]
debug1: configuration requests final Match pass
debug1: re-parsing configuration
debug1: Reading configuration data /etc/ssh/ssh_config
debug3: /etc/ssh/ssh_config line 55: Including file /etc/ssh/ssh_config.d/50-redhat.conf depth 0
debug1: Reading configuration data /etc/ssh/ssh_config.d/50-redhat.conf
debug2: checking match for 'final all' host ourserver originally ourserver
debug3: /etc/ssh/ssh_config.d/50-redhat.conf line 3: matched 'final'
debug2: match found
debug3: /etc/ssh/ssh_config.d/50-redhat.conf line 5: Including file /etc/crypto-policies/back-ends/openssh.config depth 1
debug1: Reading configuration data /etc/crypto-policies/back-ends/openssh.config
debug3: gss kex names ok: [gss-curve25519-sha256-,gss-nistp256-sha256-,gss-group14-sha256-,gss-group16-sha512-]
debug3: kex names ok: [curve25519-sha256,
curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group14-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512]
debug3: expanded UserKnownHostsFile '~/.ssh/known_hosts' -> '/root/.ssh/known_hosts'
debug3: expanded UserKnownHostsFile '~/.ssh/known_hosts2' -> '/root/.ssh/known_hosts2'
debug1: Executing proxy command: exec /usr/bin/sss_ssh_knownhostsproxy -p 22 ourserver
debug1: identity file /root/.ssh/id_rsa type 0
debug1: identity file /root/.ssh/id_rsa-cert type -1
debug1: identity file /root/.ssh/id_dsa type -1
debug1: identity file /root/.ssh/id_dsa-cert type -1
debug1: identity file /root/.ssh/id_ecdsa type -1
debug1: identity file /root/.ssh/id_ecdsa-cert type -1
debug1: identity file /root/.ssh/id_ecdsa_sk type -1
debug1: identity file /root/.ssh/id_ecdsa_sk-cert type -1
debug1: identity file /root/.ssh/id_ed25519 type -1
debug1: identity file /root/.ssh/id_ed25519-cert type -1
debug1: identity file /root/.ssh/id_ed25519_sk type -1
debug1: identity file /root/.ssh/id_ed25519_sk-cert type -1
debug1: identity file /root/.ssh/id_xmss type -1
debug1: identity file /root/.ssh/id_xmss-cert type -1
debug1: Local version string SSH-2.0-OpenSSH_8.4
debug1: Remote protocol version 2.0, remote software version OpenSSH_8.4
debug1: match: OpenSSH_8.4 pat OpenSSH* compat 0x04000000
debug2: fd 5 setting O_NONBLOCK
debug2: fd 4 setting O_NONBLOCK
debug1: Authenticating to ourserver:22 as 'ouruser'
debug3: hostkeys_foreach: reading file "/root/.ssh/known_hosts"
debug3: record_hostkey: found key type ECDSA in file /root/.ssh/known_hosts:31
debug3: load_hostkeys: loaded 1 keys from ourserver
debug3: hostkeys_foreach: reading file "/var/lib/sss/pubconf/known_hosts"
debug3: order_hostkeyalgs: have matching best-preference key type
ecdsa-sha2-nistp256-cert-v01@openssh.com, using HostkeyAlgorithms verbatim
debug3: send packet: type 20
debug1: SSH2_MSG_KEXINIT sent
debug3: receive packet: type 20
debug1: SSH2_MSG_KEXINIT received
debug2: local client KEXINIT proposal
debug2: KEX algorithms: curve25519-sha256,
curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group14-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,ext-info-c
debug2: host key algorithms:
ecdsa-sha2-nistp256-cert-v01@openssh.com,
ecdsa-sha2-nistp384-cert-v01@openssh.com,
ecdsa-sha2-nistp521-cert-v01@openssh.com,
sk-ecdsa-sha2-nistp256-cert-v01@openssh.com,
ssh-ed25519-cert-v01@openssh.com,
sk-ssh-ed25519-cert-v01@openssh.com,
rsa-sha2-512-cert-v01@openssh.com,
rsa-sha2-256-cert-v01@openssh.com,
ssh-rsa-cert-v01@openssh.com,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,
sk-ecdsa-sha2-nistp256@openssh.com,ssh-ed25519,
sk-ssh-ed25519@openssh.com,rsa-sha2-512,rsa-sha2-256,ssh-rsa
debug2: ciphers ctos:
aes256-gcm@openssh.com,
chacha20-poly1305@openssh.com,aes256-ctr,
aes128-gcm@openssh.com,aes128-ctr
debug2: ciphers stoc:
aes256-gcm@openssh.com,
chacha20-poly1305@openssh.com,aes256-ctr,
aes128-gcm@openssh.com,aes128-ctr
debug2: MACs ctos:
hmac-sha2-256-etm@openssh.com,
hmac-sha1-etm@openssh.com,
umac-128-etm@openssh.com,
hmac-sha2-512-etm@openssh.com,hmac-sha2-256,hmac-sha1,
umac-128@openssh.com,hmac-sha2-512
debug2: MACs stoc:
hmac-sha2-256-etm@openssh.com,
hmac-sha1-etm@openssh.com,
umac-128-etm@openssh.com,
hmac-sha2-512-etm@openssh.com,hmac-sha2-256,hmac-sha1,
umac-128@openssh.com,hmac-sha2-512
debug2: compression ctos: none,
zlib@openssh.com,zlib
debug2: compression stoc: none,
zlib@openssh.com,zlib
debug2: languages ctos:
debug2: languages stoc:
debug2: first_kex_follows 0
debug2: reserved 0
debug2: peer server KEXINIT proposal
debug2: KEX algorithms: curve25519-sha256,
curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group14-sha256
debug2: host key algorithms: rsa-sha2-512,rsa-sha2-256,ssh-rsa,ecdsa-sha2-nistp256,ssh-ed25519
debug2: ciphers ctos:
chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,
aes128-gcm@openssh.com,
aes256-gcm@openssh.comdebug2: ciphers stoc:
chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,
aes128-gcm@openssh.com,
aes256-gcm@openssh.comdebug2: MACs ctos:
umac-64-etm@openssh.com,
umac-128-etm@openssh.com,
hmac-sha2-256-etm@openssh.com,
hmac-sha2-512-etm@openssh.com,
hmac-sha1-etm@openssh.com,
umac-64@openssh.com,
umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: MACs stoc:
umac-64-etm@openssh.com,
umac-128-etm@openssh.com,
hmac-sha2-256-etm@openssh.com,
hmac-sha2-512-etm@openssh.com,
hmac-sha1-etm@openssh.com,
umac-64@openssh.com,
umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: compression ctos: none,
zlib@openssh.comdebug2: compression stoc: none,
zlib@openssh.comdebug2: languages ctos:
debug2: languages stoc:
debug2: first_kex_follows 0
debug2: reserved 0
debug1: kex: algorithm: curve25519-sha256
debug1: kex: host key algorithm: ecdsa-sha2-nistp256
debug1: kex: server->client cipher:
aes256-gcm@openssh.com MAC: <implicit> compression: none
debug1: kex: client->server cipher:
aes256-gcm@openssh.com MAC: <implicit> compression: none
debug1: kex: curve25519-sha256 need=32 dh_need=32
debug1: kex: curve25519-sha256 need=32 dh_need=32
debug3: send packet: type 30
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug3: receive packet: type 31
debug1: Server host key: ecdsa-sha2-nistp256 SHA256:yVm8V2ODZo0nAuvr9k2ydTJv0RtOgkl8Sp5Mkmp/F0M
debug3: hostkeys_foreach: reading file "/root/.ssh/known_hosts"
debug3: record_hostkey: found key type ECDSA in file /root/.ssh/known_hosts:31
debug3: load_hostkeys: loaded 1 keys from ourserver
debug3: hostkeys_foreach: reading file "/var/lib/sss/pubconf/known_hosts"
debug1: Host 'ourserver' is known and matches the ECDSA host key.
debug1: Found key in /root/.ssh/known_hosts:31
debug3: send packet: type 21
debug2: set_newkeys: mode 1
debug1: rekey out after 4294967296 blocks
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug3: receive packet: type 21
debug1: SSH2_MSG_NEWKEYS received
debug2: set_newkeys: mode 0
debug1: rekey in after 4294967296 blocks
debug1: Will attempt key: /root/.ssh/id_rsa RSA SHA256:Sena4XB1wVt7x+o55Y9EI5WnQIyZ/SLFk+t6tmBFMYA
debug1: Will attempt key: /root/.ssh/id_dsa
debug1: Will attempt key: /root/.ssh/id_ecdsa
debug1: Will attempt key: /root/.ssh/id_ecdsa_sk
debug1: Will attempt key: /root/.ssh/id_ed25519
debug1: Will attempt key: /root/.ssh/id_ed25519_sk
debug1: Will attempt key: /root/.ssh/id_xmss
debug2: pubkey_prepare: done
debug3: send packet: type 5
debug3: receive packet: type 7
debug1: SSH2_MSG_EXT_INFO received
debug1: kex_input_ext_info: server-sig-algs=<ssh-ed25519,
sk-ssh-ed25519@openssh.com,ssh-rsa,rsa-sha2-256,rsa-sha2-512,ssh-dss,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,
sk-ecdsa-sha2-nistp256@openssh.com,
webauthn-sk-ecdsa-sha2-nistp256@openssh.com>
debug3: receive packet: type 6
debug2: service_accept: ssh-userauth
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug3: send packet: type 50
debug3: receive packet: type 51
debug1: Authentications that can continue: publickey,gssapi-keyex,gssapi-with-mic,password,keyboard-interactive
debug3: start over, passed a different list publickey,gssapi-keyex,gssapi-with-mic,password,keyboard-interactive
debug3: preferred gssapi-with-mic,keyboard-interactive,password
debug3: authmethod_lookup gssapi-with-mic
debug3: remaining preferred: keyboard-interactive,password
debug3: authmethod_is_enabled gssapi-with-mic
debug1: Next authentication method: gssapi-with-mic
debug1: Unspecified GSS failure. Minor code may provide more information
Server host/
ourserver.edu@ourserver.edu not found in Kerberos database
And here are the server logs from /var/log secure and you can see sssd is being used:
Feb 10 14:36:24 ourserver sshd[3024290]: debug1: Forked child 3084339.
Feb 10 14:36:24 ourserver sshd[3084339]: debug1: Set /proc/self/oom_score_adj to 0
Feb 10 14:36:24 ourserver sshd[3084339]: debug1: rexec start in 4 out 4 newsock 4 pipe 6 sock 7
Feb 10 14:36:24 ourserver sshd[3084339]: debug1: inetd sockets after dupping: 4, 4
Feb 10 14:36:24 ourserver sshd[3084339]: Connection from x.x.x.x port 34160 on 150.108.68.128 port 22 rdomain ""
Feb 10 14:36:24 ourserver sshd[3084339]: debug1: Local version string SSH-2.0-OpenSSH_8.4
Feb 10 14:36:24 ourserver sshd[3084339]: debug1: Remote protocol version 2.0, remote software version OpenSSH_8.4
Feb 10 14:36:24 ourserver sshd[3084339]: debug1: match: OpenSSH_8.4 pat OpenSSH* compat 0x04000000
Feb 10 14:36:24 ourserver sshd[3084339]: debug1: SELinux support disabled [preauth]
Feb 10 14:36:24 ourserver sshd[3084339]: debug1: permanently_set_uid: 74/74 [preauth]
Feb 10 14:36:24 ourserver sshd[3084339]: debug1: list_hostkey_types: rsa-sha2-512,rsa-sha2-256,ssh-rsa,ecdsa-sha2-nistp256,ssh-ed25519 [preauth]
Feb 10 14:36:24 ourserver sshd[3084339]: debug1: SSH2_MSG_KEXINIT sent [preauth]
Feb 10 14:36:24 ourserver sshd[3084339]: debug1: SSH2_MSG_KEXINIT received [preauth]
Feb 10 14:36:24 ourserver sshd[3084339]: debug1: kex: algorithm: curve25519-sha256 [preauth]
Feb 10 14:36:24 ourserver sshd[3084339]: debug1: kex: host key algorithm: ecdsa-sha2-nistp256 [preauth]
Feb 10 14:36:24 ourserver sshd[3084339]: debug1: kex: client->server cipher:
aes256-gcm@openssh.com MAC: <implicit> compression: none [preauth]
Feb 10 14:36:24 ourserver sshd[3084339]: debug1: kex: server->client cipher:
aes256-gcm@openssh.com MAC: <implicit> compression: none [preauth]
Feb 10 14:36:24 ourserver sshd[3084339]: debug1: kex: curve25519-sha256 need=32 dh_need=32 [preauth]
Feb 10 14:36:24 ourserver sshd[3084339]: debug1: kex: curve25519-sha256 need=32 dh_need=32 [preauth]
Feb 10 14:36:24 ourserver sshd[3084339]: debug1: expecting SSH2_MSG_KEX_ECDH_INIT [preauth]
Feb 10 14:36:24 ourserver sshd[3084339]: debug1: rekey out after 4294967296 blocks [preauth]
Feb 10 14:36:24 ourserver sshd[3084339]: debug1: SSH2_MSG_NEWKEYS sent [preauth]
Feb 10 14:36:24 ourserver sshd[3084339]: debug1: Sending SSH2_MSG_EXT_INFO [preauth]
Feb 10 14:36:24 ourserver sshd[3084339]: debug1: expecting SSH2_MSG_NEWKEYS [preauth]
Feb 10 14:36:24 ourserver sshd[3084339]: debug1: SSH2_MSG_NEWKEYS received [preauth]
Feb 10 14:36:24 ourserver sshd[3084339]: debug1: rekey in after 4294967296 blocks [preauth]
Feb 10 14:36:24 ourserver sshd[3084339]: debug1: KEX done [preauth]
Feb 10 14:36:24 ourserver sshd[3084339]: debug1: userauth-request for user ouruser service ssh-connection method none [preauth]
Feb 10 14:36:24 ourserver sshd[3084339]: debug1: attempt 0 failures 0 [preauth]
Feb 10 14:36:24 ourserver sshd[3084339]: debug1: connection from x.x.x.x matched 'Address 192.168.0.*,127.0.0.1,10.10.1.*' at line 158
Feb 10 14:36:24 ourserver sshd[3084339]: debug1: PAM: initializing for "ouruser"
Feb 10 14:36:24 ourserver sshd[3084339]: debug1: PAM: setting PAM_RHOST to "x.x.x.x"
Feb 10 14:36:24 ourserver sshd[3084339]: debug1: PAM: setting PAM_TTY to "ssh"
Feb 10 14:36:24 ourserver sshd[3084339]: debug1: userauth-request for user ouruser service ssh-connection method gssapi-with-mic [preauth]
Feb 10 14:36:24 ourserver sshd[3084339]: debug1: attempt 1 failures 0 [preauth]
Feb 10 14:36:24 ourserver sshd[3084339]: debug1: userauth-request for user ouruser service ssh-connection method keyboard-interactive [preauth]
Feb 10 14:36:24 ourserver sshd[3084339]: debug1: attempt 2 failures 0 [preauth]
Feb 10 14:36:24 ourserver sshd[3084339]: debug1: keyboard-interactive devs [preauth]
Feb 10 14:36:24 ourserver sshd[3084339]: debug1: auth2_challenge: user=ouruser devs= [preauth]
Feb 10 14:36:24 ourserver sshd[3084339]: debug1: kbdint_alloc: devices 'pam' [preauth]
Feb 10 14:36:24 ourserver sshd[3084339]: debug1: auth2_challenge_start: trying authentication method 'pam' [preauth]
Feb 10 14:36:24 ourserver sshd[3084339]: Postponed keyboard-interactive for ouruser from x.x.x.x port 34160 ssh2 [preauth]
Feb 10 14:36:28 ourserver sshd[3084344]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=x.x.x.x user=ouruser
Feb 10 14:36:28 ourserver proxy_child: pam_unix(sssd-shadowutils:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=x.x.x.x user=ouruser
Feb 10 14:36:31 ourserver sshd[3084344]: pam_sss(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=x.x.x.x user=ouruser
Feb 10 14:36:31 ourserver sshd[3084344]: pam_sss(sshd:auth): received for user ouruser: 7 (Authentication failure)
Feb 10 14:36:33 ourserver sshd[3084339]: error: PAM: Authentication failure for ouruser from x.x.x.x
Feb 10 14:36:33 ourserver sshd[3084339]: Failed keyboard-interactive/pam for ouruser from x.x.x.x port 34160 ssh2
Feb 10 14:36:33 ourserver sshd[3084339]: debug1: userauth-request for user ouruser service ssh-connection method keyboard-interactive [preauth]
Feb 10 14:36:33 ourserver sshd[3084339]: debug1: attempt 3 failures 1 [preauth]
Feb 10 14:36:33 ourserver sshd[3084339]: debug1: keyboard-interactive devs [preauth]
Feb 10 14:36:33 ourserver sshd[3084339]: debug1: auth2_challenge: user=ouruser devs= [preauth]
Feb 10 14:36:33 ourserver sshd[3084339]: debug1: kbdint_alloc: devices 'pam' [preauth]
Feb 10 14:36:33 ourserver sshd[3084339]: debug1: auth2_challenge_start: trying authentication method 'pam' [preauth]
Feb 10 14:36:33 ourserver sshd[3084339]: Postponed keyboard-interactive for ouruser from x.x.x.x port 34160 ssh2 [preauth]
I verified the FreeIPA password in both the GUI and via ipa user-mod. The only time the user is able to log in is using the NIS password. ldapsearch -x -D and kinit username work successfully. klist displays the user details correctly.
Feb 10 14:56:39 ourserver sshd[3085147]: debug1: userauth-request for user ouruser service ssh-connection method none [preauth]
Feb 10 14:56:39 ourserver sshd[3085147]: debug1: attempt 0 failures 0 [preauth]
Feb 10 14:56:39 ourserver sshd[3085147]: debug1: connection from x.x.x.x matched 'Address 192.168.0.*,127.0.0.1,10.10.1.*' at line 158
Feb 10 14:56:39 ourserver sshd[3085147]: debug1: PAM: initializing for "ouruser"
Feb 10 14:56:39 ourserver sshd[3085147]: debug1: PAM: setting PAM_RHOST to "x.x.x.x"
Feb 10 14:56:39 ourserver sshd[3085147]: debug1: PAM: setting PAM_TTY to "ssh"
Feb 10 14:56:39 ourserver sshd[3085147]: debug1: userauth-request for user ouruser service ssh-connection method gssapi-with-mic [preauth]
Feb 10 14:56:39 ourserver sshd[3085147]: debug1: attempt 1 failures 0 [preauth]
Feb 10 14:56:39 ourserver sshd[3085147]: debug1: userauth-request for user ouruser service ssh-connection method keyboard-interactive [preauth]
Feb 10 14:56:39 ourserver sshd[3085147]: debug1: attempt 2 failures 0 [preauth]
Feb 10 14:56:39 ourserver sshd[3085147]: debug1: keyboard-interactive devs [preauth]
Feb 10 14:56:39 ourserver sshd[3085147]: debug1: auth2_challenge: user=ouruser devs= [preauth]
Feb 10 14:56:39 ourserver sshd[3085147]: debug1: kbdint_alloc: devices 'pam' [preauth]
Feb 10 14:56:39 ourserver sshd[3085147]: debug1: auth2_challenge_start: trying authentication method 'pam' [preauth]
Feb 10 14:56:39 ourserver sshd[3085147]: Postponed keyboard-interactive for ouruser from x.x.x.x port 35046 ssh2 [preauth]
Feb 10 14:56:42 ourserver sshd[3085152]: debug1: do_pam_account: called
Feb 10 14:56:42 ourserver sshd[3085147]: debug1: PAM: num PAM env strings 2
Feb 10 14:56:42 ourserver sshd[3085147]: Postponed keyboard-interactive/pam for ouruser from x.x.x.x port 35046 ssh2 [preauth]
Feb 10 14:56:42 ourserver sshd[3085147]: debug1: do_pam_account: called