I believe we've made some progress but not quite there yet. Just to recap, any NEW user created via CLI or GUI can connect via ssh. All imported NIS users can only log in with their NIS password. I change the user's password in the UI and check the Password checkbox in User authentication type and click Save. I successfully added a client:

ipa host-add-managedby --hosts=ourdomain.edu client.ourdomain.edu
Host name: client.ourdomain.edu
Platform: x86_64
Operating system: 5.10.9-201.fc33.x86_64
Principal name: host/client.ourdomain.edu@OURDOMAIN.EDU
Principal alias: host/client.ourdomain.edu@OURDOMAIN.EDU
Managed by: client.ourdomain.edu, ourdomain.edu
-------------------------
Number of members added 1
-------------------------
[root@ourdomain ~]# ipa-getkeytab -s ourdomain.edu -p host/client.ourdomain.edu -k /tmp/client.keytab

Keytab successfully retrieved and stored in: /tmp/client.keytab

Based on this SF discussion, I changed: in /etc/krb5.conf

default_ccache_name = FILE:/tmp/krb5cc_%{uid}

Before the change in the krb5.log file I would see:

LOOKING_UP_SERVER: authtime 0, etypes {rep=UNSUPPORTED:(0)} host/ourdomain.edu@OURDOMAIN.EDU for nfs/client.ourdomain.edu@OUDOMAIN.EDU, Server not found in Kerberos database

Now I see:

Mar 11 13:38:28 ourdomain.edu u krb5kdc[369141](info): AS_REQ (6 etypes {aes256-cts-hmac-sha1-96(18), aes128-cts-hmac-sha1-96(17), aes256-cts-hmac-sha384-192(20), camellia256-cts-cmac(26), aes128-cts-hmac-sha256-128(19), camellia128-cts-cmac(25)}) 150.108.64.55: NEEDED_PREAUTH: host/client. ourdomain.edu .edu@OURDOMAIN.EDU for krbtgt/ OURDOMAIN.EDU @ OURDOMAIN.EDU , Additional pre-authentication required
Mar 11 13:38:28 ourdomain.edu krb5kdc[369141](info): closing down fd 11
Mar 11 13:38:28 ourdomain.edu krb5kdc[369144](info): preauth (spake) verify failure: Preauthentication failed

I had run kinit admin a few times but it seems to expire with the old 1969 date but another kinit admin fixes that:

klist
Ticket cache: FILE:/tmp/krb5cc_0
Default principal: host/ourdomain.edu@OURDOMAIN.EDU

Valid starting Expires Service principal
12/31/69 19:00:00 12/31/69 19:00:00 Encrypted/Credentials/v1@X-GSSPROXY:

[root@ourdomain ~]# kinit admin
Password for admin@OURDOMAIN.EDU:

[root@ourdomain ~]# klist
Ticket cache: FILE:/tmp/krb5cc_0
Default principal: admin@OURDOMAIN.EDU

Valid starting Expires Service principal
03/11/21 13:38:27 03/12/21 13:38:24 krbtgt/OURDOMAIN.EDU@OURDOMAIN.EDU

ipa -vv pwpolicy-show global_policy
ipa: INFO: Request: {
"id": 0,
"method": "pwpolicy_show/1",
"params": [
[
"global_policy"
],
{
"version": "2.240"
}
]
}
ipa: INFO: Response: {
"error": null,
"id": 0,
"principal": "admin@OURDOMAIN.EDU",
"result": {
"result": {
"cn": [
"global_policy"
],
"dn": "cn=global_policy,cn=OURDOMAIN.EDU,cn=kerberos,dc=ourdomain,dc=edu",
"krbmaxpwdlife": [
"90"
],
"krbminpwdlife": [
"1"
],
"krbpwdfailurecountinterval": [
"60"
],
"krbpwdhistorylength": [
"0"
],
"krbpwdlockoutduration": [
"600"
],
"krbpwdmaxfailure": [
"6"
],
"krbpwdmindiffchars": [
"0"
],
"krbpwdminlength": [
"8"
]
},
"summary": null,
"value": "global_policy"
},
"version": "4.9.2"
}
Group: global_policy
Max lifetime (days): 90
Min lifetime (hours): 1
History size: 0
Character classes: 0
Min length: 8
Max failures: 6
Failure reset interval: 60
Lockout duration: 600

Alas ssh -k fails again with these SSH server logs:
Mar 11 13:38:58 ourdomain sshd[372719]: debug1: attempt 2 failures 1 [preauth]
Mar 11 13:38:58 ourdomain sshd[372719]: debug1: keyboard-interactive devs [preauth]
Mar 11 13:38:58 ourdomain sshd[372719]: debug1: auth2_challenge: user=ouruser devs= [preauth]
Mar 11 13:38:58 ourdomain sshd[372719]: debug1: kbdint_alloc: devices 'pam' [preauth]
Mar 11 13:38:58 ourdomain sshd[372719]: debug1: auth2_challenge_start: trying authentication method 'pam' [preauth]
Mar 11 13:38:58 ourdomain sshd[372719]: Postponed keyboard-interactive for ouruser from xx.xx.xx.xx port 41834 ssh2 [preauth]
Mar 11 13:39:01 ourdomain sshd[372726]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=xx.xx.xx.xx user=ouruser
Mar 11 13:39:01 ourdomain sshd[372726]: pam_sss(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=xx.xx.xx.xx user=ouruser
Mar 11 13:39:01 ourdomain sshd[372726]: pam_sss(sshd:auth): received for user ouruser: 9 (Authentication service cannot retrieve authentication info)
Mar 11 13:39:04 ourdomain sshd[372719]: error: PAM: Authentication failure for ouruser from xx.xx.xx.xx

And with ssh -vv I still see:

debug1: Next authentication method: gssapi-with-mic
debug1: Unspecified GSS failure. Minor code may provide more information
No Kerberos credentials available (default cache: KEYRING:persistent:0)
debug1: Unspecified GSS failure. Minor code may provide more information
No Kerberos credentials available (default cache: KEYRING:persistent:0)
debug2: we did not send a packet, disable method
debug1: Next authentication method: publickey
debug1: Offering public key: /root/.ssh/id_rsa RSA SHA256:2ucGhU53Ue6Z8BbwowH5U3ykOoVL8F8oN1NbPUCt2vU
debug2: we sent a publickey packet, wait for reply
debug1: Authentications that can continue: publickey,gssapi-keyex,gssapi-with-mic,password,keyboard-interactive

What am I missing?

On Wed, Mar 3, 2021 at 2:24 PM Robert Kudyba <rkudyba@fordham.edu> wrote:

have you enabled the migration mode with

ipa config-mod --enable-migration=True

I've tried it with True and False. At what point should this be changed to False?

With this authentication with SSSD should fall back to LDAP
authentication if the Kerberos keys are not available and this would
trigger a creation of the Kerberos keys for the user trying to log in.

The fallback appears to be NIS.

The good news is the user can log in to the GUI, i.e.i https://ourserver/ipa/ui and change their password but I do see this error in the Apache error log which sounds like this issue:
[Wed Mar 03 13:53:07.526386 2021] [wsgi:error] [pid 16169:tid 16554] [remote xx.xx.xx.xx:63098] ipa: DEBUG: Destroyed connection context.ldap2_140265125387520
[Wed Mar 03 13:53:07.563873 2021] [:warn] [pid 16174:tid 16239] [client xx.xx.xx.xx:63098] failed to set perms (3140) on file (/run/ipa/ccaches/admin@OURDOMAIN.EDU-jhCS0U)!, referer: https://ourdomain.edu/ipa/ui/
[Wed Mar 03 13:53:07.564720 2021] [wsgi:error] [pid 16170:tid 16545] [remote xx.xx.xx.xx:63098] ipa: DEBUG: WSGI wsgi_dispatch.__call__:
[Wed Mar 03 13:53:07.564838 2021] [wsgi:error] [pid 16170:tid 16545] [remote xx.xx.xx.xx:63098] ipa: DEBUG: WSGI jsonserver_session.__call__:
[Wed Mar 03 13:53:07.570164 2021] [:warn] [pid 16174:tid 16285] [client xx.xx.xx.xx:63076] failed to set perms (3140) on file (/run/ipa/ccaches/adminOURDOMAIN.EDU-jhCS0U)!, referer: https://ourdomain.edu/ipa/ui/

Now from ssh -vvv -k this is what we see:
debug1: Authentications that can continue: publickey,gssapi-keyex,gssapi-with-mic,password,keyboard-interactive
debug3: start over, passed a different list publickey,gssapi-keyex,gssapi-with-mic,password,keyboard-interactive
debug3: preferred gssapi-with-mic,publickey,keyboard-interactive,password
debug3: authmethod_lookup gssapi-with-mic
debug3: remaining preferred: publickey,keyboard-interactive,password
debug3: authmethod_is_enabled gssapi-with-mic
debug1: Next authentication method: gssapi-with-mic
debug1: Unspecified GSS failure. Minor code may provide more information
No Kerberos credentials available (default cache: KEYRING:persistent:0)
debug1: Unspecified GSS failure. Minor code may provide more information
No Kerberos credentials available (default cache: KEYRING:persistent:0)
debug2: we did not send a packet, disable method
debug3: authmethod_lookup publickey
debug3: remaining preferred: keyboard-interactive,password
debug3: authmethod_is_enabled publickey
debug1: Next authentication method: publickey
debug1: Offering public key: /ouruser/.ssh/id_rsa RSA SHA256:2ucGhU53Ue6Z8BbwowH5U3ykOoVL8F8oN1NbPUCt2vU
debug3: send packet: type 50
debug2: we sent a publickey packet, wait for reply
debug3: receive packet: type 51
debug1: Authentications that can continue: publickey,gssapi-keyex,gssapi-with-mic,password,keyboard-interactive
debug1: Trying private key: /ouruser/.ssh/id_dsa
debug3: no such identity: /ouruser/.ssh/id_dsa: No such file or directory
debug1: Trying private key: /ouruser/.ssh/id_ecdsa
debug3: no such identity: /ouruser/.ssh/id_ecdsa: No such file or directory
debug1: Trying private key: /ouruser/.ssh/id_ecdsa_sk
debug3: no such identity: /ouruser/.ssh/id_ecdsa_sk: No such file or directory
debug1: Trying private key: /ouruser/.ssh/id_ed25519
debug3: no such identity: /ouruser/.ssh/id_ed25519: No such file or directory
debug1: Trying private key: /ouruser/.ssh/id_ed25519_sk
debug3: no such identity: /ouruser/.ssh/id_ed25519_sk: No such file or directory
debug1: Trying private key: /ouruser/.ssh/id_xmss
debug3: no such identity: /ouruser/.ssh/id_xmss: No such file or directory
debug2: we did not send a packet, disable method
debug3: authmethod_lookup keyboard-interactive
debug3: remaining preferred: password
debug3: authmethod_is_enabled keyboard-interactive
debug1: Next authentication

Is this a clue?
Unspecified GSS failure. Minor code may provide more information
No Kerberos credentials available (default cache: KEYRING:persistent:0)

From the ssh server logs, set to debug:
Mar 3 14:00:46 ourserver sshd[79161]: debug1: attempt 0 failures 0 [preauth]
Mar 3 14:00:46 ourserver sshd[79161]: debug1: PAM: initializing for "ouruser"
Mar 3 14:00:46 ourserver sshd[79161]: debug1: PAM: setting PAM_RHOST to "x.x.x.x"
Mar 3 14:00:46 ourserver sshd[79161]: debug1: PAM: setting PAM_TTY to "ssh"
Mar 3 14:00:46 ourserver sshd[79161]: debug1: userauth-request for user ouruser service ssh-connection method publickey [preauth]
Mar 3 14:00:46 ourserver sshd[79161]: debug1: attempt 1 failures 0 [preauth]
Mar 3 14:00:46 ourserver sshd[79161]: debug1: userauth_pubkey: test pkalg rsa-sha2-512 pkblob RSA SHA256:2ucGhU53Ue6Z8BbwowH5U3ykOoVL8F8oN1NbPUCt2vU [preauth]
Mar 3 14:00:46 ourserver sshd[79161]: debug1: temporarily_use_uid: 6915/200 (e=0/0)
Mar 3 14:00:46 ourserver sshd[79161]: debug1: trying public key file /home/ouruser/.ssh/authorized_keys
Mar 3 14:00:46 ourserver sshd[79161]: debug1: Could not open authorized keys '/home/ouruser/.ssh/authorized_keys': No such file or directory
Mar 3 14:00:46 ourserver sshd[79161]: debug1: restore_uid: 0/0
Mar 3 14:00:46 ourserver sshd[79161]: debug1: temporarily_use_uid: 99/99 (e=0/0)
Mar 3 14:00:46 ourserver sshd[79161]: debug1: restore_uid: 0/0
Mar 3 14:00:46 ourserver sshd[79161]: debug1: temporarily_use_uid: 99/99 (e=0/0)
Mar 3 14:00:46 ourserver sshd[79161]: debug1: restore_uid: 0/0
Mar 3 14:00:46 ourserver sshd[79161]: Failed publickey for ouruser from x.x.x.x port 40248 ssh2: RSA SHA256:2ucGhU53Ue6Z8BbwowH5U3ykOoVL8F8oN1NbPUCt2vU
Mar 3 14:00:46 ourserver sshd[79161]: debug1: userauth-request for user ouruser service ssh-connection method keyboard-interactive [preauth]
Mar 3 14:00:46 ourserver sshd[79161]: debug1: attempt 2 failures 1 [preauth]
Mar 3 14:00:46 ourserver sshd[79161]: debug1: keyboard-interactive devs [preauth]
Mar 3 14:00:46 ourserver sshd[79161]: debug1: auth2_challenge: user=ouruser devs= [preauth]
Mar 3 14:00:46 ourserver sshd[79161]: debug1: kbdint_alloc: devices 'pam' [preauth]
Mar 3 14:00:46 ourserver sshd[79161]: debug1: auth2_challenge_start: trying authentication method 'pam' [preauth]
Mar 3 14:00:46 ourserver sshd[79161]: Postponed keyboard-interactive for ouruser from x.x.x.x port 40248 ssh2 [preauth]
Mar 3 14:00:50 ourserver sshd[79168]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=x.x.x.x user=ouruser
Mar 3 14:00:50 ourserver sshd[79168]: pam_sss(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=x.x.x.x user=ouruser
Mar 3 14:00:50 ourserver sshd[79168]: pam_sss(sshd:auth): received for user ouruser: 9 (Authentication service cannot retrieve authentication info)
Mar 3 14:00:52 ourserver sshd[79161]: error: PAM: Authentication failure for ouruser from x.x.x.x
Mar 3 14:00:52 ourserver sshd[79161]: Failed keyboard-interactive/pam for ouruser from x.x.x.x port 40248 ssh2
Mar 3 14:00:52 ourserver sshd[79161]: debug1: userauth-request for user ouruser service ssh-connection method keyboard-interactive [preauth]
Mar 3 14:00:52 ourserver sshd[79161]: debug1: attempt 3 failures 2 [preauth]
Mar 3 14:00:52 ourserver sshd[79161]: debug1: keyboard-interactive devs [preauth]
Mar 3 14:00:52 ourserver sshd[79161]: debug1: auth2_challenge: user=ouruser devs= [preauth]
Mar 3 14:00:52 ourserver sshd[79161]: debug1: kbdint_alloc: devices 'pam' [preauth]
Mar 3 14:00:52 ourserver sshd[79161]: debug1: auth2_challenge_start: trying authentication method 'pam' [preauth]

Here are the server logs on a successful login with the NIS password:
Mar 3 14:06:09 ourserver sshd[79292]: Accepted keyboard-interactive/pam for ouruser from xx.xx.xx.xx port 40252 ssh2
Mar 3 14:06:09 ourserver sshd[79292]: debug1: monitor_child_preauth: ouruser has been authenticated by privileged process
Mar 3 14:06:09 ourserver sshd[79292]: debug1: monitor_read_log: child log fd closed
Mar 3 14:06:09 ourserver sshd[79292]: debug1: audit_event: unhandled event 2
Mar 3 14:06:09 ourserver sshd[79292]: debug1: temporarily_use_uid: 6915/200 (e=0/0)
Mar 3 14:06:09 ourserver sshd[79292]: debug1: ssh_gssapi_storecreds: Not a GSSAPI mechanism
Mar 3 14:06:09 ourserver sshd[79292]: debug1: restore_uid: 0/0
Mar 3 14:06:09 ourserver sshd[79292]: debug1: SELinux support disabled
Mar 3 14:06:09 ourserver sshd[79292]: debug1: PAM: establishing credentials
Mar 3 14:06:09 ourserver systemd[79307]: pam_unix(systemd-user:session): session opened for user ouruser(uid=6915) by (uid=0)
Mar 3 14:06:10 ourserver sshd[79292]: pam_unix(sshd:session): session opened for user ouruser(uid=6915) by (uid=0)
Mar 3 14:06:10 ourserver sshd[79292]: User child is on pid 79320
Mar 3 14:06:10 ourserver sshd[79320]: debug1: PAM: establishing credentials

So it clearly says it's "Not a GSSAPI mechanism".

/etc/nsswitch.conf (which is a symbolic link to /etc/authselect/nsswitch.conf)
passwd: sss files systemd
group: sss files systemd
netgroup: sss files
automount: sss files
services: sss files
sudoers: files sss
shadow: files nis
hosts: files nis mdns4_minimal [NOTFOUND=return] dns myhostname mymachines

And /etc/authselect/user-nsswitch.conf has
passwd: files nis systemd
shadow: files nis
group: files nis systemd
hosts: files nis mdns4_minimal [NOTFOUND=return] dns myhostname mymachines

Why is the Kerberos login failing?