Posting this here before I file a bug because there are a few similar bugs which have been closed, and so I don't know if I'm seeing one of them or whether this is a new one.
On the server, before I SSH in, my credentials cache collection looks like this:
$ printenv KRB5CCNAME KCM:
$ klist -l Principal name Cache name -------------- ---------- user@EXAMPLE.COM KCM:1011:48623 (Expired) user@EXAMPLE.COM KCM:1011:1518 (Expired) user@EXAMPLE.COM KCM:1011:12525 (Expired) user@EXAMPLE.COM KCM:1011:95695 (Expired) user@EXAMPLE.COM KCM:1011:19607 (Expired) user@EXAMPLE.COM KCM:1011:25261 (Expired)
$ klist Ticket cache: KCM:1011:95695 Default principal: user@EXAMPLE.COM
Valid starting Expires Service principal 06/22/22 00:03:52 06/23/22 00:01:49 krbtgt/EXAMPLE.COM@EXAMPLE.COM
The TGT in the primary credentials cache is an old one that expired a few days ago.
Now I SSH into the server, using keyboard-interactive authentication, so sshd calls into pam_sss to authenticate me with Kerberos.
$ ssh -v -l user server.example.com debug1: Next authentication method: keyboard-interactive (user@server.example.com) First Factor: (user@server.example.com) Second Factor: Authenticated to server.example.com (via proxy) using "keyboard-interactive".
$ klist Ticket cache: KCM:1011:48623 Default principal: user@EXAMPLE.COM
Valid starting Expires Service principal 05/24/22 23:39:09 05/25/22 23:16:16 krbtgt/EXAMPLE.COM@EXAMPLE.COM 05/25/22 19:31:42 05/25/22 23:16:16 HTTP/server.example.com@EXAMPLE.COM
The primary credential cache has been changed to a different cache, that contains an old, expired ticket. So the I can't use my TGT to access network services.
If I then list the contents of the original primary credential cache:
$ klist -c KCM:1011:95695 Ticket cache: KCM:1011:95695 Default principal: user@EXAMPLE.COM
Valid starting Expires Service principal 09/28/22 09:37:09 09/29/22 09:36:36 krbtgt/EXAMPLE.COM@EXAMPLE.COM
SSSD put the new TGT into the former primary credential cache, and then changed the primary credential cache to one of the other caches within the collection.
I expected that, regardless of whether the new TGT goes into a new credential cache or an existing one, the destination credential cache is set as the primary so that it can be used by the user.
In particular this looks like https://github.com/SSSD/sssd/issues/4988, but a comment says that that has been fixed in RHEL 8 some time ago, nonetheless I'm seeing this problem on both RHEL 9.0 (sssd-common-2.6.2-4.el9_0.1.x86_64) and RHEL 8.6 (sssd-common-2.6.2-4.el8_6.1.x86_64).
The workaround is to identify the credential cache that contains non-expired credentials and then make it the primary:
$ klist -l Principal name Cache name -------------- ---------- user@EXAMPLE.COM KCM:1011:48623 (Expired) user@EXAMPLE.COM KCM:1011:1518 (Expired) user@EXAMPLE.COM KCM:1011:12525 (Expired) user@EXAMPLE.COM KCM:1011:19607 (Expired) user@EXAMPLE.COM KCM:1011:95695
$ kswitch -c KCM:1011:95695
If I perform the same test but using gssapi-with-mic authentication in combination with GSSAPIDelegateCredentials, then I see that the TGT is stored in a new credentials cache, which is then set as the primary, e.g. everything works!
I also think that sssd should be cleaning up these old credential caches. AFAIK, there's no perfect forward secrecy with Kerberos, so can these old tickets can be used to decrypt captured network traffic?
Regards,