New FreeIPA deployment, and i have one server that is not allowing Kerberos to handle authentication, but instead is prompting for password with a valid kerberos ticket. All other machines are working normally. I've double-checked the /etc/ssh/sshd_config file, identical between the one not working, and the one that is. Done the same for SSSD and IPA configuration info. Entering password on the machine does work, and does result in a valid ticket being issued. Below is some debug info, generated with "KRB5_TRACE=/dev/stdout ssh -vvv {hostname}", and truncated down to only parts that differ:

On a working machine:

debug1: Next authentication method: gssapi-with-mic

[28004] 1508434137.499258: ccselect can't find appropriate cache for server principal host/tc-adm01.trustcharge.net@

[28004] 1508434137.499490: Getting credentials jeremy@IPA.TRUSTCHARGE.NET -> host/tc-adm01.trustcharge.net@ using ccache KEYRING:persistent:1001:krb_ccache_MjbcsDY

[28004] 1508434137.499669: Retrieving jeremy@IPA.TRUSTCHARGE.NET -> host/tc-adm01.trustcharge.net@ from KEYRING:persistent:1001:krb_ccache_MjbcsDY with result: -1765328243/Matching credential not found

[28004] 1508434137.499768: Retrying jeremy@IPA.TRUSTCHARGE.NET -> host/tc-adm01.trustcharge.net@IPA.TRUSTCHARGE.NET with result: -1765328243/Matching credential not found

[28004] 1508434137.499778: Server has referral realm; starting with host/tc-adm01.trustcharge.net@IPA.TRUSTCHARGE.NET

[28004] 1508434137.499878: Retrieving jeremy@IPA.TRUSTCHARGE.NET -> krbtgt/IPA.TRUSTCHARGE.NET@IPA.TRUSTCHARGE.NET from KEYRING:persistent:1001:krb_ccache_MjbcsDY with result: 0/Success

[28004] 1508434137.499888: Starting with TGT for client realm: jeremy@IPA.TRUSTCHARGE.NET -> krbtgt/IPA.TRUSTCHARGE.NET@IPA.TRUSTCHARGE.NET

[28004] 1508434137.499900: Requesting tickets for host/tc-adm01.trustcharge.net@IPA.TRUSTCHARGE.NET, referrals on

[28004] 1508434137.499961: Generated subkey for TGS request: aes256-cts/B274

[28004] 1508434137.500054: etypes requested in TGS request: aes256-cts, aes128-cts, des3-cbc-sha1, rc4-hmac, camellia128-cts, camellia256-cts

[28004] 1508434137.500259: Encoding request body and padata into FAST request

[28004] 1508434137.500374: Sending request (985 bytes) to IPA.TRUSTCHARGE.NET

[28004] 1508434137.500660: Initiating TCP connection to stream 172.31.92.18:88

[28004] 1508434137.501228: Sending TCP request to stream 172.31.92.18:88

[28004] 1508434137.507122: Received answer (937 bytes) from stream 172.31.92.18:88

[28004] 1508434137.507139: Terminating TCP connection to stream 172.31.92.18:88

[28004] 1508434137.507240: Response was from master KDC

[28004] 1508434137.507273: Decoding FAST response

[28004] 1508434137.507439: FAST reply key: aes256-cts/9BE9

[28004] 1508434137.507497: TGS reply is for jeremy@IPA.TRUSTCHARGE.NET -> host/tc-adm01.trustcharge.net@IPA.TRUSTCHARGE.NET with session key aes256-cts/CD56

[28004] 1508434137.507522: TGS request result: 0/Success

[28004] 1508434137.507529: Received creds for desired service host/tc-adm01.trustcharge.net@IPA.TRUSTCHARGE.NET

[28004] 1508434137.507543: Storing jeremy@IPA.TRUSTCHARGE.NET -> host/tc-adm01.trustcharge.net@ in KEYRING:persistent:1001:krb_ccache_MjbcsDY

[28004] 1508434137.507690: Also storing jeremy@IPA.TRUSTCHARGE.NET -> host/tc-adm01.trustcharge.net@IPA.TRUSTCHARGE.NET based on ticket

[28004] 1508434137.507704: Removing jeremy@IPA.TRUSTCHARGE.NET -> host/tc-adm01.trustcharge.net@IPA.TRUSTCHARGE.NET from KEYRING:persistent:1001:krb_ccache_MjbcsDY

[28004] 1508434137.507911: Creating authenticator for jeremy@IPA.TRUSTCHARGE.NET -> host/tc-adm01.trustcharge.net@, seqnum 291429769, subkey aes256-cts/A214, session key aes256-cts/CD56

debug2: we sent a gssapi-with-mic packet, wait for reply

[28004] 1508434137.511804: ccselect can't find appropriate cache for server principal host/tc-adm01.trustcharge.net@

[28004] 1508434137.511964: Getting credentials jeremy@IPA.TRUSTCHARGE.NET -> host/tc-adm01.trustcharge.net@ using ccache KEYRING:persistent:1001:krb_ccache_MjbcsDY

[28004] 1508434137.512124: Retrieving jeremy@IPA.TRUSTCHARGE.NET -> host/tc-adm01.trustcharge.net@ from KEYRING:persistent:1001:krb_ccache_MjbcsDY with result: 0/Success

[28004] 1508434137.512197: Creating authenticator for jeremy@IPA.TRUSTCHARGE.NET -> host/tc-adm01.trustcharge.net@, seqnum 487674855, subkey aes256-cts/0383, session key aes256-cts/CD56

[28004] 1508434137.670683: Read AP-REP, time 1508434137.512205, subkey aes256-cts/2950, seqnum 529391729

debug1: Authentication succeeded (gssapi-with-mic).

On failing machine:

debug1: Next authentication method: gssapi-with-mic

[23080] 1508434210.54069: ccselect module realm chose cache FILE:/tmp/krb5cc_1001 with client principal jeremy@IPA.TRUSTCHARGE.NET for server principal host/tc-log01.trustcharge.net@IPA.TRUSTCHARGE.NET

[23080] 1508434210.54141: Retrieving jeremy@IPA.TRUSTCHARGE.NET -> krb5_ccache_conf_data/proxy_impersonator@X-CACHECONF: from FILE:/tmp/krb5cc_1001 with result: -1765328243/Matching credential not found

[23080] 1508434210.54160: Getting credentials jeremy@IPA.TRUSTCHARGE.NET -> host/tc-log01.trustcharge.net@IPA.TRUSTCHARGE.NET using ccache FILE:/tmp/krb5cc_1001

[23080] 1508434210.54207: Retrieving jeremy@IPA.TRUSTCHARGE.NET -> host/tc-log01.trustcharge.net@IPA.TRUSTCHARGE.NET from FILE:/tmp/krb5cc_1001 with result: -1765328243/Matching credential not found

[23080] 1508434210.54242: Retrieving jeremy@IPA.TRUSTCHARGE.NET -> krbtgt/IPA.TRUSTCHARGE.NET@IPA.TRUSTCHARGE.NET from FILE:/tmp/krb5cc_1001 with result: 0/Success

[23080] 1508434210.54248: Found cached TGT for service realm: jeremy@IPA.TRUSTCHARGE.NET -> krbtgt/IPA.TRUSTCHARGE.NET@IPA.TRUSTCHARGE.NET

[23080] 1508434210.54253: Requesting tickets for host/tc-log01.trustcharge.net@IPA.TRUSTCHARGE.NET, referrals on

[23080] 1508434210.54285: Generated subkey for TGS request: aes256-cts/52BF

[23080] 1508434210.54292: etypes requested in TGS request: aes256-cts, aes128-cts, des3-cbc-sha1, rc4-hmac

[23080] 1508434210.54411: Sending request (740 bytes) to IPA.TRUSTCHARGE.NET

[23080] 1508434210.54541: Initiating TCP connection to stream 172.31.92.18:88

[23080] 1508434210.54902: Sending TCP request to stream 172.31.92.18:88

[23080] 1508434210.60311: Received answer from stream 172.31.92.18:88

[23080] 1508434210.60349: Response was from master KDC

[23080] 1508434210.60409: TGS reply is for jeremy@IPA.TRUSTCHARGE.NET -> host/tc-log01.trustcharge.net@IPA.TRUSTCHARGE.NET with session key aes256-cts/98CE

[23080] 1508434210.60438: TGS request result: 0/Success

[23080] 1508434210.60444: Received creds for desired service host/tc-log01.trustcharge.net@IPA.TRUSTCHARGE.NET

[23080] 1508434210.60450: Removing jeremy@IPA.TRUSTCHARGE.NET -> host/tc-log01.trustcharge.net@IPA.TRUSTCHARGE.NET from FILE:/tmp/krb5cc_1001

[23080] 1508434210.60455: Storing jeremy@IPA.TRUSTCHARGE.NET -> host/tc-log01.trustcharge.net@IPA.TRUSTCHARGE.NET in FILE:/tmp/krb5cc_1001

[23080] 1508434210.60557: Creating authenticator for jeremy@IPA.TRUSTCHARGE.NET -> host/tc-log01.trustcharge.net@IPA.TRUSTCHARGE.NET, seqnum 77295956, subkey aes256-cts/5E8E, session key aes256-cts/98CE

debug2: we sent a gssapi-with-mic packet, wait for reply

debug3: Wrote 100 bytes for a total of 1417

debug1: Authentications that can continue: publickey,gssapi-keyex,gssapi-with-mic,password

[23080] 1508434210.62494: ccselect module realm chose cache FILE:/tmp/krb5cc_1001 with client principal jeremy@IPA.TRUSTCHARGE.NET for server principal host/tc-log01.trustcharge.net@IPA.TRUSTCHARGE.NET

[23080] 1508434210.62534: Retrieving jeremy@IPA.TRUSTCHARGE.NET -> krb5_ccache_conf_data/proxy_impersonator@X-CACHECONF: from FILE:/tmp/krb5cc_1001 with result: -1765328243/Matching credential not found

[23080] 1508434210.62542: Getting credentials jeremy@IPA.TRUSTCHARGE.NET -> host/tc-log01.trustcharge.net@IPA.TRUSTCHARGE.NET using ccache FILE:/tmp/krb5cc_1001

[23080] 1508434210.62574: Retrieving jeremy@IPA.TRUSTCHARGE.NET -> host/tc-log01.trustcharge.net@IPA.TRUSTCHARGE.NET from FILE:/tmp/krb5cc_1001 with result: 0/Success

[23080] 1508434210.62628: Getting credentials jeremy@IPA.TRUSTCHARGE.NET -> host/tc-log01.trustcharge.net@IPA.TRUSTCHARGE.NET using ccache FILE:/tmp/krb5cc_1001

[23080] 1508434210.62662: Retrieving jeremy@IPA.TRUSTCHARGE.NET -> host/tc-log01.trustcharge.net@IPA.TRUSTCHARGE.NET from FILE:/tmp/krb5cc_1001 with result: 0/Success

[23080] 1508434210.62689: Creating authenticator for jeremy@IPA.TRUSTCHARGE.NET -> host/tc-log01.trustcharge.net@IPA.TRUSTCHARGE.NET, seqnum 764360366, subkey aes256-cts/1570, session key aes256-cts/98CE

debug2: we sent a gssapi-with-mic packet, wait for reply

debug3: Wrote 100 bytes for a total of 1517

debug1: Authentications that can continue: publickey,gssapi-keyex,gssapi-with-mic,password

Any ideas what could be going wrong? I'm not real familiar with the internals of Kerberos/GSSAPI, but it seems that is where it is failing.

Jeremy