Replaced IPA server - now cannot ssh to it from ipa clients

[Russ Long]

In my homelab, where I only have a single IPA server, I have had to rebuild / replace it due to issues outside of IPA.  I was unable to restore the VM from a snapshot, however I was able to rebuild and use a backup created by ipa-backup to perform a restore.  Everything appeared to work fine, and IPA clients are working, as well as LDAP and other services.  The only issue I'm encountering is that when I try to ssh to the IPA server from one of the clients, SSH hangs here:

$ ssh -vvvvv master
OpenSSH_8.1p1, OpenSSL 1.1.1d FIPS  10 Sep 2019
debug1: Reading configuration data /etc/ssh/ssh_config
debug3: /etc/ssh/ssh_config line 51: Including file /etc/ssh/ssh_config.d/05-redhat.conf depth 0
debug1: Reading configuration data /etc/ssh/ssh_config.d/05-redhat.conf
debug2: checking match for 'final all' host master originally master
debug3: /etc/ssh/ssh_config.d/05-redhat.conf line 3: not matched 'final'
debug2: match not found
debug3: /etc/ssh/ssh_config.d/05-redhat.conf line 5: Including file /etc/crypto-policies/back-ends/openssh.config depth 1 (parse only)
debug1: Reading configuration data /etc/crypto-policies/back-ends/openssh.config
debug3: gss kex names ok: [gss-gex-sha1-,gss-group14-sha1-,gss-group1-sha1-]
debug3: kex names ok: [curve25519-sha256,[e-mail],ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group14-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1,diffie-hellman-group1-sha1]
debug1: configuration requests final Match pass
debug1: re-parsing configuration
debug1: Reading configuration data /etc/ssh/ssh_config
debug3: /etc/ssh/ssh_config line 51: Including file /etc/ssh/ssh_config.d/05-redhat.conf depth 0
debug1: Reading configuration data /etc/ssh/ssh_config.d/05-redhat.conf
debug2: checking match for 'final all' host master originally master
debug3: /etc/ssh/ssh_config.d/05-redhat.conf line 3: matched 'final'
debug2: match found
debug3: /etc/ssh/ssh_config.d/05-redhat.conf line 5: Including file /etc/crypto-policies/back-ends/openssh.config depth 1
debug1: Reading configuration data /etc/crypto-policies/back-ends/openssh.config
debug3: gss kex names ok: [gss-gex-sha1-,gss-group14-sha1-,gss-group1-sha1-]
debug3: kex names ok: [curve25519-sha256,[e-mail],ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group14-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1,diffie-hellman-group1-sha1]
debug1: Executing proxy command: exec /usr/bin/sss_ssh_knownhostsproxy -p 22 master
debug1: identity file /home/rlong/.ssh/id_rsa type -1
debug1: identity file /home/rlong/.ssh/id_rsa-cert type -1
debug1: identity file /home/rlong/.ssh/id_dsa type -1
debug1: identity file /home/rlong/.ssh/id_dsa-cert type -1
debug1: identity file /home/rlong/.ssh/id_ecdsa type -1
debug1: identity file /home/rlong/.ssh/id_ecdsa-cert type -1
debug1: identity file /home/rlong/.ssh/id_ed25519 type -1
debug1: identity file /home/rlong/.ssh/id_ed25519-cert type -1
debug1: identity file /home/rlong/.ssh/id_xmss type -1
debug1: identity file /home/rlong/.ssh/id_xmss-cert type -1
debug1: Local version string SSH-2.0-OpenSSH_8.1

I'm assuming it has something to do with the knownhostsproxy setup, but I can't find any information on how to confirm that or what to do to resolve it.  I am able to ssh to the IPA server from any system that is not enrolled as an IPA client.

Any ideas?

[Rob Crittenden]

Russ Long via FreeIPA-users wrote:
> In my homelab, where I only have a single IPA server, I have had to rebuild / replace it due to issues outside of IPA.  I was unable to restore the VM from a snapshot, however I was able to rebuild and use a backup created by ipa-backup to perform a restore.  Everything appeared to work fine, and IPA clients are working, as well as LDAP and other services.  The only issue I'm encountering is that when I try to ssh to the IPA server from one of the clients, SSH hangs here:
>
> $ ssh -vvvvv master
> OpenSSH_8.1p1, OpenSSL 1.1.1d FIPS  10 Sep 2019
> debug1: Reading configuration data /etc/ssh/ssh_config
> debug3: /etc/ssh/ssh_config line 51: Including file /etc/ssh/ssh_config.d/05-redhat.conf depth 0
> debug1: Reading configuration data /etc/ssh/ssh_config.d/05-redhat.conf
> debug2: checking match for 'final all' host master originally master
> debug3: /etc/ssh/ssh_config.d/05-redhat.conf line 3: not matched 'final'
> debug2: match not found
> debug3: /etc/ssh/ssh_config.d/05-redhat.conf line 5: Including file /etc/crypto-policies/back-ends/openssh.config depth 1 (parse only)
> debug1: Reading configuration data /etc/crypto-policies/back-ends/openssh.config
> debug3: gss kex names ok: [gss-gex-sha1-,gss-group14-sha1-,gss-group1-sha1-]
> debug3: kex names ok: [curve25519-sha256,[e-mail],ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group14-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1,diffie-hellman-group1-sha1]
> debug1: configuration requests final Match pass
> debug1: re-parsing configuration
> debug1: Reading configuration data /etc/ssh/ssh_config
> debug3: /etc/ssh/ssh_config line 51: Including file /etc/ssh/ssh_config.d/05-redhat.conf depth 0
> debug1: Reading configuration data /etc/ssh/ssh_config.d/05-redhat.conf
> debug2: checking match for 'final all' host master originally master
> debug3: /etc/ssh/ssh_config.d/05-redhat.conf line 3: matched 'final'
> debug2: match found
> debug3: /etc/ssh/ssh_config.d/05-redhat.conf line 5: Including file /etc/crypto-policies/back-ends/openssh.config depth 1
> debug1: Reading configuration data /etc/crypto-policies/back-ends/openssh.config
> debug3: gss kex names ok: [gss-gex-sha1-,gss-group14-sha1-,gss-group1-sha1-]
> debug3: kex names ok: [curve25519-sha256,[e-mail],ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group14-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1,diffie-hellman-group1-sha1]
> debug1: Executing proxy command: exec /usr/bin/sss_ssh_knownhostsproxy -p 22 master
> debug1: identity file /home/rlong/.ssh/id_rsa type -1
> debug1: identity file /home/rlong/.ssh/id_rsa-cert type -1
> debug1: identity file /home/rlong/.ssh/id_dsa type -1
> debug1: identity file /home/rlong/.ssh/id_dsa-cert type -1
> debug1: identity file /home/rlong/.ssh/id_ecdsa type -1
> debug1: identity file /home/rlong/.ssh/id_ecdsa-cert type -1
> debug1: identity file /home/rlong/.ssh/id_ed25519 type -1
> debug1: identity file /home/rlong/.ssh/id_ed25519-cert type -1
> debug1: identity file /home/rlong/.ssh/id_xmss type -1
> debug1: identity file /home/rlong/.ssh/id_xmss-cert type -1
> debug1: Local version string SSH-2.0-OpenSSH_8.1
>
> I'm assuming it has something to do with the knownhostsproxy setup, but I can't find any information on how to confirm that or what to do to resolve it.  I am able to ssh to the IPA server from any system that is not enrolled as an IPA client.
>
> Any ideas?

Maybe see if it can get the keys from the master:

sss_ssh_knownhostsproxy -k master

rob

[Russ Long]

sss_ssh_knownhostsproxy -k master - this works without issue, and appears to return the correct keys from what I can tell.

SSH to the master still fails.

[Russ Long]

Actually ssh to the master now works, it does take a long period of time.  There is a very long pause at the same location noted above, here's the rest of the debug logs:

# ssh [e-mail] -vvvv
OpenSSH_8.1p1, OpenSSL 1.1.1d FIPS  10 Sep 2019
debug1: Reading configuration data /root/.ssh/config
debug1: /root/.ssh/config line 1: Applying options for *
debug1: Reading configuration data /etc/ssh/ssh_config
debug3: /etc/ssh/ssh_config line 51: Including file /etc/ssh/ssh_config.d/05-redhat.conf depth 0
debug1: Reading configuration data /etc/ssh/ssh_config.d/05-redhat.conf
debug2: checking match for 'final all' host master.ipa.tfmm.co originally master.ipa.tfmm.co
debug3: /etc/ssh/ssh_config.d/05-redhat.conf line 3: not matched 'final'
debug2: match not found
debug3: /etc/ssh/ssh_config.d/05-redhat.conf line 5: Including file /etc/crypto-policies/back-ends/openssh.config depth 1 (parse only)
debug1: Reading configuration data /etc/crypto-policies/back-ends/openssh.config
debug3: gss kex names ok: [gss-gex-sha1-,gss-group14-sha1-,gss-group1-sha1-]
debug3: kex names ok: [curve25519-sha256,[e-mail],ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group14-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1,diffie-hellman-group1-sha1]
debug1: configuration requests final Match pass
debug1: re-parsing configuration
debug1: Reading configuration data /root/.ssh/config
debug1: /root/.ssh/config line 1: Applying options for *
debug1: Reading configuration data /etc/ssh/ssh_config
debug3: /etc/ssh/ssh_config line 51: Including file /etc/ssh/ssh_config.d/05-redhat.conf depth 0
debug1: Reading configuration data /etc/ssh/ssh_config.d/05-redhat.conf
debug2: checking match for 'final all' host master.ipa.tfmm.co originally master.ipa.tfmm.co
debug3: /etc/ssh/ssh_config.d/05-redhat.conf line 3: matched 'final'
debug2: match found
debug3: /etc/ssh/ssh_config.d/05-redhat.conf line 5: Including file /etc/crypto-policies/back-ends/openssh.config depth 1
debug1: Reading configuration data /etc/crypto-policies/back-ends/openssh.config
debug3: gss kex names ok: [gss-gex-sha1-,gss-group14-sha1-,gss-group1-sha1-]
debug3: kex names ok: [curve25519-sha256,[e-mail],ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group14-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1,diffie-hellman-group1-sha1]
debug1: Executing proxy command: exec /usr/bin/sss_ssh_knownhostsproxy -p 22 master.ipa.tfmm.co
debug1: identity file /root/.ssh/id_rsa type -1
debug1: identity file /root/.ssh/id_rsa-cert type -1
debug1: identity file /root/.ssh/id_dsa type -1
debug1: identity file /root/.ssh/id_dsa-cert type -1
debug1: identity file /root/.ssh/id_ecdsa type -1
debug1: identity file /root/.ssh/id_ecdsa-cert type -1
debug1: identity file /root/.ssh/id_ed25519 type -1
debug1: identity file /root/.ssh/id_ed25519-cert type -1
debug1: identity file /root/.ssh/id_xmss type -1
debug1: identity file /root/.ssh/id_xmss-cert type -1
debug1: Local version string SSH-2.0-OpenSSH_8.1
debug1: Remote protocol version 2.0, remote software version OpenSSH_8.1
debug1: match: OpenSSH_8.1 pat OpenSSH* compat 0x04000000
debug2: fd 7 setting O_NONBLOCK
debug2: fd 6 setting O_NONBLOCK
debug1: Authenticating to master.ipa.tfmm.co:22 as 'rlong'
debug3: hostkeys_foreach: reading file "/root/.ssh/known_hosts"
debug3: hostkeys_foreach: reading file "/var/lib/sss/pubconf/known_hosts"
debug3: record_hostkey: found key type ECDSA in file /var/lib/sss/pubconf/known_hosts:1
debug3: record_hostkey: found key type ECDSA in file /var/lib/sss/pubconf/known_hosts:2
debug3: record_hostkey: found key type ED25519 in file /var/lib/sss/pubconf/known_hosts:4
debug3: record_hostkey: found key type ED25519 in file /var/lib/sss/pubconf/known_hosts:5
debug3: record_hostkey: found key type RSA in file /var/lib/sss/pubconf/known_hosts:7
debug3: record_hostkey: found key type RSA in file /var/lib/sss/pubconf/known_hosts:8
debug3: load_hostkeys: loaded 6 keys from master.ipa.tfmm.co
debug3: order_hostkeyalgs: prefer hostkeyalgs: [e-mail],[e-mail],[e-mail],[e-mail],[e-mail],[e-mail],[e-mail],ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,ssh-ed25519,rsa-sha2-512,rsa-sha2-256,ssh-rsa
debug3: send packet: type 20
debug1: SSH2_MSG_KEXINIT sent
debug3: receive packet: type 20
debug1: SSH2_MSG_KEXINIT received
debug2: local client KEXINIT proposal
debug2: KEX algorithms: curve25519-sha256,[e-mail],ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group14-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1,diffie-hellman-group1-sha1,ext-info-c
debug2: host key algorithms: [e-mail],[e-mail],[e-mail],[e-mail],[e-mail],[e-mail],[e-mail],ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,ssh-ed25519,rsa-sha2-512,rsa-sha2-256,ssh-rsa
debug2: ciphers ctos: [e-mail],[e-mail],aes256-ctr,aes256-cbc,[e-mail],aes128-ctr,aes128-cbc
debug2: ciphers stoc: [e-mail],[e-mail],aes256-ctr,aes256-cbc,[e-mail],aes128-ctr,aes128-cbc
debug2: MACs ctos: [e-mail],[e-mail],[e-mail],[e-mail],hmac-sha2-256,hmac-sha1,[e-mail],hmac-sha2-512
debug2: MACs stoc: [e-mail],[e-mail],[e-mail],[e-mail],hmac-sha2-256,hmac-sha1,[e-mail],hmac-sha2-512
debug2: compression ctos: none,[e-mail],zlib
debug2: compression stoc: none,[e-mail],zlib
debug2: languages ctos: 
debug2: languages stoc: 
debug2: first_kex_follows 0 
debug2: reserved 0 
debug2: peer server KEXINIT proposal
debug2: KEX algorithms: curve25519-sha256,[e-mail],ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group14-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1
debug2: host key algorithms: rsa-sha2-512,rsa-sha2-256,ssh-rsa,ecdsa-sha2-nistp256,ssh-ed25519
debug2: ciphers ctos: [e-mail],[e-mail],aes256-ctr,aes256-cbc,[e-mail],aes128-ctr,aes128-cbc
debug2: ciphers stoc: [e-mail],[e-mail],aes256-ctr,aes256-cbc,[e-mail],aes128-ctr,aes128-cbc
debug2: MACs ctos: [e-mail],[e-mail],[e-mail],[e-mail],hmac-sha2-256,hmac-sha1,[e-mail],hmac-sha2-512
debug2: MACs stoc: [e-mail],[e-mail],[e-mail],[e-mail],hmac-sha2-256,hmac-sha1,[e-mail],hmac-sha2-512
debug2: compression ctos: none,[e-mail]
debug2: compression stoc: none,[e-mail]
debug2: languages ctos: 
debug2: languages stoc: 
debug2: first_kex_follows 0 
debug2: reserved 0 
debug1: kex: algorithm: curve25519-sha256
debug1: kex: host key algorithm: ecdsa-sha2-nistp256
debug1: kex: server->client cipher: [e-mail] MAC: <implicit> compression: none
debug1: kex: client->server cipher: [e-mail] MAC: <implicit> compression: none
debug1: kex: curve25519-sha256 need=32 dh_need=32
debug1: kex: curve25519-sha256 need=32 dh_need=32
debug3: send packet: type 30
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug3: receive packet: type 31
debug1: Server host key: ecdsa-sha2-nistp256 SHA256:VDPeQEW3gn8jzgkiLW0k5Gpp1/+TnG4FS+RHEiSU8Lc
debug3: hostkeys_foreach: reading file "/root/.ssh/known_hosts"
debug3: hostkeys_foreach: reading file "/var/lib/sss/pubconf/known_hosts"
debug3: record_hostkey: found key type ECDSA in file /var/lib/sss/pubconf/known_hosts:1
debug3: record_hostkey: found key type ECDSA in file /var/lib/sss/pubconf/known_hosts:2
debug3: record_hostkey: found key type ED25519 in file /var/lib/sss/pubconf/known_hosts:4
debug3: record_hostkey: found key type ED25519 in file /var/lib/sss/pubconf/known_hosts:5
debug3: record_hostkey: found key type RSA in file /var/lib/sss/pubconf/known_hosts:7
debug3: record_hostkey: found key type RSA in file /var/lib/sss/pubconf/known_hosts:8
debug3: load_hostkeys: loaded 6 keys from master.ipa.tfmm.co
debug1: Host 'master.ipa.tfmm.co' is known and matches the ECDSA host key.
debug1: Found key in /var/lib/sss/pubconf/known_hosts:1
debug3: send packet: type 21
debug2: set_newkeys: mode 1
debug1: rekey out after 4294967296 blocks
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug3: receive packet: type 21
debug1: SSH2_MSG_NEWKEYS received
debug2: set_newkeys: mode 0
debug1: rekey in after 4294967296 blocks
debug1: Will attempt key: /root/.ssh/id_rsa 
debug1: Will attempt key: /root/.ssh/id_dsa 
debug1: Will attempt key: /root/.ssh/id_ecdsa 
debug1: Will attempt key: /root/.ssh/id_ed25519 
debug1: Will attempt key: /root/.ssh/id_xmss 
debug2: pubkey_prepare: done
debug3: send packet: type 5
debug3: receive packet: type 7
debug1: SSH2_MSG_EXT_INFO received
debug1: kex_input_ext_info: server-sig-algs=<ssh-ed25519,ssh-rsa,rsa-sha2-256,rsa-sha2-512,ssh-dss,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521>
debug3: receive packet: type 6
debug2: service_accept: ssh-userauth
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug3: send packet: type 50
debug3: receive packet: type 51
debug1: Authentications that can continue: publickey,gssapi-keyex,gssapi-with-mic,password,keyboard-interactive
debug3: start over, passed a different list publickey,gssapi-keyex,gssapi-with-mic,password,keyboard-interactive
debug3: preferred gssapi-with-mic,publickey,keyboard-interactive,password
debug3: authmethod_lookup gssapi-with-mic
debug3: remaining preferred: publickey,keyboard-interactive,password
debug3: authmethod_is_enabled gssapi-with-mic
debug1: Next authentication method: gssapi-with-mic
debug1: Unspecified GSS failure.  Minor code may provide more information
No Kerberos credentials available (default cache: KCM:)

debug1: Unspecified GSS failure.  Minor code may provide more information
No Kerberos credentials available (default cache: KCM:)

debug2: we did not send a packet, disable method
debug3: authmethod_lookup publickey
debug3: remaining preferred: keyboard-interactive,password
debug3: authmethod_is_enabled publickey
debug1: Next authentication method: publickey
debug1: Trying private key: /root/.ssh/id_rsa
debug3: no such identity: /root/.ssh/id_rsa: No such file or directory
debug1: Trying private key: /root/.ssh/id_dsa
debug3: no such identity: /root/.ssh/id_dsa: No such file or directory
debug1: Trying private key: /root/.ssh/id_ecdsa
debug3: no such identity: /root/.ssh/id_ecdsa: No such file or directory
debug1: Trying private key: /root/.ssh/id_ed25519
debug3: no such identity: /root/.ssh/id_ed25519: No such file or directory
debug1: Trying private key: /root/.ssh/id_xmss
debug3: no such identity: /root/.ssh/id_xmss: No such file or directory
debug2: we did not send a packet, disable method
debug3: authmethod_lookup keyboard-interactive
debug3: remaining preferred: password
debug3: authmethod_is_enabled keyboard-interactive
debug1: Next authentication method: keyboard-interactive
debug2: userauth_kbdint
debug3: send packet: type 50
debug2: we sent a keyboard-interactive packet, wait for reply
debug3: receive packet: type 60
debug2: input_userauth_info_req
debug2: input_userauth_info_req: num_prompts 1
Password: 
debug3: send packet: type 61
ssh_dispatch_run_fatal: Connection to UNKNOWN port 65535: Broken pipe

[Rob Crittenden]

Russ Long via FreeIPA-users wrote:
> Actually ssh to the master now works, it does take a long period of time.  There is a very long pause at the same location noted above, here's the rest of the debug logs:
>
> # ssh [e-mail] -vvvv
> OpenSSH_8.1p1, OpenSSL 1.1.1d FIPS  10 Sep 2019
> debug1: Reading configuration data /root/.ssh/config
> debug1: /root/.ssh/config line 1: Applying options for *
> debug1: Reading configuration data /etc/ssh/ssh_config
> debug3: /etc/ssh/ssh_config line 51: Including file /etc/ssh/ssh_config.d/05-redhat.conf depth 0
> debug1: Reading configuration data /etc/ssh/ssh_config.d/05-redhat.conf
> debug2: checking match for 'final all' host master.ipa.tfmm.co originally master.ipa.tfmm.co
> debug3: /etc/ssh/ssh_config.d/05-redhat.conf line 3: not matched 'final'
> debug2: match not found
> debug3: /etc/ssh/ssh_config.d/05-redhat.conf line 5: Including file /etc/crypto-policies/back-ends/openssh.config depth 1 (parse only)
> debug1: Reading configuration data /etc/crypto-policies/back-ends/openssh.config
> debug3: gss kex names ok: [gss-gex-sha1-,gss-group14-sha1-,gss-group1-sha1-]
> debug3: kex names ok: [curve25519-sha256,[e-mail],ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group14-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1,diffie-hellman-group1-sha1]
> debug1: configuration requests final Match pass
> debug1: re-parsing configuration
> debug1: Reading configuration data /root/.ssh/config
> debug1: /root/.ssh/config line 1: Applying options for *
> debug1: Reading configuration data /etc/ssh/ssh_config
> debug3: /etc/ssh/ssh_config line 51: Including file /etc/ssh/ssh_config.d/05-redhat.conf depth 0
> debug1: Reading configuration data /etc/ssh/ssh_config.d/05-redhat.conf
> debug2: checking match for 'final all' host master.ipa.tfmm.co originally master.ipa.tfmm.co
> debug3: /etc/ssh/ssh_config.d/05-redhat.conf line 3: matched 'final'
> debug2: match found
> debug3: /etc/ssh/ssh_config.d/05-redhat.conf line 5: Including file /etc/crypto-policies/back-ends/openssh.config depth 1
> debug1: Reading configuration data /etc/crypto-policies/back-ends/openssh.config
> debug3: gss kex names ok: [gss-gex-sha1-,gss-group14-sha1-,gss-group1-sha1-]
> debug3: kex names ok: [curve25519-sha256,[e-mail],ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group14-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1,diffie-hellman-group1-sha1]
> debug1: Executing proxy command: exec /usr/bin/sss_ssh_knownhostsproxy -p 22 master.ipa.tfmm.co
> debug1: identity file /root/.ssh/id_rsa type -1
> debug1: identity file /root/.ssh/id_rsa-cert type -1
> debug1: identity file /root/.ssh/id_dsa type -1
> debug1: identity file /root/.ssh/id_dsa-cert type -1
> debug1: identity file /root/.ssh/id_ecdsa type -1
> debug1: identity file /root/.ssh/id_ecdsa-cert type -1
> debug1: identity file /root/.ssh/id_ed25519 type -1
> debug1: identity file /root/.ssh/id_ed25519-cert type -1
> debug1: identity file /root/.ssh/id_xmss type -1
> debug1: identity file /root/.ssh/id_xmss-cert type -1
> debug1: Local version string SSH-2.0-OpenSSH_8.1
> debug1: Remote protocol version 2.0, remote software version OpenSSH_8.1
> debug1: match: OpenSSH_8.1 pat OpenSSH* compat 0x04000000
> debug2: fd 7 setting O_NONBLOCK
> debug2: fd 6 setting O_NONBLOCK
> debug1: Authenticating to master.ipa.tfmm.co:22 as 'rlong'
> debug3: hostkeys_foreach: reading file "/root/.ssh/known_hosts"
> debug3: hostkeys_foreach: reading file "/var/lib/sss/pubconf/known_hosts"
> debug3: record_hostkey: found key type ECDSA in file /var/lib/sss/pubconf/known_hosts:1
> debug3: record_hostkey: found key type ECDSA in file /var/lib/sss/pubconf/known_hosts:2
> debug3: record_hostkey: found key type ED25519 in file /var/lib/sss/pubconf/known_hosts:4
> debug3: record_hostkey: found key type ED25519 in file /var/lib/sss/pubconf/known_hosts:5
> debug3: record_hostkey: found key type RSA in file /var/lib/sss/pubconf/known_hosts:7
> debug3: record_hostkey: found key type RSA in file /var/lib/sss/pubconf/known_hosts:8
> debug3: load_hostkeys: loaded 6 keys from master.ipa.tfmm.co
> debug3: order_hostkeyalgs: prefer hostkeyalgs: [e-mail],[e-mail],[e-mail],[e-mail],[e-mail],[e-mail],[e-mail],ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,ssh-ed25519,rsa-sha2-512,rsa-sha2-256,ssh-rsa
> debug3: send packet: type 20
> debug1: SSH2_MSG_KEXINIT sent
> debug3: receive packet: type 20
> debug1: SSH2_MSG_KEXINIT received
> debug2: local client KEXINIT proposal
> debug2: KEX algorithms: curve25519-sha256,[e-mail],ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group14-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1,diffie-hellman-group1-sha1,ext-info-c
> debug2: host key algorithms: [e-mail],[e-mail],[e-mail],[e-mail],[e-mail],[e-mail],[e-mail],ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,ssh-ed25519,rsa-sha2-512,rsa-sha2-256,ssh-rsa
> debug2: ciphers ctos: [e-mail],[e-mail],aes256-ctr,aes256-cbc,[e-mail],aes128-ctr,aes128-cbc
> debug2: ciphers stoc: [e-mail],[e-mail],aes256-ctr,aes256-cbc,[e-mail],aes128-ctr,aes128-cbc
> debug2: MACs ctos: [e-mail],[e-mail],[e-mail],[e-mail],hmac-sha2-256,hmac-sha1,[e-mail],hmac-sha2-512
> debug2: MACs stoc: [e-mail],[e-mail],[e-mail],[e-mail],hmac-sha2-256,hmac-sha1,[e-mail],hmac-sha2-512
> debug2: compression ctos: none,[e-mail],zlib
> debug2: compression stoc: none,[e-mail],zlib
> debug2: languages ctos: 
> debug2: languages stoc: 
> debug2: first_kex_follows 0 
> debug2: reserved 0 
> debug2: peer server KEXINIT proposal
> debug2: KEX algorithms: curve25519-sha256,[e-mail],ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group14-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1
> debug2: host key algorithms: rsa-sha2-512,rsa-sha2-256,ssh-rsa,ecdsa-sha2-nistp256,ssh-ed25519
> debug2: ciphers ctos: [e-mail],[e-mail],aes256-ctr,aes256-cbc,[e-mail],aes128-ctr,aes128-cbc
> debug2: ciphers stoc: [e-mail],[e-mail],aes256-ctr,aes256-cbc,[e-mail],aes128-ctr,aes128-cbc
> debug2: MACs ctos: [e-mail],[e-mail],[e-mail],[e-mail],hmac-sha2-256,hmac-sha1,[e-mail],hmac-sha2-512
> debug2: MACs stoc: [e-mail],[e-mail],[e-mail],[e-mail],hmac-sha2-256,hmac-sha1,[e-mail],hmac-sha2-512
> debug2: compression ctos: none,[e-mail]
> debug2: compression stoc: none,[e-mail]
> debug2: languages ctos: 
> debug2: languages stoc: 
> debug2: first_kex_follows 0 
> debug2: reserved 0 
> debug1: kex: algorithm: curve25519-sha256
> debug1: kex: host key algorithm: ecdsa-sha2-nistp256
> debug1: kex: server->client cipher: [e-mail] MAC: <implicit> compression: none
> debug1: kex: client->server cipher: [e-mail] MAC: <implicit> compression: none
> debug1: kex: curve25519-sha256 need=32 dh_need=32
> debug1: kex: curve25519-sha256 need=32 dh_need=32
> debug3: send packet: type 30
> debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
> debug3: receive packet: type 31
> debug1: Server host key: ecdsa-sha2-nistp256 SHA256:VDPeQEW3gn8jzgkiLW0k5Gpp1/+TnG4FS+RHEiSU8Lc
> debug3: hostkeys_foreach: reading file "/root/.ssh/known_hosts"
> debug3: hostkeys_foreach: reading file "/var/lib/sss/pubconf/known_hosts"
> debug3: record_hostkey: found key type ECDSA in file /var/lib/sss/pubconf/known_hosts:1
> debug3: record_hostkey: found key type ECDSA in file /var/lib/sss/pubconf/known_hosts:2
> debug3: record_hostkey: found key type ED25519 in file /var/lib/sss/pubconf/known_hosts:4
> debug3: record_hostkey: found key type ED25519 in file /var/lib/sss/pubconf/known_hosts:5
> debug3: record_hostkey: found key type RSA in file /var/lib/sss/pubconf/known_hosts:7
> debug3: record_hostkey: found key type RSA in file /var/lib/sss/pubconf/known_hosts:8
> debug3: load_hostkeys: loaded 6 keys from master.ipa.tfmm.co
> debug1: Host 'master.ipa.tfmm.co' is known and matches the ECDSA host key.
> debug1: Found key in /var/lib/sss/pubconf/known_hosts:1
> debug3: send packet: type 21
> debug2: set_newkeys: mode 1
> debug1: rekey out after 4294967296 blocks
> debug1: SSH2_MSG_NEWKEYS sent
> debug1: expecting SSH2_MSG_NEWKEYS
> debug3: receive packet: type 21
> debug1: SSH2_MSG_NEWKEYS received
> debug2: set_newkeys: mode 0
> debug1: rekey in after 4294967296 blocks
> debug1: Will attempt key: /root/.ssh/id_rsa 
> debug1: Will attempt key: /root/.ssh/id_dsa 
> debug1: Will attempt key: /root/.ssh/id_ecdsa 
> debug1: Will attempt key: /root/.ssh/id_ed25519 
> debug1: Will attempt key: /root/.ssh/id_xmss 
> debug2: pubkey_prepare: done
> debug3: send packet: type 5
> debug3: receive packet: type 7
> debug1: SSH2_MSG_EXT_INFO received
> debug1: kex_input_ext_info: server-sig-algs=<ssh-ed25519,ssh-rsa,rsa-sha2-256,rsa-sha2-512,ssh-dss,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521>
> debug3: receive packet: type 6
> debug2: service_accept: ssh-userauth
> debug1: SSH2_MSG_SERVICE_ACCEPT received
> debug3: send packet: type 50
> debug3: receive packet: type 51
> debug1: Authentications that can continue: publickey,gssapi-keyex,gssapi-with-mic,password,keyboard-interactive
> debug3: start over, passed a different list publickey,gssapi-keyex,gssapi-with-mic,password,keyboard-interactive
> debug3: preferred gssapi-with-mic,publickey,keyboard-interactive,password
> debug3: authmethod_lookup gssapi-with-mic
> debug3: remaining preferred: publickey,keyboard-interactive,password
> debug3: authmethod_is_enabled gssapi-with-mic
> debug1: Next authentication method: gssapi-with-mic
> debug1: Unspecified GSS failure.  Minor code may provide more information
> No Kerberos credentials available (default cache: KCM:)
>
> debug1: Unspecified GSS failure.  Minor code may provide more information
> No Kerberos credentials available (default cache: KCM:)
>
> debug2: we did not send a packet, disable method
> debug3: authmethod_lookup publickey
> debug3: remaining preferred: keyboard-interactive,password
> debug3: authmethod_is_enabled publickey
> debug1: Next authentication method: publickey
> debug1: Trying private key: /root/.ssh/id_rsa
> debug3: no such identity: /root/.ssh/id_rsa: No such file or directory
> debug1: Trying private key: /root/.ssh/id_dsa
> debug3: no such identity: /root/.ssh/id_dsa: No such file or directory
> debug1: Trying private key: /root/.ssh/id_ecdsa
> debug3: no such identity: /root/.ssh/id_ecdsa: No such file or directory
> debug1: Trying private key: /root/.ssh/id_ed25519
> debug3: no such identity: /root/.ssh/id_ed25519: No such file or directory
> debug1: Trying private key: /root/.ssh/id_xmss
> debug3: no such identity: /root/.ssh/id_xmss: No such file or directory
> debug2: we did not send a packet, disable method
> debug3: authmethod_lookup keyboard-interactive
> debug3: remaining preferred: password
> debug3: authmethod_is_enabled keyboard-interactive
> debug1: Next authentication method: keyboard-interactive
> debug2: userauth_kbdint
> debug3: send packet: type 50
> debug2: we sent a keyboard-interactive packet, wait for reply
> debug3: receive packet: type 60
> debug2: input_userauth_info_req
> debug2: input_userauth_info_req: num_prompts 1
> Password: 
> debug3: send packet: type 61
> ssh_dispatch_run_fatal: Connection to UNKNOWN port 65535: Broken pipe

root doesn't have a Kerberos ticket, that is going to delay things.

rob

[Russ Long]

I'm seeing the same behavior when trying to ssh from an IPA user's account on the same machine. I have also confirmed that the user has an active kerberos ticket.