SSH with ecdsa key pair asks for password since yesterday on fully updated F34 setup (both sides)

[Ranjan Maitra]

Dear friends,

I am not sure what has changed and it is possible that some configuration has inadvertently changed that has messed things up but over the past 3 hours, I have been having this problem in that my key is no longer recognized. So, here is what I get when I try to log in to remote.server as user fedusr (other things are below):

19:58:38~$ ssh remote -vvv
OpenSSH_8.6p1, OpenSSL 1.1.1l  FIPS 24 Aug 2021
debug1: Reading configuration data /home/fedusr/.ssh/config
debug1: /home/fedusr/.ssh/config line 1: Applying options for *
debug1: /home/fedusr/.ssh/config line 4: Applying options for remote
debug1: Reading configuration data /etc/ssh/ssh_config
debug3: /etc/ssh/ssh_config line 55: Including file /etc/ssh/ssh_config.d/50-redhat.conf depth 0
debug1: Reading configuration data /etc/ssh/ssh_config.d/50-redhat.conf
debug2: checking match for 'final all' host remote.server originally remote
debug3: /etc/ssh/ssh_config.d/50-redhat.conf line 3: not matched 'final'
debug2: match not found
debug3: /etc/ssh/ssh_config.d/50-redhat.conf line 5: Including file /etc/crypto-policies/back-ends/openssh.config depth 1 (parse only)
debug1: Reading configuration data /etc/crypto-policies/back-ends/openssh.config
debug3: gss kex names ok: [gss-curve25519-sha256-,gss-nistp256-sha256-,gss-group14-sha256-,gss-group16-sha512-]
debug3: kex names ok: [curve25519-sha256,[e-mail],ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group14-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512]
debug1: configuration requests final Match pass
debug1: re-parsing configuration
debug1: Reading configuration data /home/fedusr/.ssh/config
debug1: /home/fedusr/.ssh/config line 1: Applying options for *
debug1: /home/fedusr/.ssh/config line 9: Applying options for remote.server
debug1: Reading configuration data /etc/ssh/ssh_config
debug3: /etc/ssh/ssh_config line 55: Including file /etc/ssh/ssh_config.d/50-redhat.conf depth 0
debug1: Reading configuration data /etc/ssh/ssh_config.d/50-redhat.conf
debug2: checking match for 'final all' host remote.server originally remote
debug3: /etc/ssh/ssh_config.d/50-redhat.conf line 3: matched 'final'
debug2: match found
debug3: /etc/ssh/ssh_config.d/50-redhat.conf line 5: Including file /etc/crypto-policies/back-ends/openssh.config depth 1
debug1: Reading configuration data /etc/crypto-policies/back-ends/openssh.config
debug3: gss kex names ok: [gss-curve25519-sha256-,gss-nistp256-sha256-,gss-group14-sha256-,gss-group16-sha512-]
debug3: kex names ok: [curve25519-sha256,[e-mail],ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group14-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512]
debug3: expanded UserKnownHostsFile '~/.ssh/known_hosts' -> '/home/fedusr/.ssh/known_hosts'
debug3: expanded UserKnownHostsFile '~/.ssh/known_hosts2' -> '/home/fedusr/.ssh/known_hosts2'
debug2: resolving "remote.server" port 51064
debug3: ssh_connect_direct: entering
debug1: Connecting to remote.server [10.25.123.189] port 51064.
debug3: set_sock_tos: set socket 5 IP_TOS 0x48
debug2: fd 5 setting O_NONBLOCK
debug1: fd 5 clearing O_NONBLOCK
debug1: Connection established.
debug3: timeout: 599960 ms remain after connect
debug1: identity file /home/fedusr/.ssh/id_rsa type 0
debug1: identity file /home/fedusr/.ssh/id_rsa-cert type -1
debug1: identity file /home/fedusr/.ssh/id_dsa type 1
debug1: identity file /home/fedusr/.ssh/id_dsa-cert type -1
debug1: identity file /home/fedusr/.ssh/id_ecdsa type 2
debug1: identity file /home/fedusr/.ssh/id_ecdsa-cert type -1
debug1: identity file /home/fedusr/.ssh/id_ecdsa_sk type -1
debug1: identity file /home/fedusr/.ssh/id_ecdsa_sk-cert type -1
debug1: identity file /home/fedusr/.ssh/id_ed25519 type 3
debug1: identity file /home/fedusr/.ssh/id_ed25519-cert type -1
debug1: identity file /home/fedusr/.ssh/id_ed25519_sk type -1
debug1: identity file /home/fedusr/.ssh/id_ed25519_sk-cert type -1
debug1: identity file /home/fedusr/.ssh/id_xmss type -1
debug1: identity file /home/fedusr/.ssh/id_xmss-cert type -1
debug1: Local version string SSH-2.0-OpenSSH_8.6
debug1: Remote protocol version 2.0, remote software version OpenSSH_8.6
debug1: compat_banner: match: OpenSSH_8.6 pat OpenSSH* compat 0x04000000
debug2: fd 5 setting O_NONBLOCK
debug1: Authenticating to remote.server:51064 as 'fedusr'
debug3: put_host_port: [remote.server]:51064
debug3: record_hostkey: found key type ED25519 in file /home/fedusr/.ssh/known_hosts:29
debug3: record_hostkey: found key type RSA in file /home/fedusr/.ssh/known_hosts:30
debug3: record_hostkey: found key type ECDSA in file /home/fedusr/.ssh/known_hosts:31
debug3: load_hostkeys_file: loaded 3 keys from [remote.server]:51064
debug1: load_hostkeys: fopen /home/fedusr/.ssh/known_hosts2: No such file or directory
debug1: load_hostkeys: fopen /etc/ssh/ssh_known_hosts: No such file or directory
debug1: load_hostkeys: fopen /etc/ssh/ssh_known_hosts2: No such file or directory
debug3: order_hostkeyalgs: have matching best-preference key type [e-mail], using HostkeyAlgorithms verbatim
debug3: send packet: type 20
debug1: SSH2_MSG_KEXINIT sent
debug3: receive packet: type 20
debug1: SSH2_MSG_KEXINIT received
debug2: local client KEXINIT proposal
debug2: KEX algorithms: curve25519-sha256,[e-mail],ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group14-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,ext-info-c
debug2: host key algorithms: [e-mail],[e-mail],[e-mail],[e-mail],[e-mail],[e-mail],[e-mail],[e-mail],[e-mail],ssh-ed25519,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,[e-mail],[e-mail],rsa-sha2-512,rsa-sha2-256,ssh-rsa
debug2: ciphers ctos: [e-mail],[e-mail],aes256-ctr,[e-mail],aes128-ctr
debug2: ciphers stoc: [e-mail],[e-mail],aes256-ctr,[e-mail],aes128-ctr
debug2: MACs ctos: [e-mail],[e-mail],[e-mail],[e-mail],hmac-sha2-256,hmac-sha1,[e-mail],hmac-sha2-512
debug2: MACs stoc: [e-mail],[e-mail],[e-mail],[e-mail],hmac-sha2-256,hmac-sha1,[e-mail],hmac-sha2-512
debug2: compression ctos: none,[e-mail],zlib
debug2: compression stoc: none,[e-mail],zlib
debug2: languages ctos:
debug2: languages stoc:
debug2: first_kex_follows 0
debug2: reserved 0
debug2: peer server KEXINIT proposal
debug2: KEX algorithms: curve25519-sha256,[e-mail],ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group14-sha256
debug2: host key algorithms: rsa-sha2-512,rsa-sha2-256,ssh-rsa,ecdsa-sha2-nistp256,ssh-ed25519
debug2: ciphers ctos: [e-mail],aes128-ctr,aes192-ctr,aes256-ctr,[e-mail],[e-mail]
debug2: ciphers stoc: [e-mail],aes128-ctr,aes192-ctr,aes256-ctr,[e-mail],[e-mail]
debug2: MACs ctos: [e-mail],[e-mail],[e-mail],[e-mail],[e-mail],[e-mail],[e-mail],hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: MACs stoc: [e-mail],[e-mail],[e-mail],[e-mail],[e-mail],[e-mail],[e-mail],hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: compression ctos: none,[e-mail]
debug2: compression stoc: none,[e-mail]
debug2: languages ctos:
debug2: languages stoc:
debug2: first_kex_follows 0
debug2: reserved 0
debug1: kex: algorithm: curve25519-sha256
debug1: kex: host key algorithm: ssh-ed25519
debug1: kex: server->client cipher: [e-mail] MAC: <implicit> compression: none
debug1: kex: client->server cipher: [e-mail] MAC: <implicit> compression: none
debug1: kex: curve25519-sha256 need=32 dh_need=32
debug1: kex: curve25519-sha256 need=32 dh_need=32
debug3: send packet: type 30
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug3: receive packet: type 31
debug1: SSH2_MSG_KEX_ECDH_REPLY received
debug1: Server host key: ssh-ed25519 SHA256:Oog5b0gWsmIAYZgZIFWc9Rhr087UCLchoJlb5ux8LvY
debug3: put_host_port: [10.25.123.189]:51064
debug3: put_host_port: [remote.server]:51064
debug3: record_hostkey: found key type ED25519 in file /home/fedusr/.ssh/known_hosts:29
debug3: record_hostkey: found key type RSA in file /home/fedusr/.ssh/known_hosts:30
debug3: record_hostkey: found key type ECDSA in file /home/fedusr/.ssh/known_hosts:31
debug3: load_hostkeys_file: loaded 3 keys from [remote.server]:51064
debug1: load_hostkeys: fopen /home/fedusr/.ssh/known_hosts2: No such file or directory
debug1: load_hostkeys: fopen /etc/ssh/ssh_known_hosts: No such file or directory
debug1: load_hostkeys: fopen /etc/ssh/ssh_known_hosts2: No such file or directory
debug1: Host '[remote.server]:51064' is known and matches the ED25519 host key.
debug1: Found key in /home/fedusr/.ssh/known_hosts:29
debug3: send packet: type 21
debug2: set_newkeys: mode 1
debug1: rekey out after 4294967296 blocks
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug3: receive packet: type 21
debug1: SSH2_MSG_NEWKEYS received
debug2: set_newkeys: mode 0
debug1: rekey in after 4294967296 blocks
debug1: Skipping ssh-dss key /home/fedusr/.ssh/id_dsa - corresponding algo not in PubkeyAcceptedAlgorithms
debug1: Skipping ssh-dss key /home/fedusr/.ssh/id_dsa - corresponding algo not in PubkeyAcceptedAlgorithms
debug1: Will attempt key: /home/fedusr/.ssh/id_rsa RSA SHA256:SiFQrcb8BaVwMr65Mm+yfyUTqSN2grVfXTutqDXfpjA agent
debug1: Will attempt key: /home/fedusr/.ssh/id_ecdsa ECDSA SHA256:Pj5mRaoS/lINfM2bLBxYhv1xx9IbR0+2qY9sEz+Kq48 agent
debug1: Will attempt key: /home/fedusr/.ssh/id_ed25519 ED25519 SHA256:PbBA+HdHM84Al31IsGaTsQjhhqMQdjdJcrowH4LL6J0 agent
debug1: Will attempt key: /home/fedusr/.ssh/id_rsa RSA SHA256:Egveyy1ui8/cXaFoYGvF8zE63m+4zYQ1kg64WK8DM/I agent
debug1: Will attempt key: /home/fedusr/.ssh/id_ecdsa ECDSA SHA256:/844OPWwIB4ly00YnMmfOf/aZAWtlImRXWhhiFkhdTw agent
debug1: Will attempt key: /home/fedusr/.ssh/id_ecdsa_sk
debug1: Will attempt key: /home/fedusr/.ssh/id_ed25519_sk
debug1: Will attempt key: /home/fedusr/.ssh/id_xmss
debug2: pubkey_prepare: done
debug3: send packet: type 5
debug3: receive packet: type 7
debug1: SSH2_MSG_EXT_INFO received
debug1: kex_input_ext_info: server-sig-algs=<ssh-ed25519,[e-mail],ssh-rsa,rsa-sha2-256,rsa-sha2-512,ssh-dss,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,[e-mail],[e-mail]>
debug3: receive packet: type 6
debug2: service_accept: ssh-userauth
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug3: send packet: type 50
debug3: receive packet: type 51
debug1: Authentications that can continue: publickey,gssapi-keyex,gssapi-with-mic,password
debug3: start over, passed a different list publickey,gssapi-keyex,gssapi-with-mic,password
debug3: preferred publickey,keyboard-interactive,password
debug3: authmethod_lookup publickey
debug3: remaining preferred: keyboard-interactive,password
debug3: authmethod_is_enabled publickey
debug1: Next authentication method: publickey
debug1: Offering public key: /home/fedusr/.ssh/id_rsa RSA SHA256:SiFQrcb8BaVwMr65Mm+yfyUTqSN2grVfXTutqDXfpjA agent
debug3: send packet: type 50
debug2: we sent a publickey packet, wait for reply
debug3: receive packet: type 51
debug1: Authentications that can continue: publickey,gssapi-keyex,gssapi-with-mic,password
debug1: Offering public key: /home/fedusr/.ssh/id_ecdsa ECDSA SHA256:Pj5mRaoS/lINfM2bLBxYhv1xx9IbR0+2qY9sEz+Kq48 agent
debug3: send packet: type 50
debug2: we sent a publickey packet, wait for reply
debug3: receive packet: type 51
debug1: Authentications that can continue: publickey,gssapi-keyex,gssapi-with-mic,password
debug1: Offering public key: /home/fedusr/.ssh/id_ed25519 ED25519 SHA256:PbBA+HdHM84Al31IsGaTsQjhhqMQdjdJcrowH4LL6J0 agent
debug3: send packet: type 50
debug2: we sent a publickey packet, wait for reply
debug3: receive packet: type 51
debug1: Authentications that can continue: publickey,gssapi-keyex,gssapi-with-mic,password
debug1: Offering public key: /home/fedusr/.ssh/id_rsa RSA SHA256:Egveyy1ui8/cXaFoYGvF8zE63m+4zYQ1kg64WK8DM/I agent
debug3: send packet: type 50
debug2: we sent a publickey packet, wait for reply
debug3: receive packet: type 51
debug1: Authentications that can continue: publickey,gssapi-keyex,gssapi-with-mic,password
debug1: Offering public key: /home/fedusr/.ssh/id_ecdsa ECDSA SHA256:/844OPWwIB4ly00YnMmfOf/aZAWtlImRXWhhiFkhdTw agent
debug3: send packet: type 50
debug2: we sent a publickey packet, wait for reply
debug3: receive packet: type 51
debug1: Authentications that can continue: publickey,gssapi-keyex,gssapi-with-mic,password
debug1: Trying private key: /home/fedusr/.ssh/id_ecdsa_sk
debug3: no such identity: /home/fedusr/.ssh/id_ecdsa_sk: No such file or directory
debug1: Trying private key: /home/fedusr/.ssh/id_ed25519_sk
debug3: no such identity: /home/fedusr/.ssh/id_ed25519_sk: No such file or directory
debug1: Trying private key: /home/fedusr/.ssh/id_xmss
debug3: no such identity: /home/fedusr/.ssh/id_xmss: No such file or directory
debug2: we did not send a packet, disable method
debug3: authmethod_lookup password
debug3: remaining preferred: ,password
debug3: authmethod_is_enabled password
debug1: Next authentication method: password
[e-mail]'s password:

I have the following in my localhost .ssh:

-rw-r--r--  1 fedusr fedusr  196 Nov  3  2015 id_ecdsa.pub
-rw-------  1 fedusr fedusr  314 Nov  3  2015 id_ecdsa

The authorized_keys matches the id_ecdsa.pub in the local file.

However,  I can no longer do as password-less connection, it worked fine till about 30 hours ago.  What could be wrong? Both machines are automatically updated, the one at localhost at 21:00 hours CT (US) and the other (remote server) at 03:00 CT (US) everyday. But the connection did work fine even after the two machines got updated for a while, till around 12:00 yesterday (I can not pinpoint the exact time it last worked).

Any suggestions as to what I should be looking at? Oh, yes, the passphrases are old, but I tried replacing with new ones and there was no difference to this problem and the insistence on the password.

Please feel free to let me know what additional information might be helpful.

Many thanks and best wishes,
Ranjan

[Samuel Sieb]

On 9/12/21 6:12 PM, Ranjan Maitra wrote:
> However,  I can no longer do as password-less connection, it worked fine till about 30 hours ago.  What could be wrong? Both machines are automatically updated, the one at localhost at 21:00 hours CT (US) and the other (remote server) at 03:00 CT (US) everyday. But the connection did work fine even after the two machines got updated for a while, till around 12:00 yesterday (I can not pinpoint the exact time it last worked).

Check if the config has changed (most likely on the server side).  You 
say the machines are automatically updated, but are they rebooted after 
the update?  Assuming they aren't and sshd was updated, then it's hard 
to say if or when the updated server or config would come into effect. 
Check the dnf history and see if there are any relevant updates recently.

[Cameron Simpson]

On 13Sep2021 18:20, Ranjan Maitra  wrote:
> Thank you very much for your help! After hours on this, I finally 
>   found
>   that somehow in some manner, my permissions had changed on my home
>   directory  (by something I had done, I guess, that I am still trying to
>   track dwn) and which made the permissions global (for my home
>   directory). Which is not a big deal since I am the only user, but
>   selinux does not like it. So, the problem is fixed now.

ssh does not like loose permissions either, as they mean your keys etc 
etc can no have been affected by a user other than yourself. It will 
refuse to use key files in this circumstance.

Cheers,
Cameron Simpson