ssh to gnu fails after F33 update

[Per Bothner]

I don't know if this is expected or a Fedora issue, but after I updated
to Fedora 33 (in-place) I can no longer do ssh operations with cvs.savannah.gnu.org.
It is supposed to use uploaded ~/.ssh/id_rsa.pub, but instead it asks for a password:

$ cvs diff
bothner(a)cvs.savannah.gnu.org's password:

See https://savannah.gnu.org/maintenance/SshAccess/

This worked a few days ago, when I did a 'cvs ci' in the same directory.
It is possible something changed an Savannah (or I did something wrong),
but it seems updating to F33 is the most likely cause.

I updated using the process described here:
https://docs.fedoraproject.org/en-US/quick-docs/dnf-system-upgrade/
but with --releasever=33.

I have not tried creating a new new ssh-key with ssh-keygen.

Otherwise Fedora 33 has been smooth sailing so far.
-- 
	--Per Bothner
per(a)bothner.com   http://per.bothner.com/

[Adam Williamson]

On Wed, 2020-10-21 at 13:37 -0700, Per Bothner wrote:
>  I don't know if this is expected or a Fedora issue, but after I
> updated
>  to Fedora 33 (in-place) I can no longer do ssh operations with cvs.savannah.gnu.org.
>  It is supposed to use uploaded ~/.ssh/id_rsa.pub, but instead it asks for a password:
>  
>  $ cvs diff
>  bothner(a)cvs.savannah.gnu.org's password:
>  
>  See https://savannah.gnu.org/maintenance/SshAccess/
>  
>  This worked a few days ago, when I did a 'cvs ci' in the same directory.
>  It is possible something changed an Savannah (or I did something wrong),
>  but it seems updating to F33 is the most likely cause.
>  
>  I updated using the process described here:
>  https://docs.fedoraproject.org/en-US/quick-docs/dnf-system-upgrade/
>  but with --releasever=33.
>  
>  I have not tried creating a new new ssh-key with ssh-keygen. 
Try ssh -vvv or ssh -vvvv , it will give you much more detail on what
mechanisms are attempted and (sometimes) why they fail. May be an
algorithm issue possibly.
-- 
Adam Williamson
Fedora QA Community Monkey
IRC: adamw | Twitter: AdamW_Fedora | XMPP: adamw AT happyassassin . net
http://www.happyassassin.net

[Per Bothner]

On 10/21/20 1:47 PM, Adam Williamson wrote:
>  Try ssh -vvv or ssh -vvvv , it will give you much more detail on
> what
>  mechanisms are attempted and (sometimes) why they fail. May be an
>  algorithm issue possibly. 
$ ssh -vvvv cvs.savannah.gnu.org
OpenSSH_8.4p1, OpenSSL 1.1.1g FIPS  21 Apr 2020
debug1: Reading configuration data /home/bothner/.ssh/config
debug1: /home/bothner/.ssh/config line 11: Applying options for *.gnu.org
debug1: Reading configuration data /etc/ssh/ssh_config
debug3: /etc/ssh/ssh_config line 55: Including file /etc/ssh/ssh_config.d/50-redhat.conf
depth 0
debug1: Reading configuration data /etc/ssh/ssh_config.d/50-redhat.conf
debug2: checking match for 'final all' host cvs.savannah.gnu.org originally
cvs.savannah.gnu.org
debug3: /etc/ssh/ssh_config.d/50-redhat.conf line 3: not matched 'final'
debug2: match not found
debug3: /etc/ssh/ssh_config.d/50-redhat.conf line 5: Including file
/etc/crypto-policies/back-ends/openssh.config depth 1 (parse only)
debug1: Reading configuration data /etc/crypto-policies/back-ends/openssh.config
debug3: gss kex names ok:
[gss-curve25519-sha256-,gss-nistp256-sha256-,gss-group14-sha256-,gss-group16-sha512-]
debug3: kex names ok:
[curve25519-sha256,[e-mail],ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group14-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512]
debug1: configuration requests final Match pass
debug1: re-parsing configuration
debug1: Reading configuration data /home/bothner/.ssh/config
debug1: /home/bothner/.ssh/config line 11: Applying options for *.gnu.org
debug1: Reading configuration data /etc/ssh/ssh_config
debug3: /etc/ssh/ssh_config line 55: Including file /etc/ssh/ssh_config.d/50-redhat.conf
depth 0
debug1: Reading configuration data /etc/ssh/ssh_config.d/50-redhat.conf
debug2: checking match for 'final all' host cvs.savannah.gnu.org originally
cvs.savannah.gnu.org
debug3: /etc/ssh/ssh_config.d/50-redhat.conf line 3: matched 'final'
debug2: match found
debug3: /etc/ssh/ssh_config.d/50-redhat.conf line 5: Including file
/etc/crypto-policies/back-ends/openssh.config depth 1
debug1: Reading configuration data /etc/crypto-policies/back-ends/openssh.config
debug3: gss kex names ok:
[gss-curve25519-sha256-,gss-nistp256-sha256-,gss-group14-sha256-,gss-group16-sha512-]
debug3: kex names ok:
[curve25519-sha256,[e-mail],ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group14-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512]
debug3: expanded UserKnownHostsFile '~/.ssh/known_hosts' ->
'/home/bothner/.ssh/known_hosts'
debug3: expanded UserKnownHostsFile '~/.ssh/known_hosts2' ->
'/home/bothner/.ssh/known_hosts2'
debug2: resolving "cvs.savannah.gnu.org" port 22
debug2: ssh_connect_direct
debug1: Connecting to cvs.savannah.gnu.org [209.51.188.81] port 22.
debug1: Connection established.
debug1: identity file /home/bothner/.ssh/id_rsa type 0
debug1: identity file /home/bothner/.ssh/id_rsa-cert type -1
debug1: identity file /home/bothner/.ssh/id_dsa type 1
debug1: identity file /home/bothner/.ssh/id_dsa-cert type -1
debug1: identity file /home/bothner/.ssh/id_ecdsa type -1
debug1: identity file /home/bothner/.ssh/id_ecdsa-cert type -1
debug1: identity file /home/bothner/.ssh/id_ecdsa_sk type -1
debug1: identity file /home/bothner/.ssh/id_ecdsa_sk-cert type -1
debug1: identity file /home/bothner/.ssh/id_ed25519 type -1
debug1: identity file /home/bothner/.ssh/id_ed25519-cert type -1
debug1: identity file /home/bothner/.ssh/id_ed25519_sk type -1
debug1: identity file /home/bothner/.ssh/id_ed25519_sk-cert type -1
debug1: identity file /home/bothner/.ssh/id_xmss type -1
debug1: identity file /home/bothner/.ssh/id_xmss-cert type -1
debug1: Local version string SSH-2.0-OpenSSH_8.4
debug1: Remote protocol version 2.0, remote software version OpenSSH_7.4p1
Debian-10+deb9u7
debug1: match: OpenSSH_7.4p1 Debian-10+deb9u7 pat
OpenSSH_7.0*,OpenSSH_7.1*,OpenSSH_7.2*,OpenSSH_7.3*,OpenSSH_7.4*,OpenSSH_7.5*,OpenSSH_7.6*,OpenSSH_7.7*
compat 0x04000002
debug2: fd 4 setting O_NONBLOCK
debug1: Authenticating to cvs.savannah.gnu.org:22 as 'bothner'
debug3: hostkeys_foreach: reading file "/home/bothner/.ssh/known_hosts"
debug3: record_hostkey: found key type RSA in file /home/bothner/.ssh/known_hosts:2
debug3: load_hostkeys: loaded 1 keys from cvs.savannah.gnu.org
debug3: order_hostkeyalgs: prefer hostkeyalgs:
[e-mail],[e-mail],ssh-rsa-cert-v01(a)openssh.com,rsa-sha2-512,rsa-sha2-256,ssh-rsa
debug3: send packet: type 20
debug1: SSH2_MSG_KEXINIT sent
debug3: receive packet: type 20
debug1: SSH2_MSG_KEXINIT received
debug2: local client KEXINIT proposal
debug2: KEX algorithms:
curve25519-sha256,[e-mail],ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group14-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,ext-info-c
debug2: host key algorithms:
[e-mail],[e-mail],[e-mail],rsa-sha2-512,rsa-sha2-256,ssh-rsa,[e-mail],[e-mail],[e-mail],[e-mail],[e-mail],[e-mail],ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,[e-mail],ssh-ed25519,sk-ssh-ed25519(a)openssh.com
debug2: ciphers ctos:
[e-mail],[e-mail],aes256-ctr,aes128-gcm(a)openssh.com,aes128-ctr
debug2: ciphers stoc:
[e-mail],[e-mail],aes256-ctr,aes128-gcm(a)openssh.com,aes128-ctr
debug2: MACs ctos:
[e-mail],[e-mail],[e-mail],[e-mail],hmac-sha2-256,hmac-sha1,umac-128(a)openssh.com,hmac-sha2-512
debug2: MACs stoc:
[e-mail],[e-mail],[e-mail],[e-mail],hmac-sha2-256,hmac-sha1,umac-128(a)openssh.com,hmac-sha2-512
debug2: compression ctos: zlib(a)openssh.com,zlib,none
debug2: compression stoc: zlib(a)openssh.com,zlib,none
debug2: languages ctos:
debug2: languages stoc:
debug2: first_kex_follows 0
debug2: reserved 0
debug2: peer server KEXINIT proposal
debug2: KEX algorithms:
curve25519-sha256,[e-mail],ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group14-sha256,diffie-hellman-group14-sha1
debug2: host key algorithms:
ssh-rsa,rsa-sha2-512,rsa-sha2-256,ecdsa-sha2-nistp256,ssh-ed25519
debug2: ciphers ctos:
[e-mail],aes128-ctr,aes192-ctr,aes256-ctr,[e-mail],aes256-gcm(a)openssh.com
debug2: ciphers stoc:
[e-mail],aes128-ctr,aes192-ctr,aes256-ctr,[e-mail],aes256-gcm(a)openssh.com
debug2: MACs ctos:
[e-mail],[e-mail],[e-mail],[e-mail],[e-mail],[e-mail],umac-128(a)openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: MACs stoc:
[e-mail],[e-mail],[e-mail],[e-mail],[e-mail],[e-mail],umac-128(a)openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: compression ctos: none,zlib(a)openssh.com
debug2: compression stoc: none,zlib(a)openssh.com
debug2: languages ctos:
debug2: languages stoc:
debug2: first_kex_follows 0
debug2: reserved 0
debug1: kex: algorithm: curve25519-sha256
debug1: kex: host key algorithm: rsa-sha2-512
debug1: kex: server->client cipher: aes256-gcm(a)openssh.com MAC: <implicit>
compression: zlib(a)openssh.com
debug1: kex: client->server cipher: aes256-gcm(a)openssh.com MAC: <implicit>
compression: zlib(a)openssh.com
debug1: kex: curve25519-sha256 need=32 dh_need=32
debug1: kex: curve25519-sha256 need=32 dh_need=32
debug3: send packet: type 30
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug3: receive packet: type 31
debug1: Server host key: ssh-rsa SHA256:FYkx0iik+iBeCLRzvUyUSTRT98TEBBJoYuQsTXbyGL8
debug3: hostkeys_foreach: reading file "/home/bothner/.ssh/known_hosts"
debug3: record_hostkey: found key type RSA in file /home/bothner/.ssh/known_hosts:2
debug3: load_hostkeys: loaded 1 keys from cvs.savannah.gnu.org
debug3: hostkeys_foreach: reading file "/home/bothner/.ssh/known_hosts"
debug3: record_hostkey: found key type RSA in file /home/bothner/.ssh/known_hosts:32
debug3: load_hostkeys: loaded 1 keys from 209.51.188.81
debug1: Host 'cvs.savannah.gnu.org' is known and matches the RSA host key.
debug1: Found key in /home/bothner/.ssh/known_hosts:2
debug3: send packet: type 21
debug2: set_newkeys: mode 1
debug1: rekey out after 4294967296 blocks
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug3: receive packet: type 21
debug1: SSH2_MSG_NEWKEYS received
debug2: set_newkeys: mode 0
debug1: rekey in after 4294967296 blocks
debug1: Skipping ssh-dss key /home/bothner/.ssh/id_dsa - not in PubkeyAcceptedKeyTypes
debug1: Will attempt key: /home/bothner/.ssh/id_rsa RSA
SHA256:UrZs1kSmA0JMUjJTlkP4Roth6eI8g5rCNiLWj8xX99E agent
debug1: Will attempt key: /home/bothner/.ssh/id_ecdsa
debug1: Will attempt key: /home/bothner/.ssh/id_ecdsa_sk
debug1: Will attempt key: /home/bothner/.ssh/id_ed25519
debug1: Will attempt key: /home/bothner/.ssh/id_ed25519_sk
debug1: Will attempt key: /home/bothner/.ssh/id_xmss
debug2: pubkey_prepare: done
debug3: send packet: type 5
debug3: receive packet: type 7
debug1: SSH2_MSG_EXT_INFO received
debug1: kex_input_ext_info:
server-sig-algs=<ssh-ed25519,ssh-rsa,ssh-dss,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521>
debug3: receive packet: type 6
debug2: service_accept: ssh-userauth
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug3: send packet: type 50
debug3: receive packet: type 51
debug1: Authentications that can continue: publickey,password
debug3: start over, passed a different list publickey,password
debug3: preferred gssapi-with-mic,publickey,keyboard-interactive,password
debug3: authmethod_lookup publickey
debug3: remaining preferred: keyboard-interactive,password
debug3: authmethod_is_enabled publickey
debug1: Next authentication method: publickey
debug1: Offering public key: /home/bothner/.ssh/id_rsa RSA
SHA256:UrZs1kSmA0JMUjJTlkP4Roth6eI8g5rCNiLWj8xX99E agent
debug1: send_pubkey_test: no mutual signature algorithm
debug1: Trying private key: /home/bothner/.ssh/id_ecdsa
debug3: no such identity: /home/bothner/.ssh/id_ecdsa: No such file or directory
debug1: Trying private key: /home/bothner/.ssh/id_ecdsa_sk
debug3: no such identity: /home/bothner/.ssh/id_ecdsa_sk: No such file or directory
debug1: Trying private key: /home/bothner/.ssh/id_ed25519
debug3: no such identity: /home/bothner/.ssh/id_ed25519: No such file or directory
debug1: Trying private key: /home/bothner/.ssh/id_ed25519_sk
debug3: no such identity: /home/bothner/.ssh/id_ed25519_sk: No such file or directory
debug1: Trying private key: /home/bothner/.ssh/id_xmss
debug3: no such identity: /home/bothner/.ssh/id_xmss: No such file or directory
debug2: we did not send a packet, disable method
debug3: authmethod_lookup password
debug3: remaining preferred: ,password
debug3: authmethod_is_enabled password
debug1: Next authentication method: password
bothner(a)cvs.savannah.gnu.org&#39;s password:

~/.ssh/config is:

Compression yes
Host *.redhat.com
Protocol 2
ForwardX11 no
#Host sourceware.org
#Protocol 2
#ForwardX11 no
Host gcc.gnu.org
Protocol 2
ForwardX11 no
Host *.gnu.org
Protocol 2
ForwardX11 no

-- 
	--Per Bothner
per(a)bothner.com   http://per.bothner.com/

[Adam Williamson]

On Wed, 2020-10-21 at 13:53 -0700, Per Bothner wrote:
>  debug1: Next authentication method: publickey
>  debug1: Offering public key: /home/bothner/.ssh/id_rsa RSA
> SHA256:UrZs1kSmA0JMUjJTlkP4Roth6eI8g5rCNiLWj8xX99E agent
>  debug1: send_pubkey_test: no mutual signature algorithm 
that looks to be the problem: "no mutual signature algorithm".

The Fedora crypto policies are updated every release, so this is
probably what changed. Changing to the LEGACY policy might make it work
again, at the cost of reduced security, I guess. See `man update-
crypto-policies` (which also documents how you can override the policy
for ssh).
-- 
Adam Williamson
Fedora QA Community Monkey
IRC: adamw | Twitter: AdamW_Fedora | XMPP: adamw AT happyassassin . net
http://www.happyassassin.net

[Per Bothner]

On 10/21/20 1:59 PM, Adam Williamson wrote:
>  The Fedora crypto policies are updated every release, so this is
>  probably what changed. Changing to the LEGACY policy might make it work
>  again, at the cost of reduced security, I guess. See `man update-
>  crypto-policies` (which also documents how you can override the policy
>  for ssh). 
I was able to get things working again by adding this line to the
Host *.gnu.org section of ~/.ssh/config

PubkeyAcceptedKeyTypes +rsa-sha2-256,rsa-sha2-512

(There were some web-searching and mis-steps on the way ...)
-- 
	--Per Bothner
per(a)bothner.com   http://per.bothner.com/

[Gordon Messmer]

On 10/21/20 5:51 PM, Per Bothner wrote:
>  I was able to get things working again by adding this line to the
>  Host *.gnu.org section of ~/.ssh/config
>
>  PubkeyAcceptedKeyTypes +rsa-sha2-256,rsa-sha2-512  

"rsa-sha2-256,rsa-sha2-51" are already part of the Fedora 33 crypto 
policy.  This configuration works (as I understand it) because the '+' 
value adds key types to the default set, rather than the local policy.  
The effect is that the configuration above downgrades the crypto policy 
so that it includes 'ssh-rsa' again, and that's the key type that ends 
up used with OpenSSH 7.4 servers.

[José Abílio Matos]

On Wednesday, October 21, 2020 9:53:04 PM WEST Per Bothner wrote:
>  
>  ~/.ssh/config is:
>  
>  Compression yes
>  Host *.redhat.com
>  Protocol 2
>  ForwardX11 no
>  #Host sourceware.org
>  #Protocol 2
>  #ForwardX11 no
>  Host gcc.gnu.org
>  Protocol 2
>  ForwardX11 no
>  Host *.gnu.org
>  Protocol 2
>  ForwardX11 no 
I had similar problems and my solution was to add to .ssh/config:

Host *
   PubkeyAcceptedKeyTypes +rsa-sha2-256,rsa-sha2-512

Of course, instead of applying this to all addresses you can add this just to 
the addresses that need it.

Regards,
-- 
José Abílio

[Per Bothner]

Looks like our emails crossed ...

On 10/21/20 5:28 PM, José Abílio Matos wrote:
>  I had similar problems and my solution was to add to .ssh/config:
>  
>  
>  Host *
>  
>      PubkeyAcceptedKeyTypes +rsa-sha2-256,rsa-sha2-512
>  
>  
>  Of course, instead of applying this to all addresses you can add this just to the
> addresses that need it. 
Indeed I did:

Host *.gnu.org
Protocol 2
ForwardX11 no
PubkeyAcceptedKeyTypes +rsa-sha2-256,rsa-sha2-512

-- 
	--Per Bothner
per(a)bothner.com   http://per.bothner.com/

[Gordon Messmer]

On 10/21/20 1:37 PM, Per Bothner wrote:
>  I don't know if this is expected or a Fedora issue, but after I
> updated
>  to Fedora 33 (in-place) I can no longer do ssh operations with 
>  cvs.savannah.gnu.org.  

This is a bug in the OpenSSH 7.4 server they're using.  I provided a 
patch to work around it in the client, here:
https://bugzilla.redhat.com/show_bug.cgi?id=1881301