bastion ssh host key change 2023-03-29

[Kevin Fenzi]

So, as part of our outages yesterday I reinstalled bastion01 (and 02 a
few days before) with rhel9. This means it's ssh host key changed.

However, if you are setup right this should be a non event. :) 

There's at least 2 ways you can confirm the new new is right:

1. Enable sshfp:

Add in your .ssh/config the following to the entry for
bastion/fedora-infrastructure hosts: 

   VerifyHostKeyDNS yes

This will get the ssh fingerprint from dns and confirm it matches.

2. Add our ssh cert authority to your ~/.ssh/known_hosts file.
This can be found at: 
https://admin.fedoraproject.org/ssh_known_hosts
Just add those lines to your known_hosts and ssh will verify and trust
any ssh host key thats signed by those certificate authorities.
(which we do for all hosts).

Sorry for any trouble.

kevin

[Frank Ch. Eigler]

Hi -

>  So, as part of our outages yesterday I reinstalled bastion01 (and 02
> a
>  few days before) with rhel9. This means it's ssh host key changed. 
There was no way of saving & restoring the former host key?


>  1. Enable sshfp:
>  Add in your .ssh/config the following to the entry for
>  bastion/fedora-infrastructure hosts: 
>     VerifyHostKeyDNS yes
>  This will get the ssh fingerprint from dns and confirm it matches. 
(This is enabled but f37 ssh still rejects the change.)


- FChE

[Kevin Fenzi]

On Thu, Mar 30, 2023 at 11:57:45AM -0400, Frank Ch. Eigler wrote:
>  Hi -
>  
>  > So, as part of our outages yesterday I reinstalled bastion01 (and 02 a
>  > few days before) with rhel9. This means it's ssh host key changed.
>  
>  There was no way of saving & restoring the former host key? 
I could get it back. But at this point I think it's better to figure out
how to never hit it again. ;) 

>  > 1. Enable sshfp:
>  > Add in your .ssh/config the following to the entry for
>  > bastion/fedora-infrastructure hosts: 
>  >    VerifyHostKeyDNS yes
>  > This will get the ssh fingerprint from dns and confirm it matches.
>  
>  (This is enabled but f37 ssh still rejects the change.) 
Odd. It definitely used to work. I can look more when I get a chance. 
I did update the SSHFP records. 

kevin

[Todd Zullinger]

Kevin Fenzi wrote:
>  On Thu, Mar 30, 2023 at 11:57:45AM -0400, Frank Ch. Eigler wrote:
> > Hi -
> > 
> >> So, as part of our outages yesterday I reinstalled bastion01 (and 02 a
> >> few days before) with rhel9. This means it's ssh host key changed.
> > 
> > There was no way of saving & restoring the former host key?
>  
>  I could get it back. But at this point I think it's better to figure out
>  how to never hit it again. ;) 
>  
> >> 1. Enable sshfp:
> >> Add in your .ssh/config the following to the entry for
> >> bastion/fedora-infrastructure hosts: 
> >>    VerifyHostKeyDNS yes
> >> This will get the ssh fingerprint from dns and confirm it matches.
> > 
> > (This is enabled but f37 ssh still rejects the change.)
>  
>  Odd. It definitely used to work. I can look more when I get a chance. 
>  I did update the SSHFP records.  
The VerifyHostKeyDNS does require secure DNS to avoid any
confirmation prompt.  Without DNSSEC, `VerifyHostKeyDNS yes`
is the same as `VerifyHostKeyDNS ask`.

Perhaps that's the issue in Frank's case?

-- 
Todd

[Frank Ch. Eigler]

Hi -

>  The VerifyHostKeyDNS does require secure DNS to avoid any
>  confirmation prompt.  Without DNSSEC, `VerifyHostKeyDNS yes`
>  is the same as `VerifyHostKeyDNS ask`. 
OK, that's one thing to check/fix.

>  Perhaps that's the issue in Frank's case? 
Plus: bastion-iad01.fedoraproject.org. appears to lack the SSHFP records.

- FChE

[Kevin Fenzi]

On Thu, Mar 30, 2023 at 04:11:45PM -0400, Frank Ch. Eigler wrote:
>  Hi -
>  
>  > The VerifyHostKeyDNS does require secure DNS to avoid any
>  > confirmation prompt.  Without DNSSEC, `VerifyHostKeyDNS yes`
>  > is the same as `VerifyHostKeyDNS ask`.
>  
>  OK, that's one thing to check/fix. 
Ah yeah, that could well be the case.

>  > Perhaps that's the issue in Frank's case?
>  
>  Plus: bastion-iad01.fedoraproject.org. appears to lack the SSHFP records. 
We should drop that from dns. It was only used when we were moving from
phx2 to iad2. There should be 3 things in dns:

bastion01.fedoraproject.org
bastion02.fedoraproject.org
bastion.fedoraproject.org

with the last one being a CNAME for whichever one is default
(currently 01).

Anyhow, the ssh access SOP should be updated with all this info. 

Probibly sshfp isn't worth doing these days and we should just stick
with the certificate signed config.

kevin

[Fabian Arrotin]

On 30/03/2023 22:25, Kevin Fenzi wrote:
>  On Thu, Mar 30, 2023 at 04:11:45PM -0400, Frank Ch. Eigler wrote:
> > Hi -
> >
> >> The VerifyHostKeyDNS does require secure DNS to avoid any
> >> confirmation prompt.  Without DNSSEC, `VerifyHostKeyDNS yes`
> >> is the same as `VerifyHostKeyDNS ask`.
> >
> > OK, that's one thing to check/fix.
>  
>  Ah yeah, that could well be the case.
>  
> >> Perhaps that's the issue in Frank's case?
> >
> > Plus: bastion-iad01.fedoraproject.org. appears to lack the SSHFP records.
>  
>  We should drop that from dns. It was only used when we were moving from
>  phx2 to iad2. There should be 3 things in dns:
>  
>  bastion01.fedoraproject.org
>  bastion02.fedoraproject.org
>  bastion.fedoraproject.org
>  
>  with the last one being a CNAME for whichever one is default
>  (currently 01).
>  
>  Anyhow, the ssh access SOP should be updated with all this info.
>  
>  Probibly sshfp isn't worth doing these days and we should just stick
>  with the certificate signed config.
>  
+1 to that ..
While it's possible to automate SSHFP (to a kind of delegated/dynamic 
zone), I prefer just relying on signed host certificate : just trust the 
CA once and enjoy forever :)

FWIW, that's also what we do for centos.org infra

-- 
Fabian Arrotin
gpg key: 17F3B7A1

[Aurelien Bompard]

>  We should drop that from dns. [...]
>  Anyhow, the ssh access SOP should be updated with all this info. 
I looked for the SOP and found this:
https://docs.fedoraproject.org/en-US/infra/sysadmin_guide/sshaccess/
It still mentions bastion-iad01. Am I on the wrong docs? It looks like
the right place.

I also stumbled upon https://fedora-infra-docs.readthedocs.io/ in my
bookmarks and it's been squatted. It's built from
https://github.com/primitivea/fedora-infra-docs, but that's a
different issue altogether.


Aurélien

[Kevin Fenzi]

On Fri, Mar 31, 2023 at 10:30:41AM +0200, Aurelien Bompard wrote:
>  > We should drop that from dns. [...]
>  > Anyhow, the ssh access SOP should be updated with all this info.
>  
>  I looked for the SOP and found this:
>  https://docs.fedoraproject.org/en-US/infra/sysadmin_guide/sshaccess/
>  It still mentions bastion-iad01. Am I on the wrong docs? It looks like
>  the right place. 
Thats the right place, no one updated it. ;( 

https://pagure.io/infra-docs-fpo/pull-request/195

adds a bunch of things. Reviews welcome. :)

>  I also stumbled upon https://fedora-infra-docs.readthedocs.io/ in my
>  bookmarks and it's been squatted. It's built from
>  https://github.com/primitivea/fedora-infra-docs, but that's a
>  different issue altogether. 
Ugh. Well, we used to use that, but retired it.

Is there any way we could ask them to remove it?
Or are we just out of luck since we dropped it?

kevin