router anomoly

Sat Feb 7 04:35:56 UTC 2015

Jumping in because some bad guesswork is annoying me.

On 06Feb2015 17:56, Jim Lewis <jim at jklewis.com> wrote:
>> On 02/06/2015 08:25 PM, Ed Greshko wrote:
>>> On 02/07/15 11:14, Geoffrey Leach wrote:
>>>> Turns out that I can't ssh to the other F21 system, so we can confine
>>>> the analysis to F21.  So here's the ssh output from ssh to that that
>>>> system.
>>>>
>>>> geoff at puget[38]->ssh -vv webster
>>>> OpenSSH_6.6.1, OpenSSL 1.0.1k-fips 8 Jan 2015
>>>> debug1: Reading configuration data /etc/ssh/ssh_config
>>>> debug1: /etc/ssh/ssh_config line 56: Applying options for *
>>>> debug2: ssh_connect: needpriv 0
>>>> debug1: Connecting to webster [192.168.10.5] port 22.
>>>> debug1: Connection established.

This says that the machines have correct routing between them: the connection 
completed, requiring the standard TCP three packet handshake, which passes 
packets in both directions. I'd be quite surprised if they could not ping each 
other.

>>>> debug1: identity file /home/geoff/.ssh/id_rsa type 1
>>>> debug1: identity file /home/geoff/.ssh/id_rsa-cert type -1
>>>> debug1: identity file /home/geoff/.ssh/id_dsa type -1
>>>> debug1: identity file /home/geoff/.ssh/id_dsa-cert type -1
>>>> debug1: identity file /home/geoff/.ssh/id_ecdsa type -1
>>>> debug1: identity file /home/geoff/.ssh/id_ecdsa-cert type -1
>>>> debug1: identity file /home/geoff/.ssh/id_ed25519 type -1
>>>> debug1: identity file /home/geoff/.ssh/id_ed25519-cert type -1
>>>> debug1: Enabling compatibility mode for protocol 2.0
>>>> debug1: Local version string SSH-2.0-OpenSSH_6.6.1

All of this happens at the client before receiving anything from the server.

>>>> ssh_exchange_identification: read: Connection reset by peer

This is EOF from the server.

The commonest situation I see this in (which is _not_ what the OP is seeing) is 
connection via an existing ssh port forward (or other proxy of some kind) which 
forwards to a service which itself is not listening. Your client ssh connects 
to the proxy successfully, the proxy tries to connect at the far end and fails, 
and drops the connection back to the client. Client sees EOF as above.

The commonality here is that the connection is established, then cut off by the 
server. No ssh protocol traffic takes place.

I would be starting by checking how ssh is handled on the target ssh server. I 
do not have F21 here.

Traditionally, sshd itself directly listens on port 22 and proceeds. If this is 
the case in F21, then there should be logging in a log file of the received 
connection and the reason for dropping it. Traditionally that log file is 
/var/log/secure on RH and Fedora machines.

So: run "tail -F /var/log/secure" on the server and try your client ssh 
connection attempt again.

An alternative setup is that something else listens on port 22, receives the 
connection, and the drops it instead of invoking sshd. This should also log.

So: find out, on the server, what is listening on port 22. Example:

  netstat -anp | fgrep 22 | fgrep LISTEN

On an example server here it says:

  tcp    0   0 0.0.0.0:22    0.0.0.0:* LISTEN  2480/sshd

at the top of the listing. Therefore, "sshd" is directly listening in my 
example. The process PID is 2480 in the listing above.

One scenario is that something like systemd is doing the listening, and 
rejecting your client after connection for whatever reason.

>>>> WRT the other questions.
>>>> Yes, I did mean /etc/hosts.allow and /etc/hosts.deny
>>> OK, those files are empty on "webster"
>>>
>>> Have you tried using "webster's" IP address?

This is good. Definitiely use the serv's IP address from the client; there is 
less scope of accidents.

However, the OP's transcript says:

  debug1: Connecting to webster [192.168.10.5] port 22.

so if that is correct, no need to directly use the IP address test.

>>>> sshd logs? I can't find any.
>>> On the server side, "webster" ....
>>>
>>> journalctl -b -0 -u sshd
>>>
>> Hi Ed,
>> Does the ssh Server need to resolve client hostname and ip
>> address so that name resolves to the incoming request's ip address?
>> And if it is not able to resolve, then it drops the connection?

This is possible I suppose. Sshd can be told to use PAM, which might do any 
kind of open ended requirements on the client, and plenty of systems ship with 
"UsePAM yes" in the sshd_config. But I would still expect a log message on the 
server in /var/log/secure and I would also expect the client to do some ssh 
traffic anyway, saying access denied.

>  I am going to jump in here as I should probably know how to fix stuff
>like this. Try this (some have been already mentioned):
>-  Use the numeric IP instead of webster.

Worth checking, but the OP can check his original ssh log above as described.

>- I am assuming the user name is the same on both ends, but put it anyway
>just to humor me.

The OP is not getting as far as mentioning the username because there is no ssh 
traffic.

>- If you have an authorized_keys file on the server (in $HOME/.ssh)
>temporarily move it to somewhere else.

Again, irrelevant: the ssh and not conducting conversations with the server.

>- Can each machine ping the other?

Almost certainly, or the ssh connection would never get established.

Even if some firewall rules (eg iptables on client or server) prevented ping, 
they clearly pass ssh TCP connections because the connection gets established.

>  I just did an experiment on my network. Using my two Fedora 21 systems I
>ran "ssh -vv user at numeric-ip" to see what was different on mine:
>
>[guest1 at laptop1 ~]$ ssh -vv guest1 at 192.168.1.24
>OpenSSH_6.6.1, OpenSSL 1.0.1k-fips 8 Jan 2015
>debug1: Reading configuration data /etc/ssh/ssh_config
>debug1: /etc/ssh/ssh_config line 51: Applying options for *
>debug2: ssh_connect: needpriv 0
>debug1: Connecting to 192.168.1.24 [192.168.1.24] port 22.
>debug1: Connection established.
>debug1: identity file /home/guest1/.ssh/id_rsa type -1
>debug1: identity file /home/guest1/.ssh/id_rsa-cert type -1
>debug1: identity file /home/guest1/.ssh/id_dsa type -1
>debug1: identity file /home/guest1/.ssh/id_dsa-cert type -1
>debug1: identity file /home/guest1/.ssh/id_ecdsa type -1
>debug1: identity file /home/guest1/.ssh/id_ecdsa-cert type -1
>debug1: identity file /home/guest1/.ssh/id_ed25519 type -1
>debug1: identity file /home/guest1/.ssh/id_ed25519-cert type -1
>debug1: Enabling compatibility mode for protocol 2.0
>debug1: Local version string SSH-2.0-OpenSSH_6.6.1
>debug1: Remote protocol version 2.0, remote software version OpenSSH_6.6.1
>debug1: match: OpenSSH_6.6.1 pat OpenSSH_6.6.1* compat 0x04000000
>
>   The second to last line is different on mine. I believe this means that
>either the server couldn't send a response or the client couldn't
>receive it.

Yes.

>I think "Connection reset by peer" means the client didn't
>like where this was going and shut it down.

No. That is server-side connection closed. "peer" is the server: it is the peer 
of the client (the far end of the connection from the client).

>Maybe tcpdump could be used to check this.

Yes. But I expect you to learn little from tcpdump except confirmation of the 
basic scenario: client connects to server, server closes connection, no ssh 
protocol traffic.

1: find out if it is sshd directly listening on the server as described earlier.

2: tail various logs on the server as the client attempts connections

Regarding (2) above, if /var/log/secure is silent then get into /var/log and:

  ls -ltr $(find . -type f)

and see what files have bee recently modified (at the bottom of the listing).  
Possibly on of them is the relevant log.

Cheers,
Cameron Simpson <cs at zip.com.au>

If you 'aint falling off, you ar'nt going hard enough.  - Fred Gassit