Tracking down SSH access

[Michael Eager]

When I look at /var/log/secure or run journalctl on my workstation, I
see failed SSH login attempts from a variety of IP addresses.  The
attempts are every 3-12 minutes.

/etc/ssh/sshd_config contains:
PasswordAuthentication no

The workstation is on a LAN behind an EdgeRouter firewall.  No Internet-
accessible ports are forwarded to the workstation.  The LAN has a
variety of servers, NAS boxes, WiFi access points, WiFi-connected
laptops, etc.

A typical /var/log/secure entry looks like this:
Jan 30 12:43:50 redwood sshd[21228]: Invalid user jackiehulu from 
124.204.36.138 port 37394
Jan 30 12:43:51 redwood sshd[21228]: Received disconnect from 
124.204.36.138 port 37394:11: Bye Bye [preauth]
Jan 30 12:43:51 redwood sshd[21228]: Disconnected from invalid user 
jackiehulu 124.204.36.138 port 37394 [preauth]

The corresponding journalctl is:
Jan 30 12:43:51 redwood.eagercon.com audit[21228]: USER_ERR pid=21228 
uid=0 auid=4294967295 ses=4294967295 
subj=system_u:system_r:sshd_t:s0-s0:c0.c1023 msg='op=PAM:bad_ident 
grantors=? acct="?" exe="/usr/sbin/sshd" hostname=124.204.36.138 
addr=124.204.36.138 terminal=ssh res=failed'

I'm assuming that something on the network has been compromised,
allowing SSH login attempts on the LAN.  Other than turning off
each server/AP/laptop/etc, one at a time, to find when the accesses
stop, is there any way to find out where the SSH attempt is coming from?

-- Mike Eager

[Samuel Sieb]

On 1/30/20 1:12 PM, Michael Eager wrote:
> When I look at /var/log/secure or run journalctl on my workstation, I
> see failed SSH login attempts from a variety of IP addresses.  The
> attempts are every 3-12 minutes.
>
> The workstation is on a LAN behind an EdgeRouter firewall.  No Internet-
> accessible ports are forwarded to the workstation.  The LAN has a
> variety of servers, NAS boxes, WiFi access points, WiFi-connected
> laptops, etc.
>
> A typical /var/log/secure entry looks like this:
> Jan 30 12:43:50 redwood sshd[21228]: Invalid user jackiehulu from 
> 124.204.36.138 port 37394
> Jan 30 12:43:51 redwood sshd[21228]: Received disconnect from 
> 124.204.36.138 port 37394:11: Bye Bye [preauth]
> Jan 30 12:43:51 redwood sshd[21228]: Disconnected from invalid user 
> jackiehulu 124.204.36.138 port 37394 [preauth]
>
> I'm assuming that something on the network has been compromised,
> allowing SSH login attempts on the LAN.  Other than turning off
> each server/AP/laptop/etc, one at a time, to find when the accesses
> stop, is there any way to find out where the SSH attempt is coming from?

From the attacking IP address, unless you're in China, your computer 
must be internet accessible somehow.  That's not an IP address on your 
LAN, right?

[Jack Craig]

wireshark -> tcpdump on dst=port# src = all
??

On Thu, Jan 30, 2020 at 1:13 PM Michael Eager  wrote:
> When I look at /var/log/secure or run journalctl on my workstation, I
> see failed SSH login attempts from a variety of IP addresses.  The
> attempts are every 3-12 minutes.
>
> /etc/ssh/sshd_config contains:
> PasswordAuthentication no
>
> The workstation is on a LAN behind an EdgeRouter firewall.  No Internet-
> accessible ports are forwarded to the workstation.  The LAN has a
> variety of servers, NAS boxes, WiFi access points, WiFi-connected
> laptops, etc.
>
> A typical /var/log/secure entry looks like this:
> Jan 30 12:43:50 redwood sshd[21228]: Invalid user jackiehulu from
> 124.204.36.138 port 37394
> Jan 30 12:43:51 redwood sshd[21228]: Received disconnect from
> 124.204.36.138 port 37394:11: Bye Bye [preauth]
> Jan 30 12:43:51 redwood sshd[21228]: Disconnected from invalid user
> jackiehulu 124.204.36.138 port 37394 [preauth]
>
> The corresponding journalctl is:
> Jan 30 12:43:51 redwood.eagercon.com audit[21228]: USER_ERR pid=21228
> uid=0 auid=4294967295 ses=4294967295
> subj=system_u:system_r:sshd_t:s0-s0:c0.c1023 msg='op=PAM:bad_ident
> grantors=? acct="?" exe="/usr/sbin/sshd" hostname=124.204.36.138
> addr=124.204.36.138 terminal=ssh res=failed'
>
> I'm assuming that something on the network has been compromised,
> allowing SSH login attempts on the LAN.  Other than turning off
> each server/AP/laptop/etc, one at a time, to find when the accesses
> stop, is there any way to find out where the SSH attempt is coming from?
>
> -- Mike Eager
> _______________________________________________
> users mailing list -- [e-mail]
> To unsubscribe send an email to [e-mail]
> Fedora Code of Conduct:
> https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives:
> https://lists.fedoraproject.org/archives/list/[e-mail]

[Michael Eager]

Thanks.  I'll give that a try.

On 1/30/20 1:49 PM, Jack Craig wrote:
> wireshark -> tcpdump on dst=port# src = all
> ??
>
> On Thu, Jan 30, 2020 at 1:13 PM Michael Eager <[e-mail] 
> > wrote:
> When I look at /var/log/secure or run journalctl on my workstation, I
> see failed SSH login attempts from a variety of IP addresses.  The
> attempts are every 3-12 minutes.
>
> /etc/ssh/sshd_config contains:
> PasswordAuthentication no
>
> The workstation is on a LAN behind an EdgeRouter firewall.  No Internet-
> accessible ports are forwarded to the workstation.  The LAN has a
> variety of servers, NAS boxes, WiFi access points, WiFi-connected
> laptops, etc.
>
> A typical /var/log/secure entry looks like this:
> Jan 30 12:43:50 redwood sshd[21228]: Invalid user jackiehulu from
> 124.204.36.138 port 37394
> Jan 30 12:43:51 redwood sshd[21228]: Received disconnect from
> 124.204.36.138 port 37394:11: Bye Bye [preauth]
> Jan 30 12:43:51 redwood sshd[21228]: Disconnected from invalid user
> jackiehulu 124.204.36.138 port 37394 [preauth]
>
> The corresponding journalctl is:
> Jan 30 12:43:51 redwood.eagercon.com <http://redwood.eagercon.com>
> audit[21228]: USER_ERR pid=21228
> uid=0 auid=4294967295 ses=4294967295
> subj=system_u:system_r:sshd_t:s0-s0:c0.c1023 msg='op=PAM:bad_ident
> grantors=? acct="?" exe="/usr/sbin/sshd" hostname=124.204.36.138
> addr=124.204.36.138 terminal=ssh res=failed'
>
> I'm assuming that something on the network has been compromised,
> allowing SSH login attempts on the LAN.  Other than turning off
> each server/AP/laptop/etc, one at a time, to find when the accesses
> stop, is there any way to find out where the SSH attempt is coming from?
>
> -- Mike Eager
> _______________________________________________
> users mailing list -- [e-mail]
> <mailto:[e-mail]>
> To unsubscribe send an email to [e-mail]
> <mailto:[e-mail]>
> Fedora Code of Conduct:
> https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives:
> https://lists.fedoraproject.org/archives/list/[e-mail]
>
>
> users mailing list -- [e-mail]
> To unsubscribe send an email to [e-mail]
> Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives: https://lists.fedoraproject.org/archives/list/[e-mail]

[Jack Craig]

with some work, you can limit the filter on capture to screen out all but
the traffic you want to see.

the web should have lots of 'how to' clips.

good luck, ...

On Thu, Jan 30, 2020 at 5:12 PM Michael Eager  wrote:
> Thanks.  I'll give that a try.
>
> On 1/30/20 1:49 PM, Jack Craig wrote:
> > wireshark -> tcpdump on dst=port# src = all
> > ??
> >
> > On Thu, Jan 30, 2020 at 1:13 PM Michael Eager <[e-mail]
> > > wrote:
> > When I look at /var/log/secure or run journalctl on my workstation, I
> > see failed SSH login attempts from a variety of IP addresses.  The
> > attempts are every 3-12 minutes.
> >
> > /etc/ssh/sshd_config contains:
> > PasswordAuthentication no
> >
> > The workstation is on a LAN behind an EdgeRouter firewall.  No
>
> Internet-
> > accessible ports are forwarded to the workstation.  The LAN has a
> > variety of servers, NAS boxes, WiFi access points, WiFi-connected
> > laptops, etc.
> >
> > A typical /var/log/secure entry looks like this:
> > Jan 30 12:43:50 redwood sshd[21228]: Invalid user jackiehulu from
> > 124.204.36.138 port 37394
> > Jan 30 12:43:51 redwood sshd[21228]: Received disconnect from
> > 124.204.36.138 port 37394:11: Bye Bye [preauth]
> > Jan 30 12:43:51 redwood sshd[21228]: Disconnected from invalid user
> > jackiehulu 124.204.36.138 port 37394 [preauth]
> >
> > The corresponding journalctl is:
> > Jan 30 12:43:51 redwood.eagercon.com <http://redwood.eagercon.com>
> > audit[21228]: USER_ERR pid=21228
> > uid=0 auid=4294967295 ses=4294967295
> > subj=system_u:system_r:sshd_t:s0-s0:c0.c1023 msg='op=PAM:bad_ident
> > grantors=? acct="?" exe="/usr/sbin/sshd" hostname=124.204.36.138
> > addr=124.204.36.138 terminal=ssh res=failed'
> >
> > I'm assuming that something on the network has been compromised,
> > allowing SSH login attempts on the LAN.  Other than turning off
> > each server/AP/laptop/etc, one at a time, to find when the accesses
> > stop, is there any way to find out where the SSH attempt is coming
>
> from?
> > -- Mike Eager
> > _______________________________________________
> > users mailing list -- [e-mail]
> > <mailto:[e-mail]>
> > To unsubscribe send an email to [e-mail]
> > <mailto:[e-mail]>
> > Fedora Code of Conduct:
> > https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> > List Guidelines:
>
> https://fedoraproject.org/wiki/Mailing_list_guidelines
> > List Archives:
>
> https://lists.fedoraproject.org/archives/list/[e-mail]
> > users mailing list -- [e-mail]
> > To unsubscribe send an email to [e-mail]
> > Fedora Code of Conduct:
>
> https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> > List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> > List Archives:
>
> https://lists.fedoraproject.org/archives/list/[e-mail]
> >
>
> --
> Michael Eager    [e-mail]
> 1960 Park Blvd., Palo Alto, CA 94306
> _______________________________________________
> users mailing list -- [e-mail]
> To unsubscribe send an email to [e-mail]
> Fedora Code of Conduct:
> https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives:
> https://lists.fedoraproject.org/archives/list/[e-mail]

[Jack Craig]

one more bit, when you get to the command line ssh ... , throw in a bunch
of -v to crank up verbosity

On Thu, Jan 30, 2020 at 5:18 PM Jack Craig 
wrote:
> with some work, you can limit the filter on capture to screen out all but
> the traffic you want to see.
>
> the web should have lots of 'how to' clips.
>
> good luck, ...
>
> On Thu, Jan 30, 2020 at 5:12 PM Michael Eager  wrote:
> > Thanks.  I'll give that a try.
> >
> > On 1/30/20 1:49 PM, Jack Craig wrote:
> > > wireshark -> tcpdump on dst=port# src = all
> > > ??
> > >
> > > On Thu, Jan 30, 2020 at 1:13 PM Michael Eager <[e-mail]
> > > > wrote:
> > > When I look at /var/log/secure or run journalctl on my workstation,
> >
> > I
> > > see failed SSH login attempts from a variety of IP addresses.  The
> > > attempts are every 3-12 minutes.
> > >
> > > /etc/ssh/sshd_config contains:
> > > PasswordAuthentication no
> > >
> > > The workstation is on a LAN behind an EdgeRouter firewall.  No
> >
> > Internet-
> > > accessible ports are forwarded to the workstation.  The LAN has a
> > > variety of servers, NAS boxes, WiFi access points, WiFi-connected
> > > laptops, etc.
> > >
> > > A typical /var/log/secure entry looks like this:
> > > Jan 30 12:43:50 redwood sshd[21228]: Invalid user jackiehulu from
> > > 124.204.36.138 port 37394
> > > Jan 30 12:43:51 redwood sshd[21228]: Received disconnect from
> > > 124.204.36.138 port 37394:11: Bye Bye [preauth]
> > > Jan 30 12:43:51 redwood sshd[21228]: Disconnected from invalid user
> > > jackiehulu 124.204.36.138 port 37394 [preauth]
> > >
> > > The corresponding journalctl is:
> > > Jan 30 12:43:51 redwood.eagercon.com <http://redwood.eagercon.com>
> > > audit[21228]: USER_ERR pid=21228
> > > uid=0 auid=4294967295 ses=4294967295
> > > subj=system_u:system_r:sshd_t:s0-s0:c0.c1023 msg='op=PAM:bad_ident
> > > grantors=? acct="?" exe="/usr/sbin/sshd" hostname=124.204.36.138
> > > addr=124.204.36.138 terminal=ssh res=failed'
> > >
> > > I'm assuming that something on the network has been compromised,
> > > allowing SSH login attempts on the LAN.  Other than turning off
> > > each server/AP/laptop/etc, one at a time, to find when the accesses
> > > stop, is there any way to find out where the SSH attempt is coming
> >
> > from?
> > > -- Mike Eager
> > > _______________________________________________
> > > users mailing list -- [e-mail]
> > > <mailto:[e-mail]>
> > > To unsubscribe send an email to [e-mail]
> > > <mailto:[e-mail]>
> > > Fedora Code of Conduct:
> > > https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> > > List Guidelines:
> >
> > https://fedoraproject.org/wiki/Mailing_list_guidelines
> > > List Archives:
> >
> > https://lists.fedoraproject.org/archives/list/[e-mail]
> > > users mailing list -- [e-mail]
> > > To unsubscribe send an email to [e-mail]
> > > Fedora Code of Conduct:
> >
> > https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> > > List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> > > List Archives:
> >
> > https://lists.fedoraproject.org/archives/list/[e-mail]
> > >
> >
> > --
> > Michael Eager    [e-mail]
> > 1960 Park Blvd., Palo Alto, CA 94306
> > _______________________________________________
> > users mailing list -- [e-mail]
> > To unsubscribe send an email to [e-mail]
> > Fedora Code of Conduct:
> > https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> > List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> > List Archives:
> > https://lists.fedoraproject.org/archives/list/[e-mail]

[Roger Heflin]

Echoing what samuel says.  If you have non-local ip address from lot
of different ranges, then port 22 from internet is being forwarded by
something to this server.

I have a port 22 forwarded to a machine, and it does get almost
continuous attempts (many an hour) trying various accounts.

#1: disable root from logins like you have.
#2: whatever account you could use to login should be something
abnormal (ie not mike, john, or a common name or software product), as
then the will be attempting against non-existent accounts and never
get in.
#3: whatever password you do have needs to be something good and
something not compromised from one of your other accounts, it would be
hard for them to even use a compromised password for you so long as
they don't have a way to know this computer is yours and it gets even
harder if the account in #2 is not something that you use on anyone
else website that could be compromised and linked to the username.
#4: install and configure fail2ban and it will block ip addresses
after a few attempts, and whitelist anything on your own subnet you
fully trust.

On Thu, Jan 30, 2020 at 3:23 PM Samuel Sieb  wrote:
> On 1/30/20 1:12 PM, Michael Eager wrote:
> > When I look at /var/log/secure or run journalctl on my workstation, I
> > see failed SSH login attempts from a variety of IP addresses.  The
> > attempts are every 3-12 minutes.
> >
> > The workstation is on a LAN behind an EdgeRouter firewall.  No Internet-
> > accessible ports are forwarded to the workstation.  The LAN has a
> > variety of servers, NAS boxes, WiFi access points, WiFi-connected
> > laptops, etc.
> >
> > A typical /var/log/secure entry looks like this:
> > Jan 30 12:43:50 redwood sshd[21228]: Invalid user jackiehulu from
> > 124.204.36.138 port 37394
> > Jan 30 12:43:51 redwood sshd[21228]: Received disconnect from
> > 124.204.36.138 port 37394:11: Bye Bye [preauth]
> > Jan 30 12:43:51 redwood sshd[21228]: Disconnected from invalid user
> > jackiehulu 124.204.36.138 port 37394 [preauth]
> >
> > I'm assuming that something on the network has been compromised,
> > allowing SSH login attempts on the LAN.  Other than turning off
> > each server/AP/laptop/etc, one at a time, to find when the accesses
> > stop, is there any way to find out where the SSH attempt is coming from?
>
> From the attacking IP address, unless you're in China, your computer
> must be internet accessible somehow.  That's not an IP address on your
> LAN, right?
> _______________________________________________
> users mailing list -- [e-mail]
> To unsubscribe send an email to [e-mail]
> Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives: https://lists.fedoraproject.org/archives/list/[e-mail]

[Ed Greshko]

On 2020-01-31 05:12, Michael Eager wrote:
> When I look at /var/log/secure or run journalctl on my workstation, I
> see failed SSH login attempts from a variety of IP addresses.  The
> attempts are every 3-12 minutes.
>
> /etc/ssh/sshd_config contains:
> PasswordAuthentication no
>
> The workstation is on a LAN behind an EdgeRouter firewall.  No Internet-
> accessible ports are forwarded to the workstation.  The LAN has a
> variety of servers, NAS boxes, WiFi access points, WiFi-connected
> laptops, etc.
>
> A typical /var/log/secure entry looks like this:
> Jan 30 12:43:50 redwood sshd[21228]: Invalid user jackiehulu from 124.204.36.138 port 37394
> Jan 30 12:43:51 redwood sshd[21228]: Received disconnect from 124.204.36.138 port 37394:11: Bye Bye [preauth]
> Jan 30 12:43:51 redwood sshd[21228]: Disconnected from invalid user jackiehulu 124.204.36.138 port 37394 [preauth]
>
> The corresponding journalctl is:
> Jan 30 12:43:51 redwood.eagercon.com audit[21228]: USER_ERR pid=21228 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:sshd_t:s0-s0:c0.c1023 msg='op=PAM:bad_ident grantors=? acct="?" exe="/usr/sbin/sshd" hostname=124.204.36.138 addr=124.204.36.138 terminal=ssh res=failed'
>
> I'm assuming that something on the network has been compromised,
> allowing SSH login attempts on the LAN.  Other than turning off
> each server/AP/laptop/etc, one at a time, to find when the accesses
> stop, is there any way to find out where the SSH attempt is coming from?

FWIW, I find the "lastb" command a bit more convenient to check for bad login attempts.

I also only allow public key authentication.

[Michael Eager]

Apparently, my original post was not as clear as I thought.

Password authentication on the workstation is disabled and port 22
is not forwarded by the firewall.

Fail2ban would not answer the question of where the SSH access is coming
from on the LAN.  If something on the LAN is forwarding SSH connections,
knowing that the IP is in China does not give me info about where this
is happening.

On 1/30/20 5:41 PM, Roger Heflin wrote:
> Echoing what samuel says.  If you have non-local ip address from lot
> of different ranges, then port 22 from internet is being forwarded by
> something to this server.
>
> I have a port 22 forwarded to a machine, and it does get almost
> continuous attempts (many an hour) trying various accounts.
>
> #1: disable root from logins like you have.
> #2: whatever account you could use to login should be something
> abnormal (ie not mike, john, or a common name or software product), as
> then the will be attempting against non-existent accounts and never
> get in.
> #3: whatever password you do have needs to be something good and
> something not compromised from one of your other accounts, it would be
> hard for them to even use a compromised password for you so long as
> they don't have a way to know this computer is yours and it gets even
> harder if the account in #2 is not something that you use on anyone
> else website that could be compromised and linked to the username.
> #4: install and configure fail2ban and it will block ip addresses
> after a few attempts, and whitelist anything on your own subnet you
> fully trust.
>
> On Thu, Jan 30, 2020 at 3:23 PM Samuel Sieb  wrote:
> > On 1/30/20 1:12 PM, Michael Eager wrote:
> > > When I look at /var/log/secure or run journalctl on my workstation, I
> > > see failed SSH login attempts from a variety of IP addresses.  The
> > > attempts are every 3-12 minutes.
> > >
> > > The workstation is on a LAN behind an EdgeRouter firewall.  No Internet-
> > > accessible ports are forwarded to the workstation.  The LAN has a
> > > variety of servers, NAS boxes, WiFi access points, WiFi-connected
> > > laptops, etc.
> > >
> > > A typical /var/log/secure entry looks like this:
> > > Jan 30 12:43:50 redwood sshd[21228]: Invalid user jackiehulu from
> > > 124.204.36.138 port 37394
> > > Jan 30 12:43:51 redwood sshd[21228]: Received disconnect from
> > > 124.204.36.138 port 37394:11: Bye Bye [preauth]
> > > Jan 30 12:43:51 redwood sshd[21228]: Disconnected from invalid user
> > > jackiehulu 124.204.36.138 port 37394 [preauth]
> > >
> > > I'm assuming that something on the network has been compromised,
> > > allowing SSH login attempts on the LAN.  Other than turning off
> > > each server/AP/laptop/etc, one at a time, to find when the accesses
> > > stop, is there any way to find out where the SSH attempt is coming from?
> >
> > From the attacking IP address, unless you're in China, your computer
> > must be internet accessible somehow.  That's not an IP address on your
> > LAN, right?
> > _______________________________________________
> > users mailing list -- [e-mail]
> > To unsubscribe send an email to [e-mail]
> > Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> > List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> > List Archives: https://lists.fedoraproject.org/archives/list/[e-mail]
>
> users mailing list -- [e-mail]
> To unsubscribe send an email to [e-mail]
> Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives: https://lists.fedoraproject.org/archives/list/[e-mail]

[Ed Greshko]

On 2020-01-31 22:37, Michael Eager wrote:
> Apparently, my original post was not as clear as I thought.
>
> Password authentication on the workstation is disabled and port 22
> is not forwarded by the firewall.
>
> Fail2ban would not answer the question of where the SSH access is coming
> from on the LAN.  If something on the LAN is forwarding SSH connections,
> knowing that the IP is in China does not give me info about where this
> is happening.

Well, if the connection is not coming via the firewall and its associated Internet Connection
then it only stands to reason that another device on your network is connected/connecting
to the Internet in a different manner and allowing traffic on the LAN.

However, if you run wireshark on the server which is getting the connection you will
know not only the source IP but the MAC address which is passing it.  That should tell
you the device on your network which is sending the connection request.

[George N. White III]

On Thu, 30 Jan 2020 at 17:13, Michael Eager  wrote:
> When I look at /var/log/secure or run journalctl on my workstation, I
> see failed SSH login attempts from a variety of IP addresses.  The
> attempts are every 3-12 minutes.
>
> /etc/ssh/sshd_config contains:
> PasswordAuthentication no
>
> The workstation is on a LAN behind an EdgeRouter firewall.  No Internet-
> accessible ports are forwarded to the workstation.  The LAN has a
> variety of servers, NAS boxes, WiFi access points, WiFi-connected
> laptops, etc.

Do you manage the network so you can check things like the firewall's
software
version and have access to a list of mac addresses for devices on your
network?   Is the firewall running the current software?    Does the router
do NAT
translation?   Are there other (external 3rd party) WiFi networks
visible to systems
on your LAN?   Are there BYOD's such as smart phones that could be
connected to the internet via cellular data networks?
> A typical /var/log/secure entry looks like this:
> Jan 30 12:43:50 redwood sshd[21228]: Invalid user jackiehulu from
> 124.204.36.138 port 37394
> Jan 30 12:43:51 redwood sshd[21228]: Received disconnect from
> 124.204.36.138 port 37394:11: Bye Bye [preauth]
> Jan 30 12:43:51 redwood sshd[21228]: Disconnected from invalid user
> jackiehulu 124.204.36.138 port 37394 [preauth]
>
> The corresponding journalctl is:
> Jan 30 12:43:51 redwood.eagercon.com audit[21228]: USER_ERR pid=21228
> uid=0 auid=4294967295 ses=4294967295
> subj=system_u:system_r:sshd_t:s0-s0:c0.c1023 msg='op=PAM:bad_ident
> grantors=? acct="?" exe="/usr/sbin/sshd" hostname=124.204.36.138
> addr=124.204.36.138 terminal=ssh res=failed'
>
> I'm assuming that something on the network has been compromised,
> allowing SSH login attempts on the LAN.  Other than turning off
> each server/AP/laptop/etc, one at a time, to find when the accesses
> stop, is there any way to find out where the SSH attempt is coming from?

A compromised system is unlikely to allow access to a large number of
external sites, so it is more likely to be a device (maybe inadvertently)
bridging a cellular data network or an external system connecting to your
LAN.   There have been cases where bad actors just plugged in a small
system to gain access to a LAN.

As Ed suggested, getting the mac address is a start, but it may be spoofed
or a device you didn't know existed: personal device, IoT device (HVAC,
security
camera, etc.), or computer used to control equipment that was not supposed
to be connected to the network.

[Samuel Sieb]

On 1/31/20 6:37 AM, Michael Eager wrote:
> Apparently, my original post was not as clear as I thought.
>
> Password authentication on the workstation is disabled and port 22
> is not forwarded by the firewall.
>
> Fail2ban would not answer the question of where the SSH access is coming
> from on the LAN.  If something on the LAN is forwarding SSH connections,
> knowing that the IP is in China does not give me info about where this
> is happening.

Your original post was completely clear.  However, something is 
happening on your network that you aren't aware of.  The fact that you 
are getting connections from an external IP address means that somehow 
there is a path from the external internet to this computer.  It's 
possible that another computer on your network could somehow be routing 
incoming packets to the computer, but the outgoing ones have to be 
following the default route to the default gateway.  An interesting 
split routing.  tcpdump (or wireshark) will give you the mac address and 
if it doesn't match your gateway, you will have to track down which 
computer has that mac.

[Ed Greshko]

On 2020-02-01 04:31, Samuel Sieb wrote:
> On 1/31/20 6:37 AM, Michael Eager wrote:
> > Apparently, my original post was not as clear as I thought.
> >
> > Password authentication on the workstation is disabled and port 22
> > is not forwarded by the firewall.
> >
> > Fail2ban would not answer the question of where the SSH access is coming
> > from on the LAN.  If something on the LAN is forwarding SSH connections,
> > knowing that the IP is in China does not give me info about where this
> > is happening.
>
> Your original post was completely clear.  However, something is happening on your network that you aren't aware of.  The fact that you are getting connections from an external IP address means that somehow there is a path from the external internet to this computer.  It's possible that another computer on your network could somehow be routing incoming packets to the computer, but the outgoing ones have to be following the default route to the default gateway.  An interesting split routing.  tcpdump (or wireshark) will give you the mac address and if it doesn't match your gateway, you will have to track down which computer has that mac.

And in that regard the "arp" command may be useful.  That is if one is aware of what IP addresses on
the LAN belong to what devices.

[Samuel Sieb]

On 1/31/20 12:35 PM, Ed Greshko wrote:
> On 2020-02-01 04:31, Samuel Sieb wrote:
> > Your original post was completely clear.  However, something is happening on your network that you aren't aware of.  The fact that you are getting connections from an external IP address means that somehow there is a path from the external internet to this computer.  It's possible that another computer on your network could somehow be routing incoming packets to the computer, but the outgoing ones have to be following the default route to the default gateway.  An interesting split routing.  tcpdump (or wireshark) will give you the mac address and if it doesn't match your gateway, you will have to track down which computer has that mac.
>
> And in that regard the "arp" command may be useful.  That is if one is aware of what IP addresses on
> the LAN belong to what devices.

I thought about that, but it's only useful for mapping back from the MAC 
address and that would only work if the computers are talking directly 
using local addresses.  Only the attacking computer would have an arp 
entry for the target computer.  If the target does not normally have any 
communication with the attacker, it won't have an entry for it.  If he 
has access to the gateway computer, then that would more likely have an 
arp entry for the attacker.

One more thing I just thought of, depending on the network structure, 
the incoming packets could also be coming through the default gateway 
which would be even more difficult to track down.  But without the MAC 
address, it's all just speculation.

[Ed Greshko]

On 2020-02-01 04:56, Samuel Sieb wrote:
> On 1/31/20 12:35 PM, Ed Greshko wrote:
> > On 2020-02-01 04:31, Samuel Sieb wrote:
> > > Your original post was completely clear.  However, something is happening on your network that you aren't aware of.  The fact that you are getting connections from an external IP address means that somehow there is a path from the external internet to this computer.  It's possible that another computer on your network could somehow be routing incoming packets to the computer, but the outgoing ones have to be following the default route to the default gateway.  An interesting split routing.  tcpdump (or wireshark) will give you the mac address and if it doesn't match your gateway, you will have to track down which computer has that mac.
> >
> > And in that regard the "arp" command may be useful.  That is if one is aware of what IP addresses on
> > the LAN belong to what devices.
>
> I thought about that, but it's only useful for mapping back from the MAC address and that would only work if the computers are talking directly using local addresses.  Only the attacking computer would have an arp entry for the target computer.  If the target does not normally have any communication with the attacker, it won't have an entry for it.  If he has access to the gateway computer, then that would more likely have an arp entry for the attacker.

Well since arp is only on the LAN and since LAN communication is arp based the tcpdump packets will
have the MAC address of the device on the local network from which the ssh packets were routed through.

Normally, that would be the MAC of the gateway/firewall (assuming they are the same).  But at least one
would know the previous hop.
> One more thing I just thought of, depending on the network structure, the incoming packets could also be coming through the default gateway which would be even more difficult to track down.  But without the MAC address, it's all just speculation.

And if the incoming packets have the MAC address of the default gw and the default gw is also the firewall
then that would indicate the firewall is not doing what the OP thinks it is doing.

[Roger Heflin]

Do you have anything defined as a DMZ node/ipaddress on the firewall?

On Fri, Jan 31, 2020 at 3:53 PM Ed Greshko  wrote:
> On 2020-02-01 04:56, Samuel Sieb wrote:
> > On 1/31/20 12:35 PM, Ed Greshko wrote:
> > > On 2020-02-01 04:31, Samuel Sieb wrote:
> > > > Your original post was completely clear.  However, something is happening on your network that you aren't aware of.  The fact that you are getting connections from an external IP address means that somehow there is a path from the external internet to this computer.  It's possible that another computer on your network could somehow be routing incoming packets to the computer, but the outgoing ones have to be following the default route to the default gateway.  An interesting split routing.  tcpdump (or wireshark) will give you the mac address and if it doesn't match your gateway, you will have to track down which computer has that mac.
> > >
> > > And in that regard the "arp" command may be useful.  That is if one is aware of what IP addresses on
> > > the LAN belong to what devices.
> >
> > I thought about that, but it's only useful for mapping back from the MAC address and that would only work if the computers are talking directly using local addresses.  Only the attacking computer would have an arp entry for the target computer.  If the target does not normally have any communication with the attacker, it won't have an entry for it.  If he has access to the gateway computer, then that would more likely have an arp entry for the attacker.
>
> Well since arp is only on the LAN and since LAN communication is arp based the tcpdump packets will
> have the MAC address of the device on the local network from which the ssh packets were routed through.
>
> Normally, that would be the MAC of the gateway/firewall (assuming they are the same).  But at least one
> would know the previous hop.
> > One more thing I just thought of, depending on the network structure, the incoming packets could also be coming through the default gateway which would be even more difficult to track down.  But without the MAC address, it's all just speculation.
>
> And if the incoming packets have the MAC address of the default gw and the default gw is also the firewall
> then that would indicate the firewall is not doing what the OP thinks it is doing.
>
> --
> The key to getting good answers is to ask good questions.
> _______________________________________________
> users mailing list -- [e-mail]
> To unsubscribe send an email to [e-mail]
> Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives: https://lists.fedoraproject.org/archives/list/[e-mail]

[Samuel Sieb]

On 1/31/20 1:52 PM, Ed Greshko wrote:
> On 2020-02-01 04:56, Samuel Sieb wrote:
> > I thought about that, but it's only useful for mapping back from the MAC address and that would only work if the computers are talking directly using local addresses.  Only the attacking computer would have an arp entry for the target computer.  If the target does not normally have any communication with the attacker, it won't have an entry for it.  If he has access to the gateway computer, then that would more likely have an arp entry for the attacker.
>
> Well since arp is only on the LAN and since LAN communication is arp based the tcpdump packets will
> have the MAC address of the device on the local network from which the ssh packets were routed through.

I'm not sure what you're saying.  Yes, the packets will have the MAC 
address of the sending device.  But the local arp table will most likely 
not have an entry for that MAC address.  So you will have to try to 
track down the device only by the MAC and not by IP.  The DHCP server 
would be a good place to look for that.

An ARP lookup is only done on sending, not receiving.  Since the 
incoming IP address is not local, there will be no ARP request made for 
the reply because it will be sending it to the default gateway.  (There 
might be an ARP request for the gateway if the entry is stale.)

[Ed Greshko]

On 2020-02-01 06:16, Samuel Sieb wrote:
> On 1/31/20 1:52 PM, Ed Greshko wrote:
> > On 2020-02-01 04:56, Samuel Sieb wrote:
> > > I thought about that, but it's only useful for mapping back from the MAC address and that would only work if the computers are talking directly using local addresses.  Only the attacking computer would have an arp entry for the target computer.  If the target does not normally have any communication with the attacker, it won't have an entry for it.  If he has access to the gateway computer, then that would more likely have an arp entry for the attacker.
> >
> > Well since arp is only on the LAN and since LAN communication is arp based the tcpdump packets will
> > have the MAC address of the device on the local network from which the ssh packets were routed through.
>
> I'm not sure what you're saying.  Yes, the packets will have the MAC address of the sending device.  But the local arp table will most likely not have an entry for that MAC address.  So you will have to try to track down the device only by the MAC and not by IP.  The DHCP server would be a good place to look for that.
>
> An ARP lookup is only done on sending, not receiving.  Since the incoming IP address is not local, there will be no ARP request made for the reply because it will be sending it to the default gateway.  (There might be an ARP request for the gateway if the entry is stale.)

I see what you're saying.  Thanks for pointing it out.

I suppose I'm use to smaller LAN segments where all systems are regularly communicating.

But, it would not hurt to check to see if that MAC address did happen to show up in arp table.  If it were the
same as the firewall/gateway (again assuming they are one and the same) then one would have to check
the firewall to see why it isn't doing what it is assumed to be doing.

[Ed Greshko]

On 2020-02-01 06:16, Samuel Sieb wrote:
> An ARP lookup is only done on sending, not receiving.

Humm....  That appears to be incorrect.

I have 3 systems on a LAN.

192.168.122.1     meimei   (also the gateway)
192.168.122.2     frk
192.168.122.152   f31k

I ssh into frk and f31k from meimei and I see....

[egreshko@frk ~]$ arp
Address                  HWtype  HWaddress           Flags Mask            Iface
_gateway                 ether   52:54:00:9a:e8:49   C                     enp1s0

[egreshko@f31k ~]$ arp
Address                  HWtype  HWaddress           Flags Mask            Iface
_gateway                 ether   52:54:00:9a:e8:49   C                     enp1s0

Then I attempt ssh from frk to f31k.  I say attempt since I have password
authentication disabled and f31k doesn't have the needed authorized_keys.

[egreshko@frk ~]$ arp
Address                  HWtype  HWaddress           Flags Mask            Iface
f31k                     ether   52:54:00:9b:21:c1   C                     enp1s0
_gateway                 ether   52:54:00:9a:e8:49   C                     enp1s0

[egreshko@f31k ~]$ arp
Address                  HWtype  HWaddress           Flags Mask            Iface
_gateway                 ether   52:54:00:9a:e8:49   C                     enp1s0
frk                      ether   52:54:00:f8:08:79   C                     enp1s0

[Samuel Sieb]

On 1/31/20 8:33 PM, Ed Greshko wrote:
> On 2020-02-01 06:16, Samuel Sieb wrote:
> > An ARP lookup is only done on sending, not receiving.
>
> Humm....  That appears to be incorrect.

[snip arp test]

You're missing an important piece.  When you make a tcp connection, the 
target computer has to send packets back, so needs to arp.  In the OP's 
case, the sending IP address is not on the local subnet, so to send a 
reply, the targeted computer has to arp the gateway to send a reply.  In 
your example, all the computers are on the same subnet.

[Ed Greshko]

On 2020-02-01 12:40, Samuel Sieb wrote:
> On 1/31/20 8:33 PM, Ed Greshko wrote:
> > On 2020-02-01 06:16, Samuel Sieb wrote:
> > > An ARP lookup is only done on sending, not receiving.
> >
> > Humm....  That appears to be incorrect.
>
> [snip arp test]
>
> You're missing an important piece.  When you make a tcp connection, the target computer has to send packets back, so needs to arp.  In the OP's case, the sending IP address is not on the local subnet, so to send a reply, the targeted computer has to arp the gateway to send a reply.  In your example, all the computers are on the same subnet.

Yes, but if the packets aren't coming via the firewall as the OP contends (and he hasn't revealed if the fw and gware one and the same) then it must be coming from a rogue system with an alternate internet connection.

If that rogue system is also on the same LAN then the targeted system needs to know the ARP address ofwhere to send the rejection packets.

It has been close to 15 years, but we had that situation at a company I worked at.  When the company was bought by British Telecom they installed their networking and firewall with restrictions that chaffed atone department.  One restriction being that the firewall would not allow incoming connections.  They wantedtheir remote workers to be able to telnet in.  VPN wasn't an option either.

But the folks in that department had enough weight that they were able to order a circuit
from Chungwa Telecom for their own use without BT's knowledge.  They "goofed" and packets from their connection found their way onto the BT side.

I'm pretty sure we tracked down what happened using arp to some degree.

[Ed Greshko]

On 2020-02-01 13:26, Ed Greshko wrote:
> I'm pretty sure we tracked down what happened using arp to some degree.

OK....  Maybe it wasn't that simple.....

I just found my emails from 15 years ago.  Glad I didn't delete them.  :-)

Turns out we saw the return/reject packets at the GW/FW which had Intrusion detection SW.
This is what raised alarm bells.

This tracked backto a system which they wanted to access but didn't have routing setup
correctly.  It didn't know where to send replies to the foreign IP so it sent it to the
default route.  That then lead usto the rogue system.

So, I was mistaken.

I enjoy being wrong twice in one day.

[Bill Shirley]

Assuming the subnet is 192.168.0.0/24:
nmap -sP 192.168.0.0/24
should populate the ARP table.

Bill

On 1/31/2020 5:16 PM, Samuel Sieb wrote:
> On 1/31/20 1:52 PM, Ed Greshko wrote:
> > On 2020-02-01 04:56, Samuel Sieb wrote:
> > > I thought about that, but it's only useful for mapping back from the MAC address and that would only work if the computers 
> > > are talking directly using local addresses.  Only the attacking computer would have an arp entry for the target computer.  
> > > If the target does not normally have any communication with the attacker, it won't have an entry for it.  If he has access 
> > > to the gateway computer, then that would more likely have an arp entry for the attacker.
> >
> > Well since arp is only on the LAN and since LAN communication is arp based the tcpdump packets will
> > have the MAC address of the device on the local network from which the ssh packets were routed through.
>
> I'm not sure what you're saying.  Yes, the packets will have the MAC address of the sending device.  But the local arp table 
> will most likely not have an entry for that MAC address.  So you will have to try to track down the device only by the MAC and 
> not by IP.  The DHCP server would be a good place to look for that.
>
> An ARP lookup is only done on sending, not receiving.  Since the incoming IP address is not local, there will be no ARP 
> request made for the reply because it will be sending it to the default gateway.  (There might be an ARP request for the 
> gateway if the entry is stale.)
> _______________________________________________
> users mailing list -- [e-mail]
> To unsubscribe send an email to [e-mail]
> Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives: https://lists.fedoraproject.org/archives/list/[e-mail]

[Tim]

On Thu, 2020-01-30 at 13:12 -0800, Michael Eager wrote:
> ... The LAN has a variety of servers, NAS boxes, WiFi access points,
> WiFi-connected laptops, etc.
>
> ...
>
> I'm assuming that something on the network has been compromised,
> allowing SSH login attempts on the LAN.  Other than turning off
> each server/AP/laptop/etc, one at a time, to find when the accesses
> stop, is there any way to find out where the SSH attempt is coming
> from?

Considering the timespan of this thread, disconnecting likely devices
might have been a quicker method.

Anything that offers cloud services (doing backups, remote access to
your NAS, etc), are the first things I'd look at.  All it takes is for
one of those remote services to be exploitable, like so many are.

e.g. WiFi surveillance cameras:  They often have cloud access, and
their cloud services are often easily compromised, and so are the
devices.  Their cameras come with predefined access codes, and someone
has worked out the pattern (you don't set up a random new account, you
use the code printed on a sticker to use an account already waiting for
you).  So they step through the permutations trying to connect.

[Samuel Sieb]

On 2/1/20 12:58 AM, Bill Shirley wrote:
> Assuming the subnet is 192.168.0.0/24:
> nmap -sP 192.168.0.0/24
> should populate the ARP table.

Have you tried that?  I was surprised by it because however nmap works, 
it does not create entries in the ARP table.  Maybe because it's 
creating raw packets and bypassing the usual system paths.

[Michael Eager]

On 1/30/20 1:12 PM, Michael Eager wrote:
> When I look at /var/log/secure or run journalctl on my workstation, I
> see failed SSH login attempts from a variety of IP addresses.  The
> attempts are every 3-12 minutes.
>
> /etc/ssh/sshd_config contains:
> PasswordAuthentication no
>
> The workstation is on a LAN behind an EdgeRouter firewall.  No Internet-
> accessible ports are forwarded to the workstation.  The LAN has a
> variety of servers, NAS boxes, WiFi access points, WiFi-connected
> laptops, etc.
>
> A typical /var/log/secure entry looks like this:
> Jan 30 12:43:50 redwood sshd[21228]: Invalid user jackiehulu from 
> 124.204.36.138 port 37394
> Jan 30 12:43:51 redwood sshd[21228]: Received disconnect from 
> 124.204.36.138 port 37394:11: Bye Bye [preauth]
> Jan 30 12:43:51 redwood sshd[21228]: Disconnected from invalid user 
> jackiehulu 124.204.36.138 port 37394 [preauth]
>
> The corresponding journalctl is:
> Jan 30 12:43:51 redwood.eagercon.com audit[21228]: USER_ERR pid=21228 
> uid=0 auid=4294967295 ses=4294967295 
> subj=system_u:system_r:sshd_t:s0-s0:c0.c1023 msg='op=PAM:bad_ident 
> grantors=? acct="?" exe="/usr/sbin/sshd" hostname=124.204.36.138 
> addr=124.204.36.138 terminal=ssh res=failed'
>
> I'm assuming that something on the network has been compromised,
> allowing SSH login attempts on the LAN.  Other than turning off
> each server/AP/laptop/etc, one at a time, to find when the accesses
> stop, is there any way to find out where the SSH attempt is coming from?
>
> -- Mike Eager

Thanks for all the suggestions.

I used tcpdump to collect access to port 22, after shutting off all
other SSH connections.  Running tcpdump with -e gave me the MAC
address of the sending device, and arp led me to the right device.

It was the firewall router.  It was forwarding a non-standard port to my
workstation's port 22.  I had thought that I had disabled all port
forwarding, but that apparently was not the case.  The access attempts
were infrequent because someone had to run a port scanner and find
the non-standard port, rather than just banging on port 22.

-- Mike Eager

[Ed Greshko]

On 2020-02-02 04:05, Michael Eager wrote:
> The access attempts
> were infrequent because someone had to run a port scanner and find
> the non-standard port, rather than just banging on port 22.

No need to run a port scanner when you pick port 2222.