When I look at /var/log/secure or run journalctl on my workstation, I see failed SSH login attempts from a variety of IP addresses. The attempts are every 3-12 minutes.
/etc/ssh/sshd_config contains: PasswordAuthentication no
The workstation is on a LAN behind an EdgeRouter firewall. No Internet- accessible ports are forwarded to the workstation. The LAN has a variety of servers, NAS boxes, WiFi access points, WiFi-connected laptops, etc.
A typical /var/log/secure entry looks like this: Jan 30 12:43:50 redwood sshd[21228]: Invalid user jackiehulu from 124.204.36.138 port 37394 Jan 30 12:43:51 redwood sshd[21228]: Received disconnect from 124.204.36.138 port 37394:11: Bye Bye [preauth] Jan 30 12:43:51 redwood sshd[21228]: Disconnected from invalid user jackiehulu 124.204.36.138 port 37394 [preauth]
The corresponding journalctl is: Jan 30 12:43:51 redwood.eagercon.com audit[21228]: USER_ERR pid=21228 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:sshd_t:s0-s0:c0.c1023 msg='op=PAM:bad_ident grantors=? acct="?" exe="/usr/sbin/sshd" hostname=124.204.36.138 addr=124.204.36.138 terminal=ssh res=failed'
I'm assuming that something on the network has been compromised, allowing SSH login attempts on the LAN. Other than turning off each server/AP/laptop/etc, one at a time, to find when the accesses stop, is there any way to find out where the SSH attempt is coming from?
-- Mike Eager
On 1/30/20 1:12 PM, Michael Eager wrote:
When I look at /var/log/secure or run journalctl on my workstation, I see failed SSH login attempts from a variety of IP addresses. The attempts are every 3-12 minutes.
The workstation is on a LAN behind an EdgeRouter firewall. No Internet- accessible ports are forwarded to the workstation. The LAN has a variety of servers, NAS boxes, WiFi access points, WiFi-connected laptops, etc.
A typical /var/log/secure entry looks like this: Jan 30 12:43:50 redwood sshd[21228]: Invalid user jackiehulu from 124.204.36.138 port 37394 Jan 30 12:43:51 redwood sshd[21228]: Received disconnect from 124.204.36.138 port 37394:11: Bye Bye [preauth] Jan 30 12:43:51 redwood sshd[21228]: Disconnected from invalid user jackiehulu 124.204.36.138 port 37394 [preauth]
I'm assuming that something on the network has been compromised, allowing SSH login attempts on the LAN. Other than turning off each server/AP/laptop/etc, one at a time, to find when the accesses stop, is there any way to find out where the SSH attempt is coming from?
From the attacking IP address, unless you're in China, your computer must be internet accessible somehow. That's not an IP address on your LAN, right?
wireshark -> tcpdump on dst=port# src = all ??
On Thu, Jan 30, 2020 at 1:13 PM Michael Eager eager@eagercon.com wrote:
When I look at /var/log/secure or run journalctl on my workstation, I see failed SSH login attempts from a variety of IP addresses. The attempts are every 3-12 minutes.
/etc/ssh/sshd_config contains: PasswordAuthentication no
The workstation is on a LAN behind an EdgeRouter firewall. No Internet- accessible ports are forwarded to the workstation. The LAN has a variety of servers, NAS boxes, WiFi access points, WiFi-connected laptops, etc.
A typical /var/log/secure entry looks like this: Jan 30 12:43:50 redwood sshd[21228]: Invalid user jackiehulu from 124.204.36.138 port 37394 Jan 30 12:43:51 redwood sshd[21228]: Received disconnect from 124.204.36.138 port 37394:11: Bye Bye [preauth] Jan 30 12:43:51 redwood sshd[21228]: Disconnected from invalid user jackiehulu 124.204.36.138 port 37394 [preauth]
The corresponding journalctl is: Jan 30 12:43:51 redwood.eagercon.com audit[21228]: USER_ERR pid=21228 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:sshd_t:s0-s0:c0.c1023 msg='op=PAM:bad_ident grantors=? acct="?" exe="/usr/sbin/sshd" hostname=124.204.36.138 addr=124.204.36.138 terminal=ssh res=failed'
I'm assuming that something on the network has been compromised, allowing SSH login attempts on the LAN. Other than turning off each server/AP/laptop/etc, one at a time, to find when the accesses stop, is there any way to find out where the SSH attempt is coming from?
-- Mike Eager _______________________________________________ users mailing list -- users@lists.fedoraproject.org To unsubscribe send an email to users-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/users@lists.fedoraproject.org
Thanks. I'll give that a try.
On 1/30/20 1:49 PM, Jack Craig wrote:
wireshark -> tcpdump on dst=port# src = all ??
On Thu, Jan 30, 2020 at 1:13 PM Michael Eager <eager@eagercon.com mailto:eager@eagercon.com> wrote:
When I look at /var/log/secure or run journalctl on my workstation, I see failed SSH login attempts from a variety of IP addresses. The attempts are every 3-12 minutes. /etc/ssh/sshd_config contains: PasswordAuthentication no The workstation is on a LAN behind an EdgeRouter firewall. No Internet- accessible ports are forwarded to the workstation. The LAN has a variety of servers, NAS boxes, WiFi access points, WiFi-connected laptops, etc. A typical /var/log/secure entry looks like this: Jan 30 12:43:50 redwood sshd[21228]: Invalid user jackiehulu from 124.204.36.138 port 37394 Jan 30 12:43:51 redwood sshd[21228]: Received disconnect from 124.204.36.138 port 37394:11: Bye Bye [preauth] Jan 30 12:43:51 redwood sshd[21228]: Disconnected from invalid user jackiehulu 124.204.36.138 port 37394 [preauth] The corresponding journalctl is: Jan 30 12:43:51 redwood.eagercon.com <http://redwood.eagercon.com> audit[21228]: USER_ERR pid=21228 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:sshd_t:s0-s0:c0.c1023 msg='op=PAM:bad_ident grantors=? acct="?" exe="/usr/sbin/sshd" hostname=124.204.36.138 addr=124.204.36.138 terminal=ssh res=failed' I'm assuming that something on the network has been compromised, allowing SSH login attempts on the LAN. Other than turning off each server/AP/laptop/etc, one at a time, to find when the accesses stop, is there any way to find out where the SSH attempt is coming from? -- Mike Eager _______________________________________________ users mailing list -- users@lists.fedoraproject.org <mailto:users@lists.fedoraproject.org> To unsubscribe send an email to users-leave@lists.fedoraproject.org <mailto:users-leave@lists.fedoraproject.org> Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/users@lists.fedoraproject.org
users mailing list -- users@lists.fedoraproject.org To unsubscribe send an email to users-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/users@lists.fedoraproject.org
with some work, you can limit the filter on capture to screen out all but the traffic you want to see.
the web should have lots of 'how to' clips.
good luck, ...
On Thu, Jan 30, 2020 at 5:12 PM Michael Eager eager@eagercon.com wrote:
Thanks. I'll give that a try.
On 1/30/20 1:49 PM, Jack Craig wrote:
wireshark -> tcpdump on dst=port# src = all ??
On Thu, Jan 30, 2020 at 1:13 PM Michael Eager <eager@eagercon.com mailto:eager@eagercon.com> wrote:
When I look at /var/log/secure or run journalctl on my workstation, I see failed SSH login attempts from a variety of IP addresses. The attempts are every 3-12 minutes. /etc/ssh/sshd_config contains: PasswordAuthentication no The workstation is on a LAN behind an EdgeRouter firewall. NoInternet-
accessible ports are forwarded to the workstation. The LAN has a variety of servers, NAS boxes, WiFi access points, WiFi-connected laptops, etc. A typical /var/log/secure entry looks like this: Jan 30 12:43:50 redwood sshd[21228]: Invalid user jackiehulu from 124.204.36.138 port 37394 Jan 30 12:43:51 redwood sshd[21228]: Received disconnect from 124.204.36.138 port 37394:11: Bye Bye [preauth] Jan 30 12:43:51 redwood sshd[21228]: Disconnected from invalid user jackiehulu 124.204.36.138 port 37394 [preauth] The corresponding journalctl is: Jan 30 12:43:51 redwood.eagercon.com <http://redwood.eagercon.com> audit[21228]: USER_ERR pid=21228 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:sshd_t:s0-s0:c0.c1023 msg='op=PAM:bad_ident grantors=? acct="?" exe="/usr/sbin/sshd" hostname=124.204.36.138 addr=124.204.36.138 terminal=ssh res=failed' I'm assuming that something on the network has been compromised, allowing SSH login attempts on the LAN. Other than turning off each server/AP/laptop/etc, one at a time, to find when the accesses stop, is there any way to find out where the SSH attempt is comingfrom?
-- Mike Eager _______________________________________________ users mailing list -- users@lists.fedoraproject.org <mailto:users@lists.fedoraproject.org> To unsubscribe send an email to users-leave@lists.fedoraproject.org <mailto:users-leave@lists.fedoraproject.org> Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines:https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives:https://lists.fedoraproject.org/archives/list/users@lists.fedoraproject.org
users mailing list -- users@lists.fedoraproject.org To unsubscribe send an email to users-leave@lists.fedoraproject.org Fedora Code of Conduct:
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives:
https://lists.fedoraproject.org/archives/list/users@lists.fedoraproject.org
-- Michael Eager eager@eagercon.com 1960 Park Blvd., Palo Alto, CA 94306 _______________________________________________ users mailing list -- users@lists.fedoraproject.org To unsubscribe send an email to users-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/users@lists.fedoraproject.org
one more bit, when you get to the command line ssh ... , throw in a bunch of -v to crank up verbosity
On Thu, Jan 30, 2020 at 5:18 PM Jack Craig jack.craig.aptos@gmail.com wrote:
with some work, you can limit the filter on capture to screen out all but the traffic you want to see.
the web should have lots of 'how to' clips.
good luck, ...
On Thu, Jan 30, 2020 at 5:12 PM Michael Eager eager@eagercon.com wrote:
Thanks. I'll give that a try.
On 1/30/20 1:49 PM, Jack Craig wrote:
wireshark -> tcpdump on dst=port# src = all ??
On Thu, Jan 30, 2020 at 1:13 PM Michael Eager <eager@eagercon.com mailto:eager@eagercon.com> wrote:
When I look at /var/log/secure or run journalctl on my workstation,I
see failed SSH login attempts from a variety of IP addresses. The attempts are every 3-12 minutes. /etc/ssh/sshd_config contains: PasswordAuthentication no The workstation is on a LAN behind an EdgeRouter firewall. NoInternet-
accessible ports are forwarded to the workstation. The LAN has a variety of servers, NAS boxes, WiFi access points, WiFi-connected laptops, etc. A typical /var/log/secure entry looks like this: Jan 30 12:43:50 redwood sshd[21228]: Invalid user jackiehulu from 124.204.36.138 port 37394 Jan 30 12:43:51 redwood sshd[21228]: Received disconnect from 124.204.36.138 port 37394:11: Bye Bye [preauth] Jan 30 12:43:51 redwood sshd[21228]: Disconnected from invalid user jackiehulu 124.204.36.138 port 37394 [preauth] The corresponding journalctl is: Jan 30 12:43:51 redwood.eagercon.com <http://redwood.eagercon.com> audit[21228]: USER_ERR pid=21228 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:sshd_t:s0-s0:c0.c1023 msg='op=PAM:bad_ident grantors=? acct="?" exe="/usr/sbin/sshd" hostname=124.204.36.138 addr=124.204.36.138 terminal=ssh res=failed' I'm assuming that something on the network has been compromised, allowing SSH login attempts on the LAN. Other than turning off each server/AP/laptop/etc, one at a time, to find when the accesses stop, is there any way to find out where the SSH attempt is comingfrom?
-- Mike Eager _______________________________________________ users mailing list -- users@lists.fedoraproject.org <mailto:users@lists.fedoraproject.org> To unsubscribe send an email to users-leave@lists.fedoraproject.org <mailto:users-leave@lists.fedoraproject.org> Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines:https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives:https://lists.fedoraproject.org/archives/list/users@lists.fedoraproject.org
users mailing list -- users@lists.fedoraproject.org To unsubscribe send an email to users-leave@lists.fedoraproject.org Fedora Code of Conduct:
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives:
https://lists.fedoraproject.org/archives/list/users@lists.fedoraproject.org
-- Michael Eager eager@eagercon.com 1960 Park Blvd., Palo Alto, CA 94306 _______________________________________________ users mailing list -- users@lists.fedoraproject.org To unsubscribe send an email to users-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/users@lists.fedoraproject.org
Echoing what samuel says. If you have non-local ip address from lot of different ranges, then port 22 from internet is being forwarded by something to this server.
I have a port 22 forwarded to a machine, and it does get almost continuous attempts (many an hour) trying various accounts.
#1: disable root from logins like you have. #2: whatever account you could use to login should be something abnormal (ie not mike, john, or a common name or software product), as then the will be attempting against non-existent accounts and never get in. #3: whatever password you do have needs to be something good and something not compromised from one of your other accounts, it would be hard for them to even use a compromised password for you so long as they don't have a way to know this computer is yours and it gets even harder if the account in #2 is not something that you use on anyone else website that could be compromised and linked to the username. #4: install and configure fail2ban and it will block ip addresses after a few attempts, and whitelist anything on your own subnet you fully trust.
On Thu, Jan 30, 2020 at 3:23 PM Samuel Sieb samuel@sieb.net wrote:
On 1/30/20 1:12 PM, Michael Eager wrote:
When I look at /var/log/secure or run journalctl on my workstation, I see failed SSH login attempts from a variety of IP addresses. The attempts are every 3-12 minutes.
The workstation is on a LAN behind an EdgeRouter firewall. No Internet- accessible ports are forwarded to the workstation. The LAN has a variety of servers, NAS boxes, WiFi access points, WiFi-connected laptops, etc.
A typical /var/log/secure entry looks like this: Jan 30 12:43:50 redwood sshd[21228]: Invalid user jackiehulu from 124.204.36.138 port 37394 Jan 30 12:43:51 redwood sshd[21228]: Received disconnect from 124.204.36.138 port 37394:11: Bye Bye [preauth] Jan 30 12:43:51 redwood sshd[21228]: Disconnected from invalid user jackiehulu 124.204.36.138 port 37394 [preauth]
I'm assuming that something on the network has been compromised, allowing SSH login attempts on the LAN. Other than turning off each server/AP/laptop/etc, one at a time, to find when the accesses stop, is there any way to find out where the SSH attempt is coming from?
From the attacking IP address, unless you're in China, your computer must be internet accessible somehow. That's not an IP address on your LAN, right? _______________________________________________ users mailing list -- users@lists.fedoraproject.org To unsubscribe send an email to users-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/users@lists.fedoraproject.org
On 2020-01-31 05:12, Michael Eager wrote:
When I look at /var/log/secure or run journalctl on my workstation, I see failed SSH login attempts from a variety of IP addresses. The attempts are every 3-12 minutes.
/etc/ssh/sshd_config contains: PasswordAuthentication no
The workstation is on a LAN behind an EdgeRouter firewall. No Internet- accessible ports are forwarded to the workstation. The LAN has a variety of servers, NAS boxes, WiFi access points, WiFi-connected laptops, etc.
A typical /var/log/secure entry looks like this: Jan 30 12:43:50 redwood sshd[21228]: Invalid user jackiehulu from 124.204.36.138 port 37394 Jan 30 12:43:51 redwood sshd[21228]: Received disconnect from 124.204.36.138 port 37394:11: Bye Bye [preauth] Jan 30 12:43:51 redwood sshd[21228]: Disconnected from invalid user jackiehulu 124.204.36.138 port 37394 [preauth]
The corresponding journalctl is: Jan 30 12:43:51 redwood.eagercon.com audit[21228]: USER_ERR pid=21228 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:sshd_t:s0-s0:c0.c1023 msg='op=PAM:bad_ident grantors=? acct="?" exe="/usr/sbin/sshd" hostname=124.204.36.138 addr=124.204.36.138 terminal=ssh res=failed'
I'm assuming that something on the network has been compromised, allowing SSH login attempts on the LAN. Other than turning off each server/AP/laptop/etc, one at a time, to find when the accesses stop, is there any way to find out where the SSH attempt is coming from?
FWIW, I find the "lastb" command a bit more convenient to check for bad login attempts.
I also only allow public key authentication.
Apparently, my original post was not as clear as I thought.
Password authentication on the workstation is disabled and port 22 is not forwarded by the firewall.
Fail2ban would not answer the question of where the SSH access is coming from on the LAN. If something on the LAN is forwarding SSH connections, knowing that the IP is in China does not give me info about where this is happening.
On 1/30/20 5:41 PM, Roger Heflin wrote:
Echoing what samuel says. If you have non-local ip address from lot of different ranges, then port 22 from internet is being forwarded by something to this server.
I have a port 22 forwarded to a machine, and it does get almost continuous attempts (many an hour) trying various accounts.
#1: disable root from logins like you have. #2: whatever account you could use to login should be something abnormal (ie not mike, john, or a common name or software product), as then the will be attempting against non-existent accounts and never get in. #3: whatever password you do have needs to be something good and something not compromised from one of your other accounts, it would be hard for them to even use a compromised password for you so long as they don't have a way to know this computer is yours and it gets even harder if the account in #2 is not something that you use on anyone else website that could be compromised and linked to the username. #4: install and configure fail2ban and it will block ip addresses after a few attempts, and whitelist anything on your own subnet you fully trust.
On Thu, Jan 30, 2020 at 3:23 PM Samuel Sieb samuel@sieb.net wrote:
On 1/30/20 1:12 PM, Michael Eager wrote:
When I look at /var/log/secure or run journalctl on my workstation, I see failed SSH login attempts from a variety of IP addresses. The attempts are every 3-12 minutes.
The workstation is on a LAN behind an EdgeRouter firewall. No Internet- accessible ports are forwarded to the workstation. The LAN has a variety of servers, NAS boxes, WiFi access points, WiFi-connected laptops, etc.
A typical /var/log/secure entry looks like this: Jan 30 12:43:50 redwood sshd[21228]: Invalid user jackiehulu from 124.204.36.138 port 37394 Jan 30 12:43:51 redwood sshd[21228]: Received disconnect from 124.204.36.138 port 37394:11: Bye Bye [preauth] Jan 30 12:43:51 redwood sshd[21228]: Disconnected from invalid user jackiehulu 124.204.36.138 port 37394 [preauth]
I'm assuming that something on the network has been compromised, allowing SSH login attempts on the LAN. Other than turning off each server/AP/laptop/etc, one at a time, to find when the accesses stop, is there any way to find out where the SSH attempt is coming from?
From the attacking IP address, unless you're in China, your computer must be internet accessible somehow. That's not an IP address on your LAN, right? _______________________________________________ users mailing list -- users@lists.fedoraproject.org To unsubscribe send an email to users-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/users@lists.fedoraproject.org
users mailing list -- users@lists.fedoraproject.org To unsubscribe send an email to users-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/users@lists.fedoraproject.org
On 2020-01-31 22:37, Michael Eager wrote:
Apparently, my original post was not as clear as I thought.
Password authentication on the workstation is disabled and port 22 is not forwarded by the firewall.
Fail2ban would not answer the question of where the SSH access is coming from on the LAN. If something on the LAN is forwarding SSH connections, knowing that the IP is in China does not give me info about where this is happening.
Well, if the connection is not coming via the firewall and its associated Internet Connection then it only stands to reason that another device on your network is connected/connecting to the Internet in a different manner and allowing traffic on the LAN.
However, if you run wireshark on the server which is getting the connection you will know not only the source IP but the MAC address which is passing it. That should tell you the device on your network which is sending the connection request.
On Thu, 30 Jan 2020 at 17:13, Michael Eager eager@eagercon.com wrote:
When I look at /var/log/secure or run journalctl on my workstation, I see failed SSH login attempts from a variety of IP addresses. The attempts are every 3-12 minutes.
/etc/ssh/sshd_config contains: PasswordAuthentication no
The workstation is on a LAN behind an EdgeRouter firewall. No Internet- accessible ports are forwarded to the workstation. The LAN has a variety of servers, NAS boxes, WiFi access points, WiFi-connected laptops, etc.
Do you manage the network so you can check things like the firewall's software version and have access to a list of mac addresses for devices on your network? Is the firewall running the current software? Does the router do NAT translation? Are there other (external 3rd party) WiFi networks visible to systems on your LAN? Are there BYOD's such as smart phones that could be connected to the internet via cellular data networks?
A typical /var/log/secure entry looks like this: Jan 30 12:43:50 redwood sshd[21228]: Invalid user jackiehulu from 124.204.36.138 port 37394 Jan 30 12:43:51 redwood sshd[21228]: Received disconnect from 124.204.36.138 port 37394:11: Bye Bye [preauth] Jan 30 12:43:51 redwood sshd[21228]: Disconnected from invalid user jackiehulu 124.204.36.138 port 37394 [preauth]
The corresponding journalctl is: Jan 30 12:43:51 redwood.eagercon.com audit[21228]: USER_ERR pid=21228 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:sshd_t:s0-s0:c0.c1023 msg='op=PAM:bad_ident grantors=? acct="?" exe="/usr/sbin/sshd" hostname=124.204.36.138 addr=124.204.36.138 terminal=ssh res=failed'
I'm assuming that something on the network has been compromised, allowing SSH login attempts on the LAN. Other than turning off each server/AP/laptop/etc, one at a time, to find when the accesses stop, is there any way to find out where the SSH attempt is coming from?
A compromised system is unlikely to allow access to a large number of external sites, so it is more likely to be a device (maybe inadvertently) bridging a cellular data network or an external system connecting to your LAN. There have been cases where bad actors just plugged in a small system to gain access to a LAN.
As Ed suggested, getting the mac address is a start, but it may be spoofed or a device you didn't know existed: personal device, IoT device (HVAC, security camera, etc.), or computer used to control equipment that was not supposed to be connected to the network.
On 1/31/20 6:37 AM, Michael Eager wrote:
Apparently, my original post was not as clear as I thought.
Password authentication on the workstation is disabled and port 22 is not forwarded by the firewall.
Fail2ban would not answer the question of where the SSH access is coming from on the LAN. If something on the LAN is forwarding SSH connections, knowing that the IP is in China does not give me info about where this is happening.
Your original post was completely clear. However, something is happening on your network that you aren't aware of. The fact that you are getting connections from an external IP address means that somehow there is a path from the external internet to this computer. It's possible that another computer on your network could somehow be routing incoming packets to the computer, but the outgoing ones have to be following the default route to the default gateway. An interesting split routing. tcpdump (or wireshark) will give you the mac address and if it doesn't match your gateway, you will have to track down which computer has that mac.
On 2020-02-01 04:31, Samuel Sieb wrote:
On 1/31/20 6:37 AM, Michael Eager wrote:
Apparently, my original post was not as clear as I thought.
Password authentication on the workstation is disabled and port 22 is not forwarded by the firewall.
Fail2ban would not answer the question of where the SSH access is coming from on the LAN. If something on the LAN is forwarding SSH connections, knowing that the IP is in China does not give me info about where this is happening.
Your original post was completely clear. However, something is happening on your network that you aren't aware of. The fact that you are getting connections from an external IP address means that somehow there is a path from the external internet to this computer. It's possible that another computer on your network could somehow be routing incoming packets to the computer, but the outgoing ones have to be following the default route to the default gateway. An interesting split routing. tcpdump (or wireshark) will give you the mac address and if it doesn't match your gateway, you will have to track down which computer has that mac.
And in that regard the "arp" command may be useful. That is if one is aware of what IP addresses on the LAN belong to what devices.
On 1/31/20 12:35 PM, Ed Greshko wrote:
On 2020-02-01 04:31, Samuel Sieb wrote:
Your original post was completely clear. However, something is happening on your network that you aren't aware of. The fact that you are getting connections from an external IP address means that somehow there is a path from the external internet to this computer. It's possible that another computer on your network could somehow be routing incoming packets to the computer, but the outgoing ones have to be following the default route to the default gateway. An interesting split routing. tcpdump (or wireshark) will give you the mac address and if it doesn't match your gateway, you will have to track down which computer has that mac.
And in that regard the "arp" command may be useful. That is if one is aware of what IP addresses on the LAN belong to what devices.
I thought about that, but it's only useful for mapping back from the MAC address and that would only work if the computers are talking directly using local addresses. Only the attacking computer would have an arp entry for the target computer. If the target does not normally have any communication with the attacker, it won't have an entry for it. If he has access to the gateway computer, then that would more likely have an arp entry for the attacker.
One more thing I just thought of, depending on the network structure, the incoming packets could also be coming through the default gateway which would be even more difficult to track down. But without the MAC address, it's all just speculation.
On 2020-02-01 04:56, Samuel Sieb wrote:
On 1/31/20 12:35 PM, Ed Greshko wrote:
On 2020-02-01 04:31, Samuel Sieb wrote:
Your original post was completely clear. However, something is happening on your network that you aren't aware of. The fact that you are getting connections from an external IP address means that somehow there is a path from the external internet to this computer. It's possible that another computer on your network could somehow be routing incoming packets to the computer, but the outgoing ones have to be following the default route to the default gateway. An interesting split routing. tcpdump (or wireshark) will give you the mac address and if it doesn't match your gateway, you will have to track down which computer has that mac.
And in that regard the "arp" command may be useful. That is if one is aware of what IP addresses on the LAN belong to what devices.
I thought about that, but it's only useful for mapping back from the MAC address and that would only work if the computers are talking directly using local addresses. Only the attacking computer would have an arp entry for the target computer. If the target does not normally have any communication with the attacker, it won't have an entry for it. If he has access to the gateway computer, then that would more likely have an arp entry for the attacker.
Well since arp is only on the LAN and since LAN communication is arp based the tcpdump packets will have the MAC address of the device on the local network from which the ssh packets were routed through.
Normally, that would be the MAC of the gateway/firewall (assuming they are the same). But at least one would know the previous hop.
One more thing I just thought of, depending on the network structure, the incoming packets could also be coming through the default gateway which would be even more difficult to track down. But without the MAC address, it's all just speculation.
And if the incoming packets have the MAC address of the default gw and the default gw is also the firewall then that would indicate the firewall is not doing what the OP thinks it is doing.
Do you have anything defined as a DMZ node/ipaddress on the firewall?
On Fri, Jan 31, 2020 at 3:53 PM Ed Greshko ed.greshko@greshko.com wrote:
On 2020-02-01 04:56, Samuel Sieb wrote:
On 1/31/20 12:35 PM, Ed Greshko wrote:
On 2020-02-01 04:31, Samuel Sieb wrote:
Your original post was completely clear. However, something is happening on your network that you aren't aware of. The fact that you are getting connections from an external IP address means that somehow there is a path from the external internet to this computer. It's possible that another computer on your network could somehow be routing incoming packets to the computer, but the outgoing ones have to be following the default route to the default gateway. An interesting split routing. tcpdump (or wireshark) will give you the mac address and if it doesn't match your gateway, you will have to track down which computer has that mac.
And in that regard the "arp" command may be useful. That is if one is aware of what IP addresses on the LAN belong to what devices.
I thought about that, but it's only useful for mapping back from the MAC address and that would only work if the computers are talking directly using local addresses. Only the attacking computer would have an arp entry for the target computer. If the target does not normally have any communication with the attacker, it won't have an entry for it. If he has access to the gateway computer, then that would more likely have an arp entry for the attacker.
Well since arp is only on the LAN and since LAN communication is arp based the tcpdump packets will have the MAC address of the device on the local network from which the ssh packets were routed through.
Normally, that would be the MAC of the gateway/firewall (assuming they are the same). But at least one would know the previous hop.
One more thing I just thought of, depending on the network structure, the incoming packets could also be coming through the default gateway which would be even more difficult to track down. But without the MAC address, it's all just speculation.
And if the incoming packets have the MAC address of the default gw and the default gw is also the firewall then that would indicate the firewall is not doing what the OP thinks it is doing.
-- The key to getting good answers is to ask good questions. _______________________________________________ users mailing list -- users@lists.fedoraproject.org To unsubscribe send an email to users-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/users@lists.fedoraproject.org
On 1/31/20 1:52 PM, Ed Greshko wrote:
On 2020-02-01 04:56, Samuel Sieb wrote:
I thought about that, but it's only useful for mapping back from the MAC address and that would only work if the computers are talking directly using local addresses. Only the attacking computer would have an arp entry for the target computer. If the target does not normally have any communication with the attacker, it won't have an entry for it. If he has access to the gateway computer, then that would more likely have an arp entry for the attacker.
Well since arp is only on the LAN and since LAN communication is arp based the tcpdump packets will have the MAC address of the device on the local network from which the ssh packets were routed through.
I'm not sure what you're saying. Yes, the packets will have the MAC address of the sending device. But the local arp table will most likely not have an entry for that MAC address. So you will have to try to track down the device only by the MAC and not by IP. The DHCP server would be a good place to look for that.
An ARP lookup is only done on sending, not receiving. Since the incoming IP address is not local, there will be no ARP request made for the reply because it will be sending it to the default gateway. (There might be an ARP request for the gateway if the entry is stale.)
On 2020-02-01 06:16, Samuel Sieb wrote:
On 1/31/20 1:52 PM, Ed Greshko wrote:
On 2020-02-01 04:56, Samuel Sieb wrote:
I thought about that, but it's only useful for mapping back from the MAC address and that would only work if the computers are talking directly using local addresses. Only the attacking computer would have an arp entry for the target computer. If the target does not normally have any communication with the attacker, it won't have an entry for it. If he has access to the gateway computer, then that would more likely have an arp entry for the attacker.
Well since arp is only on the LAN and since LAN communication is arp based the tcpdump packets will have the MAC address of the device on the local network from which the ssh packets were routed through.
I'm not sure what you're saying. Yes, the packets will have the MAC address of the sending device. But the local arp table will most likely not have an entry for that MAC address. So you will have to try to track down the device only by the MAC and not by IP. The DHCP server would be a good place to look for that.
An ARP lookup is only done on sending, not receiving. Since the incoming IP address is not local, there will be no ARP request made for the reply because it will be sending it to the default gateway. (There might be an ARP request for the gateway if the entry is stale.)
I see what you're saying. Thanks for pointing it out.
I suppose I'm use to smaller LAN segments where all systems are regularly communicating.
But, it would not hurt to check to see if that MAC address did happen to show up in arp table. If it were the same as the firewall/gateway (again assuming they are one and the same) then one would have to check the firewall to see why it isn't doing what it is assumed to be doing.
On 2020-02-01 06:16, Samuel Sieb wrote:
An ARP lookup is only done on sending, not receiving.
Humm.... That appears to be incorrect.
I have 3 systems on a LAN.
192.168.122.1 meimei (also the gateway) 192.168.122.2 frk 192.168.122.152 f31k
I ssh into frk and f31k from meimei and I see....
[egreshko@frk ~]$ arp Address HWtype HWaddress Flags Mask Iface _gateway ether 52:54:00:9a:e8:49 C enp1s0
[egreshko@f31k ~]$ arp Address HWtype HWaddress Flags Mask Iface _gateway ether 52:54:00:9a:e8:49 C enp1s0
Then I attempt ssh from frk to f31k. I say attempt since I have password authentication disabled and f31k doesn't have the needed authorized_keys.
[egreshko@frk ~]$ arp Address HWtype HWaddress Flags Mask Iface f31k ether 52:54:00:9b:21:c1 C enp1s0 _gateway ether 52:54:00:9a:e8:49 C enp1s0
[egreshko@f31k ~]$ arp Address HWtype HWaddress Flags Mask Iface _gateway ether 52:54:00:9a:e8:49 C enp1s0 frk ether 52:54:00:f8:08:79 C enp1s0
On 1/31/20 8:33 PM, Ed Greshko wrote:
On 2020-02-01 06:16, Samuel Sieb wrote:
An ARP lookup is only done on sending, not receiving.
Humm.... That appears to be incorrect.
[snip arp test]
You're missing an important piece. When you make a tcp connection, the target computer has to send packets back, so needs to arp. In the OP's case, the sending IP address is not on the local subnet, so to send a reply, the targeted computer has to arp the gateway to send a reply. In your example, all the computers are on the same subnet.
On 2020-02-01 12:40, Samuel Sieb wrote:
On 1/31/20 8:33 PM, Ed Greshko wrote:
On 2020-02-01 06:16, Samuel Sieb wrote:
An ARP lookup is only done on sending, not receiving.
Humm.... That appears to be incorrect.
[snip arp test]
You're missing an important piece. When you make a tcp connection, the target computer has to send packets back, so needs to arp. In the OP's case, the sending IP address is not on the local subnet, so to send a reply, the targeted computer has to arp the gateway to send a reply. In your example, all the computers are on the same subnet.
Yes, but if the packets aren't coming via the firewall as the OP contends (and he hasn't revealed if the fw and gware one and the same) then it must be coming from a rogue system with an alternate internet connection.
If that rogue system is also on the same LAN then the targeted system needs to know the ARP address ofwhere to send the rejection packets.
It has been close to 15 years, but we had that situation at a company I worked at. When the company was bought by British Telecom they installed their networking and firewall with restrictions that chaffed atone department. One restriction being that the firewall would not allow incoming connections. They wantedtheir remote workers to be able to telnet in. VPN wasn't an option either.
But the folks in that department had enough weight that they were able to order a circuit from Chungwa Telecom for their own use without BT's knowledge. They "goofed" and packets from their connection found their way onto the BT side.
I'm pretty sure we tracked down what happened using arp to some degree.
On 2020-02-01 13:26, Ed Greshko wrote:
I'm pretty sure we tracked down what happened using arp to some degree.
OK.... Maybe it wasn't that simple.....
I just found my emails from 15 years ago. Glad I didn't delete them. :-)
Turns out we saw the return/reject packets at the GW/FW which had Intrusion detection SW. This is what raised alarm bells.
This tracked backto a system which they wanted to access but didn't have routing setup correctly. It didn't know where to send replies to the foreign IP so it sent it to the default route. That then lead usto the rogue system.
So, I was mistaken.
I enjoy being wrong twice in one day.
Assuming the subnet is 192.168.0.0/24: nmap -sP 192.168.0.0/24 should populate the ARP table.
Bill
On 1/31/2020 5:16 PM, Samuel Sieb wrote:
On 1/31/20 1:52 PM, Ed Greshko wrote:
On 2020-02-01 04:56, Samuel Sieb wrote:
I thought about that, but it's only useful for mapping back from the MAC address and that would only work if the computers are talking directly using local addresses. Only the attacking computer would have an arp entry for the target computer. If the target does not normally have any communication with the attacker, it won't have an entry for it. If he has access to the gateway computer, then that would more likely have an arp entry for the attacker.
Well since arp is only on the LAN and since LAN communication is arp based the tcpdump packets will have the MAC address of the device on the local network from which the ssh packets were routed through.
I'm not sure what you're saying. Yes, the packets will have the MAC address of the sending device. But the local arp table will most likely not have an entry for that MAC address. So you will have to try to track down the device only by the MAC and not by IP. The DHCP server would be a good place to look for that.
An ARP lookup is only done on sending, not receiving. Since the incoming IP address is not local, there will be no ARP request made for the reply because it will be sending it to the default gateway. (There might be an ARP request for the gateway if the entry is stale.) _______________________________________________ users mailing list -- users@lists.fedoraproject.org To unsubscribe send an email to users-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/users@lists.fedoraproject.org
On Thu, 2020-01-30 at 13:12 -0800, Michael Eager wrote:
... The LAN has a variety of servers, NAS boxes, WiFi access points, WiFi-connected laptops, etc.
...
I'm assuming that something on the network has been compromised, allowing SSH login attempts on the LAN. Other than turning off each server/AP/laptop/etc, one at a time, to find when the accesses stop, is there any way to find out where the SSH attempt is coming from?
Considering the timespan of this thread, disconnecting likely devices might have been a quicker method.
Anything that offers cloud services (doing backups, remote access to your NAS, etc), are the first things I'd look at. All it takes is for one of those remote services to be exploitable, like so many are.
e.g. WiFi surveillance cameras: They often have cloud access, and their cloud services are often easily compromised, and so are the devices. Their cameras come with predefined access codes, and someone has worked out the pattern (you don't set up a random new account, you use the code printed on a sticker to use an account already waiting for you). So they step through the permutations trying to connect.
On 2/1/20 12:58 AM, Bill Shirley wrote:
Assuming the subnet is 192.168.0.0/24: nmap -sP 192.168.0.0/24 should populate the ARP table.
Have you tried that? I was surprised by it because however nmap works, it does not create entries in the ARP table. Maybe because it's creating raw packets and bypassing the usual system paths.
On 1/30/20 1:12 PM, Michael Eager wrote:
When I look at /var/log/secure or run journalctl on my workstation, I see failed SSH login attempts from a variety of IP addresses. The attempts are every 3-12 minutes.
/etc/ssh/sshd_config contains: PasswordAuthentication no
The workstation is on a LAN behind an EdgeRouter firewall. No Internet- accessible ports are forwarded to the workstation. The LAN has a variety of servers, NAS boxes, WiFi access points, WiFi-connected laptops, etc.
A typical /var/log/secure entry looks like this: Jan 30 12:43:50 redwood sshd[21228]: Invalid user jackiehulu from 124.204.36.138 port 37394 Jan 30 12:43:51 redwood sshd[21228]: Received disconnect from 124.204.36.138 port 37394:11: Bye Bye [preauth] Jan 30 12:43:51 redwood sshd[21228]: Disconnected from invalid user jackiehulu 124.204.36.138 port 37394 [preauth]
The corresponding journalctl is: Jan 30 12:43:51 redwood.eagercon.com audit[21228]: USER_ERR pid=21228 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:sshd_t:s0-s0:c0.c1023 msg='op=PAM:bad_ident grantors=? acct="?" exe="/usr/sbin/sshd" hostname=124.204.36.138 addr=124.204.36.138 terminal=ssh res=failed'
I'm assuming that something on the network has been compromised, allowing SSH login attempts on the LAN. Other than turning off each server/AP/laptop/etc, one at a time, to find when the accesses stop, is there any way to find out where the SSH attempt is coming from?
-- Mike Eager
Thanks for all the suggestions.
I used tcpdump to collect access to port 22, after shutting off all other SSH connections. Running tcpdump with -e gave me the MAC address of the sending device, and arp led me to the right device.
It was the firewall router. It was forwarding a non-standard port to my workstation's port 22. I had thought that I had disabled all port forwarding, but that apparently was not the case. The access attempts were infrequent because someone had to run a port scanner and find the non-standard port, rather than just banging on port 22.
-- Mike Eager
On 2020-02-02 04:05, Michael Eager wrote:
The access attempts were infrequent because someone had to run a port scanner and find the non-standard port, rather than just banging on port 22.
No need to run a port scanner when you pick port 2222.