Once again I find myself trying to help someone piece together how an intruder managed to get into their system. The system was way out of date (FC6) so it is no surprise that they got compromised. What I can tell, the intruder managed to get root which allowed them to remove the iptables file and lower the protection on ssh to allow unix passwords. The attacker then installed an ssh-probing client that was installed in /root. That lowered ssh security allowed a second intrusion at user level (probably by password guessing) where an IRC bot was installed and run from cron with normal user permissions.
I would have been nice to know when and how they initially got in. The site runs a handful of daemons (postix, named, ntp, apache, dovecot), so any of them could have allowed the initial intrution. They didn't have selinux enabled, so that compounded problems. Clearly the top level answer is to just impress upon them the fact that they need to stay current and keep selinux enabled. It still would be nice to know how the attackers got in though.
The real issue is that there isn't a good activity log. While I can install tripwire to watch for changed files, it probably won't tell me how they got in. Is there something that addresses that problem? Some poor sucker always has to be the first victim of a new attack. It would be nice to know which service to disable or reconfigure until a fix is distributed. Is there some way to track intruders that I'm missing?
-wolfgang
On 01/25/2011 04:34 PM, Wolfgang S. Rupprecht wrote:
Once again I find myself trying to help someone piece together how an intruder managed to get into their system. The system was way out of date (FC6) so it is no surprise that they got compromised. What I can tell, the intruder managed to get root which allowed them to remove the iptables file and lower the protection on ssh to allow unix passwords. The attacker then installed an ssh-probing client that was installed in /root. That lowered ssh security allowed a second intrusion at user level (probably by password guessing) where an IRC bot was installed and run from cron with normal user permissions.
I would have been nice to know when and how they initially got in. The site runs a handful of daemons (postix, named, ntp, apache, dovecot), so any of them could have allowed the initial intrution. They didn't have selinux enabled, so that compounded problems. Clearly the top level answer is to just impress upon them the fact that they need to stay current and keep selinux enabled. It still would be nice to know how the attackers got in though.
The real issue is that there isn't a good activity log. While I can install tripwire to watch for changed files, it probably won't tell me how they got in. Is there something that addresses that problem? Some poor sucker always has to be the first victim of a new attack. It would be nice to know which service to disable or reconfigure until a fix is distributed. Is there some way to track intruders that I'm missing?
-wolfgang
I like OSSEC. It's pretty good at detecting break in attempts and file system changes. At the very least, OSSEC would have said something as the intruder made changes that would disable it.
Joe Zeff joe@zeff.us writes:
On 01/25/2011 02:34 PM, Wolfgang S. Rupprecht wrote:
That lowered ssh security allowed a second intrusion at user level (probably by password guessing)
No need. Once they had root they could add a user and use that for their user-level work.
I understand. I believe they were two unrelated breakins by different people, the second breakin was enabled by the lax security caused by the first.
-wolfgang
On Tuesday 25 January 2011 22:34:16 Wolfgang S. Rupprecht wrote:
Once again I find myself trying to help someone piece together how an intruder managed to get into their system. The system was way out of date (FC6) so it is no surprise that they got compromised. What I can tell, the intruder managed to get root which allowed them to remove the iptables file and lower the protection on ssh to allow unix passwords. The attacker then installed an ssh-probing client that was installed in /root. That lowered ssh security allowed a second intrusion at user level (probably by password guessing) where an IRC bot was installed and run from cron with normal user permissions.
Shouldn't this be the other way around? I mean, ordinary user gets compromized first, and then root gets compromized later?
If the intruder has root access, he has absolutely no need to brute-force the user passwords through ssh. It is enough to change the password interactively or by modifying /etc/shadow. That is, unless the intruder is just plain stupid. ;-)
The real issue is that there isn't a good activity log. While I can install tripwire to watch for changed files, it probably won't tell me how they got in. Is there something that addresses that problem? Some poor sucker always has to be the first victim of a new attack. It would be nice to know which service to disable or reconfigure until a fix is distributed. Is there some way to track intruders that I'm missing?
The only safe way to track and analyze intrusion details of a live system is to have the machine log all activities to another machine on the net. That way the logs are physically append-only, and even after the intrusion happens, the intruder has no way of deleting the logs and otherwise covering up how the machine got compromized.
Other than that, once the intruder becomes root, all bets are off, there is no safe way to know anything about the intrusion and what exactly happened. The only thing you can do is wipe the hard disk and reinstall the system from scratch. Forensic research of a rooted system is (a) very painful and tough job (even for experts) and (b) almost impossible, in most cases.
If you are into intrusion detection research, you can set up a honeypot machine, make an exact cloned copy of the hard disk, log all activity to another server, monitor all network traffic with a transparent machine-in-the- middle, and then sit and wait for the machine to get hacked. Then take it off the net, do a diff of the entire hard disk against the initial copy, analyze logs and network traffic, etc. Those are the "laboratory conditions" in which you can do proper forensics.
Other than that, the only thing that can give you a trustworthy clue what happened is the remote log server, if you have one set up. If you don't, well, the only thing you can do is to keep guessing what happened... ;-)
HTH, :-) Marko
Marko Vojinovic vvmarko@gmail.com writes:
Shouldn't this be the other way around? I mean, ordinary user gets compromized first, and then root gets compromized later?
Oh, I'm sure there was an initial user-level attack that I haven't found yet and probably won't. Apache will all that dynamic stuff run from the web server scares me. I don't think the guys running the system appreciated how much of a security vulnarability that is.
If the intruder has root access, he has absolutely no need to brute-force the user passwords through ssh. It is enough to change the password interactively or by modifying /etc/shadow. That is, unless the intruder is just plain stupid. ;-)
It was most likely a double infection with the second intruder just taking advantage of the now lower security.
The only safe way to track and analyze intrusion details of a live system is to have the machine log all activities to another machine on the net. That way the logs are physically append-only, and even after the intrusion happens, the intruder has no way of deleting the logs and otherwise covering up how the machine got compromized.
Agreed.
I was just wondering if there is a package that already does that or something close.
Other than that, once the intruder becomes root, all bets are off, there is no safe way to know anything about the intrusion and what exactly happened. The only thing you can do is wipe the hard disk and reinstall the system from scratch. Forensic research of a rooted system is (a) very painful and tough job (even for experts) and (b) almost impossible, in most cases.
I'll certainy take a snapshot of the disk and pick at it as inspiration strikes. If it was a server attack (apache? dovecot?), there might be probes in log file. They might have been lazy and not deleted the log entries.
-wolfgang
On 01/26/2011 01:06 PM, Wolfgang S. Rupprecht wrote:
Oh, I'm sure there was an initial user-level attack that I haven't found yet and probably won't.
Check /etc/passwd for users you don't recognize.
grep -v nologin /etc/passwd
will give you a list of users who can log in. The few who aren't regular users, such as halt and shutdown will probably have obvious "shells." On my system, the only such "user" with /bin/bash is mysql. If one of the intruders did create a new account, it should jump out at you. And, of course, if you haven't changed the root password, do it now!
On 26.01.2011, Wolfgang S. Rupprecht wrote:
The real issue is that there isn't a good activity log. While I can install tripwire to watch for changed files
I would have used "aide" instead of tripwire.
it probably won't tell me how they got in. Is there something that addresses that problem?
No way. Once the attacker has become root, all your logs could be deleted and/or manipulated. You can't rely on them any longer.
On 01/26/2011 07:00 AM, Heinz Diehl wrote:
On 26.01.2011, Wolfgang S. Rupprecht wrote:
The real issue is that there isn't a good activity log. While I can install tripwire to watch for changed files
I would have used "aide" instead of tripwire.
it probably won't tell me how they got in. Is there something that addresses that problem?
No way. Once the attacker has become root, all your logs could be deleted and/or manipulated. You can't rely on them any longer.
There are patches to the bash, korn and c shells that log every command line entered to syslog. Get those and install them, then set up syslog to log to a remote logging server.
It ain't perfect, but we've nabbed a couple of baddies that way. ---------------------------------------------------------------------- - Rick Stevens, Systems Engineer, C2 Hosting ricks@nerd.com - - AIM/Skype: therps2 ICQ: 22643734 Yahoo: origrps2 - - - - I'm afraid my karma just ran over your dogma - ----------------------------------------------------------------------
-
Heinz Diehl -
Joe Zeff -
Marko Vojinovic -
Rick Stevens -
SternData -
Wolfgang S. Rupprecht