Hi.
Got an old FC system that was hacked. - I know - my own fault (perhaps) should have updated, etc,,,
The system is vital, need to extract all/as much of the files from the 700G drive as possible. I'm going to blow away the corrupt system, replacing the system with centos 6.5.
On the corrupted machine, when it was setup, the partitions where such that most of the data was written to the /apps partition - so hopefully most of what I really need will be there. However, I'm pretty sure that a chunk of other useful/critical stuff was placed on other dirs within the drive.
I'd like comments/suggestions on my approach to resolve the issue:
-Setup new machine with a couple of drive bays -take the corrupted drive, insert it in new machine's drive bay -insert clean 750G drive in the other drive bay -from the new machine, do a complete "find" on the corrupted drive to get a "complete" list of files/dirs/tree -go down the list, identifying the initial dirs/files that are important/data, that aren't part of the OS --copy these dirs/files to a tmp area on the clean drive, maintaining the dir structure -repeat this process untill I pretty much get the data files (txt/py/pl/php/etc..) --go through a complete process, trying to identify all the apps/functions that were added to the corrupt system. -identify these apps, as well as the rpms required to generate the functions -create a script to auto install these apps/functions from the associated centos/associated centos repos -handle all mysql stuff by doing a mysqldump from the good machine, reading the mysql data from the corrupted drive, and then copying reinserting the mysql data into the new mysql on the clean/tmp machine -identify any dev languages/environments (py/gearman/perl/php/etc..) and the required rpms to install or run to recreate the env on the clean/tmp machine
-identify all of the "services" running on the corrupt system/drive, and clean/install the rpms/services on the clean/tmp machine/drive -change all ssh keys for the new clean/tmp drive/machine.. -change all passwds on the new machine -for any web sites, change all passwds
-the goal is to recreate the file system/dirs/files from the corrupt machine/drive on the new clean/tmp machine as much as possible
-however, once I've gone through all of the above, I still need to know how to lock down services, how to harden the overall system..
so, the more comments that are on point the better.
thanks
See comments inline.
On 12/18/2013 12:49 PM, bruce wrote:
Hi.
Got an old FC system that was hacked. - I know - my own fault (perhaps) should have updated, etc,,,
How do you know you were hacked?
The system is vital, need to extract all/as much of the files from the 700G drive as possible. I'm going to blow away the corrupt system, replacing the system with centos 6.5.
No backups? I'd be /very/ careful pulling data off the old system. Depending on how recent the intrusion is, I'd almost go back to the most recent /safe/ backup and start from there.
On the corrupted machine, when it was setup, the partitions where such that most of the data was written to the /apps partition - so hopefully most of what I really need will be there. However, I'm pretty sure that a chunk of other useful/critical stuff was placed on other dirs within the drive.
I'd like comments/suggestions on my approach to resolve the issue:
-Setup new machine with a couple of drive bays
What I would do is setup the 'new' machine to boot from a flash drive (like a live CD of CentOS) in order to recover the data. This way you can at least be certain the installed OS on the new machine will never see or use the hacked drive. (See comment above about backups.)
-take the corrupted drive, insert it in new machine's drive bay -insert clean 750G drive in the other drive bay -from the new machine, do a complete "find" on the corrupted drive to get a "complete" list of files/dirs/tree
I don't know about anyone else, but I know what and where my critical data resides. (Not a helpful comment as much as an FYI for the future.)
-go down the list, identifying the initial dirs/files that are important/data, that aren't part of the OS --copy these dirs/files to a tmp area on the clean drive, maintaining the dir structure -repeat this process untill I pretty much get the data files (txt/py/pl/php/etc..) --go through a complete process, trying to identify all the apps/functions that were added to the corrupt system. -identify these apps, as well as the rpms required to generate the functions -create a script to auto install these apps/functions from the associated centos/associated centos repos -handle all mysql stuff by doing a mysqldump from the good machine,
IIRC, you'll need to attach those DBS to a mysql instance to do a dump, which means potentially exposing your new system. If that's the case, LiveCDs are the way to go. Be careful here though, MySQL versions should be the same or close to it or they won't play nice.
reading the mysql data from the corrupted drive, and then copying reinserting the mysql data into the new mysql on the clean/tmp machine -identify any dev languages/environments (py/gearman/perl/php/etc..) and the required rpms to install or run to recreate the env on the clean/tmp machine
-identify all of the "services" running on the corrupt system/drive, and clean/install the rpms/services on the clean/tmp machine/drive -change all ssh keys for the new clean/tmp drive/machine.. -change all passwds on the new machine -for any web sites, change all passwds
You don't know what services you run on the hacked system? If you don't, I'd probably start with the bare minimum of what I recalled I ran and go from there.
-the goal is to recreate the file system/dirs/files from the corrupt machine/drive on the new clean/tmp machine as much as possible
-however, once I've gone through all of the above, I still need to know how to lock down services, how to harden the overall system..
I'll be glad to help offlist if you like. I've rebuilt my share of hacked systems, so I might be able to offer some ideas.
Hi Mark,
Thanks for the reply. 'ppreciate it.
Basically, I'll be able to keep the corrupted system up/running, offline for a bit, so I'll be able to (more or less) really be able to see/recall what apps/functions are running, or should be running.
And I'll be keeping a copy of the drive in a drawer just in case I get that "oh Christ moment months from now!!)
A critical part of this whole process however, is that we're going to be creating a system of connected boxes/VMs that need to be tightly secured. IE, the systems will form a distributed network of boxes, each connecting back to the mother/master system via either a webservice process, or via ssh, so I'm going to be looking for a good approach to really securing the overall architecture.
You up for a more indepth conversation on this??!
Thanks
ps. are you in the us/canada?
-bruce
On Wed, Dec 18, 2013 at 1:03 PM, Mark Haney mhaney@practichem.com wrote:
See comments inline.
On 12/18/2013 12:49 PM, bruce wrote:
Hi.
Got an old FC system that was hacked. - I know - my own fault (perhaps) should have updated, etc,,,
How do you know you were hacked?
The system is vital, need to extract all/as much of the files from the 700G drive as possible. I'm going to blow away the corrupt system, replacing the system with centos 6.5.
No backups? I'd be /very/ careful pulling data off the old system. Depending on how recent the intrusion is, I'd almost go back to the most recent /safe/ backup and start from there.
On the corrupted machine, when it was setup, the partitions where such that most of the data was written to the /apps partition - so hopefully most of what I really need will be there. However, I'm pretty sure that a chunk of other useful/critical stuff was placed on other dirs within the drive.
I'd like comments/suggestions on my approach to resolve the issue:
-Setup new machine with a couple of drive bays
What I would do is setup the 'new' machine to boot from a flash drive (like a live CD of CentOS) in order to recover the data. This way you can at least be certain the installed OS on the new machine will never see or use the hacked drive. (See comment above about backups.)
-take the corrupted drive, insert it in new machine's drive bay -insert clean 750G drive in the other drive bay -from the new machine, do a complete "find" on the corrupted drive to get a "complete" list of files/dirs/tree
I don't know about anyone else, but I know what and where my critical data resides. (Not a helpful comment as much as an FYI for the future.)
-go down the list, identifying the initial dirs/files that are important/data, that aren't part of the OS --copy these dirs/files to a tmp area on the clean drive, maintaining the dir structure -repeat this process untill I pretty much get the data files (txt/py/pl/php/etc..) --go through a complete process, trying to identify all the apps/functions that were added to the corrupt system. -identify these apps, as well as the rpms required to generate the functions -create a script to auto install these apps/functions from the associated centos/associated centos repos -handle all mysql stuff by doing a mysqldump from the good machine,
IIRC, you'll need to attach those DBS to a mysql instance to do a dump, which means potentially exposing your new system. If that's the case, LiveCDs are the way to go. Be careful here though, MySQL versions should be the same or close to it or they won't play nice.
reading the mysql data from the corrupted drive, and then copying reinserting the mysql data into the new mysql on the clean/tmp machine -identify any dev languages/environments (py/gearman/perl/php/etc..) and the required rpms to install or run to recreate the env on the clean/tmp machine
-identify all of the "services" running on the corrupt system/drive, and clean/install the rpms/services on the clean/tmp machine/drive -change all ssh keys for the new clean/tmp drive/machine.. -change all passwds on the new machine -for any web sites, change all passwds
You don't know what services you run on the hacked system? If you don't, I'd probably start with the bare minimum of what I recalled I ran and go from there.
-the goal is to recreate the file system/dirs/files from the corrupt machine/drive on the new clean/tmp machine as much as possible
-however, once I've gone through all of the above, I still need to know how to lock down services, how to harden the overall system..
I'll be glad to help offlist if you like. I've rebuilt my share of hacked systems, so I might be able to offer some ideas.
-- Mark Haney Network Administrator/IT Support Practichem W:919-714-8428
-- users mailing list users@lists.fedoraproject.org To unsubscribe or change subscription options: https://admin.fedoraproject.org/mailman/listinfo/users Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct Guidelines: http://fedoraproject.org/wiki/Mailing_list_guidelines Have a question? Ask away: http://ask.fedoraproject.org
On Wed, 18 Dec 2013, bruce wrote:
Hi.
Got an old FC system that was hacked. - I know - my own fault (perhaps) should have updated, etc,,,
[snip]
I can't tell you how to restore because it's impossible to know what the intrusion was, what was done, and how you know what was or was not modified. All you are really telling me is that you have some corrupt files -- which, by the way, may not be the result of an intrusion. What is and is not now "corrupt" in the sense of being a trojan or something like that isn't something most folk can help you with unless there's some more information.
That being said, here's my idiot's guide to keeping a relatively clean box:
1) Backup on a regular basis. There are lots of philosophies about backing up, whether your should always do full backups, incremental backups, etc. It's both religious and a function of how much data you have. It may not be reasonable for Google to do full nightly backups, for instance, but it's certainly reasonable for *me* to do full backups periodically.
So, what I do is do a nightly incremental backup to a machine in a second location. In addition, I do a full backup every week and keep my last few backups as well as a couple of old ones. Disks are cheap nowadays.
Remember that if you are really the victim of an intrusion, unless you *know* what you are doing the chances are the intrusion happened well before the bad stuff started happening. Thus, if you have to retrieve stuff, you may want to go back until before you are pretty confident you had been hit.
2) Do a periodic clean install. One of the really nice things about Linux is that it's free and it's easy to install. That means that you can do a clean install on your machine quickly. One of the things I hate about Windows is that you are pretty much stuck with the same intstall, accumulating crap, for as long as you have that version of Windows. Doh. That's asking for trouble.
Going back to the idea of the cheap disks, I usually mirror my disk as soon as I install a new OS and get all the configs right. Then, every few weeks, I just mirror it back and restore the non-os data. That way if there's a rootkit on I didn't know about, it will be gone.
Similarly, every few months, I simply install the OS from scratch, usually bouncing between a Ubuntu-like OS such as Mint and a Red Hat like one such as Fedora/CentOS/Mageia.
Since it's all pretty mechanical, most of this is turn-it-on-and-go-to-bed kind of stuff, so it's not like I'm sitting around staring at the screen for hours.
3) Standard network security:
a) I like the old timey concepts of choke and bastion firewalls, with the "crown jewels" way back away from the server, per se. You may have to compromise on that if you have bandwidth/efficiency problems, but most folk do OK. Google on "choke bastion DMZ firewall tutorial"
b) If you have choke and bastion firewalls, you probably have a DMZ. Again, I'm a fan of having a separate box for each service -- an email server, a web server, etc., for the obvious reason that if one gets hit, the others may not. I break this rule for my home network, but I've always done it when I had a "real" network to run.
c) Don't serve anything you don't have to.
d) Don't listen at ports you don't have to.
e) Don't do stuff open text. Turn off telnet except when you need to use it to test stuff (and I find netcat works just as well most of the time anyway), etc.
4) Basic intrusion detection and avoidance
a) Read your logs.
b) Read your logs.
c) Read your logs.
d) Keep track of who should be on your machines when. It may be that a zillion people go to your web site, but only a few would go to your ftp server and only *you* should log in to your email server -- and you know when you do that.
e) Use brute force tripwires for ssh, ftp, etc that look for and stop script kiddies.
f) Everybody tells me to use tripwire, so I'll throw that in, but I have to say that I get so many false positives on it that I pretty much ignore it...
g) Iptables is your friend. If you know that you only want to talk to certain places, then just block everything else out. I don't know anybody in China or Taiwan or Korea. So screw it. I'll just use iptables to block everything except the US, Canada, and Europe. That won't stop the assholes in Russia, or the assholes in China that own boxes in France, but it helps. Yeah, yeah, firewalld is all the cat's meow, but I haven't learned it. I like iptables. Sue me.
h) Encrypt stuff as much as you can stand it. I don't encrypt drives, but I encrypt directories and files. You know that there are lots of files you only access every once in a blue moon. Encrypt them. What the hell. It's easy, and I've been surprised how much I didn't want certain emails made public that I also didn't need to re-read every freaking day.
billo
-
bruce -
Mark Haney -
William Oliver