Notes-to-self for a potential new guide on Disaster Recovery

[Roger Baran, Owner]

Hi all!
I have been working on putting together some 'notes-to-self' for a
potential client to assist me in giving them a presentation on Disaster
Recovery.

In the process, I thought that what I have is a good start for a
potential Disaster Recovery guide for Fedora.

If you have the time, and/or inclination, please provide feedback on the
attached file, so that I can take that into consideration when it comes
time to actually put the guide together -- if that is something Fedora
Docs could use.

Thank you all very much in advance!

-- 
Warm Regards,
Roger Baran, Owner
Rapid Recovery IT Services
San Antonio, TX


-------------- next part --------------
Tools and Processes of the Trade:
### At this point, this file contains 'Notes to Self' and should not be used unless all of the device/file locations are adapted to your own situation. It has been written with the idea of *someday* adapting it for others to use.

dd - the old standby
ddrescue - to copy the bad drive with the greatest chance of success
Use both dd and ddrescue on read only mounts --- In other words, from a booted USB or by unmounting and remounting as read only, as required.
Obviously, active system partitions cannot be used, so if you cannot move the drive to another system (for whatever reason) booting from a USB/CD/DVD will be required.

Any Live distro with ddrescue, lzip, foremost, your favorite AV package, and autopsy (if desired) is all that is required.
----------------------
Most distros automount insertable media as rw (except CD/DVD's).
PLEASE SEE END NOTES FOR FORENSIC MOUNTING GUIDELINES AND "GOTCHA'S" if that is a concern:
Example of mounting a single partition as read only:

Mounting the desired recovery partition

$ sudo mkdir /mnt/recovery
$ sudo mount /dev/sda4 -r /mnt/recovery

You can just as easily mount an entire drive in the same way:

$ sudo mkdir /mnt/recovery
$ sudo mount /dev/sda -r /mnt/recovery

After mounting the drive, you can copy it using dd or ddrescue.
The resulting file will be a 'raw'* image file of the partition or full disk
* -- meaning that it is a disk-based copy [sector-by-sector] and not a file-based [file-by-file] copy
-----------------------
Using dd to copy the mounted disk:

Go to root, then...
~# fdisk -l /dev/sdX?
To know what the block size is of the device/partition you are going to use dd on.
~# exit

~$ mkdir recovery
~$ dd if=/dev/sdX? of=/home/pentest/recovery/backup.img conv=noerror,sync {bs=XXX <-- put in the correct block size here, if other than 512 (default).}

NOTE:
{If the output file is going to be *huge* you can specify an external drive and adopt the following commands accordingly for the correct file locations.  I.E. dd if=/dev/sdX? of=/dev/sdx?/backup.img conv=noerror,sync bs=XXX}
----------
For best possible results, you should have an external drive that has been prepped appropriately for data recovery:
The drive should be mounted rw
The drive should be wiped
The drive size should be approx. double the size (3x is used for forensics) of the required data recovery area
IF the data recovery is critical then an additional drive (used to store the master initial copy along with its logfile) is required.
This second external drive needs to be mounted rw and wiped as well.
In cases where the total size of the desired recovery partition/drive is much smaller, a USB can be used.
Drive can be pre-partitioned as necessary to hold individual partitions of recovery data. (See notes for ddrescue and Autopsy)
----------
Copy the output file (backup.img) to another file/location
~$ cp ~/recovery/backup.img ~/recovery/do_not_touch.backup.img
Next, unmount the drive from which you just copied the data from.
~$ umount /mnt/recovery
-----------------------
Mount the backup.img file in its place: (see below to mount a single partition from within the image)
Go to root, then...

# fdisk -l /home/pentest/recovery/backup.img

To show you the block-size and the start-block of the partition.
Sample Output:

Disk Stick.img: 3984 MB, 3984588800 bytes
249 heads, 6 sectors/track, 5209 cylinders, total 7782400 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x0004bfaa

    Device Boot      Start         End      Blocks   Id  System
Stick.img1   *         128     8015999     4007936    b  W95 FAT32

So, in this example, I have a block-size of 512 bytes and the start-block is 128.
Therefore, the mount command would be:

# mount -t auto -o ro loop,offset=$((512*128)) /home/pentest/recovery/backup.img /mnt/recovery

Note that we are mounting the image as *read only* (-o ro) -- just to make sure we don't accidently write to it.

-----------------------------------------------------------
To mount a single partition inside the disk image:

# fdisk -l /home/pentest/recovery/backup.img

This will retrieve the block size and starting block number of the partition you want to mount.

Then mount the device with loopback and an offset option equal to blocksize*startingblock.

If, for example, the blocksize is 512, and the starting block is 40890...

# mount -t auto -o ro loop,offset=$((40890*512)) /home/pentest/recovery/backup.img /mnt/recovery

------------------------------------------------------------
Once the disk/partition has been mounted 
Run foremost to retrieve all deleted files possible.
Run fsck against the mount to effect repairs of the file structure as needed.
Run AV against the mount.
After that process is complete you can use a GUI file manager to copy out the undeleted, desired contents
This is also a good point at which to run anti-virus software on the recovered, deleted files.
**Do not run AV on the mounted image until AFTER you have recovered deleted files.**
----------
At this point, you can just save all of the required data to another location and then transfer it to wherever, -- OR --
Once deleted files have been recovered, the image has been scanned for viruses, and FSCK has been run against the disk/partitions image, the entire repaired, recovery image can be restored a another drive of the same geometry and used to replace the failed drive.
dd can be used to restore the image.
------------------------------------------------------------ 

Using DDRESCUE:
ddrescue provides an output file letting you know what went well and what went not-so-well.
Best tool. Too much info to include at this time. Here are the links:

ddrescue documentation: http://www.forensicswiki.org/wiki/Ddrescue , https://www.gnu.org/software/ddrescue/manual/ddrescue_manual.html
foremost documentation: http://linux.die.net/man/1/foremost 
lzip recovery documentation: http://www.nongnu.org/lzip/manual/lziprecover_manual.html

----------------------
VERY IMPORTANT:
Make an initial image copy using ddrescue first.
Then copy that file to a second location!
----
The first is the master copy and the second is your "working copy"
If anything goes wrong with your working copy you still have the master to fall back on.
----------------------
Run foremost to attempt all deleted files recovery.
Attempt to repair the working image copy using 'fsck'on the image file.
Run Anti-virus on the image file.
Using the same procedures outlined above, mount the file ro so that a standard GUI file manager can be used to browse and copy out the remaining desired files.
----------------------
If all else fails, you will have to use a hex editor to manually yank out the data or use a file recovery tool like photorec or Autopsy.
----------------------
lzip - is a compression tool that also has recovery capabilities; via "merge" which uses multiple copies of the same data from multiple sources.
See link above for details.
----------------------
Very Immportant Notes!

A failing drive tends to develop more and more errors as time passes. Because of this, you should rescue the data from a drive as soon as you notice the first error. Be diligent because every time a physically damaged drive powers up and is able to output some data, it may be the very last time that it ever will.

You should make a copy of the failing drive with ddrescue, and then try to repair the copy. If your data is really important, use the first copy as a master for a second copy, and try to repair the second copy. If something goes wrong, you have the master intact to try again.

If you are trying to rescue a whole partition, first repair the copy with fsck or some other tool appropriate for the type of partition you are trying to rescue, then mount the repaired copy somewhere and try to recover the files in it.

If the drive is so damaged that the file system in the rescued partition can't be repaired or mounted, you will have to browse the rescued data with an hex editor and extract the desired parts by hand or use a file recovery tool like photorec.

If the partition table is damaged, you may try to rescue the whole disc, then try to repair the partition table and the partitions on the copy.

If the damaged drive is not listed in /dev, then you cannot rescue it. At least not with ddrescue. 
----------------------
Mounting file systems in a forensically sound manner

To mount various file system types in a forensically sound manner the following methods can be used:

1. Loopback devices in read-only mode: can be switched on using «ro,loop» mount options (for example: «mount -o ro,loop /dev/sda1 /mnt/sda1»);
2. Block devices for disks and partitions in read-only mode: you can switch any block device to read-only mode by using blockdev tool (for example: «blockdev --setro /dev/sda1»);
3. You can disable all journal recovery actions for Ext3 and Ext4 file systems by using «ext2» file system type during the mount. Unfortunately, not all file systems support such kind of write protection — for example, there are no working write protection mount options for XFS (however, this was fixed in very recent kernel versions).

It should be noted that mounting damaged Ext3 and Ext4 file systems using these ways is only possible by utilizing alternate superblocks. You can get alternate superblocks locations by using «mke2fs -n» command (for example: «mke2fs -n /dev/sda1»). 

-----------------------
To mount or not to mount?
If not involved with forensics, it is easier/faster and safe to mount as read only.
Drive images do not need to be mounted for using Autopsy. 
You can only work on individual partitions with Autopsy - so plan accordingly. 
Mounting does alter the journal, but allows browsing and moving/copying the files to a different location with a GUI.
-----------------------

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 473 bytes
Desc: This is a digitally signed message part
URL: <http://lists.fedoraproject.org/pipermail/docs/attachments/20141009/cf7673bf/attachment.sig>