I have a 1TB drive that is getting I/O errors enough that while I was out of the country, it hung and my wife was NOT happy. So it is time to replace it. And with a 2TB file.
This is a server with the drive as EXT4, boot partitions, swap partition, and a few data (/boot, /, etc.) partitions.
Given that I am experiencing I/O errors, it would seem that a dd is not advisable. I can dd the boot portion onto a new drive, I can build the partitions on a new drive. I can boot the system with a live CD.
What is a good way to move the OS and all the data? I used Clonezilla some many years ago, but that does not seem to be the way to go when there are bad sections on the source drive?
Thank you
On Mon, Nov 21, 2016 at 8:49 AM, Dave Ihnat dihnat@dminet.com wrote:
Likely not, but you _can_ try ddrescue.
ddrescue would be an excellent option. Also, if the I/O's errors are consistent sectors then perhaps you can "fsck" the disk to repair the bad sectors to buy you enough time to format a new disk and do a copy of the data to it.
-- Fred
On Mon, Nov 21, 2016 at 08:21:01AM -0500, Robert Moskowitz wrote:
I have a 1TB drive that is getting I/O errors enough that while I What is a good way to move the OS and all the data? I used Clonezilla some many years ago, but that does not seem to be the way to go when there are bad sections on the source drive?
Not helpful _now_, but consider making sure you have backups once you start using the new system. Then, next time this happens, forget worrying about transferring from a failing drive and just restore your backup.
If this is a desktop system with room for it, consider buying *two* drives and mirroring them. That's not a backup (since errors and mistakes get propagated instantly, and because it's in the same physical location), but makes situations like this rare (at least, if you replace as soon as you start seeing errors instead of waiting for a double failure).
On 11/21/2016 09:20 AM, Matthew Miller wrote:
On Mon, Nov 21, 2016 at 08:21:01AM -0500, Robert Moskowitz wrote:
I have a 1TB drive that is getting I/O errors enough that while I What is a good way to move the OS and all the data? I used Clonezilla some many years ago, but that does not seem to be the way to go when there are bad sections on the source drive?
Not helpful _now_, but consider making sure you have backups once you start using the new system. Then, next time this happens, forget worrying about transferring from a failing drive and just restore your backup.
The data is backed up. In fact it was one of the rsync cron tasks that wedged the system while I was gone. The I/O did not seem to be in the data, but perhaps some temp or other storage needed.
The challenge is the OS and software configs. Actually the system is a ClearOS file server and there are all those config information scattered around by ClearOS. I back up the whole /var/flexshare, but there is a lot more to their setup and no 'clear' directions from them. Building a new server from scratch would be painful. Last year (when I had time while building up my contract work) I looked into building my own Samba 4 server, but dropped it. Anyway, I have yet to find directions on a good way to maintain a running backup of the OS for a straight rebuild.
If this is a desktop system with room for it, consider buying *two* drives and mirroring them. That's not a backup (since errors and mistakes get propagated instantly, and because it's in the same physical location), but makes situations like this rare (at least, if you replace as soon as you start seeing errors instead of waiting for a double failure).
I could never figure out what mirroring failing sectors meant.
Bob
On Mon, Nov 21, 2016 at 12:59 PM, Robert Moskowitz rgm@htt-consult.com wrote:
On 11/21/2016 09:20 AM, Matthew Miller wrote:
On Mon, Nov 21, 2016 at 08:21:01AM -0500, Robert Moskowitz wrote:
I have a 1TB drive that is getting I/O errors enough that while I What is a good way to move the OS and all the data? I used Clonezilla some many years ago, but that does not seem to be the way to go when there are bad sections on the source drive?
Not helpful _now_, but consider making sure you have backups once you start using the new system. Then, next time this happens, forget worrying about transferring from a failing drive and just restore your backup.
The data is backed up. In fact it was one of the rsync cron tasks that wedged the system while I was gone. The I/O did not seem to be in the data, but perhaps some temp or other storage needed.
Backups should periodically read the entire disk, so may encounter bad sectors in areas that are not often read in normal operation. Backups that take too long to run are often the first indication of a failing disk.
The challenge is the OS and software configs. Actually the system is a ClearOS file server and there are all those config information scattered around by ClearOS. I back up the whole /var/flexshare, but there is a lot more to their setup and no 'clear' directions from them. Building a new server from scratch would be painful. Last year (when I had time while building up my contract work) I looked into building my own Samba 4 server, but dropped it. Anyway, I have yet to find directions on a good way to maintain a running backup of the OS for a straight rebuild.
After many experiences with having to reinstall the OS in order to install the backup client needed to restore user data, I make it a practice to keep a base OS with current updates handy. Without that I end up reinstalling the OS and then doing multiple updates. If you do a few updates a week and hit a problem then it usually isn't hard to figure out which package caused the issue, but on a system that has been in service for a few years with 100's updates, getting form a fresh install from the original media back to an up-to-date running system may not be easy. With systems that don't have restrictive licenses, you can just dd the base system to a fresh disk, change the system name and add the user(s), then install the backup client, restore the user data, and reinstall end-user applications.
ddrescue often recovers nearly all the files, so even if the resulting image isn't fully working you may be able to recover most of the configuration data.
If this is a desktop system with room for it, consider buying *two*
drives and mirroring them. That's not a backup (since errors and mistakes get propagated instantly, and because it's in the same physical location), but makes situations like this rare (at least, if you replace as soon as you start seeing errors instead of waiting for a double failure).
I could never figure out what mirroring failing sectors meant.
Not sure of the context, but the first sign of a failing disk is often a massive decrease in performance which I'm told is due to multiple re-reads of a sector. When a sector shows signs of failing, disk firmware copies the data to a spare sector. If the new disk doesn't arrive quickly you then see disk failure with a code that translates to "no spare sectors left on device".
George N. White III aa056@chebucto.ns.ca Head of St. Margarets Bay, Nova Scotia
On 11/21/2016 01:42 PM, George N. White III wrote:
On Mon, Nov 21, 2016 at 12:59 PM, Robert Moskowitz <rgm@htt-consult.com mailto:rgm@htt-consult.com> wrote:
On 11/21/2016 09:20 AM, Matthew Miller wrote: On Mon, Nov 21, 2016 at 08:21:01AM -0500, Robert Moskowitz wrote: I have a 1TB drive that is getting I/O errors enough that while I What is a good way to move the OS and all the data? I used Clonezilla some many years ago, but that does not seem to be the way to go when there are bad sections on the source drive? Not helpful _now_, but consider making sure you have backups once you start using the new system. Then, next time this happens, forget worrying about transferring from a failing drive and just restore your backup. The data is backed up. In fact it was one of the rsync cron tasks that wedged the system while I was gone. The I/O did not seem to be in the data, but perhaps some temp or other storage needed.Backups should periodically read the entire disk, so may encounter bad sectors in areas that are not often read in normal operation. Backups that take too long to run are often the first indication of a failing disk.
The challenge is the OS and software configs. Actually the system is a ClearOS file server and there are all those config information scattered around by ClearOS. I back up the whole /var/flexshare, but there is a lot more to their setup and no 'clear' directions from them. Building a new server from scratch would be painful. Last year (when I had time while building up my contract work) I looked into building my own Samba 4 server, but dropped it. Anyway, I have yet to find directions on a good way to maintain a running backup of the OS for a straight rebuild.After many experiences with having to reinstall the OS in order to install the backup client needed to restore user data, I make it a practice to keep a base OS with current updates handy.
How does on 'backup the entire disk' and/or build a 'base OS with current updates'? Can you point me to some guidance?
thanks
Without that I end up reinstalling the OS and then doing multiple updates. If you do a few updates a week and hit a problem then it usually isn't hard to figure out which package caused the issue, but on a system that has been in service for a few years with 100's updates, getting form a fresh install from the original media back to an up-to-date running system may not be easy. With systems that don't have restrictive licenses, you can just dd the base system to a fresh disk, change the system name and add the user(s), then install the backup client, restore the user data, and reinstall end-user applications.
ddrescue often recovers nearly all the files, so even if the resulting image isn't fully working you may be able to recover most of the configuration data.
If this is a desktop system with room for it, consider buying *two* drives and mirroring them. That's not a backup (since errors and mistakes get propagated instantly, and because it's in the same physical location), but makes situations like this rare (at least, if you replace as soon as you start seeing errors instead of waiting for a double failure). I could never figure out what mirroring failing sectors meant.Not sure of the context, but the first sign of a failing disk is often a massive decrease in performance which I'm told is due to multiple re-reads of a sector. When a sector shows signs of failing, disk firmware copies the data to a spare sector. If the new disk doesn't arrive quickly you then see disk failure with a code that translates to "no spare sectors left on device".
George N. White III <aa056@chebucto.ns.ca mailto:aa056@chebucto.ns.ca> Head of St. Margarets Bay, Nova Scotia
users mailing list -- users@lists.fedoraproject.org To unsubscribe send an email to users-leave@lists.fedoraproject.org
A couple of things that might work. I have maintained the g4l disk imaging project since 2004, and it's main options use dd to either clone a disk/partition or make an image file, but it would also have issues with bad sectors. But I have sometimes found that booting to it rather than the running OS, does allow it to sometimes make images with no issues, but that is rare.
A second option. The g4l also includes fsarchiver program, which I added at the request of a user. It is not a program that I've written, and only used it a few times to confirm that it did work. It is a file based imager, so it creates images of the files on the partitions. If the errors are in non-used space it might be able to make images of each partition and restore to another disk.
A third option is dd_rescue, which is also on the g4l disk that doesn't stop when it finds bad sectors, but will continue on, and then try again to copy them.
It has also been a while, but I have also used spinrite in the past on disk to repair disk, and it many times could recovere some data. But don't know if it supports the latest disks, and it does work on the original disks and makes changes.
On 21 Nov 2016 at 15:16, Robert Moskowitz wrote:
Subject: Backing up the OS - Re: Rebuilding a failing drive To: Community support for Fedora users users@lists.fedoraproject.org From: Robert Moskowitz rgm@htt-consult.com Date sent: Mon, 21 Nov 2016 15:16:15 -0500 Send reply to: Community support for Fedora users users@lists.fedoraproject.org
On 11/21/2016 01:42 PM, George N. White III wrote: On Mon, Nov 21, 2016 at 12:59 PM, Robert Moskowitz rgm@htt-consult.com wrote:
On 11/21/2016 09:20 AM, Matthew Miller wrote: On Mon, Nov 21, 2016 at 08:21:01AM -0500, Robert Moskowitz wrote: I have a 1TB drive that is getting I/O errors enough that while I What is a good way to move the OS and all the data? I used Clonezilla some many years ago, but that does not seem to be the way to go when there are bad sections on the source drive? Not helpful _now_, but consider making sure you have backups once you start using the new system. Then, next time this happens, forget worrying about transferring from a failing drive and just restore your backup. The data is backed up. In fact it was one of the rsync cron tasks that wedged the system while I was gone. The I/O did not seem to be in the data, but perhaps some temp or other storage needed. Backups should periodically read the entire disk, so may encounter bad sectors in areas that are not often read in normal operation. Backups that take too long to run are often the first indication of a failing disk. The challenge is the OS and software configs. Actually the system is a ClearOS file server and there are all those config information scattered around by ClearOS. I back up the whole /var/flexshare, but there is a lot more to their setup and no 'clear' directions from them. Building a new server from scratch would be painful. Last year (when I had time while building up my contract work) I looked into building my own Samba 4 server, but dropped it. Anyway, I have yet to find directions on a good way to maintain a running backup of the OS for a straight rebuild. After many experiences with having to reinstall the OS in order to install the backup client needed to restore user data, I make it a practice to keep a base OS with current updates handy.How does on 'backup the entire disk' and/or build a 'base OS with current updates'? Can you point me to some guidance?
thanks
Without that I end up reinstalling the OS and then doing multiple updates. If you do a few updates a week and hit a problem then it usually isn't hard to figure out which package caused the issue, but on a system that has been in service for a few years with 100's updates, getting form a fresh install from the original media back to an up-to-date running system may not be easy. With systems that don't have restrictive licenses, you can just dd the base system to a fresh disk, change the system name and add the user(s), then install the backup client, restore the user data, and reinstall end-user applications. ddrescue often recovers nearly all the files, so even if the resulting image isn't fully working you may be able to recover most of the configuration data. If this is a desktop system with room for it, consider buying *two* drives and mirroring them. That's not a backup (since errors and mistakes get propagated instantly, and because it's in the same physical location), but makes situations like this rare (at least, if you replace as soon as you start seeing errors instead of waiting for a double failure). I could never figure out what mirroring failing sectors meant. Not sure of the context, but the first sign of a failing disk is often a massive decrease in performance which I'm told is due to multiple re-reads of a sector. When a sector shows signs of failing, disk firmware copies the data to a spare sector. If the new disk doesn't arrive quickly you then see disk failure with a code that translates to "no spare sectors left on device". George N. White III <aa056@chebucto.ns.ca> Head of St. Margarets Bay, Nova Scotia _______________________________________________ users mailing list -- users@lists.fedoraproject.org To unsubscribe send an email to users-leave@lists.fedoraproject.org
+----------------------------------------------------------+ Michael D. Setzer II - Computer Science Instructor Guam Community College Computer Center mailto:mikes@kuentos.guam.net mailto:msetzerii@gmail.com Guam - Where America's Day Begins G4L Disk Imaging Project maintainer http://sourceforge.net/projects/g4l/ +----------------------------------------------------------+
http://setiathome.berkeley.edu (Original) Number of Seti Units Returned: 19,471 Processing time: 32 years, 290 days, 12 hours, 58 minutes (Total Hours: 287,489)
BOINC@HOME CREDITS ABC 16613838.513356 | EINSTEIN 120103112.288695 ROSETTA 52630912.921798 | SETI 95928248.603101
On Mon, Nov 21, 2016 at 08:21:01AM -0500, Robert Moskowitz wrote:
I have a 1TB drive that is getting I/O errors enough that while I was out of the country, it hung and my wife was NOT happy. So it is time to replace it. And with a 2TB file.
This is a server with the drive as EXT4, boot partitions, swap partition, and a few data (/boot, /, etc.) partitions.
Given that I am experiencing I/O errors, it would seem that a dd is not advisable. I can dd the boot portion onto a new drive, I can build the partitions on a new drive. I can boot the system with a live CD.
What is a good way to move the OS and all the data? I used Clonezilla some many years ago, but that does not seem to be the way to go when there are bad sections on the source drive?
I'd try ddrescue. I used it on my wife's win7 box a couple years ago and it just quietly did the job, despite bad spots on the HD.
On 11/21/2016 08:32 PM, Fred Smith wrote:
On Mon, Nov 21, 2016 at 08:21:01AM -0500, Robert Moskowitz wrote:
I have a 1TB drive that is getting I/O errors enough that while I was out of the country, it hung and my wife was NOT happy. So it is time to replace it. And with a 2TB file.
This is a server with the drive as EXT4, boot partitions, swap partition, and a few data (/boot, /, etc.) partitions.
Given that I am experiencing I/O errors, it would seem that a dd is not advisable. I can dd the boot portion onto a new drive, I can build the partitions on a new drive. I can boot the system with a live CD.
What is a good way to move the OS and all the data? I used Clonezilla some many years ago, but that does not seem to be the way to go when there are bad sections on the source drive?
I'd try ddrescue. I used it on my wife's win7 box a couple years ago and it just quietly did the job, despite bad spots on the HD.
Thanks for the information. Now to go out and buy a new drive. And read the man on ddrescue.
On 11/21/2016 05:21 AM, Robert Moskowitz wrote:
Given that I am experiencing I/O errors, it would seem that a dd is not advisable. I can dd the boot portion onto a new drive, I can build the partitions on a new drive. I can boot the system with a live CD.
What is a good way to move the OS and all the data? I used Clonezilla some many years ago, but that does not seem to be the way to go when there are bad sections on the source drive?
If the disk is not too far gone, you can very carefully overwrite the bad blocks which causes the disk to remap them. This might buy you enough time to copy the filesystem somewhere else using normal means. You should do the overwriting with a live image and should probably run an fsck after doing the overwrites to make sure you don't get filesystem errors while doing the copying.
-
Dave Ihnat -
fred roller -
Fred Smith -
George N. White III -
Matthew Miller -
Michael D. Setzer II -
Robert Moskowitz -
Samuel Sieb