I've been playing around with RAID config and may have finally messed it up.
Initially, I created the array with 3 300G drives: # mdadm --create --verbose /dev/md0 --level=5 --raid-devices=3 /dev/sda4 /dev/sdb1 /dev/sdc1 # mkfs.ext4 -v -m 0.01 -E stride=16,stripe-width=32 /dev/md0
It's been working nicely so far, and I decided to add a 4th 300G drive: # mdadm --grow --raid-devices=4 --backup-file=/root/grow_md0.bak /dev/md0
That finished overnight, while I looked around and found that chunk size of 512 should work better. I unmounted the FS and ran # mdadm --grow -c 512 --backup-file=/root/grow_md0_rechunk.bak /dev/md0 mdadm: component size 293033536K is not a multiple of chunksize 512K
so I sized it down a bit: # mdadm --grow -z 293033472 --backup-file=/root/grow_md0_size.bak /dev/md0
and then back to resizing chunks: mdadm --grow -c 512 --backup-file=/root/grow_md0_rechunk.bak /dev/md0
It's running right now: # cat /proc/mdstat Personalities : [raid6] [raid5] [raid4] md0 : active raid5 sdd1[3] sda4[0] sdc1[2] sdb1[1] 293033472 blocks super 0.91 level 5, 64k chunk, algorithm 2 [4/4] [UUUU] [====>................] reshape = 22.7% (66540032/293033472) finish=947.3min speed=3984K/sec
But just now I tried to mount the filesystem and it's failing: EXT4-fs (md0): bad geometry: block count 146516768 exceeds size of device (73258368 blocks)
Here's the question, then: am I royally screwed or is my data still there? How do I recover?
Konstantin Svist wrote:
I've been playing around with RAID config and may have finally messed it up.
Initially, I created the array with 3 300G drives: # mdadm --create --verbose /dev/md0 --level=5 --raid-devices=3 /dev/sda4 /dev/sdb1 /dev/sdc1 # mkfs.ext4 -v -m 0.01 -E stride=16,stripe-width=32 /dev/md0
It's been working nicely so far, and I decided to add a 4th 300G drive: # mdadm --grow --raid-devices=4 --backup-file=/root/grow_md0.bak /dev/md0
That finished overnight, while I looked around and found that chunk size of 512 should work better. I unmounted the FS and ran # mdadm --grow -c 512 --backup-file=/root/grow_md0_rechunk.bak /dev/md0 mdadm: component size 293033536K is not a multiple of chunksize 512K
so I sized it down a bit: # mdadm --grow -z 293033472 --backup-file=/root/grow_md0_size.bak /dev/md0
and then back to resizing chunks: mdadm --grow -c 512 --backup-file=/root/grow_md0_rechunk.bak /dev/md0
It's running right now: # cat /proc/mdstat Personalities : [raid6] [raid5] [raid4] md0 : active raid5 sdd1[3] sda4[0] sdc1[2] sdb1[1] 293033472 blocks super 0.91 level 5, 64k chunk, algorithm 2 [4/4] [UUUU] [====>................] reshape = 22.7% (66540032/293033472) finish=947.3min speed=3984K/sec
But just now I tried to mount the filesystem and it's failing: EXT4-fs (md0): bad geometry: block count 146516768 exceeds size of device (73258368 blocks)
Here's the question, then: am I royally screwed or is my data still there? How do I recover?
May I suggest trying this on the linux-raid list? Not that's it's off-topic here, just that people there may have faster/better answers.
On 07/18/2010 08:03 PM, Bill Davidsen wrote:
Konstantin Svist wrote:
I've been playing around with RAID config and may have finally messed it up.
Initially, I created the array with 3 300G drives: # mdadm --create --verbose /dev/md0 --level=5 --raid-devices=3 /dev/sda4 /dev/sdb1 /dev/sdc1 # mkfs.ext4 -v -m 0.01 -E stride=16,stripe-width=32 /dev/md0
It's been working nicely so far, and I decided to add a 4th 300G drive: # mdadm --grow --raid-devices=4 --backup-file=/root/grow_md0.bak /dev/md0
That finished overnight, while I looked around and found that chunk size of 512 should work better. I unmounted the FS and ran # mdadm --grow -c 512 --backup-file=/root/grow_md0_rechunk.bak /dev/md0 mdadm: component size 293033536K is not a multiple of chunksize 512K
so I sized it down a bit: # mdadm --grow -z 293033472 --backup-file=/root/grow_md0_size.bak /dev/md0
and then back to resizing chunks: mdadm --grow -c 512 --backup-file=/root/grow_md0_rechunk.bak /dev/md0
It's running right now: # cat /proc/mdstat Personalities : [raid6] [raid5] [raid4] md0 : active raid5 sdd1[3] sda4[0] sdc1[2] sdb1[1] 293033472 blocks super 0.91 level 5, 64k chunk, algorithm 2 [4/4] [UUUU] [====>................] reshape = 22.7% (66540032/293033472) finish=947.3min speed=3984K/sec
But just now I tried to mount the filesystem and it's failing: EXT4-fs (md0): bad geometry: block count 146516768 exceeds size of device (73258368 blocks)
Here's the question, then: am I royally screwed or is my data still there? How do I recover?
May I suggest trying this on the linux-raid list? Not that's it's off-topic here, just that people there may have faster/better answers.
Yeah, I thought of exactly the same thing after posting this message.
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
On Sun, 18 Jul 2010, Konstantin Svist wrote:
Here's the question, then: am I royally screwed or is my data still there? How do I recover?
Did you run resize2fs?
Gabriel
- --
// Gabriel VLASIU // // OpenGPG-KeyID : 0xE684206E // OpenGPG-Fingerprint: 0C3D 9F8B 725D E243 CB3C 8428 796A DB1F E684 206E // OpenGPG-URL : http://www.vlasiu.net/public.key
On 07/18/2010 11:59 PM, Gabriel VLASIU wrote:
Here's the question, then: am I royally screwed or is my data still there? How do I recover?
Did you run resize2fs?
No, not yet. I know it's the "next step" after growing the array, but I thought I'd re-do the chunks first.
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
On Mon, 19 Jul 2010, Konstantin Svist wrote:
No, not yet. I know it's the "next step" after growing the array, but I thought I'd re-do the chunks first.
I'm not sure but I think you have to run resize2fs before trying to mount your partition.
Gabriel
- --
// Gabriel VLASIU // // OpenGPG-KeyID : 0xE684206E // OpenGPG-Fingerprint: 0C3D 9F8B 725D E243 CB3C 8428 796A DB1F E684 206E // OpenGPG-URL : http://www.vlasiu.net/public.key
On 07/19/2010 12:10 AM, Gabriel VLASIU wrote:
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
On Mon, 19 Jul 2010, Konstantin Svist wrote:
No, not yet. I know it's the "next step" after growing the array, but I thought I'd re-do the chunks first.
I'm not sure but I think you have to run resize2fs before trying to mount your partition.
The "I'm not sure" part kind of worries me :)
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
On Mon, 19 Jul 2010, Konstantin Svist wrote:
The "I'm not sure" part kind of worries me :)
Well, this is way people usually do a full backup before. :-)
Gabriel
- --
// Gabriel VLASIU // // OpenGPG-KeyID : 0xE684206E // OpenGPG-Fingerprint: 0C3D 9F8B 725D E243 CB3C 8428 796A DB1F E684 206E // OpenGPG-URL : http://www.vlasiu.net/public.key
Konstantin Svist wrote:
so I sized it down a bit: # mdadm --grow -z 293033472 --backup-file=/root/grow_md0_size.bak /dev/md0
And there is your error. You resized the device without first resizing the fileystem.
The filesystem is *in* the device. So if you want to enlarge, you enlarge the device and then the filesystem. If you want to shrink, you shrink the filesystem and then the device.
You basically destroyed a few random blocks from the end of the filesystem. I don't know how serious the damage is. fsck will tell you.
It is also impossible to undo the error because you have reshaped the RAID after the shrinking, so the "lost" blocks are not easily reincludeable (with an opposite --grow, I mean).
The small number of missing blocks and their position at the end of the disk give you a reasonable confidence that you will save your data. I'm not sure if it is better to run an fsck and pray or it is better to reenlarge the device with some zeroed blocks to avoid the "read beyond partition size" condition (I personally would think about linearly concatenation of your RAID with another small zeroed partition (or file) just to exceed the size of the "contained" filesystem).
On 07/19/2010 05:09 AM, Roberto Ragusa wrote:
Konstantin Svist wrote:
so I sized it down a bit: # mdadm --grow -z 293033472 --backup-file=/root/grow_md0_size.bak /dev/md0
And there is your error. You resized the device without first resizing the fileystem.
The filesystem is *in* the device. So if you want to enlarge, you enlarge the device and then the filesystem. If you want to shrink, you shrink the filesystem and then the device.
You basically destroyed a few random blocks from the end of the filesystem. I don't know how serious the damage is. fsck will tell you.
It is also impossible to undo the error because you have reshaped the RAID after the shrinking, so the "lost" blocks are not easily reincludeable (with an opposite --grow, I mean).
The small number of missing blocks and their position at the end of the disk give you a reasonable confidence that you will save your data. I'm not sure if it is better to run an fsck and pray or it is better to reenlarge the device with some zeroed blocks to avoid the "read beyond partition size" condition (I personally would think about linearly concatenation of your RAID with another small zeroed partition (or file) just to exceed the size of the "contained" filesystem).
My thinking process was this: I just added a new drive and md0 was rebuilt but FS was not resized -- which implies the end of the drives should've been empty. I'm pretty sure I gave the right command to mdadm, but for whatever reason it didn't listen to me.
After I changed the chunk size back to 64, I told it to resize to max size, which should've reverted everything:
# mdadm --grow -z max --backup-file=/root/grow_md0_size_back.bak /dev/md0 mdadm: component size of /dev/md0 has been set to 293033536K # cat /proc/mdstat Personalities : [raid6] [raid5] [raid4] md0 : active raid5 sdd1[3] sda4[0] sdc1[2] sdb1[1] 293033472 blocks level 5, 64k chunk, algorithm 2 [4/4] [UUUU]
After stopping the array and re-assembling it, 3 of 4 drives came back up (because of a typo in /etc/mdadm.conf) and the array had proper size, though degraded. I found 2 mistakes -- / was missing from /dev/sdd1 in /etc/mdadm.conf, and partition type of /dev/sdd1 was '83 Linux' instead of 'fd Linux raid autodetect'
Right now the 4th drive has been added and the array is running recovery.