Inactive raid1 partition failure

[Alex]

Hi,
I have a fedora21 system with three md RAID1 arrays. I had used this
system to temporarily access an external array connected to an LSI
controller in the system. So I inserted the LSI controller, accessed
the data on the disks connected to it, and removed the controller from
the system.

Now, all three md RAID arrays are in some type of degraded state:

# cat /proc/mdstat
Personalities : [raid1]
md2 : inactive sdb1[1](S) sdc1[4](S)
      1953519616 blocks super 1.2

md0 : active raid1 sda1[0]
      511988 blocks super 1.0 [2/1] [U_]

md1 : active raid1 sda2[0]
      237566840 blocks super 1.2 [2/1] [U_]
      bitmap: 2/2 pages [8KB], 65536KB chunk

Thankfully the system boots. The inactive array, md2, is connected to
/var/backup, so I just commented it out of /etc/fstab to get the
system to boot.

It appears the devices were somehow impacted by the LSI controller,
and now the partitions/devices have been reordered.

How do I realign these slices to rebuild the array?

Thanks,
Alex

[Chris Murphy]

It looks like a device is completely missing?

blkid > blkid.txt

Then for each md device UUID pair (the md UUID will show up twice in
blkid for each of the three arrays):

mdadm -E <dev1> <dev2>  >> mdXraidstats.txt

Where X=the md device from mdstat. That way you get the superblock
from each device making up each array into an array specific text
file. They can go all in one file, but it's easier to keep things
separate.

journalctl -b -l -o short-monotonic > journal.txt
/etc/mdadm/mdadm.conf

Post those files somewhere and the URL here.

The journal should say why it wasn't assembled. But the main thing is
you should get copies of all device superblocks before you do anything
else, especially reading around the internet where often people get
the idea that they need to use -C to recreate an array which
obliterates the old metadata and almost invariably kills their data.




Chris Murphy

[Sam Varshavchik]

Alex writes:

>  Hi,
>  I have a fedora21 system with three md RAID1 arrays. I had used this
>  system to temporarily access an external array connected to an LSI
>  controller in the system. So I inserted the LSI controller, accessed
>  the data on the disks connected to it, and removed the controller from
>  the system.
>
>  Now, all three md RAID arrays are in some type of degraded state:
>
>  # cat /proc/mdstat
>  Personalities : [raid1]
>  md2 : inactive sdb1[1](S) sdc1[4](S)
>        1953519616 blocks super 1.2
>
>  md0 : active raid1 sda1[0]
>        511988 blocks super 1.0 [2/1] [U_]
>
>  md1 : active raid1 sda2[0]
>        237566840 blocks super 1.2 [2/1] [U_]
>        bitmap: 2/2 pages [8KB], 65536KB chunk
>
>  Thankfully the system boots. The inactive array, md2, is connected to
>  /var/backup, so I just commented it out of /etc/fstab to get the
>  system to boot.
>
>  It appears the devices were somehow impacted by the LSI controller,
>  and now the partitions/devices have been reordered.
>
>  How do I realign these slices to rebuild the array? 
The first step is to see what

mdadm --detail /dev/md0

says. Repeat for md1, et al…

Each one of your arrays is either missing a drive, or the other drive is  
marked as faulty.

If it's missing, use --add to add each drive to it's array. If mdadm says  
the drive has failed, try to --remove, then --add it back in.

[Alex]

Hi,

>  It looks like a device is completely missing?
>
>  blkid > blkid.txt
>
>  Then for each md device UUID pair (the md UUID will show up twice in
>  blkid for each of the three arrays):
>
>  mdadm -E <dev1> <dev2>  >> mdXraidstats.txt
>
>  Where X=the md device from mdstat. That way you get the superblock
>  from each device making up each array into an array specific text
>  file. They can go all in one file, but it's easier to keep things
>  separate.
>
>  journalctl -b -l -o short-monotonic > journal.txt
>  /etc/mdadm/mdadm.conf
>
>  Post those files somewhere and the URL here.
>
>  The journal should say why it wasn't assembled. But the main thing is
>  you should get copies of all device superblocks before you do anything
>  else, especially reading around the internet where often people get
>  the idea that they need to use -C to recreate an array which
>  obliterates the old metadata and almost invariably kills their data. 
Thanks so much for your help. In the process of disconnecting the LSI
controller and power cable connected to the disks, somehow the power
cable at the power supply side was loose for one of the other system
drives. Using these commands, I was able to figure out there really
was a device physically missing.

I panicked a bit, and didn't want to do something stupid (like mdadm
-C) before passing it by at least one other person. After connecting
the power cable, I was able to easily "mdadm --add" the drives back
and rebuild both degraded arrays. The /var/backup partition came up
automatically.

Another great mdadm command to know is --details.

Thanks again guys,
Alex

[Chris Murphy]

On Tue, Aug 4, 2015 at 6:38 PM, Alex <mysqlstudent(a)gmail.com&gt; wrote:

>  I panicked a bit, and didn't want to do something stupid (like
> mdadm
>  -C) before passing it by at least one other person. 
I definitely suggest you make a backup of each device superblock and
distribute it somewhere safe in the unlikely event two devices in a
mirrored array stop cooperating. The superblocks do come in handy from
time to time when they can't be acquired once problems happen.

The other thing is, if you're panicked, it kinda suggests no recent
backup. I can't tell you how many sad panda faces I see on many lists,
including the linux-raid@ list (all things RAID on Linux but mainly
md/mdadm) and it's like, really no backup? The brutal response is, if
you don't have a backup the data isn't important.


> After connecting
>  the power cable, I was able to easily "mdadm --add" the drives back
>  and rebuild both degraded arrays. The /var/backup partition came up
>  automatically. 
Great!


>  Another great mdadm command to know is --details. 
Yeah it's easy to get -D and -E confused. -D is pointed at the running
array and -E is pointed at devices. But they're functional
equivalents.


-- 
Chris Murphy

[Alex]

Hi,

> > I panicked a bit, and didn't want to do something stupid
> (like mdadm
> > -C) before passing it by at least one other person.
>
>  I definitely suggest you make a backup of each device superblock and
>  distribute it somewhere safe in the unlikely event two devices in a
>  mirrored array stop cooperating. The superblocks do come in handy from
>  time to time when they can't be acquired once problems happen. 
Do you have any suggestions on how to do this?

>  The other thing is, if you're panicked, it kinda suggests no
> recent
>  backup. I can't tell you how many sad panda faces I see on many lists,
>  including the linux-raid@ list (all things RAID on Linux but mainly
>  md/mdadm) and it's like, really no backup? The brutal response is, if
>  you don't have a backup the data isn't important. 
I do have a backup, but it would be a pain to rebuild everything. It
would be a major inconvenience. No one really ever wants to go through
that.

Thanks,
Alex

[Chris Murphy]

On Tue, Aug 4, 2015 at 9:02 PM, Alex <mysqlstudent(a)gmail.com&gt; wrote:
>  Hi,
>
> >> I panicked a bit, and didn't want to do something stupid (like mdadm
> >> -C) before passing it by at least one other person.
> >
> > I definitely suggest you make a backup of each device superblock and
> > distribute it somewhere safe in the unlikely event two devices in a
> > mirrored array stop cooperating. The superblocks do come in handy from
> > time to time when they can't be acquired once problems happen.
>
>  Do you have any suggestions on how to do this? 
mdadm -E for each device, and point it to a file using >> - for the
purposes of the backup you could write all devices to one file, it's
up to you.


-- 
Chris Murphy

[Alex]

Hi,

> >> I definitely suggest you make a backup of each device
> superblock and
> >> distribute it somewhere safe in the unlikely event two devices in a
> >> mirrored array stop cooperating. The superblocks do come in handy from
> >> time to time when they can't be acquired once problems happen.
> >
> > Do you have any suggestions on how to do this?
>
>  mdadm -E for each device, and point it to a file using >> - for the
>  purposes of the backup you could write all devices to one file, it's
>  up to you. 
Ah, I thought you were referring to actually storing information from
the disk itself, such as the bootsector or MBR, not just a list of the
partition info.

There's also a great resource on recovering a failed array here:

https://raid.wiki.kernel.org/index.php/RAID_Recovery

Thanks,
Alex