Expert Help needed (raid1+lvm) Did I destroy 2TB of backups? (Solved!)

Dean S. Messing deanm at sharplabs.com
Sat Nov 13 01:42:28 UTC 2010


Laurentiu Coica wrote:
<Long, detailed technical demo snipped>

Thanks VERY much, Laurentiu, for all the work you went through!

I must apologise for not including more details of what I did.

There are some differences between what I did and what you simulated
which makes your conclusion not quite applicable.

1) /mnt/deanm (using your names) was unmounted when I stopped the array.
   (This is probably minor.)

2) I did _not_ remove devices from the RAID1; I stopped the RAID with
   "mdadm  --stop <device>"

3) The LVM was not de-activated first (which is what caused the
   problem). The "live" device on which the LV sat went dead.

There were a few other minor differences not worth mentioning.

But in any case, thanks to your post, I was motivated to experiment with
a toy array on a couple of small identically sized partitions and
discovered the solution:

I was able to assemble the stopped toy array on _another_ md device
(/dev/md2).  It would not let me assemble on /dev/md1 for some reason.

Then I was used:  (I called my experimental VG "vg_exp")

# vgmknodes --refresh vg_exp
  /dev/md1: read failed after 0 of 4096 at 0: Input/output error
  /dev/vg_exp/lv_exp: read failed after 0 of 4096 at 2092892160: Input/output error
  /dev/vg_exp/lv_exp: read failed after 0 of 4096 at 2092949504: Input/output error
  /dev/vg_exp/lv_exp: read failed after 0 of 4096 at 0: Input/output error
  /dev/vg_exp/lv_exp: read failed after 0 of 4096 at 4096: Input/output error
  /dev/md1: read failed after 0 of 4096 at 2096234496: Input/output error
  /dev/md1: read failed after 0 of 4096 at 2096345088: Input/output error
  /dev/md1: read failed after 0 of 4096 at 4096: Input/output error

# vgmknodes --refresh vg_exp


Note that the first invocation gives a boatload of I/O errors just as I
have been seeing with my 2TB RAID.  But the 2nd invocation doesn't.
Magically the system figured out that the RAID1 was now on /dev/md2 and
properly associated the VG and LV with that device (which, of course,
had the physical extents of it, /dev/md1 being stopped.

Also /dev/md1  disappeared from /dev (a side benefit).
I can mount /dev/vg_exp/lv_exp  onto /mnt/exp
and see what I wrote there before.

Thanks again for your post, which motivated me to spend a couple hours
"experimenting".  My data has been saved!

Dean



More information about the users mailing list