Check your /etc/default/grub, if you use raid 1.

[Sam Varshavchik]

There's a long standing combination of two bugs: the list of rd.md.uuid boot  
parameters generated by anaconda for /etc/default/grub may not include the  
raid uuid of non-stock partitions like /home; and although the ramfs  
initscript autodiscovers all raid volumes present, sometimes (not always,  
I'll estimate 5% of the time) if a uuid is not enumerated in the boot  
parameters, one of the drives in the raid 1 volume may not get assembled at  
boot.

There's probably a third bug in here: mdmonitor should've mailed me when an  
array came up degraded at boot (I suspect that because mdmonitor gets  
started so early in the boot process, not all the moving pieces are there  
for mail delivery to happen). Eventually, you'll boot again with both drives  
in the array somehow, except they'll be out of sync, resulting in massive  
corruption. If you're lucky, you'll boot just with the other drive, and  
discover that your filesystem's contents are weeks/months out of date, and  
maybe you'll be lucky enough to figure out what happen, and switch back to  
the other drive and resync. But, not everyone's so lucky.

This first started happening in F16. It took me a while to figure out the  
cause for an occasionally raid assembly failure at boot. Fixed it, and  
forgot about it. Well, looks like the F17 anaconda brought back the broken  
/etc/default/grub, which found its way into my grub.cfg, and I just lost a  
full day, cleaning up this mess.

So, if you use raid 1 and upgraded to F17, you may need to fix this before  
it's too late: put back the missing uuid into /etc/default/grub, and into  
every entry in grub.cfg

Pissed.

[Bruno Wolff III]

On Sun, Jul 29, 2012 at 10:02:00 -0400,
   Sam Varshavchik <mrsam(a)courier-mta.com&gt; wrote:
> There's a long standing combination of two bugs: the list of 
> rd.md.uuid boot parameters generated by anaconda for 
> /etc/default/grub may not include the raid uuid of non-stock 
> partitions like /home; and although the ramfs initscript 
> autodiscovers all raid volumes present, sometimes (not always, I'll 
> estimate 5% of the time) if a uuid is not enumerated in the boot 
> parameters, one of the drives in the raid 1 volume may not get 
> assembled at boot. 
My raid info is /etc/mdadm.conf and that is what gets used by dracut when 
building an initramfs as far as I can tell.

> There's probably a third bug in here: mdmonitor should've
> mailed me 
> when an array came up degraded at boot (I suspect that because 
> mdmonitor gets started so early in the boot process, not all the 
> moving pieces are there for mail delivery to happen). Eventually, 
> you'll boot again with both drives in the array somehow, except 
> they'll be out of sync, resulting in massive corruption. If you're 
> lucky, you'll boot just with the other drive, and discover that your 
> filesystem's contents are weeks/months out of date, and maybe you'll 
> be lucky enough to figure out what happen, and switch back to the 
> other drive and resync. But, not everyone's so lucky. 
That doesn't sound right. You might come up using the incorrect raid member, 
but you should come up with two out of sync drives. (Maybe this could happen 
with some non-default setups, where the elements aren't labelled.)

There was a recent bug with raid arrays that could result in some elements 
failing when shutting down. It doesn't directly corrupt the data though. 
There is information about this bug here: 
http://neil.brown.name/blog/20120615073245

[Reindl Harald]

Am 29.07.2012 16:19, schrieb Bruno Wolff III:
>  On Sun, Jul 29, 2012 at 10:02:00 -0400,
>    Sam Varshavchik <mrsam(a)courier-mta.com&gt; wrote:
> > There's a long standing combination of two bugs: the list of rd.md.uuid boot
> parameters generated by anaconda for
> > /etc/default/grub may not include the raid uuid of non-stock partitions like /home;
> and although the ramfs
> > initscript autodiscovers all raid volumes present, sometimes (not always, I'll
> estimate 5% of the time) if a uuid
> > is not enumerated in the boot parameters, one of the drives in the raid 1 volume may
> not get assembled at boot.
>  
>  My raid info is /etc/mdadm.conf and that is what gets used by dracut when building an
> initramfs as far as I can tell. 

in theory

sam is right and the problem exists since F16

you have to add all your UUIDs from /etc/mdadm.conf with MD_UUID=<youruuid>
entries to the kernel line or you are randomly boot with degrared arrays

it does not help to have the driver in initramfs if the arrays are not s
tarted correct, it must also be used properly from the system

maybe tehre are people affected even not recognize their degraded arrays
until it is too late

[Harald Hoyer]

Am 29.07.2012 16:25, schrieb Reindl Harald:
>  
>  
>  Am 29.07.2012 16:19, schrieb Bruno Wolff III:
> > On Sun, Jul 29, 2012 at 10:02:00 -0400,
> >   Sam Varshavchik <mrsam(a)courier-mta.com&gt; wrote:
> >> There's a long standing combination of two bugs: the list of rd.md.uuid boot
> parameters generated by anaconda for
> >> /etc/default/grub may not include the raid uuid of non-stock partitions like
> /home; and although the ramfs
> >> initscript autodiscovers all raid volumes present, sometimes (not always,
> I'll estimate 5% of the time) if a uuid
> >> is not enumerated in the boot parameters, one of the drives in the raid 1 volume
> may not get assembled at boot.
> >
> > My raid info is /etc/mdadm.conf and that is what gets used by dracut when building an
> initramfs as far as I can tell.
>  
>  
>  in theory
>  
>  sam is right and the problem exists since F16
>  
>  you have to add all your UUIDs from /etc/mdadm.conf with MD_UUID=<youruuid>
>  entries to the kernel line or you are randomly boot with degrared arrays
>  
>  it does not help to have the driver in initramfs if the arrays are not s
>  tarted correct, it must also be used properly from the system
>  
>  maybe tehre are people affected even not recognize their degraded arrays
>  until it is too late
>   
I would consider it a dracut bug, if rd.md.uuid is specified and other raid
arrays are assembled in the initramfs.

[Sam Varshavchik]

Bruno Wolff III writes:

>  On Sun, Jul 29, 2012 at 10:02:00 -0400,
>    Sam Varshavchik <mrsam(a)courier-mta.com&gt; wrote:
> > There's a long standing combination of two bugs: the list of rd.md.uuid boot  
> > parameters generated by anaconda for /etc/default/grub may not include the  
> > raid uuid of non-stock partitions like /home; and although the ramfs  
> > initscript autodiscovers all raid volumes present, sometimes (not always,  
> > I'll estimate 5% of the time) if a uuid is not enumerated in the boot  
> > parameters, one of the drives in the raid 1 volume may not get assembled at  
> > boot.
>
>  My raid info is /etc/mdadm.conf and that is what gets used by dracut when  
>  building an initramfs as far as I can tell. 
All I know is in F16 I discovered that a raid 1 volume whose uuid does not  
get enumerated in the rd.md.uuid kernel boot parameters will come up with  
one drive not in the array, maybe 5% of the time. I wasn't the only one  
affected, there was another list member that reported the same bug, and that  
putting the uuid back into grub.cfg and /etc/default/grub fixed it.

> > There's probably a third bug in here: mdmonitor should've
> mailed me when an  
> > array came up degraded at boot (I suspect that because mdmonitor gets  
> > started so early in the boot process, not all the moving pieces are there  
> > for mail delivery to happen). Eventually, you'll boot again with both drives  
> > in the array somehow, except they'll be out of sync, resulting in massive  
> > corruption. If you're lucky, you'll boot just with the other drive, and  
> > discover that your filesystem's contents are weeks/months out of date, and  
> > maybe you'll be lucky enough to figure out what happen, and switch back to  
> > the other drive and resync. But, not everyone's so lucky.
>
>  That doesn't sound right. You might come up using the incorrect raid member,  
>  but you should come up with two out of sync drives. (Maybe this could happen  
>  with some non-default setups, where the elements aren't labelled.) 
According to mdadm --detail, I have a "Name" label on it.

All I know is that I spent half a day wondering why, every time I fscked  
this partition I found more crap. The other half of the day was spent  
resyncing the volume, after I figured out that the drives were not synced.

And, fail/remove/add did not resync the drive. Because the volume uses an  
internal bitmap: oh, the newly-added drive has a valid bitmap, apparently  
from the same volume, so let's add this drive without resyncing it!

That, I think is a bug. Failing a drive should zero its superblock, to force  
a real resync if it gets added back to the array.

>  There was a recent bug with raid arrays that could result in some
> elements  
>  failing when shutting down. It doesn't directly corrupt the data though.  
>  There is information about this bug here: http://neil.brown.name/blog/ 
>  20120615073245 
Since F17 is on 3.4, and this seems to indicate that only some 3.2 and 3.3  
kernels might have an issue, doesn't look like this is related.

[Bruno Wolff III]

On Sun, Jul 29, 2012 at 10:42:26 -0400,
   Sam Varshavchik <mrsam(a)courier-mta.com&gt; wrote:
> All I know is in F16 I discovered that a raid 1 volume whose uuid 
> does not get enumerated in the rd.md.uuid kernel boot parameters will 
> come up with one drive not in the array, maybe 5% of the time. I 
> wasn't the only one affected, there was another list member that 
> reported the same bug, and that putting the uuid back into grub.cfg 
> and /etc/default/grub fixed it. 
> That, I think is a bug. Failing a drive should zero its superblock, 
> to force a real resync if it gets added back to the array. 
Do you know if there are bug tracker entries for either of these bugs?

I'm interesting in following them.

[Sam Varshavchik]

Bruno Wolff III writes:

>  On Sun, Jul 29, 2012 at 10:42:26 -0400,
>    Sam Varshavchik <mrsam(a)courier-mta.com&gt; wrote:
> >
> > All I know is in F16 I discovered that a raid 1 volume whose uuid does not  
> > get enumerated in the rd.md.uuid kernel boot parameters will come up with  
> > one drive not in the array, maybe 5% of the time. I wasn't the only one  
> > affected, there was another list member that reported the same bug, and that  
> > putting the uuid back into grub.cfg and /etc/default/grub fixed it.
>
> > That, I think is a bug. Failing a drive should zero its superblock, to force  
> > a real resync if it gets added back to the array.
>
>  Do you know if there are bug tracker entries for either of these bugs?
>
>  I'm interesting in following them. 
Not clearing the superblock when a drive fails is something that I just ran  
into. This one should probably be discussed upstream.

I think I remember seeing someone reporting the array assembly bug when it's  
not enumerated in rd.md.uuid, but I can't find the bug right now.

[Bruno Wolff III]

On Sun, Jul 29, 2012 at 11:23:51 -0400,
   Sam Varshavchik <mrsam(a)courier-mta.com&gt; wrote:
> I think I remember seeing someone reporting the array assembly bug 
> when it's not enumerated in rd.md.uuid, but I can't find the bug 
> right now.
>  
I found two related bugs. The second is listed as closed.

mdadm raid10 incremental assembly fails at boot, creating a degraded array:
https://bugzilla.redhat.com/show_bug.cgi?id=823597

make mdadm --add not do stupid things:
https://bugzilla.redhat.com/show_bug.cgi?id=808774

[Reindl Harald]

Am 29.07.2012 16:42, schrieb Sam Varshavchik:
>  And, fail/remove/add did not resync the drive. Because the volume
> uses an internal bitmap: oh, the newly-added
>  drive has a valid bitmap, apparently from the same volume, so let's add this drive
> without resyncing it!
>  
>  That, I think is a bug. Failing a drive should zero its superblock, to force a real
> resync if it gets added back to
>  the array. 
no, no and no!
if you have some mechanic problems that could end in zero the superblocks of all disks

in my last tests with a RAID1 i pulled the power from one SATA disk
and after reboot i was not able to re-add the drive before manually
zero the superblock which is the right behavior

this was some months ago and played with both physical drives
(pull, reboot, zero, re-add, resync)

[Sam Varshavchik]

Reindl Harald writes:

>  Am 29.07.2012 16:42, schrieb Sam Varshavchik:
>  > And, fail/remove/add did not resync the drive. Because the volume uses an  
>  internal bitmap: oh, the newly-added
>  > drive has a valid bitmap, apparently from the same volume, so let's add  
>  this drive without resyncing it!
>  >
>  > That, I think is a bug. Failing a drive should zero its superblock, to  
>  force a real resync if it gets added back to
>  > the array.
>
>  no, no and no!
>  if you have some mechanic problems that could end in zero the superblocks of  
>  all disks
>
>  in my last tests with a RAID1 i pulled the power from one SATA disk
>  and after reboot i was not able to re-add the drive before manually
>  zero the superblock which is the right behavior 
I'm talking about a manual --fail with mdadm.

[Michael Cronenworth]

Sam Varshavchik wrote:
>  So, if you use raid 1 and upgraded to F17, you may need to fix this
>  before it's too late: put back the missing uuid into /etc/default/grub,
>  and into every entry in grub.cfg 
I took my F16 RAID1 server to F17 this past weekend using preupgrade. My
UUIDs are correct in my /etc/default/grub file. I do not seem to be
affected.

Thanks though.

[Sam Varshavchik]

Michael Cronenworth writes:

>  Sam Varshavchik wrote:
>  > So, if you use raid 1 and upgraded to F17, you may need to fix this
>  > before it's too late: put back the missing uuid into /etc/default/grub,
>  > and into every entry in grub.cfg
>
>  I took my F16 RAID1 server to F17 this past weekend using preupgrade. My
>  UUIDs are correct in my /etc/default/grub file. I do not seem to be
>  affected.
>
>  Thanks though. 
Do you remember what you told the installer to do, whether you told it to  
write out a new bootloader entry (it's one of the last steps).

[Michael Cronenworth]

Sam Varshavchik wrote:
>  Do you remember what you told the installer to do, whether you told
> it
>  to write out a new bootloader entry (it's one of the last steps). 
Preupgrade doesn't ask any questions and the whole update happens
without any user interaction.