Moving Fedora 22 to RAID 0 - Trouble with initramfs

[Samuel Rakitničan]

I have moved / partition to another partition formed in RAID 0
consisting of two SSDs. I have updated fstab with new partition UUID,
reinstalled GRUB2 and rebuild initramfs using dracut -f. Now computer
boots fine from RAID partition but hangs on when initramfs needs to
boot kernel from / partition found on RAID, because it can't find
partition with UUID that I've put in fstab.

Now from what I have understood when it drops me to dracut prompt in
initramfs boot process I indeed can't find the RAID assembled BUT I
can assemble it manually by using "mdadm -I /dev/sda" and "mdadm -I
/dev/sdb". If I boot from old hard drive the RAID is assembled
normally in kernel on boot time:

$ journalctl -b | grep "kernel: md"

Kol 30 01:20:53 computername kernel: md: bind<sdb>
Kol 30 01:20:53 computername kernel: md: bind<sda>
Kol 30 01:20:53 computername kernel: md: bind<sdb>
Kol 30 01:20:53 computername kernel: md: bind<sda>
Kol 30 01:20:53 computername kernel: md: raid0 personality registered
for level 0
Kol 30 01:20:53 computername kernel: md/raid0:md126: md_size is
500129792 sectors.
Kol 30 01:20:53 computername kernel: md: RAID0 configuration for md126 - 1 zone
Kol 30 01:20:53 computername kernel: md: zone0=[sda/sdb]
Kol 30 01:20:53 computername kernel:       zone-offset=         0KB,
device-offset=         0KB,
Kol 30 01:20:53 computername kernel:
Kol 30 01:20:53 computername kernel: md126: detected capacity change
from 0 to 256066453504
Kol 30 01:20:53 computername kernel:  md126: p1 p2 p3
Kol 30 01:20:53 computername kernel: md: export_rdev(sdb)
Kol 30 01:20:53 computername kernel: md: export_rdev(sda)
Kol 30 01:20:53 computername kernel: md: export_rdev(sdb)
Kol 30 01:20:53 computername kernel: md: export_rdev(sda)

The RAID is formed using onboard southbridge Intel controller. I have
managed to extract data from initramfs but I am not sure what to look
for, in particular what brings up RAID assembly.
/etc/udev/rules.d/65-md-incremental-imsm.rules seems like it's
responsible to assemble, but I am not sure.

Any thoughts why imsm RAID is not assembled in initramfs on boot?

[Tom Horsley]

On Sun, 30 Aug 2015 23:09:15 +0200
Samuel Rakitničan wrote:
> Any thoughts why imsm RAID is not assembled in initramfs on boot?

I have no idea what the problem was, but I had the same
issue when I tried to move an existing f22 partition to
a raid (or maybe it was f20 or 21, I forget).

It started working after I booted off a live CD
and did a dnf update that installed a new kernel while
chrooted into the partition that previously wouldn't
boot.

I had previously tried doing "dracut --force" in the
chroot, but that didn't seem to fix things.

[Gordon Messmer]

On 08/30/2015 02:09 PM, Samuel Rakitničan wrote:
> Any thoughts why imsm RAID is not assembled in initramfs on boot?

Try updating /etc/mdadm.conf and rebuilding the initramfs, again.

[Chris Murphy]

Post URL for the grub.cfg.

Chris Murphy

[Chris Murphy]

On Sun, Aug 30, 2015 at 3:09 PM, Samuel Rakitničan
 wrote:
> I have moved / partition to another partition formed in RAID 0
> consisting of two SSDs. I have updated fstab with new partition UUID,

Using blkid, make sure you use the UUID= value for the *filesystem
volume*, not PARTUUID=.
> reinstalled GRUB2 and rebuild initramfs using dracut -f. Now computer
> boots fine from RAID partition but hangs on when initramfs needs to
> boot kernel from / partition found on RAID, because it can't find
> partition with UUID that I've put in fstab.

Dracut should first look for the mduuid (which is either baked into
the initramfs via /etc/mdadm.conf; or hinted using rd.md.uuid= as a
boot parameter, this uuid is the mdadm uuid), which causes the array
to be assembled, and only after that does the fs volume UUID become
available which systemd is looking for based on the boot param
root=UUID=.
> Now from what I have understood when it drops me to dracut prompt in
> initramfs boot process I indeed can't find the RAID assembled BUT I
> can assemble it manually by using "mdadm -I /dev/sda" and "mdadm -I
> /dev/sdb". If I boot from old hard drive the RAID is assembled
> normally in kernel on boot time.

Huh. That's nice but I don't know why it works with a generic
initramfs. Is this mdadm metadata v0.9? If so that's kernel
autodetect. mdadm v.1.x is initramfs autodetect.

Anyway, try making changes per above, and if that doesn't work then
boot with parameters rd.debug rd.shell and figure out a way to write
out the journal somewhere like a USB stick mounted at /sysroot.

journalctl -b -l -o short-monotonic > /sysroot/journal.txt
> The RAID is formed using onboard southbridge Intel controller.

Oh in that case this is imsm metadata, not mdadm metadata, so ignore
0.9 vs 1.x.
> I have
> managed to extract data from initramfs but I am not sure what to look
> for, in particular what brings up RAID assembly.
> /etc/udev/rules.d/65-md-incremental-imsm.rules seems like it's
> responsible to assemble, but I am not sure.

Yep sounds right.
> Any thoughts why imsm RAID is not assembled in initramfs on boot?

rd.debug rd.shell post a URL for the journal.

[Samuel Rakitničan]

2015-08-31 3:08 GMT+02:00 Chris Murphy :
> On Sun, Aug 30, 2015 at 3:09 PM, Samuel Rakitničan
>  wrote:
> > I have moved / partition to another partition formed in RAID 0
> > consisting of two SSDs. I have updated fstab with new partition UUID,
>
> Using blkid, make sure you use the UUID= value for the *filesystem
> volume*, not PARTUUID=.

I believe I am, but the issue is that it can't even look for it
because is not assembled (I think) anyway:

$ blkid
/dev/sda: TYPE="isw_raid_member"
/dev/sdb: TYPE="isw_raid_member"
/dev/sdc1: LABEL="gedora" UUID="24d33c42-c8d5-4351-a89c-08cfa70b3115"
TYPE="ext4" PARTUUID="c35c32fe-01"
/dev/sdc2: LABEL="swap" UUID="50ba9ee0-2e73-4d89-8997-024a0d4b91da"
TYPE="swap" PARTUUID="c35c32fe-02"
/dev/sdc3: LABEL="homie" UUID="9946e484-fe87-403b-8b79-1ff15214f262"
TYPE="ext4" PARTUUID="c35c32fe-03"
/dev/sdd: UUID="cc3a22dc-21ea-4081-8b0b-21007c981eb0" TYPE="crypto_LUKS"
/dev/sde1: LABEL="Transcend" UUID="A784-A6A0" TYPE="vfat" PARTUUID="c3072e18-01"
/dev/sdf1: UUID="E250-35EB" TYPE="vfat" PARTUUID="f6aa9f3a-01"
/dev/sdg1: UUID="574B-489E" TYPE="vfat" PARTUUID="000eb8bf-01"
/dev/md126p2: LABEL="gedora"
UUID="a7a06541-ea6f-42e5-8e95-b7bdcafb33cf" TYPE="xfs"
PARTUUID="ccd4b28b-a762-4b0d-b157-9bacd3d8411d"
/dev/md126p3: LABEL="homie"
UUID="51020653-40f4-44bf-a1f2-a796758fac60" TYPE="xfs"
PARTUUID="6c017e9e-1d47-427e-9293-b4ebe368fcc7"

# cat /etc/fstab

#
# /etc/fstab
# Created by anaconda on Wed Feb  4 19:50:51 2015
#
# Accessible filesystems, by reference, are maintained under '/dev/disk'
# See man pages fstab(5), findfs(8), mount(8) and/or blkid(8) for more info
#
UUID=a7a06541-ea6f-42e5-8e95-b7bdcafb33cf /                       xfs
  defaults,lazytime        1 1
#UUID=9946e484-fe87-403b-8b79-1ff15214f262 /home
ext4    defaults,lazytime        1 2
#UUID=50ba9ee0-2e73-4d89-8997-024a0d4b91da swap
swap    defaults        0 0

And sure enough I can see exact same UUID in not found message on boot time.
> > reinstalled GRUB2 and rebuild initramfs using dracut -f. Now computer
> > boots fine from RAID partition but hangs on when initramfs needs to
> > boot kernel from / partition found on RAID, because it can't find
> > partition with UUID that I've put in fstab.
>
> Dracut should first look for the mduuid (which is either baked into
> the initramfs via /etc/mdadm.conf; or hinted using rd.md.uuid= as a
> boot parameter, this uuid is the mdadm uuid), which causes the array
> to be assembled, and only after that does the fs volume UUID become
> available which systemd is looking for based on the boot param
> root=UUID=.

I have no experience with mdadm before, but I tried to create and
remade initramfs using "mdadm --verbose --detail --scan >
/etc/mdadm.conf".

# cat /etc/mdadm.conf
ARRAY /dev/md/imsm0 level=container num-devices=2 metadata=imsm
UUID=03afe6a1:b37c71b2:f9b7eb4e:9322a561
   devices=/dev/sda,/dev/sdb
ARRAY /dev/md/TrancendPowa_0 level=raid0 num-devices=2
container=/dev/md/imsm0 member=0
UUID=1d2f6a80:99e713e3:73601439:2cdb477f
   devices=/dev/sda,/dev/sdb

But it doesn't work anyway.
> > Now from what I have understood when it drops me to dracut prompt in
> > initramfs boot process I indeed can't find the RAID assembled BUT I
> > can assemble it manually by using "mdadm -I /dev/sda" and "mdadm -I
> > /dev/sdb". If I boot from old hard drive the RAID is assembled
> > normally in kernel on boot time.
>
> Huh. That's nice but I don't know why it works with a generic
> initramfs. Is this mdadm metadata v0.9? If so that's kernel
> autodetect. mdadm v.1.x is initramfs autodetect.

No, I don't know if it assembles in generic initramfs (probably not),
but it assembles in normal kernel, the full one.
> Anyway, try making changes per above, and if that doesn't work then
> boot with parameters rd.debug rd.shell and figure out a way to write
> out the journal somewhere like a USB stick mounted at /sysroot.
>
> journalctl -b -l -o short-monotonic > /sysroot/journal.txt

Here it is: https://www.dropbox.com/s/fnt2r46izuhpbvw/journal.txt.gz?dl=0
> > The RAID is formed using onboard southbridge Intel controller.
>
> Oh in that case this is imsm metadata, not mdadm metadata, so ignore
> 0.9 vs 1.x.
> > I have
> > managed to extract data from initramfs but I am not sure what to look
> > for, in particular what brings up RAID assembly.
> > /etc/udev/rules.d/65-md-incremental-imsm.rules seems like it's
> > responsible to assemble, but I am not sure.
>
> Yep sounds right.
> > Any thoughts why imsm RAID is not assembled in initramfs on boot?
>
> rd.debug rd.shell post a URL for the journal.

Thank you and everyone else!
> --
> Chris Murphy
> --
> users mailing list
> [e-mail]
> To unsubscribe or change subscription options:
> https://admin.fedoraproject.org/mailman/listinfo/users
> Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct
> Guidelines: http://fedoraproject.org/wiki/Mailing_list_guidelines
> Have a question? Ask away: http://ask.fedoraproject.org

[Chris Murphy]

[    1.345397] computername dracut-pre-trigger[217]: rd.md=0: removing
MD RAID activation

and

[    1.354112] computername dracut-pre-trigger[217]:
///lib/dracut/hooks/pre-trigger/30-parse-md.sh@32(source): '[' -e
/etc/mdadm.conf ']'
[    1.354178] computername dracut-pre-trigger[217]:
///lib/dracut/hooks/pre-trigger/30-parse-md.sh@32(source): getargbool
1 rd.md.conf -d -n rd_NO_MDADMCONF

I'm getting the idea, possibly incorrectly, that there is no
mdadm.conf in the initramfs. You can use lsinitrd to see if there's
one inside and what the size, time stamp is, to see if it's current.

It's definitely not assembling the array is the problem. I'm not sure
if boot param rd.md.uuid= is sufficient because IMSM RAID is actually
a container plus RAID, so there are two UUIDs that apply. You could
try:

rd.md.uuid=03afe6a1:b37c71b2:f9b7eb4e:9322a561
rd.md.uuid=1d2f6a80:99e713e3:73601439:2cdb477f

And see if dracut figures out which is which and assembles the
container first then the array. I also don't know for sure if : should
be replaced with - as is typically the case for filesystems.

Chris Murphy