I have this system that I have been using that was running Fedora 21. I just updated it today (Monday) to Fedora 22. It is an old Dell Optiplex GX620 with two 2TB disk drives set up with two partitions each in a RAID One configuration. LVM2 runs on top of the second partition with several logical volumes. This system had been running just fine before I did the upgrade. After the upgrade, the system will not boot using one of the new Fedora 22 kernels. It boots just fine using one of the old Fedora 21 kernels that did not get replaced during the upgrade.
I can get the system up using one of the new kernels, but I have to wait until it stops at a Dracut prompt. I then manual run "mdadm -- assemble --scan" to get the RAID One partitions assembled, and "lvm vgchange -a y" to get the Volume Group built. After that, I issue an "exit" and the system completes the boot process. I need to figure out why this system will not boot without intervention. Any suggestions would be appreciated.
I have tried to rebuild the initramfs with dracut, no joy doing that.
Here is some of the data:
# cat /etc/mdadm.conf MAILADDR root AUTO +imsm +1.x -all ARRAY /dev/md0 metadata=0.90 UUID=c45384af:aa8fc7ca:7eebaeb8:f0ca1523 ARRAY /dev/md1 metadata=1.2 name=test3.localnet:md1 UUID=524079c6:521ca76f:126164a9:3232e116
# cat /etc/fstab UUID=5a1848ce-cff6-4c1f-a622-dc1fe5f06d55 /boot ext3 defaults 1 2 UUID=762b53d0-b5a0-41dc-b804-d339fa58b119 swap swap defaults 0 0 /dev/mapper/vg0-lv1 / ext4 defaults 1 1 /dev/mapper/vg0-lv2 /usr/local ext4 defaults 1 2 /dev/mapper/vg0-lv3 /var ext4 defaults 1 2 /dev/mapper/vg0-lv4 /tmp ext4 defaults 1 2 /dev/mapper/vg0-lv5 /opt ext4 defaults 1 2 /dev/mapper/vg0-lv6 /home ext4 defaults 1 2 /dev/mapper/vg0-lv7 /work ext4 defaults 1 2 /dev/sr0 /cd0 auto noauto,owner,user,ro 0 0
# uname -a Linux redwood 4.4.9-200.fc22.x86_64 #1 SMP Fri May 6 17:42:20 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
Other versions of kernels that work just fine: 3.18.9-200 & 4.1.13-100 both from the Fedora 21 distribution.
On Mon, May 23, 2016 at 11:13 PM, Bill Perkins bp241@grnwood.net wrote:
I have this system that I have been using that was running Fedora 21. I just updated it today (Monday) to Fedora 22. It is an old Dell Optiplex GX620 with two 2TB disk drives set up with two partitions each in a RAID One configuration. LVM2 runs on top of the second partition with several logical volumes. This system had been running just fine before I did the upgrade. After the upgrade, the system will not boot using one of the new Fedora 22 kernels. It boots just fine using one of the old Fedora 21 kernels that did not get replaced during the upgrade.
I can get the system up using one of the new kernels, but I have to wait until it stops at a Dracut prompt. I then manual run "mdadm -- assemble --scan" to get the RAID One partitions assembled, and "lvm vgchange -a y" to get the Volume Group built. After that, I issue an "exit" and the system completes the boot process. I need to figure out why this system will not boot without intervention. Any suggestions would be appreciated.
I have tried to rebuild the initramfs with dracut, no joy doing that.
Here is some of the data:
# cat /etc/mdadm.conf MAILADDR root AUTO +imsm +1.x -all ARRAY /dev/md0 metadata=0.90 UUID=c45384af:aa8fc7ca:7eebaeb8:f0ca1523 ARRAY /dev/md1 metadata=1.2 name=test3.localnet:md1 UUID=524079c6:521ca76f:126164a9:3232e116
# cat /etc/fstab UUID=5a1848ce-cff6-4c1f-a622-dc1fe5f06d55 /boot ext3 defaults 1 2 UUID=762b53d0-b5a0-41dc-b804-d339fa58b119 swap swap defaults 0 0 /dev/mapper/vg0-lv1 / ext4 defaults 1 1 /dev/mapper/vg0-lv2 /usr/local ext4 defaults 1 2 /dev/mapper/vg0-lv3 /var ext4 defaults 1 2 /dev/mapper/vg0-lv4 /tmp ext4 defaults 1 2 /dev/mapper/vg0-lv5 /opt ext4 defaults 1 2 /dev/mapper/vg0-lv6 /home ext4 defaults 1 2 /dev/mapper/vg0-lv7 /work ext4 defaults 1 2 /dev/sr0 /cd0 auto noauto,owner,user,ro 0 0
# uname -a Linux redwood 4.4.9-200.fc22.x86_64 #1 SMP Fri May 6 17:42:20 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
Can you put up journalctl -b somewhere?
On 2016.05.24 15:45, Chris Murphy wrote:
On Mon, May 23, 2016 at 11:13 PM, Bill Perkins bp241@grnwood.net wrote:
I have this system that I have been using that was running Fedora
I just updated it today (Monday) to Fedora 22. It is an old Dell Optiplex GX620 with two 2TB disk drives set up with two partitions each in a RAID One configuration. LVM2 runs on top of the second partition with several logical volumes. This system had been
running
just fine before I did the upgrade. After the upgrade, the system
will
not boot using one of the new Fedora 22 kernels. It boots just
fine
using one of the old Fedora 21 kernels that did not get replaced
during
the upgrade.
I can get the system up using one of the new kernels, but I have to wait until it stops at a Dracut prompt. I then manual run "mdadm
--
assemble --scan" to get the RAID One partitions assembled, and "lvm vgchange -a y" to get the Volume Group built. After that, I issue
an
"exit" and the system completes the boot process. I need to figure
out
why this system will not boot without intervention. Any
suggestions
would be appreciated.
I have tried to rebuild the initramfs with dracut, no joy doing
that.
Here is some of the data:
# cat /etc/mdadm.conf MAILADDR root AUTO +imsm +1.x -all ARRAY /dev/md0 metadata=0.90
UUID=c45384af:aa8fc7ca:7eebaeb8:f0ca1523
ARRAY /dev/md1 metadata=1.2 name=test3.localnet:md1 UUID=524079c6:521ca76f:126164a9:3232e116
# cat /etc/fstab UUID=5a1848ce-cff6-4c1f-a622-dc1fe5f06d55 /boot ext3 defaults 1 2 UUID=762b53d0-b5a0-41dc-b804-d339fa58b119 swap swap defaults 0 0 /dev/mapper/vg0-lv1 / ext4 defaults 1 1 /dev/mapper/vg0-lv2 /usr/local ext4 defaults 1 2 /dev/mapper/vg0-lv3 /var ext4 defaults 1 2 /dev/mapper/vg0-lv4 /tmp ext4 defaults 1 2 /dev/mapper/vg0-lv5 /opt ext4 defaults 1 2 /dev/mapper/vg0-lv6 /home ext4 defaults 1 2 /dev/mapper/vg0-lv7 /work ext4 defaults 1 2 /dev/sr0 /cd0 auto noauto,owner,user,ro 0 0
# uname -a Linux redwood 4.4.9-200.fc22.x86_64 #1 SMP Fri May 6 17:42:20 UTC
2016
x86_64 x86_64 x86_64 GNU/Linux
Can you put up journalctl -b somewhere?
-- Chris Murphy
Chris,
This link should work:
http://roo.grnwood.net/redwood-journal-b-160524.gz
Gzip file is 692,498 bytes. Original file is 11,702,620 bytes.
Thank you for looking at this problem.
Bill
May 24 00:07:00 redwood.localnet kernel: Command line: BOOT_IMAGE=/vmlinuz-4.4.9-200.fc22.x86_64 root=/dev/mapper/vg0-lv1 ro rd.md.uuid=5a1848ce:cff64c1f:a622dc1f:e5f06d55 LANG=en_US.UTF-8 SYSFONT=True rd.lvm.lv=vg0/lv0 rd.lvm.lv=vg0/lv1 KEYTABLE=us rd.luks=0 rd.dm=0
There is an rd.md.uuid= parameter, so that's good.
May 24 00:10:08 redwood.localnet dracut-initqueue[347]: Warning: Could not boot. May 24 00:10:08 redwood.localnet systemd[1]: Received SIGRTMIN+20 from PID 349 (plymouthd). May 24 00:10:08 redwood.localnet dracut-initqueue[347]: Warning: /dev/mapper/vg0-lv1 does not exist May 24 00:10:08 redwood.localnet dracut-initqueue[347]: Warning: /dev/vg0/lv0 does not exist May 24 00:10:08 redwood.localnet dracut-initqueue[347]: Warning: /dev/vg0/lv1 does not exist May 24 00:10:08 redwood.localnet systemd[1]: Starting Dracut Emergency Shell...
OK so yeah, it's basically not finding rootfs. 29 seconds later...
May 24 00:10:37 redwood.localnet kernel: md: md0 stopped. May 24 00:10:37 redwood.localnet kernel: md: bind<sda1> May 24 00:10:37 redwood.localnet kernel: md: bind<sdb1> May 24 00:10:37 redwood.localnet kernel: md: raid1 personality registered for level 1 May 24 00:10:37 redwood.localnet kernel: md/raid1:md0: active with 2 out of 2 mirrors May 24 00:10:37 redwood.localnet kernel: md0: detected capacity change from 0 to 262078464 May 24 00:10:37 redwood.localnet kernel: md: md1 stopped. May 24 00:10:37 redwood.localnet kernel: md: bind<sda2> May 24 00:10:37 redwood.localnet kernel: md: bind<sdb2> May 24 00:10:37 redwood.localnet kernel: md/raid1:md1: active with 2 out of 2 mirrors May 24 00:10:37 redwood.localnet kernel: created bitmap (15 pages) for device md1 May 24 00:10:37 redwood.localnet kernel: md1: bitmap initialized from disk: read 1 pages, set 0 of 29804 bits May 24 00:10:37 redwood.localnet kernel: md1: detected capacity change from 0 to 2000068632576 May 24 00:10:54 redwood.localnet systemd[1]: Found device /dev/mapper/vg0-lv1. May 24 00:10:54 redwood.localnet systemd[1]: Starting File System Check on /dev/mapper/vg0-lv1... May 24 00:10:54 redwood.localnet systemd-fsck[2685]: /: clean, 529693/1048576 files, 3263599/4194304 blocks May 24 00:10:54 redwood.localnet systemd[1]: Started File System Check on /dev/mapper/vg0-lv1.
So now it's found, I'm gonna guess this is your manual attempt at starting up the array?
What do you get for cat /proc/mdstat at the dracut prompt before you do anything else? And also the results from
mdstat -E /dev/sda1 mdstate -E /dev/sdb1 blkid
These can be redirected to fpaste or a file. If you have network access this early it's easiest to redirect to fpaste but I'm not sure if it's in the initramfs.
From your mdadm.conf, you have UUIDs that do not match what's on the command line.
cmdline: rd.md.uuid=5a1848ce:cff64c1f:a622dc1f:e5f06d55
mdadm.conf UUID=c45384af:aa8fc7ca:7eebaeb8:f0ca1523 UUID=524079c6:521ca76f:126164a9:3232e116
Looks like the mdadm.conf is stale and should be recreated, and recreate the initramfs. Thing is, the rd.md.uuid ought to be enough for dracut to startup that array, the mdadm.conf is really just about assigning a particular md array to a particular /dev/mdX assignment, which by the way has changed recently so there might be confusion going on there.
You might just try renaming the current mdadm.conf, or commenting out all of the lines, except for one:
ARRAY /dev/md0 UUID=5a1848ce:cff64c1f:a622dc1f:e5f06d55
And after saving it, use dracut -f.
Chris Murphy
On Wed, May 25, 2016 at 1:25 PM, Chris Murphy lists@colorremedies.com wrote:
May 24 00:07:00 redwood.localnet kernel: Command line: BOOT_IMAGE=/vmlinuz-4.4.9-200.fc22.x86_64 root=/dev/mapper/vg0-lv1 ro rd.md.uuid=5a1848ce:cff64c1f:a622dc1f:e5f06d55 LANG=en_US.UTF-8 SYSFONT=True rd.lvm.lv=vg0/lv0 rd.lvm.lv=vg0/lv1 KEYTABLE=us rd.luks=0 rd.dm=0
There is an rd.md.uuid= parameter, so that's good.
May 24 00:10:08 redwood.localnet dracut-initqueue[347]: Warning: Could not boot. May 24 00:10:08 redwood.localnet systemd[1]: Received SIGRTMIN+20 from PID 349 (plymouthd). May 24 00:10:08 redwood.localnet dracut-initqueue[347]: Warning: /dev/mapper/vg0-lv1 does not exist May 24 00:10:08 redwood.localnet dracut-initqueue[347]: Warning: /dev/vg0/lv0 does not exist May 24 00:10:08 redwood.localnet dracut-initqueue[347]: Warning: /dev/vg0/lv1 does not exist May 24 00:10:08 redwood.localnet systemd[1]: Starting Dracut Emergency Shell...
OK so yeah, it's basically not finding rootfs. 29 seconds later...
May 24 00:10:37 redwood.localnet kernel: md: md0 stopped. May 24 00:10:37 redwood.localnet kernel: md: bind<sda1> May 24 00:10:37 redwood.localnet kernel: md: bind<sdb1> May 24 00:10:37 redwood.localnet kernel: md: raid1 personality registered for level 1 May 24 00:10:37 redwood.localnet kernel: md/raid1:md0: active with 2 out of 2 mirrors May 24 00:10:37 redwood.localnet kernel: md0: detected capacity change from 0 to 262078464 May 24 00:10:37 redwood.localnet kernel: md: md1 stopped. May 24 00:10:37 redwood.localnet kernel: md: bind<sda2> May 24 00:10:37 redwood.localnet kernel: md: bind<sdb2> May 24 00:10:37 redwood.localnet kernel: md/raid1:md1: active with 2 out of 2 mirrors May 24 00:10:37 redwood.localnet kernel: created bitmap (15 pages) for device md1 May 24 00:10:37 redwood.localnet kernel: md1: bitmap initialized from disk: read 1 pages, set 0 of 29804 bits May 24 00:10:37 redwood.localnet kernel: md1: detected capacity change from 0 to 2000068632576 May 24 00:10:54 redwood.localnet systemd[1]: Found device /dev/mapper/vg0-lv1. May 24 00:10:54 redwood.localnet systemd[1]: Starting File System Check on /dev/mapper/vg0-lv1... May 24 00:10:54 redwood.localnet systemd-fsck[2685]: /: clean, 529693/1048576 files, 3263599/4194304 blocks May 24 00:10:54 redwood.localnet systemd[1]: Started File System Check on /dev/mapper/vg0-lv1.
So now it's found, I'm gonna guess this is your manual attempt at starting up the array?
What do you get for cat /proc/mdstat at the dracut prompt before you do anything else? And also the results from
mdstat -E /dev/sda1 mdstate -E /dev/sdb1 blkid
These can be redirected to fpaste or a file. If you have network access this early it's easiest to redirect to fpaste but I'm not sure if it's in the initramfs.
From your mdadm.conf, you have UUIDs that do not match what's on the command line.
cmdline: rd.md.uuid=5a1848ce:cff64c1f:a622dc1f:e5f06d55
mdadm.conf UUID=c45384af:aa8fc7ca:7eebaeb8:f0ca1523 UUID=524079c6:521ca76f:126164a9:3232e116
Looks like the mdadm.conf is stale and should be recreated, and recreate the initramfs. Thing is, the rd.md.uuid ought to be enough for dracut to startup that array, the mdadm.conf is really just about assigning a particular md array to a particular /dev/mdX assignment, which by the way has changed recently so there might be confusion going on there.
You might just try renaming the current mdadm.conf, or commenting out all of the lines, except for one:
ARRAY /dev/md0 UUID=5a1848ce:cff64c1f:a622dc1f:e5f06d55
And after saving it, use dracut -f.
Chris Murphy
I am having TONS of problems with f22 (af
ter reinstalling it from DVD media, due to system corruption). FOr example, after using dnf to install all the mate packages that were previously installed (I had saved the names of those packages in a different drive). I had no desktop control. Mouse and KB did not work. The one app that automatically came up (due to settings of the mate env) was firefox , and for some reason KB within the FF window worked, but nowhere else. yara ... yara ...yara..
When it rains problems, it pours :)
Slight change in suggestions:
On Wed, May 25, 2016 at 1:25 PM, Chris Murphy lists@colorremedies.com wrote:
What do you get for cat /proc/mdstat at the dracut prompt before you do anything else?
Do this still. Then assemble the array manually as you have been and continue the boot normally.
And also the results from
mdstat -E /dev/sda1 mdstate -E /dev/sdb1 blkid
These are easier done after basic target is reached, rather than at the dracut shell. Also, the first two have bad typos so here's the revised version:
mdadm -E /dev/sda[12] mdadm -D /dev/md[01] blkid
You might just try renaming the current mdadm.conf, or commenting out all of the lines, except for one:
ARRAY /dev/md0 UUID=5a1848ce:cff64c1f:a622dc1f:e5f06d55
And after saving it, use dracut -f.
Skip this for now. I'd rather just collect info rather than change anything.
On 2016.05.25 15:25, Chris Murphy wrote:
May 24 00:07:00 redwood.localnet kernel: Command line: BOOT_IMAGE=/vmlinuz-4.4.9-200.fc22.x86_64 root=/dev/mapper/vg0-lv1 ro rd.md.uuid=5a1848ce:cff64c1f:a622dc1f:e5f06d55 LANG=en_US.UTF-8 SYSFONT=True rd.lvm.lv=vg0/lv0 rd.lvm.lv=vg0/lv1 KEYTABLE=us rd.luks=0 rd.dm=0
There is an rd.md.uuid= parameter, so that's good.
May 24 00:10:08 redwood.localnet dracut-initqueue[347]: Warning: Could not boot. May 24 00:10:08 redwood.localnet systemd[1]: Received SIGRTMIN+20 from PID 349 (plymouthd). May 24 00:10:08 redwood.localnet dracut-initqueue[347]: Warning: /dev/mapper/vg0-lv1 does not exist May 24 00:10:08 redwood.localnet dracut-initqueue[347]: Warning: /dev/vg0/lv0 does not exist May 24 00:10:08 redwood.localnet dracut-initqueue[347]: Warning: /dev/vg0/lv1 does not exist May 24 00:10:08 redwood.localnet systemd[1]: Starting Dracut Emergency Shell...
OK so yeah, it's basically not finding rootfs. 29 seconds later...
May 24 00:10:37 redwood.localnet kernel: md: md0 stopped. May 24 00:10:37 redwood.localnet kernel: md: bind<sda1> May 24 00:10:37 redwood.localnet kernel: md: bind<sdb1> May 24 00:10:37 redwood.localnet kernel: md: raid1 personality registered for level 1 May 24 00:10:37 redwood.localnet kernel: md/raid1:md0: active with 2 out of 2 mirrors May 24 00:10:37 redwood.localnet kernel: md0: detected capacity change from 0 to 262078464 May 24 00:10:37 redwood.localnet kernel: md: md1 stopped. May 24 00:10:37 redwood.localnet kernel: md: bind<sda2> May 24 00:10:37 redwood.localnet kernel: md: bind<sdb2> May 24 00:10:37 redwood.localnet kernel: md/raid1:md1: active with 2 out of 2 mirrors May 24 00:10:37 redwood.localnet kernel: created bitmap (15 pages) for device md1 May 24 00:10:37 redwood.localnet kernel: md1: bitmap initialized from disk: read 1 pages, set 0 of 29804 bits May 24 00:10:37 redwood.localnet kernel: md1: detected capacity change from 0 to 2000068632576 May 24 00:10:54 redwood.localnet systemd[1]: Found device /dev/mapper/vg0-lv1. May 24 00:10:54 redwood.localnet systemd[1]: Starting File System Check on /dev/mapper/vg0-lv1... May 24 00:10:54 redwood.localnet systemd-fsck[2685]: /: clean, 529693/1048576 files, 3263599/4194304 blocks May 24 00:10:54 redwood.localnet systemd[1]: Started File System Check on /dev/mapper/vg0-lv1.
So now it's found, I'm gonna guess this is your manual attempt at starting up the array?
What do you get for cat /proc/mdstat at the dracut prompt before you do anything else? And also the results from
mdstat -E /dev/sda1 mdstate -E /dev/sdb1 blkid
These can be redirected to fpaste or a file. If you have network access this early it's easiest to redirect to fpaste but I'm not sure if it's in the initramfs.
From your mdadm.conf, you have UUIDs that do not match what's on the
command line.
cmdline: rd.md.uuid=5a1848ce:cff64c1f:a622dc1f:e5f06d55
mdadm.conf UUID=c45384af:aa8fc7ca:7eebaeb8:f0ca1523 UUID=524079c6:521ca76f:126164a9:3232e116
Looks like the mdadm.conf is stale and should be recreated, and recreate the initramfs. Thing is, the rd.md.uuid ought to be enough for dracut to startup that array, the mdadm.conf is really just about assigning a particular md array to a particular /dev/mdX assignment, which by the way has changed recently so there might be confusion going on there.
You might just try renaming the current mdadm.conf, or commenting out all of the lines, except for one:
ARRAY /dev/md0 UUID=5a1848ce:cff64c1f:a622dc1f:e5f06d55
And after saving it, use dracut -f.
Chris Murphy
Okay. I have figured this out after a lot of digging through Google.
I finally found this page and a post by eileon (Comment: 10)
https://bugzilla.redhat.com/show_bug.cgi?id=1085922
The problem turned on differences between a file:
/etc/default/grub
and the output of: dracut --print-cmdline
As eileon stated what is given by the output above by dracut, needed to be reflected in the /etc/default/grub file before you did ran:
grub2-mkconfig -o /boot/grub2/grub.cfg
and then:
dracut -f
The problem I had was in the GRUB_CMDLINE_LINUX line in the /etc/default/grub file. The first rd.md.uuid was supposed to be the uuid for the Linux RAID member disks that make up the mdraid devices from which you want to boot.
The second rd.md.uuid that pointed to the RAID member disks used for the root filesystems and LVM was missing!
Correcting both rd.md.uuid entries to reflect the correct devices in that file and running dracut -f before rebooting resolved my problem.
Thank you Chris for your assistance and pointing me in the right direction.
Cheers! Bill
On Wed, May 25, 2016 at 10:13 PM, Bill Perkins bp241@grnwood.net wrote:
The problem I had was in the GRUB_CMDLINE_LINUX line in the /etc/default/grub file. The first rd.md.uuid was supposed to be the uuid for the Linux RAID member disks that make up the mdraid devices from which you want to boot.
The second rd.md.uuid that pointed to the RAID member disks used for the root filesystems and LVM was missing!
Yes, I figured it'd be something like that once I realized there really were two mdraid arrays, but only one rd.md.uuid parameter. What's curious is that its absence wasn't causing a problem before the upgrade.
Correcting both rd.md.uuid entries to reflect the correct devices in that file and running dracut -f before rebooting resolved my problem.
Well in the end that's all that matters.
On 2016.05.25 15:51, Chris Murphy wrote:
Did the hostname change since the array was created?
-- Chris Murphy
No.... The hostname did not change that I can see, and I did not knowingly change it.
I am wondering why you ask the question?
The host name is redwood.
Chris, I am still working on a reply to your previous message.
On Wed, May 25, 2016 at 2:07 PM, Bill Perkins bp241@grnwood.net wrote:
No.... The hostname did not change that I can see, and I did not knowingly change it.
I am wondering why you ask the question?
It's embedded in the mdadm superblock, and it might be initrd based auto-activation (metadata 1.x and higher) may fail if the actual host name and the one in the superblock don't match. There is a way to update this information at assemble time with mdadm --assemble --update=
But that might be a distraction.
-
Bill Perkins -
Chris Murphy -
JD