F23 dracut can't find disk

[CLOSE Dave]

I have five machines which were fresh-installed with F23 back in 
February and all have been booted successfully a few times since. Today, 
booting of all of them fails in exactly the same way: dracut says it 
can't find the disk filesystems. The kernel boots as it should, and of 
course that comes from the disk, but then dracut comes along and says it 
can't find any of the filesystems. Not the root or home filesystems 
which are on LVM or the boot filesystem on a primary disk partition.

Everything on the disk is ok. I've checked by booting Anaconda from a 
thumb drive and mounting manually. Anaconda troubleshooting mode says it 
can't find the filesystems either ("you have no Linux partitions"), but 
running vgchange -ay and a few mounts gets a proper chroot image.

There must be something that is causing both dracut and Anaconda to fail 
to find the filesystems. I've tried following the instructions on 
<https://fedoraproject.org/wiki/How_to_debug_Dracut_problems>;, but that 
isn't helping:

dracut:/# parted /dev/sda -s p
sh: parted: command not found
dracut:/# lvm vgscan
File descriptor 98 (/dev/console) leaked on lvm invocation. Parent PID 
2679: sh
File descriptor 99 (/dev/console) leaked on lvm invocation. Parent PID 
2679: sh
   Reading all physical volumes. This may take a while...
dracut:/# lvm vgchange -ay
File descriptor 98 (/dev/console) leaked on lvm invocation. Parent PID 
2679: sh
File descriptor 99 (/dev/console) leaked on lvm invocation. Parent PID 
2679: sh
dracut:/# blkid
dracut:/#

Note that when booted from the thumb drive, vgchange finds the LVM 
volumes just fine.
-- 
Dave Close

[Rick Stevens]

On 04/07/2016 04:50 PM, CLOSE Dave wrote:
>  I have five machines which were fresh-installed with F23 back in
>  February and all have been booted successfully a few times since. Today,
>  booting of all of them fails in exactly the same way: dracut says it
>  can't find the disk filesystems. The kernel boots as it should, and of
>  course that comes from the disk, but then dracut comes along and says it
>  can't find any of the filesystems. Not the root or home filesystems
>  which are on LVM or the boot filesystem on a primary disk partition.
>
>  Everything on the disk is ok. I've checked by booting Anaconda from a
>  thumb drive and mounting manually. Anaconda troubleshooting mode says it
>  can't find the filesystems either ("you have no Linux partitions"), but
>  running vgchange -ay and a few mounts gets a proper chroot image.
>
>  There must be something that is causing both dracut and Anaconda to fail
>  to find the filesystems. I've tried following the instructions on
>  <https://fedoraproject.org/wiki/How_to_debug_Dracut_problems>;, but that
>  isn't helping:
>
>  dracut:/# parted /dev/sda -s p
>  sh: parted: command not found
>  dracut:/# lvm vgscan
>  File descriptor 98 (/dev/console) leaked on lvm invocation. Parent PID
>  2679: sh
>  File descriptor 99 (/dev/console) leaked on lvm invocation. Parent PID
>  2679: sh
>      Reading all physical volumes. This may take a while...
>  dracut:/# lvm vgchange -ay
>  File descriptor 98 (/dev/console) leaked on lvm invocation. Parent PID
>  2679: sh
>  File descriptor 99 (/dev/console) leaked on lvm invocation. Parent PID
>  2679: sh
>  dracut:/# blkid
>  dracut:/#
>
>  Note that when booted from the thumb drive, vgchange finds the LVM
>  volumes just fine. 
Have you tried booting the previous kernel? It may be that the ramdisk
for the new kernel is missing the LVM stuff for some reason. If the old
kernel boots and finds things, bring it up and:

	dracut -f /boot/initramfs-<kernelversion>.fc23.x86_64.img <kernelversion>

where "<kernelversion>" is the desired (new) kernel version. Then try
the new kernel again. I've seen times where a new kernel install doesn't
build a correct initramfs image. Never sorted out why, but I've used the
above BFH on it and it seems to fix it (BFH = "big freaking hammer" for
those who were curious).
----------------------------------------------------------------------
- Rick Stevens, Systems Engineer, AllDigital    ricks(a)alldigital.com -
- AIM/Skype: therps2        ICQ: 226437340           Yahoo: origrps2 -
-                                                                    -
-                   To err is human, to moo bovine.                  -
----------------------------------------------------------------------

[CLOSE Dave]

I wrote:

>  I have five machines which were fresh-installed with F23 back in
>  February and all have been booted successfully a few times since.
>  Today, booting of all of them fails in exactly the same way: dracut
>  says it can't find the disk filesystems. The kernel boots as it
>  should, and of course that comes from the disk, but then dracut comes
>  along and says it can't find any of the filesystems. Not the root or
>  home filesystems which are on LVM or the boot filesystem on a primary
>  disk partition.
>
>  Everything on the disk is ok. I've checked by booting Anaconda from
>  a thumb drive and mounting manually. Anaconda troubleshooting mode
>  says it can't find the filesystems either ("you have no Linux
>  partitions"), but running vgchange -ay and a few mounts gets a proper
>  chroot image.
>
>  There must be something that is causing both dracut and Anaconda to
>  fail to find the filesystems. I've tried following the instructions
>  on <https://fedoraproject.org/wiki/How_to_debug_Dracut_problems>;, but
>  that isn't helping:
>
>  dracut:/# parted /dev/sda -s p
>  sh: parted: command not found
>  dracut:/# lvm vgscan
>  File descriptor 98 (/dev/console) leaked on lvm invocation. Parent PID 2679: sh
>  File descriptor 99 (/dev/console) leaked on lvm invocation. Parent PID 2679: sh
>  Reading all physical volumes. This may take a while...
>  dracut:/# lvm vgchange -ay
>  File descriptor 98 (/dev/console) leaked on lvm invocation. Parent PID 2679: sh
>  File descriptor 99 (/dev/console) leaked on lvm invocation. Parent PID 2679: sh
>  dracut:/# blkid
>  dracut:/#
>
>  Note that when booted from the thumb drive, vgchange finds the LVM
>  volumes just fine.
>  
On 04/07/16 05:03 PM, Rick Stevens answered:

>  Have you tried booting the previous kernel? It may be that the
>  ramdisk for the new kernel is missing the LVM stuff for some reason.
>  If the old kernel boots and finds things, bring it up and:
>
>  dracut -f /boot/initramfs-<kernelversion>.fc23.x86_64.img
>  <kernelversion>
>
>  where "<kernelversion>" is the desired (new) kernel version. Then
>  try the new kernel again. I've seen times where a new kernel install
>  doesn't build a correct initramfs image. Never sorted out why, but
>  I've used the above BFH on it and it seems to fix it (BFH = "big
>  freaking hammer" for those who were curious).
>  
An excellent suggestion, Rick. Thanks. Indeed, there was a new kernel 
installed on these machines. Forcing the previous one works.

That doesn't explain why Anaconda also couldn't find things. But that's 
not a concern now that boot works for me. Thanks, again!
-- 
Dave Close

[Rick Stevens]

On 04/07/2016 05:21 PM, CLOSE Dave wrote:
>  I wrote:
>
> > I have five machines which were fresh-installed with F23 back in
> > February and all have been booted successfully a few times since.
> > Today, booting of all of them fails in exactly the same way: dracut
> > says it can't find the disk filesystems. The kernel boots as it
> > should, and of course that comes from the disk, but then dracut comes
> > along and says it can't find any of the filesystems. Not the root or
> > home filesystems which are on LVM or the boot filesystem on a primary
> > disk partition.
> >
> > Everything on the disk is ok. I've checked by booting Anaconda from
> > a thumb drive and mounting manually. Anaconda troubleshooting mode
> > says it can't find the filesystems either ("you have no Linux
> > partitions"), but running vgchange -ay and a few mounts gets a proper
> > chroot image.
> >
> > There must be something that is causing both dracut and Anaconda to
> > fail to find the filesystems. I've tried following the instructions
> > on <https://fedoraproject.org/wiki/How_to_debug_Dracut_problems>;, but
> > that isn't helping:
> >
> > dracut:/# parted /dev/sda -s p
> > sh: parted: command not found
> > dracut:/# lvm vgscan
> > File descriptor 98 (/dev/console) leaked on lvm invocation. Parent PID 2679: sh
> > File descriptor 99 (/dev/console) leaked on lvm invocation. Parent PID 2679: sh
> > Reading all physical volumes. This may take a while...
> > dracut:/# lvm vgchange -ay
> > File descriptor 98 (/dev/console) leaked on lvm invocation. Parent PID 2679: sh
> > File descriptor 99 (/dev/console) leaked on lvm invocation. Parent PID 2679: sh
> > dracut:/# blkid
> > dracut:/#
> >
> > Note that when booted from the thumb drive, vgchange finds the LVM
> > volumes just fine.
>
>  On 04/07/16 05:03 PM, Rick Stevens answered:
>
> > Have you tried booting the previous kernel? It may be that the
> > ramdisk for the new kernel is missing the LVM stuff for some reason.
> > If the old kernel boots and finds things, bring it up and:
> >
> > dracut -f /boot/initramfs-<kernelversion>.fc23.x86_64.img
> > <kernelversion>
> >
> > where "<kernelversion>" is the desired (new) kernel version. Then
> > try the new kernel again. I've seen times where a new kernel install
> > doesn't build a correct initramfs image. Never sorted out why, but
> > I've used the above BFH on it and it seems to fix it (BFH = "big
> > freaking hammer" for those who were curious).
>
>  An excellent suggestion, Rick. Thanks. Indeed, there was a new kernel
>  installed on these machines. Forcing the previous one works.
>
>  That doesn't explain why Anaconda also couldn't find things. But that's
>  not a concern now that boot works for me. Thanks, again! 
Glad to help. Did the rebuild of the ramdisk work? Are you running the
new kernel? Just curious as this may happen on the next update, too.

I agree it's odd. I've seen it a few times--only on single machines
however. I've never had time to sort out why it failed. I'd just get
out the hammer, whack it and move along. Later I'd curse myself that I
didn't figure out what when wrong in the first place.
----------------------------------------------------------------------
- Rick Stevens, Systems Engineer, AllDigital    ricks(a)alldigital.com -
- AIM/Skype: therps2        ICQ: 226437340           Yahoo: origrps2 -
-                                                                    -
-          Consciousness: that annoying time between naps.           -
----------------------------------------------------------------------

[CLOSE Dave]

On 04/07/16 05:56 PM, Rick Stevens wrote:

>  Glad to help. Did the rebuild of the ramdisk work? Are you running
>  the new kernel? Just curious as this may happen on the next update,
>  too.
>  
I hope to have time for that tomorrow.
-- 
Dave Close

[CLOSE Dave]

I wrote:

>  I have five machines which were fresh-installed with F23 back in
>  February and all have been booted successfully a few times since.
>  Today, booting of all of them fails in exactly the same way: dracut
>  says it can't find the disk filesystems. The kernel boots as it
>  should, and of course that comes from the disk, but then dracut comes
>  along and says it can't find any of the filesystems. Not the root or
>  home filesystems which are on LVM or the boot filesystem on a primary
>  disk partition.
>
>  Everything on the disk is ok. I've checked by booting Anaconda from
>  a thumb drive and mounting manually. Anaconda troubleshooting mode
>  says it can't find the filesystems either ("you have no Linux
>  partitions"), but running vgchange -ay and a few mounts gets a proper
>  chroot image.
>
>  There must be something that is causing both dracut and Anaconda to
>  fail to find the filesystems. I've tried following the instructions
>  on <https://fedoraproject.org/wiki/How_to_debug_Dracut_problems>;, but
>  that isn't helping:
>
>  dracut:/# parted /dev/sda -s p
>  sh: parted: command not found
>  dracut:/# lvm vgscan
>  File descriptor 98 (/dev/console) leaked on lvm invocation. Parent PID 2679: sh
>  File descriptor 99 (/dev/console) leaked on lvm invocation. Parent PID 2679: sh
>  Reading all physical volumes. This may take a while...
>  dracut:/# lvm vgchange -ay
>  File descriptor 98 (/dev/console) leaked on lvm invocation. Parent PID 2679: sh
>  File descriptor 99 (/dev/console) leaked on lvm invocation. Parent PID 2679: sh
>  dracut:/# blkid
>  dracut:/#
>
>  Note that when booted from the thumb drive, vgchange finds the LVM
>  volumes just fine.
>  
On 04/07/16 05:03 PM, Rick Stevens answered:

>  Have you tried booting the previous kernel? It may be that the
>  ramdisk for the new kernel is missing the LVM stuff for some reason.
>  If the old kernel boots and finds things, bring it up and:
>
>  dracut -f /boot/initramfs-<kernelversion>.fc23.x86_64.img
>  <kernelversion>
>
>  where "<kernelversion>" is the desired (new) kernel version. Then
>  try the new kernel again. I've seen times where a new kernel install
>  doesn't build a correct initramfs image. Never sorted out why, but
>  I've used the above BFH on it and it seems to fix it (BFH = "big
>  freaking hammer" for those who were curious).
>  
Sorry it took me so long, Rick, but I did finally have a chance to try 
your suggestion. The new ramdisk created by the dracut command contained 
34 additional files that weren't in the original. Mostly stuff from 
/usr/lib/modules but also a few files from /usr/lib/firmware. Using the 
new ramdisk, the new kernel boots just like it should. So I can report 
that you have saved me. Thanks!

I suspect the problem comes from the fact that the machine on which the 
new kernel was built has a few differences from the one which would not 
boot. Both are x86_64 machines but the build machine does not have the 
Broadcom Ethernet ports that are in the target. Those were the firmware 
files missing. There is probably some way I can tweak the build to 
insure these files are initially included.
-- 
	Dave Close

[Rick Stevens]

On 04/19/2016 06:07 PM, CLOSE Dave wrote:
>  I wrote:
>
> > I have five machines which were fresh-installed with F23 back in
> > February and all have been booted successfully a few times since.
> > Today, booting of all of them fails in exactly the same way: dracut
> > says it can't find the disk filesystems. The kernel boots as it
> > should, and of course that comes from the disk, but then dracut comes
> > along and says it can't find any of the filesystems. Not the root or
> > home filesystems which are on LVM or the boot filesystem on a primary
> > disk partition.
> >
> > Everything on the disk is ok. I've checked by booting Anaconda from
> > a thumb drive and mounting manually. Anaconda troubleshooting mode
> > says it can't find the filesystems either ("you have no Linux
> > partitions"), but running vgchange -ay and a few mounts gets a proper
> > chroot image.
> >
> > There must be something that is causing both dracut and Anaconda to
> > fail to find the filesystems. I've tried following the instructions
> > on <https://fedoraproject.org/wiki/How_to_debug_Dracut_problems>;, but
> > that isn't helping:
> >
> > dracut:/# parted /dev/sda -s p
> > sh: parted: command not found
> > dracut:/# lvm vgscan
> > File descriptor 98 (/dev/console) leaked on lvm invocation. Parent PID 2679: sh
> > File descriptor 99 (/dev/console) leaked on lvm invocation. Parent PID 2679: sh
> > Reading all physical volumes. This may take a while...
> > dracut:/# lvm vgchange -ay
> > File descriptor 98 (/dev/console) leaked on lvm invocation. Parent PID 2679: sh
> > File descriptor 99 (/dev/console) leaked on lvm invocation. Parent PID 2679: sh
> > dracut:/# blkid
> > dracut:/#
> >
> > Note that when booted from the thumb drive, vgchange finds the LVM
> > volumes just fine.
>
>  On 04/07/16 05:03 PM, Rick Stevens answered:
>
> > Have you tried booting the previous kernel? It may be that the
> > ramdisk for the new kernel is missing the LVM stuff for some reason.
> > If the old kernel boots and finds things, bring it up and:
> >
> > dracut -f /boot/initramfs-<kernelversion>.fc23.x86_64.img
> > <kernelversion>
> >
> > where "<kernelversion>" is the desired (new) kernel version. Then
> > try the new kernel again. I've seen times where a new kernel install
> > doesn't build a correct initramfs image. Never sorted out why, but
> > I've used the above BFH on it and it seems to fix it (BFH = "big
> > freaking hammer" for those who were curious).
>
>  Sorry it took me so long, Rick, but I did finally have a chance to try
>  your suggestion. The new ramdisk created by the dracut command contained
>  34 additional files that weren't in the original. Mostly stuff from
>  /usr/lib/modules but also a few files from /usr/lib/firmware. Using the
>  new ramdisk, the new kernel boots just like it should. So I can report
>  that you have saved me. Thanks! 
No worries. When you see things like missing filesystems or the
inability to see disks, the initramfs is the first place to suspect
since you're probably missing the drivers needed.

Installing the kernel RPM typically does a dracut run--I didn't realize
that you had just copied the kernel and /lib/modules/<kernelversion>
tree from another system.

>  I suspect the problem comes from the fact that the machine on which
> the
>  new kernel was built has a few differences from the one which would not
>  boot. Both are x86_64 machines but the build machine does not have the
>  Broadcom Ethernet ports that are in the target. Those were the firmware
>  files missing. There is probably some way I can tweak the build to
>  insure these files are initially included. 
Yes, you can. You can add modules in two ways. On the command line via:

	dracut --<add-drivers|force-drivers> "module0 module1" \
	    /boot/initramfs-<kernelversion>.fc23.x86_64.img \
	    <kernelversion>

Or by entries in your /etc/dracut.conf or
/etc/dracut.conf.d/<somename>.conf files:

	<add_drivers|force_drivers>+=module1 module2 module3 ... moduleN

The module lists are space-separated names of modules without the ".ko"
suffix. On the command line, the list must be enclosed in double quotes.
In the config files, they don't need to be in quotes but they have to
be on one line.

The "add" version of the commands just includes them in the ramdisk,
while the "force" version is supposed to ensure they're loaded early in
the boot process by modprobe. I'd suggest the "force" version for disk
drivers. Network drivers typically can be loaded later. Up to you.
----------------------------------------------------------------------
- Rick Stevens, Systems Engineer, AllDigital    ricks(a)alldigital.com -
- AIM/Skype: therps2        ICQ: 226437340           Yahoo: origrps2 -
-                                                                    -
-    If Windows isn't a virus, then it sure as hell is a carrier!    -
----------------------------------------------------------------------

[Harald Hoyer]

On 20.04.2016 18:58, Rick Stevens wrote:
>  On 04/19/2016 06:07 PM, CLOSE Dave wrote:
> > I wrote:
> >
> >> I have five machines which were fresh-installed with F23 back in
> >> February and all have been booted successfully a few times since.
> >> Today, booting of all of them fails in exactly the same way: dracut
> >> says it can't find the disk filesystems. The kernel boots as it
> >> should, and of course that comes from the disk, but then dracut comes
> >> along and says it can't find any of the filesystems. Not the root or
> >> home filesystems which are on LVM or the boot filesystem on a primary
> >> disk partition.
> >>
> >> Everything on the disk is ok. I've checked by booting Anaconda from
> >> a thumb drive and mounting manually. Anaconda troubleshooting mode
> >> says it can't find the filesystems either ("you have no Linux
> >> partitions"), but running vgchange -ay and a few mounts gets a proper
> >> chroot image.
> >>
> >> There must be something that is causing both dracut and Anaconda to
> >> fail to find the filesystems. I've tried following the instructions
> >> on <https://fedoraproject.org/wiki/How_to_debug_Dracut_problems>;, but
> >> that isn't helping:
> >>
> >> dracut:/# parted /dev/sda -s p
> >> sh: parted: command not found
> >> dracut:/# lvm vgscan
> >> File descriptor 98 (/dev/console) leaked on lvm invocation. Parent PID 2679: sh
> >> File descriptor 99 (/dev/console) leaked on lvm invocation. Parent PID 2679: sh
> >> Reading all physical volumes. This may take a while...
> >> dracut:/# lvm vgchange -ay
> >> File descriptor 98 (/dev/console) leaked on lvm invocation. Parent PID 2679: sh
> >> File descriptor 99 (/dev/console) leaked on lvm invocation. Parent PID 2679: sh
> >> dracut:/# blkid
> >> dracut:/#
> >>
> >> Note that when booted from the thumb drive, vgchange finds the LVM
> >> volumes just fine.
> >
> > On 04/07/16 05:03 PM, Rick Stevens answered:
> >
> >> Have you tried booting the previous kernel? It may be that the
> >> ramdisk for the new kernel is missing the LVM stuff for some reason.
> >> If the old kernel boots and finds things, bring it up and:
> >>
> >> dracut -f /boot/initramfs-<kernelversion>.fc23.x86_64.img
> >> <kernelversion>
> >>
> >> where "<kernelversion>" is the desired (new) kernel version.
> Then
> >> try the new kernel again. I've seen times where a new kernel install
> >> doesn't build a correct initramfs image. Never sorted out why, but
> >> I've used the above BFH on it and it seems to fix it (BFH = "big
> >> freaking hammer" for those who were curious).
> >
> > Sorry it took me so long, Rick, but I did finally have a chance to try
> > your suggestion. The new ramdisk created by the dracut command contained
> > 34 additional files that weren't in the original. Mostly stuff from
> > /usr/lib/modules but also a few files from /usr/lib/firmware. Using the
> > new ramdisk, the new kernel boots just like it should. So I can report
> > that you have saved me. Thanks!
>  
>  No worries. When you see things like missing filesystems or the
>  inability to see disks, the initramfs is the first place to suspect
>  since you're probably missing the drivers needed.
>  
>  Installing the kernel RPM typically does a dracut run--I didn't realize
>  that you had just copied the kernel and /lib/modules/<kernelversion>
>  tree from another system.
>  
> > I suspect the problem comes from the fact that the machine on which the
> > new kernel was built has a few differences from the one which would not
> > boot. Both are x86_64 machines but the build machine does not have the
> > Broadcom Ethernet ports that are in the target. Those were the firmware
> > files missing. There is probably some way I can tweak the build to
> > insure these files are initially included.
>  
>  Yes, you can. You can add modules in two ways. On the command line via:
>  
>      dracut --<add-drivers|force-drivers> "module0 module1" \
>          /boot/initramfs-<kernelversion>.fc23.x86_64.img \
>          <kernelversion>
>  
>  Or by entries in your /etc/dracut.conf or
>  /etc/dracut.conf.d/<somename>.conf files:
>  
>      <add_drivers|force_drivers>+=module1 module2 module3 ... moduleN
>  
>  The module lists are space-separated names of modules without the ".ko"
>  suffix. On the command line, the list must be enclosed in double quotes.
>  In the config files, they don't need to be in quotes but they have to
>  be on one line.
>  
>  The "add" version of the commands just includes them in the ramdisk,
>  while the "force" version is supposed to ensure they're loaded early in
>  the boot process by modprobe. I'd suggest the "force" version for disk
>  drivers. Network drivers typically can be loaded later. Up to you. 
If you change hardware or want to use the initramfs on another machine,
it is easier to build a generic initramfs.

The generic mode can be activated by:
*        -N, --no-hostonly
           Disable Host-Only mode

or

* hostonly=no
  in a dracut configuration file

or

* by installing the dracut-config-generic rpm

[CLOSE Dave]

On 04/21/16 05:05 AM, Harald Hoyer wrote:

>  If you change hardware or want to use the initramfs on another
>  machine, it is easier to build a generic initramfs.
>
>  The generic mode can be activated by:
>  *        -N, --no-hostonly
>              Disable Host-Only mode
>
>  or
>
>  * hostonly=no
>     in a dracut configuration file
>
>  or
>
>  * by installing the dracut-config-generic rpm
>  
Wow, that's the biggest package I've ever installed! (Not!) But it does 
the trick. I confirm that a package built this way works as intended.
-- 
Dave Close