F17 boot with lvm does not create all the device nodes

[Deron Meranda]

I have an F17 server that will no longer boot.  When entering emergency
mode, it looks like not all of the device file nodes are being created.
 All of the physician volumes and volume groups are okay, but only some of
the logical volumes have their device files created.  In particular the
volume containing my /var filesystem doesn't show up.

When I enter 'lvs', all the logical volumes are listed, and they are
active.  These are all on one physical volume.

If I enter 'vgscan --mknodes' then the missing device files do get created.
 I can subsequently get to all my data (first using cryptsetup to decrypt
the LUKS volume that is stored inside the logical volume, and then mount).

However I can not get the system to continue booting beyond the emergency
mode, and upon the next reboot, again, some of the lvm device files are
missing.

Any ideas?

Thanks
-- 
Deron Meranda
http://deron.meranda.us/

[Deron Meranda]

Continuing the exploration of my problem...

Quick problem summary:  I could not boot past emergency mode because it
could not mount /var; which is a separate filesystem than /.  My /var
filesystem is a LUKS-encrypted logical volume, which happens to be on
/dev/dm-11.  Although the logical volume itself was present and active
(shows up in lvdisplay); the lvm device file was never getting created for
it.


It is the 'udev' subsystem that is responsible for creating the device
symlinks for all the logical volumes.  I suspect that my missing device
nodes were being excluded by the "/lib/udev/rules.d/10-dm.rules" file
because the "dmsetup info" command inside it was failing.  This then caused
the "11-dm-lvm.rules" file, which normally makes the decision about
creating the symlink, to skip that device.

When I ran "dmsetup info /dev/dm-11" it failed, saying the device didn't
exist!  Yet the device file "/dev/dm-11" was present, and clearly was
usable as I could manually access the data on it, by a process such as:

  ln -s /dev/vg_xyz/lv_var  /dev/dm-11
  cryptsetup luksOpen /dev/vg_xyz/lv_var /dev/mapper/luks-xyz
  mount /dev/mapper/luks-xyz /var

So /dev/dm-11 clearly works, yet "dmsetup info" was failing for it!


Since I could manually get /var mounted, and I fortunately had enough free
space, I went ahead copied the entire filesystem into /.  [Copying details
-- I first excluded any security-sensitive files as I was going transfering
to a non-encrypted filesystem, and copied the rest of the files while
preserving SELinux contexts, and also renaming mountpoints as appropriate
so that I ended up with /var on root being an identical copy of what used
to be in the separate /var filesystem.]

After removing /var from my fstab, a reboot worked completely.
 Furthermore, when I look into the /dev/mapper directory or the device
directory for the volume group I see ALL the expected device symlinks exist
- including my old /var as well as the other missing logical volumes.  Also
"dmsetup info" now works on all of them.


I am still quite curious as to what was going on, and am wondering if I can
go back to using a separate /var logical volume.

Deron



On Mon, Nov 18, 2013 at 2:50 AM, Deron Meranda <deron.meranda(a)gmail.com&gt;wrote:

>  Here's some more information.  It looks like all the LVM device
> files that
>  correspond to a device mapper minor number of 10 or greater are missing.  I
>  have four such devices (logical volumes).
>
>  After boot the lvm devices look like,
>
>  lvhome -> ../dm-9
>  lvroot -> ../dm-1
>  ...
>  and so on.
>
>  All the devices with minor numbers < 10 appear as I expect.  However all
>  the those with minor number >= 10 are missing.
>
>  If I query dm directly (dmsetup ls), or just look for all dm devices (ls
>  /dev/dm-*) then all of the devices show up.  It is only the symbolic links
>  for the volume group that are missing.
>
>  Is there a particular dm bug, or is this just a coincidence?   How can I
>  get the system to boot?
>
>  Thanks
>  --
>  Deron Meranda
>  http://deron.meranda.us/
>  


-- 
Deron Meranda
http://deron.meranda.us/

[Daniel J Walsh]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 11/18/2013 05:50 AM, Deron Meranda wrote:
>  Continuing the exploration of my problem...
>  
>  Quick problem summary:  I could not boot past emergency mode because it
>  could not mount /var; which is a separate filesystem than /.  My /var
>  filesystem is a LUKS-encrypted logical volume, which happens to be on
>  /dev/dm-11.  Although the logical volume itself was present and active
>  (shows up in lvdisplay); the lvm device file was never getting created for
>  it.
>  
>  
>  It is the 'udev' subsystem that is responsible for creating the device
>  symlinks for all the logical volumes.  I suspect that my missing device
>  nodes were being excluded by the "/lib/udev/rules.d/10-dm.rules" file
>  because the "dmsetup info" command inside it was failing.  This then caused
>  the "11-dm-lvm.rules" file, which normally makes the decision about
>  creating the symlink, to skip that device.
>  
>  When I ran "dmsetup info /dev/dm-11" it failed, saying the device didn't
>  exist! Yet the device file "/dev/dm-11" was present, and clearly was usable
>  as I could manually access the data on it, by a process such as:
>  
>  ln -s /dev/vg_xyz/lv_var  /dev/dm-11 cryptsetup luksOpen /dev/vg_xyz/lv_var
>  /dev/mapper/luks-xyz mount /dev/mapper/luks-xyz /var
>  
>  So /dev/dm-11 clearly works, yet "dmsetup info" was failing for it!
>  
>  
>  Since I could manually get /var mounted, and I fortunately had enough free 
>  space, I went ahead copied the entire filesystem into /.  [Copying details
>  -- I first excluded any security-sensitive files as I was going transfering
>  to a non-encrypted filesystem, and copied the rest of the files while
>  preserving SELinux contexts, and also renaming mountpoints as appropriate
>  so that I ended up with /var on root being an identical copy of what used
>  to be in the separate /var filesystem.]
>  
>  After removing /var from my fstab, a reboot worked completely.
>  Furthermore, when I look into the /dev/mapper directory or the device
>  directory for the volume group I see ALL the expected device symlinks exist
>  - including my old /var as well as the other missing logical volumes.  Also
>  "dmsetup info" now works on all of them.
>  
>  
>  I am still quite curious as to what was going on, and am wondering if I can
>  go back to using a separate /var logical volume.
>  
>  Deron
>  
>  
>  
>  On Mon, Nov 18, 2013 at 2:50 AM, Deron Meranda <deron.meranda(a)gmail.com 
>  <mailto:[e-mail]>> wrote:
>  
>  Here's some more information.  It looks like all the LVM device files that 
>  correspond to a device mapper minor number of 10 or greater are missing.
>  I have four such devices (logical volumes).
>  
>  After boot the lvm devices look like,
>  
>  lvhome -> ../dm-9 lvroot -> ../dm-1 ... and so on.
>  
>  All the devices with minor numbers < 10 appear as I expect.  However all
>  the those with minor number >= 10 are missing.
>  
>  If I query dm directly (dmsetup ls), or just look for all dm devices (ls 
>  /dev/dm-*) then all of the devices show up.  It is only the symbolic links 
>  for the volume group that are missing.
>  
>  Is there a particular dm bug, or is this just a coincidence?   How can I
>  get the system to boot?
>  
>  Thanks -- Deron Meranda http://deron.meranda.us/
>  
>  
>  
>  
>  -- Deron Meranda http://deron.meranda.us/
>  
>   
Fedora 17 is not longer supported.
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v1.4.15 (GNU/Linux)
> Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/
>
> iEYEARECAAYFAlKKG/MACgkQrlYvE4MpobPs2QCgr+MuIBqbUX/4qxYUA1ZBEkYs
> yrcAoM1Wcvni9x/95wXAP/8oJGF3KeFp
> =xY6p
> -----END PGP SIGNATURE-----

[Deron Meranda]

Here's some more information.  It looks like all the LVM device files that
correspond to a device mapper minor number of 10 or greater are missing.  I
have four such devices (logical volumes).

After boot the lvm devices look like,

lvhome -> ../dm-9
lvroot -> ../dm-1
...
and so on.

All the devices with minor numbers < 10 appear as I expect.  However all
the those with minor number >= 10 are missing.

If I query dm directly (dmsetup ls), or just look for all dm devices (ls
/dev/dm-*) then all of the devices show up.  It is only the symbolic links
for the volume group that are missing.

Is there a particular dm bug, or is this just a coincidence?   How can I
get the system to boot?

Thanks
-- 
Deron Meranda
http://deron.meranda.us/