I have an F17 server that will no longer boot. When entering emergency mode, it looks like not all of the device file nodes are being created. All of the physician volumes and volume groups are okay, but only some of the logical volumes have their device files created. In particular the volume containing my /var filesystem doesn't show up.
When I enter 'lvs', all the logical volumes are listed, and they are active. These are all on one physical volume.
If I enter 'vgscan --mknodes' then the missing device files do get created. I can subsequently get to all my data (first using cryptsetup to decrypt the LUKS volume that is stored inside the logical volume, and then mount).
However I can not get the system to continue booting beyond the emergency mode, and upon the next reboot, again, some of the lvm device files are missing.
Any ideas?
Thanks
Here's some more information. It looks like all the LVM device files that correspond to a device mapper minor number of 10 or greater are missing. I have four such devices (logical volumes).
After boot the lvm devices look like,
lvhome -> ../dm-9 lvroot -> ../dm-1 ... and so on.
All the devices with minor numbers < 10 appear as I expect. However all the those with minor number >= 10 are missing.
If I query dm directly (dmsetup ls), or just look for all dm devices (ls /dev/dm-*) then all of the devices show up. It is only the symbolic links for the volume group that are missing.
Is there a particular dm bug, or is this just a coincidence? How can I get the system to boot?
Thanks
Continuing the exploration of my problem...
Quick problem summary: I could not boot past emergency mode because it could not mount /var; which is a separate filesystem than /. My /var filesystem is a LUKS-encrypted logical volume, which happens to be on /dev/dm-11. Although the logical volume itself was present and active (shows up in lvdisplay); the lvm device file was never getting created for it.
It is the 'udev' subsystem that is responsible for creating the device symlinks for all the logical volumes. I suspect that my missing device nodes were being excluded by the "/lib/udev/rules.d/10-dm.rules" file because the "dmsetup info" command inside it was failing. This then caused the "11-dm-lvm.rules" file, which normally makes the decision about creating the symlink, to skip that device.
When I ran "dmsetup info /dev/dm-11" it failed, saying the device didn't exist! Yet the device file "/dev/dm-11" was present, and clearly was usable as I could manually access the data on it, by a process such as:
ln -s /dev/vg_xyz/lv_var /dev/dm-11 cryptsetup luksOpen /dev/vg_xyz/lv_var /dev/mapper/luks-xyz mount /dev/mapper/luks-xyz /var
So /dev/dm-11 clearly works, yet "dmsetup info" was failing for it!
Since I could manually get /var mounted, and I fortunately had enough free space, I went ahead copied the entire filesystem into /. [Copying details -- I first excluded any security-sensitive files as I was going transfering to a non-encrypted filesystem, and copied the rest of the files while preserving SELinux contexts, and also renaming mountpoints as appropriate so that I ended up with /var on root being an identical copy of what used to be in the separate /var filesystem.]
After removing /var from my fstab, a reboot worked completely. Furthermore, when I look into the /dev/mapper directory or the device directory for the volume group I see ALL the expected device symlinks exist - including my old /var as well as the other missing logical volumes. Also "dmsetup info" now works on all of them.
I am still quite curious as to what was going on, and am wondering if I can go back to using a separate /var logical volume.
Deron
On Mon, Nov 18, 2013 at 2:50 AM, Deron Meranda deron.meranda@gmail.comwrote:
Here's some more information. It looks like all the LVM device files that correspond to a device mapper minor number of 10 or greater are missing. I have four such devices (logical volumes).
After boot the lvm devices look like,
lvhome -> ../dm-9 lvroot -> ../dm-1 ... and so on.
All the devices with minor numbers < 10 appear as I expect. However all the those with minor number >= 10 are missing.
If I query dm directly (dmsetup ls), or just look for all dm devices (ls /dev/dm-*) then all of the devices show up. It is only the symbolic links for the volume group that are missing.
Is there a particular dm bug, or is this just a coincidence? How can I get the system to boot?
Thanks
Deron Meranda http://deron.meranda.us/
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
On 11/18/2013 05:50 AM, Deron Meranda wrote:
Continuing the exploration of my problem...
Quick problem summary: I could not boot past emergency mode because it could not mount /var; which is a separate filesystem than /. My /var filesystem is a LUKS-encrypted logical volume, which happens to be on /dev/dm-11. Although the logical volume itself was present and active (shows up in lvdisplay); the lvm device file was never getting created for it.
It is the 'udev' subsystem that is responsible for creating the device symlinks for all the logical volumes. I suspect that my missing device nodes were being excluded by the "/lib/udev/rules.d/10-dm.rules" file because the "dmsetup info" command inside it was failing. This then caused the "11-dm-lvm.rules" file, which normally makes the decision about creating the symlink, to skip that device.
When I ran "dmsetup info /dev/dm-11" it failed, saying the device didn't exist! Yet the device file "/dev/dm-11" was present, and clearly was usable as I could manually access the data on it, by a process such as:
ln -s /dev/vg_xyz/lv_var /dev/dm-11 cryptsetup luksOpen /dev/vg_xyz/lv_var /dev/mapper/luks-xyz mount /dev/mapper/luks-xyz /var
So /dev/dm-11 clearly works, yet "dmsetup info" was failing for it!
Since I could manually get /var mounted, and I fortunately had enough free space, I went ahead copied the entire filesystem into /. [Copying details -- I first excluded any security-sensitive files as I was going transfering to a non-encrypted filesystem, and copied the rest of the files while preserving SELinux contexts, and also renaming mountpoints as appropriate so that I ended up with /var on root being an identical copy of what used to be in the separate /var filesystem.]
After removing /var from my fstab, a reboot worked completely. Furthermore, when I look into the /dev/mapper directory or the device directory for the volume group I see ALL the expected device symlinks exist
- including my old /var as well as the other missing logical volumes. Also
"dmsetup info" now works on all of them.
I am still quite curious as to what was going on, and am wondering if I can go back to using a separate /var logical volume.
Deron
On Mon, Nov 18, 2013 at 2:50 AM, Deron Meranda <deron.meranda@gmail.com mailto:deron.meranda@gmail.com> wrote:
Here's some more information. It looks like all the LVM device files that correspond to a device mapper minor number of 10 or greater are missing. I have four such devices (logical volumes).
After boot the lvm devices look like,
lvhome -> ../dm-9 lvroot -> ../dm-1 ... and so on.
All the devices with minor numbers < 10 appear as I expect. However all the those with minor number >= 10 are missing.
If I query dm directly (dmsetup ls), or just look for all dm devices (ls /dev/dm-*) then all of the devices show up. It is only the symbolic links for the volume group that are missing.
Is there a particular dm bug, or is this just a coincidence? How can I get the system to boot?
Thanks -- Deron Meranda http://deron.meranda.us/
-- Deron Meranda http://deron.meranda.us/
Fedora 17 is not longer supported.
-
Daniel J Walsh -
Deron Meranda