Re: Update FC22 - doesn't find root fs and doesn't boot

[poma]

On 28.05.2015 14:19, Gerhard Wiesinger wrote:
> On 27.05.2015 15:27, Harald Hoyer wrote:
> > Am 27.05.2015 um 15:21 schrieb Gerhard Wiesinger:
> > > On 27.05.2015 15:07, Harald Hoyer wrote:
> > > > Am 27.05.2015 um 15:03 schrieb Gerhard Wiesinger:
> > > > > On 27.05.2015 14:43, Harald Hoyer wrote:
> > > > > > Am 27.05.2015 um 14:39 schrieb Gerhard Wiesinger:
> > > > > > > On 27.05.2015 14:25, Harald Hoyer wrote:
> > > > > > >> Am 27.05.2015 um 14:13 schrieb Andrei Borzenkov:
> > > > > > >>> On Wed, May 27, 2015 at 3:09 PM, Gerhard Wiesinger 
> > > > > > >>> wrote:
> > > > > > >>>> On 27.05.2015 13:56, Harald Hoyer wrote:
> > > > > > >>>>> Am 27.05.2015 um 13:28 schrieb Gerhard Wiesinger:
> > > > > > >>>>>> Hello,
> > > > > > >>>>>>
> > > > > > >>>>>> After upgrading one machine it does't boot anymore. Booting the old
> > > > > > >>>>>> kernel
> > > > > > >>>>>> works well.
> > > > > > >>>>>>
> > > > > > >>>>>> It is a RAID, LUKS, LVM setup. Find relevant details below.
> > > > > > >>>>>>
> > > > > > >>>>>> So it looks RAID assembling doesn't work any more.
> > > > > > >>>>>>
> > > > > > >>>>>> Any ideas?
> > > > > > >>>>>>
> > > > > > >>>>>> Thank you.
> > > > > > >>>>>>
> > > > > > >>>>>> Ciao,
> > > > > > >>>>>> Gerhard
> > > > > > >>>>>>
> > > > > > >>>>>> blkid
> > > > > > >>>>>> /dev/sda1: UUID="56e9bde4-5cd1-630d-188c-22a54c1c8c37"
> > > > > > >>>>>> TYPE="linux_raid_member"
> > > > > > >>>>>> PARTUUID="5ec6792c-01"
> > > > > > >>>>>> /dev/sdb1: UUID="56e9bde4-5cd1-630d-188c-22a54c1c8c37"
> > > > > > >>>>>> TYPE="linux_raid_member"
> > > > > > >>>>>> PARTUUID="1c7cffaf-01"
> > > > > > >>>>>> /dev/sdc1: UUID="56e9bde4-5cd1-630d-188c-22a54c1c8c37"
> > > > > > >>>>>> TYPE="linux_raid_member"
> > > > > > >>>>>> PARTUUID="0079dfc3-01"
> > > > > > >>>>>> /dev/sdd1: UUID="eb295015-e20b-e133-15db-5c9130487300"
> > > > > > >>>>>> TYPE="linux_raid_member"
> > > > > > >>>>>> PARTUUID="fadeaffe-01"
> > > > > > >>>>>> /dev/sde1: UUID="eb295015-e20b-e133-15db-5c9130487300"
> > > > > > >>>>>> TYPE="linux_raid_member"
> > > > > > >>>>>> PARTUUID="539b5814-01"
> > > > > > >>>>>> /dev/sdf1: UUID="eb295015-e20b-e133-15db-5c9130487300"
> > > > > > >>>>>> TYPE="linux_raid_member"
> > > > > > >>>>>> PARTUUID="badeaffe-01"
> > > > > > >>>>>> /dev/sdh1: UUID="51e8e983-8786-b868-dbc3-87357e1a9534"
> > > > > > >>>>>> TYPE="linux_raid_member"
> > > > > > >>>>>> PARTUUID="15631562-01"
> > > > > > >>>>>> /dev/sdg1: UUID="51e8e983-8786-b868-dbc3-87357e1a9534"
> > > > > > >>>>>> TYPE="linux_raid_member"
> > > > > > >>>>>> PARTUUID="162f0d0b-01"
> > > > > > >>>>>>
> > > > > > >>>>>>
> > > > > > >>>>>> blkid -o udev
> > > > > > >>>>>> ID_FS_UUID=56e9bde4-5cd1-630d-188c-22a54c1c8c37
> > > > > > >>>>>> ID_FS_UUID_ENC=56e9bde4-5cd1-630d-188c-22a54c1c8c37
> > > > > > >>>>>> ID_FS_TYPE=linux_raid_member
> > > > > > >>>>>> ID_FS_PARTUUID=5ec6792c-01
> > > > > > >>>>>>
> > > > > > >>>>>> ID_FS_UUID=56e9bde4-5cd1-630d-188c-22a54c1c8c37
> > > > > > >>>>>> ID_FS_UUID_ENC=56e9bde4-5cd1-630d-188c-22a54c1c8c37
> > > > > > >>>>>> ID_FS_TYPE=linux_raid_member
> > > > > > >>>>>> ID_FS_PARTUUID=1c7cffaf-01
> > > > > > >>>>>>
> > > > > > >>>>>> ID_FS_UUID=56e9bde4-5cd1-630d-188c-22a54c1c8c37
> > > > > > >>>>>> ID_FS_UUID_ENC=56e9bde4-5cd1-630d-188c-22a54c1c8c37
> > > > > > >>>>>> ID_FS_TYPE=linux_raid_member
> > > > > > >>>>>> ID_FS_PARTUUID=0079dfc3-01
> > > > > > >>>>>>
> > > > > > >>>>>> ID_FS_UUID=eb295015-e20b-e133-15db-5c9130487300
> > > > > > >>>>>> ID_FS_UUID_ENC=eb295015-e20b-e133-15db-5c9130487300
> > > > > > >>>>>> ID_FS_TYPE=linux_raid_member
> > > > > > >>>>>> ID_FS_PARTUUID=fadeaffe-01
> > > > > > >>>>>>
> > > > > > >>>>>> ID_FS_UUID=eb295015-e20b-e133-15db-5c9130487300
> > > > > > >>>>>> ID_FS_UUID_ENC=eb295015-e20b-e133-15db-5c9130487300
> > > > > > >>>>>> ID_FS_TYPE=linux_raid_member
> > > > > > >>>>>> ID_FS_PARTUUID=539b5814-01
> > > > > > >>>>>>
> > > > > > >>>>>> ID_FS_UUID=eb295015-e20b-e133-15db-5c9130487300
> > > > > > >>>>>> ID_FS_UUID_ENC=eb295015-e20b-e133-15db-5c9130487300
> > > > > > >>>>>> ID_FS_TYPE=linux_raid_member
> > > > > > >>>>>> ID_FS_PARTUUID=badeaffe-01
> > > > > > >>>>>>
> > > > > > >>>>>> ID_FS_UUID=51e8e983-8786-b868-dbc3-87357e1a9534
> > > > > > >>>>>> ID_FS_UUID_ENC=51e8e983-8786-b868-dbc3-87357e1a9534
> > > > > > >>>>>> ID_FS_TYPE=linux_raid_member
> > > > > > >>>>>> ID_FS_PARTUUID=15631562-01
> > > > > > >>>>>>
> > > > > > >>>>>> ID_FS_UUID=51e8e983-8786-b868-dbc3-87357e1a9534
> > > > > > >>>>>> ID_FS_UUID_ENC=51e8e983-8786-b868-dbc3-87357e1a9534
> > > > > > >>>>>> ID_FS_TYPE=linux_raid_member
> > > > > > >>>>>> ID_FS_PARTUUID=162f0d0b-01
> > > > > > >>>>>>
> > > > > > >>>>>> [    2.479836] myhost systemd[1]: Device
> > > > > > >>>>>> dev-disk-by\x2did-usb\x2dUSBDisk_RunDisk_ABCDEFGH\x2d0:0.device appeared
> > > > > > >>>>>> twice
> > > > > > >>>>>> with different sysfs paths /sys
> > > > > > >>>>>>
> > > > > > >>>>>> /devices/pci0000:00/0000:00:04.1/usb2/2-5/2-5:1.0/host9/target9:0:0/9:0:0:0/block/sdh
> > > > > > >>>>>>
> > > > > > >>>>>>
> > > > > > >>>>>>
> > > > > > >>>>>> and /sys/devices/pci0000:00/0000:00:02.1/usb1/1-5/1-5:1.0/host8/target
> > > > > > >>>>>> 8:0:0/8:0:0:0/block/sdg
> > > > > > >>>>>> [    2.500826] myhost systemd[1]: Device
> > > > > > >>>>>> dev-disk-by\x2did-usb\x2dUSBDisk_RunDisk_ABCDEFGH\x2d0:0\x2dpart1.device
> > > > > > >>>>>> appeared twice with different sysfs p
> > > > > > >>>>>> aths
> > > > > > >>>>>>
> > > > > > >>>>>> /sys/devices/pci0000:00/0000:00:04.1/usb2/2-5/2-5:1.0/host9/target9:0:0/9:0:0:0/block/sdh/sdh1
> > > > > > >>>>>>
> > > > > > >>>>>>
> > > > > > >>>>>>
> > > > > > >>>>>> and /sys/devices/pci0000:00/0000:00:02.1/usb1/1-5/1-5:1.
> > > > > > >>>>>> 0/host8/target8:0:0/8:0:0:0/block/sdg/sdg1
> > > > > > >>>>>> [    2.632754] myhost kernel: Switched to clocksource tsc
> > > > > > >>>>>> [   91.162986] myhost systemd[1]: Job dev-md0.device/start timed out.
> > > > > > >>>>>> [   91.163138] myhost systemd[1]: Timed out waiting for device
> > > > > > >>>>>> dev-md0.device.
> > > > > > >>>>>> [   91.163251] myhost systemd[1]: Dependency failed for Cryptography
> > > > > > >>>>>> Setup for
> > > > > > >>>>>> luks-md0.
> > > > > > >>>>>> [   91.163360] myhost systemd[1]: Dependency failed for
> > > > > > >>>>>> dev-mapper-luks\x2dmd0.device.
> > > > > > >>>>>> [   91.163474] myhost systemd[1]: Job
> > > > > > >>>>>> dev-mapper-luks\x2dmd0.device/start
> > > > > > >>>>>> failed with result 'dependency'.
> > > > > > >>>>>> [   91.163586] myhost systemd[1]: Dependency failed for Encrypted
> > > > > > >>>>>> Volumes.
> > > > > > >>>>>> [   91.163694] myhost systemd[1]: Job cryptsetup.target/start failed
> > > > > > >>>>>> with
> > > > > > >>>>>> result 'dependency'.
> > > > > > >>>>>> [   91.163801] myhost systemd[1]: Job
> > > > > > >>>>>> systemd-cryptsetup@luks\x2dmd0.service/start failed with result
> > > > > > >>>>>> 'dependency'.
> > > > > > >>>>>> [   91.163910] myhost systemd[1]: Job dev-md0.device/start failed with
> > > > > > >>>>>> result
> > > > > > >>>>>> 'timeout'.
> > > > > > >>>>>> [   91.164021] myhost systemd[1]: Reached target System Initialization.
> > > > > > >>>>>> [   91.164129] myhost systemd[1]: Starting System Initialization.
> > > > > > >>>>>> [   91.164236] myhost systemd[1]: Reached target Basic System.
> > > > > > >>>>>> [   91.164346] myhost systemd[1]: Starting Basic System.
> > > > > > >>>>>> [  185.740238] myhost dracut-initqueue[340]: Warning: Could not boot.
> > > > > > >>>>>> [  185.741266] myhost systemd[1]: Received SIGRTMIN+20 from PID 341
> > > > > > >>>>>> (plymouthd).
> > > > > > >>>>>> [  185.841257] myhost dracut-initqueue[340]: Warning:
> > > > > > >>>>>> /dev/VolGroup00/root does
> > > > > > >>>>>> not exist
> > > > > > >>>>>> [  185.841403] myhost systemd[1]: Starting Dracut Emergency Shell...
> > > > > > >>>>>> [  185.859366] myhost systemd[1]: Received SIGRTMIN+21 from PID 341
> > > > > > >>>>>> (plymouthd).
> > > > > > >>>>>>
> > > > > > >>>>>> dracut:/# cat "/run/initramfs/rdsosreport.txt"
> > > > > > >>>>>> + cat /lib/dracut/dracut-041-10.fc22.1
> > > > > > >>>>>> dracut-041-10.fc22.1
> > > > > > >>>>>> + cat /proc/cmdline
> > > > > > >>>>>> BOOT_IMAGE=/vmlinuz-4.0.4-301.fc22.x86_64 root=/dev/VolGroup00/root ro
> > > > > > >>>>>> quiet
> > > > > > >>>>>> console=ttyS0,115200n8 rd_NO_PLYMOUTH SYSFONT=latarcyrheb-sun16
> > > > > > >>>>>> LANG=en_US.UTF-
> > > > > > >>>>>> 8 KEYTABLE=de-latin1
> > > > > > >>>>> your kernel cmdline seems to be missing the raid configuration
> > > > > > >>>>>
> > > > > > >>>>> rd.md.uuid=56e9bde4-5cd1-630d-188c-22a54c1c8c37
> > > > > > >>>>> rd.md.uuid=eb295015-e20b-e133-15db-5c9130487300
> > > > > > >>>>> rd.md.uuid=51e8e983-8786-b868-dbc3-87357e1a9534
> > > > > > >>>>>
> > > > > > >>>>> ... whatever is needed to boot your system.
> > > > > > >>>>>
> > > > > > >>>>> # dracut --print-cmdline
> > > > > > >>>>>
> > > > > > >>>>> should tell you what kernel cmdline parameters are needed
> > > > > > >>>> Thank you for the fast reply.
> > > > > > >>>>
> > > > > > >>>> # comment: 3 separate lines!!!!! (maybe a problem)
> > > > > > >>>> dracut --print-cmdline
> > > > > > >>>>      rd.luks.uuid=luks-0c66fc77-a8a0-49e6-b96c-55e886a91f09
> > > > > > >>>> rd.lvm.lv=VolGroup00/root
> > > > > > >>>>      rd.lvm.lv=VolGroup00/swap
> > > > > > >>>>      rd.md.uuid=51e8e983:8786b868:dbc38735:7e1a9534
> > > > > > >>>> rd.md.uuid=56e9bde4:5cd1630d:188c22a5:4c1c8c37
> > > > > > >>>> resume=/dev/mapper/VolGroup00-swap root=/dev/mapper/VolGroup00-root
> > > > > > >>>> rootflags=rw,relatime,data=ordered rootfstype=ext4
> > > > > > >>>>
> > > > > > >>>> dracut --force 4.0.4-301.fc22.x86_64
> > > > > > >>>>
> > > > > > >>>> Still no correct entry in /etc/grub2.cfg
> > > > > > >>> I would be very surprised if dracut made any changes to grub.cfg. As I
> > > > > > >>> understand it, you either build host-specific dracut that will include
> > > > > > >>> this emulated command line or you have to add lines manually to your
> > > > > > >>> boot loader configuration.
> > > > > > >>>
> > > > > > >>>> linux16 /vmlinuz-4.0.4-301.fc22.x86_64 root=/dev/VolGroup00/root ro quiet
> > > > > > >>>> console=ttyS0,115200n8 rd_NO_PLYMOUTH SYSFONT=latarcyrheb-sun16
> > > > > > >>>> LANG=en_US.UTF-8 KEYTABLE=de-latin1
> > > > > > >>>>
> > > > > > >>>> Looks like a bug to me.
> > > > > > >> You have to add the options manually to the grub conf and all in the same
> > > > > > >> line.
> > > > > > >>
> > > > > > >> dracut does not modify your grub configuration.
> > > > > > > OK, I thought it was modifying it, at least in the past.
> > > > > > >
> > > > > > > Added it manually and it worked well.
> > > > > > >
> > > > > > > What's still strange:
> > > > > > >
> > > > > > > Last lines after startup, luks-md0 has already been started as the root fs is
> > > > > > > there ....
> > > > > > > Starting Cryptography Setup for luks-md0...
> > > > > > >             Starting Authorization Manager...
> > > > > > >
> > > > > > > Fedora release 22 (Twenty Two)
> > > > > > > Kernel 4.0.4-301.fc22.x86_64 on an x86_64 (ttyS0)
> > > > > > >
> > > > > > > # Rebooting, why is a passphrase necessary here?
> > > > > > > reboot
> > > > > > > Please enter passphrase for disk luks-md0!
> > > > > > >
> > > > > > > Possible bug here?
> > > > > > >
> > > > > > > Ciao,
> > > > > > > Gerhard
> > > > > > >
> > > > > > Did you add "  rd.luks.uuid=luks-0c66fc77-a8a0-49e6-b96c-55e886a91f09 " also?
> > > > >
> > > > > Yes, of course. :-)
> > > > > > What is your /etc/crypttab ?
> > > > >
> > > > > cat /etc/crypttab
> > > > > luks-md0                /dev/md0        none
> > > > > luks-md1                /dev/md1        /etc/crypto-key-md1
> > > > >
> > > > > BTW: Why did it work in previous versions without the long command line and
> > > > > predefined "hint" parameters?
> > > > >
> > > > > Ciao,
> > > > > Gerhard
> > > >
> > > > Oh... please, please do not use kernel names like "md0" ... better use UUID=
> > >
> > > changed it, but didn't help. This machine was updated since around FC5. :-)
> > >
> > > Booting:
> > > [FAILED] Failed to start Cryptography Setup for luks-md0.
> > > See "systemctl status 'systemd-cryptsetup@luks\x2dmd0.service'" for details.
> > > [DEPEND] Dependency failed for Encrypted Volumes.
> > > ...
> > > [FAILED] Failed to start Cryptography Setup for luks-md0.
> > > See "systemctl status 'systemd-cryptsetup@luks\x2dmd0.service'" for details.
> > > [DEPEND] Dependency failed for Encrypted Volumes.
> > > ...
> > >           Starting Cryptography Setup for luks-md0...
> > >           Starting Authorization Manager...
> > >
> > > Fedora release 22 (Twenty Two)
> > >
> > > cat /etc/crypttab
> > > luks-md0 UUID="0c66fc77-a8a0-49e6-b96c-55e886a91f09"        none
> > > luks-md1 UUID="aaa6ec1e-e54f-4864-a793-cc95d5fa59d2" /etc/crypto-key-md1
> > >
> > > Ciao,
> > > Gerhard
> >
> > Does "dracut -f" help to get the update crypttab in the initramfs?
>
> Removing quotes from the UUID helped.
>
> Thank you.
>
> Ciao,
> Gerhard

Without even mention these UUIDs spaghetti.

$ grep ^kernel /etc/dracut.conf.d/custom.conf
kernel_cmdline="rd.auto=1"

# lsinitrd -f

$ lsinitrd -f etc/cmdline.d/01-default.conf
rd.auto=1

This is -not- necessary for MD/RAID setups on Fedora 21:
$ rpm -q dracut
dracut-038-39.git20150518.fc21.x86_64

But it has become -mandatory- on MD/RAID setups on Fedora 22 and Fedora 23:
$ rpm -q dracut
dracut-041-10.fc22.1.x86_64

$ rpm -q dracut
dracut-041-10.git20150219.fc23.x86_64

And already we have fought for this user's side simplification, but it seems developers are very persistent when they do not need to be.

Update FC22 - doesn't find root fs and doesn't boot

[Bruno Wolff III]

> And already we have fought for this user's side simplification, but it seems developers are very persistent when they do not need to be.

In the past changes with this effect were reverted. This time they were not. 
This generally just affects people upgrading, as anaconda has been adding 
the needed stuff on the kernel command line for a while now.

You can also hostonly_cmdline=yes to the dracut config or set hostonly=no 
if you are willing to have a larger initramfs file. The latter will also help 
if you switch out hardware.

I am not sure why hostonly_cmdline=yes is not the default when hostonly=yes. 
That would seem to be a better default.

Update FC22 - doesn't find root fs and doesn't boot

[poma]

On 31.05.2015 20:08, Bruno Wolff III wrote:
> > And already we have fought for this user's side simplification, but it seems developers are very persistent when they do not need to be.
>
> In the past changes with this effect were reverted. This time they were not. 
> This generally just affects people upgrading, as anaconda has been adding 
> the needed stuff on the kernel command line for a while now.
>
> You can also hostonly_cmdline=yes to the dracut config or set hostonly=no 
> if you are willing to have a larger initramfs file. The latter will also help 
> if you switch out hardware.
>
> I am not sure why hostonly_cmdline=yes is not the default when hostonly=yes. 
> That would seem to be a better default.

dracut.conf.d/fedora.conf.example
+hostonly_cmdline="no"

http://pkgs.fedoraproject.org/cgit/dracut.git/commit/0001-fedora.conf-do-not...
http://git.kernel.org/cgit/boot/dracut/dracut.git/commit/dracut.conf.d/fedor...

no explanation why this was done

Update FC22 - doesn't find root fs and doesn't boot

[Harald Hoyer]

On 11.06.2015 11:06, poma wrote:
> On 31.05.2015 20:08, Bruno Wolff III wrote:
> > > And already we have fought for this user's side simplification, but it seems developers are very persistent when they do not need to be.
> >
> > In the past changes with this effect were reverted. This time they were not. 
> > This generally just affects people upgrading, as anaconda has been adding 
> > the needed stuff on the kernel command line for a while now.
> >
> > You can also hostonly_cmdline=yes to the dracut config or set hostonly=no 
> > if you are willing to have a larger initramfs file. The latter will also help 
> > if you switch out hardware.
> >
> > I am not sure why hostonly_cmdline=yes is not the default when hostonly=yes. 
> > That would seem to be a better default.
>
> dracut.conf.d/fedora.conf.example
> +hostonly_cmdline="no"
>
> http://pkgs.fedoraproject.org/cgit/dracut.git/commit/0001-fedora.conf-do-not...
> http://git.kernel.org/cgit/boot/dracut/dracut.git/commit/dracut.conf.d/fedor...
>
> no explanation why this was done

Basically it's because compiled in arguments are hard to override and people
could not boot after they changed disk configuration.

I guess, s.th. like "rd.cmdline=clean rd.auto" or "rd.helpIwanttobootnow" is
the better solution.

Update FC22 - doesn't find root fs and doesn't boot

[poma]

On 15.06.2015 11:42, Harald Hoyer wrote:
> On 11.06.2015 11:06, poma wrote:
> > On 31.05.2015 20:08, Bruno Wolff III wrote:
> > > > And already we have fought for this user's side simplification, but it seems developers are very persistent when they do not need to be.
> > >
> > > In the past changes with this effect were reverted. This time they were not. 
> > > This generally just affects people upgrading, as anaconda has been adding 
> > > the needed stuff on the kernel command line for a while now.
> > >
> > > You can also hostonly_cmdline=yes to the dracut config or set hostonly=no 
> > > if you are willing to have a larger initramfs file. The latter will also help 
> > > if you switch out hardware.
> > >
> > > I am not sure why hostonly_cmdline=yes is not the default when hostonly=yes. 
> > > That would seem to be a better default.
> >
> > dracut.conf.d/fedora.conf.example
> > +hostonly_cmdline="no"
> >
> > http://pkgs.fedoraproject.org/cgit/dracut.git/commit/0001-fedora.conf-do-not...
> > http://git.kernel.org/cgit/boot/dracut/dracut.git/commit/dracut.conf.d/fedor...
> >
> > no explanation why this was done
>
> Basically it's because compiled in arguments are hard to override and people
> could not boot after they changed disk configuration.
>
> I guess, s.th. like "rd.cmdline=clean rd.auto" or "rd.helpIwanttobootnow" is
> the better solution.

rd.OhLordwontyoubuymeaMercedesBenzMyfriendsalldrivePorschesImustmakeamends