Bricked system after migrating to thin LVMs

[Sreyan Chakravarty]

Hi,

I have cloned my original root LVM, and then migrated to the newer thin
LVMs using partclone.

Both backup and restore went off without causing any problems.

But when I am now trying to boot into my system, it asks for the encryption
LUKS passphrase but after that I am getting:

dracut-initqueue[300]: Warning: dracut-initqueue timeout  -
starting timeout scripts

I have not changed anything. Even my logical volume name is the same as
before.

These are the backup and restore commands that I have used from the Fedora
live environment:
        (Creating the backup)
         #   partclone.ext4 -c -s /dev/vgfedora/fedora | gzip -c --best >
/run/media/liveuser/externalhdd/fedora.tar.gz
         (Restoring the backup)
        #  gzip -c -d /run/media/externalhdd/fedora.tar.gz | partclone.ext4
-r -o /dev/vgfedora/fedora

These commands were run as root from Fedora live environment and prior to
taking my backup I had set my logical volume to read-only via:

#  lvchange -pr vgfedora/fedora

This is my current volume group and logical volumes:

#  lvs

LV     VG       Attr       LSize   Pool   Origin Data%  Meta%  Move Log
Cpy%Sync Convert
  fedora vgfedora Vwi-a-tz-- 700.00g pool00        16.61

pool00 vgfedora twi-aotz-- 929.75g               12.50  16.61

#  vgs

VG       #PV #LV #SN Attr   VSize   VFree
  vgfedora   1   2   0 wz--n- 929.99g 8.00m

This is my /etc/fstab:

/dev/mapper/vgfedora-fedora /                       ext4
 defaults,x-systemd.device-timeout=0 1 1
UUID=4d112d8e-080d-44f2-8962-9c13f20a7885 /boot                   ext4
 defaults        1 2
UUID=041A-E511          /boot/efi               vfat
 defaults,uid=0,gid=0,umask=077,shortname=winnt 0 2
/fedora.swap none swap defaults 0 2

I have not changed the /etc/fstab after the migration since nothing has
changed.
/dev/mapper/vgfedora-fedora still exists. The only difference is that it is
a thin LVM volume.

This is my GRUB_CMDLINE:

GRUB_CMDLINE_LINUX="rd.luks.uuid=luks-2ec7f1ae-6f9b-4896-a7b2-be7809e9d2f4
rd.lvm.lv=vgfedora/fedora
resume=/dev/disk/by-uuid/03aef3ba-dca1-4cba-a3f5-36c5c0fe948e
resume_offset=1449984"

The UUID of the LUKS encryption is still the same, as I can confirm from
here :

#  ll /dev/disk/by-uuid/
                 #  lrwxrwxrwx. 1 root root 10 Nov 24 03:38
2ec7f1ae-6f9b-4896-a7b2-be7809e9d2f4 -> ../../sda3

I don't know what else I have to change, or what is going wrong. Grub is
asking for the password for LUKS and is setting up the encrypted partition,
but after that nothing happens.

I guess it is not able to load the root logical volume.

Do I have to re-generate my initramfs ? I didn't think that would be
necessary since the LVM thin tools were already installed.

Please let me know if any other info is required from my side.

Need some help as I am unable to use my laptop.

[Sreyan Chakravarty]

Ok,

I think I have found the problem:

Nov 24 10:44:00 localhost.HPNotebook dracut-initqueue[861]: Scanning
devices dm-0  for LVM logical volumes vgfedora/fedora
Nov 24 10:44:00 localhost.HPNotebook dracut-initqueue[886]: inactive
'/dev/vgfedora/pool00' [929.75 GiB] inherit
Nov 24 10:44:00 localhost.HPNotebook dracut-initqueue[886]: inactive
'/dev/vgfedora/fedora' [700.00 GiB] inherit
Nov 24 10:44:00 localhost.HPNotebook dracut-initqueue[888]: modprobe:
FATAL: Module dm-thin-pool not found in directory
/lib/modules/5.8.18-200.fc32.x86_64
Nov 24 10:44:00 localhost.HPNotebook dracut-initqueue[888]:
/usr/sbin/modprobe failed: 1
Nov 24 10:44:00 localhost.HPNotebook dracut-initqueue[888]: Can't process
LV vgfedora/fedora: thin target support missing from kernel?

But I have no idea how to solve it.

I have marked the logical volumes as active via :

lvchange -kn vgfedora/pool00
             lvchange -kn vgfedora/fedora

But no change.

How will I add the dm-thin-pool module ?

I cannot even boot.

On Tue, Nov 24, 2020 at 2:33 PM Sreyan Chakravarty 
wrote:
> Hi,
>
> I have cloned my original root LVM, and then migrated to the newer thin
> LVMs using partclone.
>
> Both backup and restore went off without causing any problems.
>
> But when I am now trying to boot into my system, it asks for the
> encryption LUKS passphrase but after that I am getting:
>           dracut-initqueue[300]: Warning: dracut-initqueue timeout  -
>
> starting timeout scripts
>
> I have not changed anything. Even my logical volume name is the same as
> before.
>
> These are the backup and restore commands that I have used from the Fedora
> live environment:
>         (Creating the backup)
>          #   partclone.ext4 -c -s /dev/vgfedora/fedora | gzip -c --best >
> /run/media/liveuser/externalhdd/fedora.tar.gz
>          (Restoring the backup)
>         #  gzip -c -d /run/media/externalhdd/fedora.tar.gz |
> partclone.ext4 -r -o /dev/vgfedora/fedora
>
> These commands were run as root from Fedora live environment and prior to
> taking my backup I had set my logical volume to read-only via:
>            #  lvchange -pr vgfedora/fedora
>
> This is my current volume group and logical volumes:
>
> #  lvs
>
> LV     VG       Attr       LSize   Pool   Origin Data%  Meta%  Move Log
> Cpy%Sync Convert
>   fedora vgfedora Vwi-a-tz-- 700.00g pool00        16.61
>
> pool00 vgfedora twi-aotz-- 929.75g               12.50  16.61
>
> #  vgs
>
> VG       #PV #LV #SN Attr   VSize   VFree
>   vgfedora   1   2   0 wz--n- 929.99g 8.00m
>
> This is my /etc/fstab:
>
> /dev/mapper/vgfedora-fedora /                       ext4
>  defaults,x-systemd.device-timeout=0 1 1
> UUID=4d112d8e-080d-44f2-8962-9c13f20a7885 /boot                   ext4
>  defaults        1 2
> UUID=041A-E511          /boot/efi               vfat
>  defaults,uid=0,gid=0,umask=077,shortname=winnt 0 2
> /fedora.swap none swap defaults 0 2
>
> I have not changed the /etc/fstab after the migration since nothing has
> changed.
> /dev/mapper/vgfedora-fedora still exists. The only difference is that it
> is a thin LVM volume.
>
> This is my GRUB_CMDLINE:
>
> GRUB_CMDLINE_LINUX="rd.luks.uuid=luks-2ec7f1ae-6f9b-4896-a7b2-be7809e9d2f4
> rd.lvm.lv=vgfedora/fedora
> resume=/dev/disk/by-uuid/03aef3ba-dca1-4cba-a3f5-36c5c0fe948e
> resume_offset=1449984"
>
> The UUID of the LUKS encryption is still the same, as I can confirm from
> here :
>              #  ll /dev/disk/by-uuid/
>              #  lrwxrwxrwx. 1 root root 10 Nov 24 03:38
>
> 2ec7f1ae-6f9b-4896-a7b2-be7809e9d2f4 -> ../../sda3
>
> I don't know what else I have to change, or what is going wrong. Grub is
> asking for the password for LUKS and is setting up the encrypted partition,
> but after that nothing happens.
>
> I guess it is not able to load the root logical volume.
>
> Do I have to re-generate my initramfs ? I didn't think that would be
> necessary since the LVM thin tools were already installed.
>
> Please let me know if any other info is required from my side.
> Need some help as I am unable to use my laptop.
>
> Regards,
> Sreyan Chakravarty

[Ed Greshko]

On 24/11/2020 20:19, Sreyan Chakravarty wrote:
> I think I have found the problem:
>
> Nov 24 10:44:00 localhost.HPNotebook dracut-initqueue[861]: Scanning devices dm-0  for LVM logical volumes vgfedora/fedora
> Nov 24 10:44:00 localhost.HPNotebook dracut-initqueue[886]: inactive '/dev/vgfedora/pool00' [929.75 GiB] inherit
> Nov 24 10:44:00 localhost.HPNotebook dracut-initqueue[886]: inactive '/dev/vgfedora/fedora' [700.00 GiB] inherit
> Nov 24 10:44:00 localhost.HPNotebook dracut-initqueue[888]: modprobe: FATAL: Module dm-thin-pool not found in directory /lib/modules/5.8.18-200.fc32.x86_64
> Nov 24 10:44:00 localhost.HPNotebook dracut-initqueue[888]: /usr/sbin/modprobe failed: 1
> Nov 24 10:44:00 localhost.HPNotebook dracut-initqueue[888]: Can't process LV vgfedora/fedora: thin target support missing from kernel?
>
> But I have no idea how to solve it.
>
> I have marked the logical volumes as active via :
>
> lvchange -kn vgfedora/pool00
>              lvchange -kn vgfedora/fedora
>
> But no change.
>
> How will I add the dm-thin-pool module ?
>
> I cannot even boot.

Well, the module is located here in my f32 systems.

[egreshko@f32k ~]$ locate dm-thin-pool
/usr/lib/modules/5.8.13-200.fc32.x86_64/kernel/drivers/md/dm-thin-pool.ko.xz
/usr/lib/modules/5.8.14-200.fc32.x86_64/kernel/drivers/md/dm-thin-pool.ko.xz
/usr/lib/modules/5.9.8-100.fc32.x86_64/kernel/drivers/md/dm-thin-pool.ko.xz

You can try adding modules_load=dm-thin-pool to the kernel parameters.

---
The key to getting good answers is to ask good questions.

[Roger Heflin]

If you have a "rescue" option in grub then boot it, the rescue install
should be built with hostonly=no and is expected to include all
drivers including that one.  I verified on one of my machines that one
is included in my rescue.   The normal initrd is hostonly=yes and only
includes the critical modules needed to boot the system, and thin was
not previously need and so not included.

Once booted rebuild the other initrd and that should include the thin
module now.

lsinitrd <initrdname> | grep -i thin    should show you if it gets put
into the initrd after the rebuild and can be used to confirm that it
is not in any initramfs except rescue.

On Tue, Nov 24, 2020 at 6:19 AM Sreyan Chakravarty  wrote:
> Ok,
>
> I think I have found the problem:
>
> Nov 24 10:44:00 localhost.HPNotebook dracut-initqueue[861]: Scanning devices dm-0  for LVM logical volumes vgfedora/fedora
> Nov 24 10:44:00 localhost.HPNotebook dracut-initqueue[886]: inactive '/dev/vgfedora/pool00' [929.75 GiB] inherit
> Nov 24 10:44:00 localhost.HPNotebook dracut-initqueue[886]: inactive '/dev/vgfedora/fedora' [700.00 GiB] inherit
> Nov 24 10:44:00 localhost.HPNotebook dracut-initqueue[888]: modprobe: FATAL: Module dm-thin-pool not found in directory /lib/modules/5.8.18-200.fc32.x86_64
> Nov 24 10:44:00 localhost.HPNotebook dracut-initqueue[888]: /usr/sbin/modprobe failed: 1
> Nov 24 10:44:00 localhost.HPNotebook dracut-initqueue[888]: Can't process LV vgfedora/fedora: thin target support missing from kernel?
>
> But I have no idea how to solve it.
>
> I have marked the logical volumes as active via :
>          lvchange -kn vgfedora/pool00
>          lvchange -kn vgfedora/fedora
>
> But no change.
>
> How will I add the dm-thin-pool module ?
>
> I cannot even boot.
>
> On Tue, Nov 24, 2020 at 2:33 PM Sreyan Chakravarty  wrote:
> > Hi,
> >
> > I have cloned my original root LVM, and then migrated to the newer thin LVMs using partclone.
> >
> > Both backup and restore went off without causing any problems.
> >
> > But when I am now trying to boot into my system, it asks for the encryption LUKS passphrase but after that I am getting:
> >           dracut-initqueue[300]: Warning: dracut-initqueue timeout  -  starting timeout scripts
> >
> > I have not changed anything. Even my logical volume name is the same as before.
> >
> > These are the backup and restore commands that I have used from the Fedora live environment:
> >         (Creating the backup)
> >          #   partclone.ext4 -c -s /dev/vgfedora/fedora | gzip -c --best > /run/media/liveuser/externalhdd/fedora.tar.gz
> >          (Restoring the backup)
> >         #  gzip -c -d /run/media/externalhdd/fedora.tar.gz | partclone.ext4 -r -o /dev/vgfedora/fedora
> >
> > These commands were run as root from Fedora live environment and prior to taking my backup I had set my logical volume to read-only via:
> >            #  lvchange -pr vgfedora/fedora
> >
> > This is my current volume group and logical volumes:
> >
> > #  lvs
> >
> > LV     VG       Attr       LSize   Pool   Origin Data%  Meta%  Move Log Cpy%Sync Convert
> >   fedora vgfedora Vwi-a-tz-- 700.00g pool00        16.61
> >   pool00 vgfedora twi-aotz-- 929.75g               12.50  16.61
> >
> > #  vgs
> >
> > VG       #PV #LV #SN Attr   VSize   VFree
> >   vgfedora   1   2   0 wz--n- 929.99g 8.00m
> >
> > This is my /etc/fstab:
> >
> > /dev/mapper/vgfedora-fedora /                       ext4    defaults,x-systemd.device-timeout=0 1 1
> > UUID=4d112d8e-080d-44f2-8962-9c13f20a7885 /boot                   ext4    defaults        1 2
> > UUID=041A-E511          /boot/efi               vfat    defaults,uid=0,gid=0,umask=077,shortname=winnt 0 2
> > /fedora.swap none swap defaults 0 2
> >
> > I have not changed the /etc/fstab after the migration since nothing has changed.
> > /dev/mapper/vgfedora-fedora still exists. The only difference is that it is a thin LVM volume.
> >
> > This is my GRUB_CMDLINE:
> >
> > GRUB_CMDLINE_LINUX="rd.luks.uuid=luks-2ec7f1ae-6f9b-4896-a7b2-be7809e9d2f4 rd.lvm.lv=vgfedora/fedora resume=/dev/disk/by-uuid/03aef3ba-dca1-4cba-a3f5-36c5c0fe948e resume_offset=1449984"
> >
> > The UUID of the LUKS encryption is still the same, as I can confirm from here :
> >              #  ll /dev/disk/by-uuid/
> >              #  lrwxrwxrwx. 1 root root 10 Nov 24 03:38 2ec7f1ae-6f9b-4896-a7b2-be7809e9d2f4 -> ../../sda3
> >
> > I don't know what else I have to change, or what is going wrong. Grub is asking for the password for LUKS and is setting up the encrypted partition, but after that nothing happens.
> >
> > I guess it is not able to load the root logical volume.
> >
> > Do I have to re-generate my initramfs ? I didn't think that would be necessary since the LVM thin tools were already installed.
> >
> > Please let me know if any other info is required from my side.
> > Need some help as I am unable to use my laptop.
> >
> > Regards,
> > Sreyan Chakravarty
>
> --
> Regards,
> Sreyan Chakravarty
> _______________________________________________
> users mailing list -- [e-mail]
> To unsubscribe send an email to [e-mail]
> Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives: https://lists.fedoraproject.org/archives/list/[e-mail]

[Sreyan Chakravarty]

On Tue, Nov 24, 2020 at 9:23 PM Roger Heflin  wrote:
> If you have a "rescue" option in grub then boot it, the rescue install
> should be built with hostonly=no and is expected to include all
> drivers including that one.  I verified on one of my machines that one
> is included in my rescue.   The normal initrd is hostonly=yes and only
> includes the critical modules needed to boot the system, and thin was
> not previously need and so not included.
>
> Once booted rebuild the other initrd and that should include the thin
> module now.
>
> lsinitrd <initrdname> | grep -i thin    should show you if it gets put
> into the initrd after the rebuild and can be used to confirm that it
> is not in any initramfs except rescue.

I went the dumb path.

I made a regular logical volume, used partclone to restore my back up and
then booted into the OS.

Once booted in I added the following to /etc/dracut.conf:

add_drivers+=" dm-thin-pool "

and then rebuild by initramfs by

dracut -f

and then booted into my other two fedora kernels and did the same.

After that, I created the thread pool and thin volumes again and restored
my backup again, and logged in.

[Sreyan Chakravarty]

On Tue, Nov 24, 2020 at 6:12 PM Ed Greshko  wrote:
> Well, the module is located here in my f32 systems.
>
> [egreshko@f32k ~]$ locate dm-thin-pool
>
> /usr/lib/modules/5.8.13-200.fc32.x86_64/kernel/drivers/md/dm-thin-pool.ko.xz
>
> /usr/lib/modules/5.8.14-200.fc32.x86_64/kernel/drivers/md/dm-thin-pool.ko.xz
> /usr/lib/modules/5.9.8-100.fc32.x86_64/kernel/drivers/md/dm-thin-pool.ko.xz
>
> You can try adding modules_load=dm-thin-pool to the kernel parameters.

I did not add it over here, instead restored my backups booted in and then
regenerated my initramfs after adding the parameters.

[Ed Greshko]

On 25/11/2020 16:42, Sreyan Chakravarty wrote:
> On Tue, Nov 24, 2020 at 6:12 PM Ed Greshko <[e-mail] > wrote:
> Well, the module is located here in my f32 systems.
>
> [egreshko@f32k ~]$ locate dm-thin-pool
> /usr/lib/modules/5.8.13-200.fc32.x86_64/kernel/drivers/md/dm-thin-pool.ko.xz
> /usr/lib/modules/5.8.14-200.fc32.x86_64/kernel/drivers/md/dm-thin-pool.ko.xz
> /usr/lib/modules/5.9.8-100.fc32.x86_64/kernel/drivers/md/dm-thin-pool.ko.xz
>
> You can try adding modules_load=dm-thin-pool to the kernel parameters.
>
>
> I did not add it over here, instead restored my backups booted in and then regenerated my initramfs after adding the parameters.

That was the correct way to resolve the issue.

I thought about that later, but didn't have the chance to make that suggestion.  Glad you figured it out.

---
The key to getting good answers is to ask good questions.