Bricked system after migrating to thin LVMs

[Sreyan Chakravarty]

Hi,

I have cloned my original root LVM, and then migrated to the newer thin
LVMs using partclone.

Both backup and restore went off without causing any problems.

But when I am now trying to boot into my system, it asks for the encryption
LUKS passphrase but after that I am getting:

              dracut-initqueue[300]: Warning: dracut-initqueue timeout  -
starting timeout scripts

I have not changed anything. Even my logical volume name is the same as
before.

These are the backup and restore commands that I have used from the Fedora
live environment:
        (Creating the backup)
         #   partclone.ext4 -c -s /dev/vgfedora/fedora | gzip -c --best >
/run/media/liveuser/externalhdd/fedora.tar.gz
         (Restoring the backup)
        #  gzip -c -d /run/media/externalhdd/fedora.tar.gz | partclone.ext4
-r -o /dev/vgfedora/fedora

These commands were run as root from Fedora live environment and prior to
taking my backup I had set my logical volume to read-only via:

               #  lvchange -pr vgfedora/fedora

This is my current volume group and logical volumes:

#  lvs

 LV     VG       Attr       LSize   Pool   Origin Data%  Meta%  Move Log
Cpy%Sync Convert
  fedora vgfedora Vwi-a-tz-- 700.00g pool00        16.61

  pool00 vgfedora twi-aotz-- 929.75g               12.50  16.61


#  vgs

  VG       #PV #LV #SN Attr   VSize   VFree
  vgfedora   1   2   0 wz--n- 929.99g 8.00m


This is my /etc/fstab:

/dev/mapper/vgfedora-fedora /                       ext4
 defaults,x-systemd.device-timeout=0 1 1
UUID=4d112d8e-080d-44f2-8962-9c13f20a7885 /boot                   ext4
 defaults        1 2
UUID=041A-E511          /boot/efi               vfat
 defaults,uid=0,gid=0,umask=077,shortname=winnt 0 2
/fedora.swap none swap defaults 0 2

I have not changed the /etc/fstab after the migration since nothing has
changed.
/dev/mapper/vgfedora-fedora still exists. The only difference is that it is
a thin LVM volume.

This is my GRUB_CMDLINE:

GRUB_CMDLINE_LINUX="rd.luks.uuid=luks-2ec7f1ae-6f9b-4896-a7b2-be7809e9d2f4
rd.lvm.lv=vgfedora/fedora
resume=/dev/disk/by-uuid/03aef3ba-dca1-4cba-a3f5-36c5c0fe948e
resume_offset=1449984"

The UUID of the LUKS encryption is still the same, as I can confirm from
here :

                 #  ll /dev/disk/by-uuid/
                 #  lrwxrwxrwx. 1 root root 10 Nov 24 03:38
2ec7f1ae-6f9b-4896-a7b2-be7809e9d2f4 -> ../../sda3

I don't know what else I have to change, or what is going wrong. Grub is
asking for the password for LUKS and is setting up the encrypted partition,
but after that nothing happens.

I guess it is not able to load the root logical volume.

Do I have to re-generate my initramfs ? I didn't think that would be
necessary since the LVM thin tools were already installed.

Please let me know if any other info is required from my side.

Need some help as I am unable to use my laptop.
-- 
Regards,
Sreyan Chakravarty

[Sreyan Chakravarty]

Ok,

I think I have found the problem:

Nov 24 10:44:00 localhost.HPNotebook dracut-initqueue[861]: Scanning
devices dm-0  for LVM logical volumes vgfedora/fedora
Nov 24 10:44:00 localhost.HPNotebook dracut-initqueue[886]: inactive
'/dev/vgfedora/pool00' [929.75 GiB] inherit
Nov 24 10:44:00 localhost.HPNotebook dracut-initqueue[886]: inactive
'/dev/vgfedora/fedora' [700.00 GiB] inherit
Nov 24 10:44:00 localhost.HPNotebook dracut-initqueue[888]: modprobe:
FATAL: Module dm-thin-pool not found in directory
/lib/modules/5.8.18-200.fc32.x86_64
Nov 24 10:44:00 localhost.HPNotebook dracut-initqueue[888]:
/usr/sbin/modprobe failed: 1
Nov 24 10:44:00 localhost.HPNotebook dracut-initqueue[888]: Can't process
LV vgfedora/fedora: thin target support missing from kernel?

But I have no idea how to solve it.

I have marked the logical volumes as active via :

             lvchange -kn vgfedora/pool00
             lvchange -kn vgfedora/fedora

But no change.

How will I add the dm-thin-pool module ?

I cannot even boot.


On Tue, Nov 24, 2020 at 2:33 PM Sreyan Chakravarty <sreyan32(a)gmail.com&gt;
wrote:

>  Hi,
>
>  I have cloned my original root LVM, and then migrated to the newer thin
>  LVMs using partclone.
>
>  Both backup and restore went off without causing any problems.
>
>  But when I am now trying to boot into my system, it asks for the
>  encryption LUKS passphrase but after that I am getting:
>
>                dracut-initqueue[300]: Warning: dracut-initqueue timeout  -
>  starting timeout scripts
>
>  I have not changed anything. Even my logical volume name is the same as
>  before.
>
>  These are the backup and restore commands that I have used from the Fedora
>  live environment:
>          (Creating the backup)
>           #   partclone.ext4 -c -s /dev/vgfedora/fedora | gzip -c --best >
>  /run/media/liveuser/externalhdd/fedora.tar.gz
>           (Restoring the backup)
>          #  gzip -c -d /run/media/externalhdd/fedora.tar.gz |
>  partclone.ext4 -r -o /dev/vgfedora/fedora
>
>  These commands were run as root from Fedora live environment and prior to
>  taking my backup I had set my logical volume to read-only via:
>
>                 #  lvchange -pr vgfedora/fedora
>
>  This is my current volume group and logical volumes:
>
>  #  lvs
>
>   LV     VG       Attr       LSize   Pool   Origin Data%  Meta%  Move Log
>  Cpy%Sync Convert
>    fedora vgfedora Vwi-a-tz-- 700.00g pool00        16.61
>
>    pool00 vgfedora twi-aotz-- 929.75g               12.50  16.61
>
>
>  #  vgs
>
>    VG       #PV #LV #SN Attr   VSize   VFree
>    vgfedora   1   2   0 wz--n- 929.99g 8.00m
>
>
>  This is my /etc/fstab:
>
>  /dev/mapper/vgfedora-fedora /                       ext4
>   defaults,x-systemd.device-timeout=0 1 1
>  UUID=4d112d8e-080d-44f2-8962-9c13f20a7885 /boot                   ext4
>   defaults        1 2
>  UUID=041A-E511          /boot/efi               vfat
>   defaults,uid=0,gid=0,umask=077,shortname=winnt 0 2
>  /fedora.swap none swap defaults 0 2
>
>  I have not changed the /etc/fstab after the migration since nothing has
>  changed.
>  /dev/mapper/vgfedora-fedora still exists. The only difference is that it
>  is a thin LVM volume.
>
>  This is my GRUB_CMDLINE:
>
>  GRUB_CMDLINE_LINUX="rd.luks.uuid=luks-2ec7f1ae-6f9b-4896-a7b2-be7809e9d2f4
>  rd.lvm.lv=vgfedora/fedora
>  resume=/dev/disk/by-uuid/03aef3ba-dca1-4cba-a3f5-36c5c0fe948e
>  resume_offset=1449984"
>
>  The UUID of the LUKS encryption is still the same, as I can confirm from
>  here :
>
>                   #  ll /dev/disk/by-uuid/
>                   #  lrwxrwxrwx. 1 root root 10 Nov 24 03:38
>  2ec7f1ae-6f9b-4896-a7b2-be7809e9d2f4 -> ../../sda3
>
>  I don't know what else I have to change, or what is going wrong. Grub is
>  asking for the password for LUKS and is setting up the encrypted partition,
>  but after that nothing happens.
>
>  I guess it is not able to load the root logical volume.
>
>  Do I have to re-generate my initramfs ? I didn't think that would be
>  necessary since the LVM thin tools were already installed.
>
>  Please let me know if any other info is required from my side.
>
>  Need some help as I am unable to use my laptop.
>  --
>  Regards,
>  Sreyan Chakravarty
>  

-- 
Regards,
Sreyan Chakravarty

[Ed Greshko]

On 24/11/2020 20:19, Sreyan Chakravarty wrote:
>  I think I have found the problem:
>
>  Nov 24 10:44:00 localhost.HPNotebook dracut-initqueue[861]: Scanning devices dm-0  for
> LVM logical volumes vgfedora/fedora
>  Nov 24 10:44:00 localhost.HPNotebook dracut-initqueue[886]: inactive
> '/dev/vgfedora/pool00' [929.75 GiB] inherit
>  Nov 24 10:44:00 localhost.HPNotebook dracut-initqueue[886]: inactive
> '/dev/vgfedora/fedora' [700.00 GiB] inherit
>  Nov 24 10:44:00 localhost.HPNotebook dracut-initqueue[888]: modprobe: FATAL: Module
> dm-thin-pool not found in directory /lib/modules/5.8.18-200.fc32.x86_64
>  Nov 24 10:44:00 localhost.HPNotebook dracut-initqueue[888]: /usr/sbin/modprobe failed: 1
>  Nov 24 10:44:00 localhost.HPNotebook dracut-initqueue[888]: Can't process LV
> vgfedora/fedora: thin target support missing from kernel?
>
>  But I have no idea how to solve it.
>
>  I have marked the logical volumes as active via :
>
>               lvchange -kn vgfedora/pool00
>               lvchange -kn vgfedora/fedora
>
>  But no change.
>
>  How will I add the dm-thin-pool module ?
>
>  I cannot even boot. 
Well, the module is located here in my f32 systems.

[egreshko@f32k ~]$ locate dm-thin-pool
/usr/lib/modules/5.8.13-200.fc32.x86_64/kernel/drivers/md/dm-thin-pool.ko.xz
/usr/lib/modules/5.8.14-200.fc32.x86_64/kernel/drivers/md/dm-thin-pool.ko.xz
/usr/lib/modules/5.9.8-100.fc32.x86_64/kernel/drivers/md/dm-thin-pool.ko.xz

You can try adding modules_load=dm-thin-pool to the kernel parameters.


---
The key to getting good answers is to ask good questions.

[Sreyan Chakravarty]

On Tue, Nov 24, 2020 at 6:12 PM Ed Greshko <ed.greshko(a)greshko.com&gt; wrote:

>  Well, the module is located here in my f32 systems.
>
>  [egreshko@f32k ~]$ locate dm-thin-pool
>
>  /usr/lib/modules/5.8.13-200.fc32.x86_64/kernel/drivers/md/dm-thin-pool.ko.xz
>
>  /usr/lib/modules/5.8.14-200.fc32.x86_64/kernel/drivers/md/dm-thin-pool.ko.xz
>  /usr/lib/modules/5.9.8-100.fc32.x86_64/kernel/drivers/md/dm-thin-pool.ko.xz
>
>  You can try adding modules_load=dm-thin-pool to the kernel parameters.
>  
I did not add it over here, instead restored my backups booted in and then
regenerated my initramfs after adding the parameters.
-- 
Regards,
Sreyan Chakravarty

[Ed Greshko]

On 25/11/2020 16:42, Sreyan Chakravarty wrote:
>  On Tue, Nov 24, 2020 at 6:12 PM Ed Greshko <ed.greshko(a)greshko.com
> <mailto:[e-mail]>> wrote:
>
>      Well, the module is located here in my f32 systems.
>
>      [egreshko@f32k ~]$ locate dm-thin-pool
>      /usr/lib/modules/5.8.13-200.fc32.x86_64/kernel/drivers/md/dm-thin-pool.ko.xz
>      /usr/lib/modules/5.8.14-200.fc32.x86_64/kernel/drivers/md/dm-thin-pool.ko.xz
>      /usr/lib/modules/5.9.8-100.fc32.x86_64/kernel/drivers/md/dm-thin-pool.ko.xz
>
>      You can try adding modules_load=dm-thin-pool to the kernel parameters.
>
>
>  I did not add it over here, instead restored my backups booted in and then regenerated my
> initramfs after adding the parameters.
>  
That was the correct way to resolve the issue.

I thought about that later, but didn't have the chance to make that suggestion.  Glad
you figured it out.

---
The key to getting good answers is to ask good questions.

[Roger Heflin]

If you have a "rescue" option in grub then boot it, the rescue install
should be built with hostonly=no and is expected to include all
drivers including that one.  I verified on one of my machines that one
is included in my rescue.   The normal initrd is hostonly=yes and only
includes the critical modules needed to boot the system, and thin was
not previously need and so not included.

Once booted rebuild the other initrd and that should include the thin
module now.

lsinitrd <initrdname> | grep -i thin    should show you if it gets put
into the initrd after the rebuild and can be used to confirm that it
is not in any initramfs except rescue.



On Tue, Nov 24, 2020 at 6:19 AM Sreyan Chakravarty <sreyan32(a)gmail.com&gt; wrote:
>
> Ok,
>
> I think I have found the problem:
>
> Nov 24 10:44:00 localhost.HPNotebook dracut-initqueue[861]: Scanning devices dm-0 
for LVM logical volumes vgfedora/fedora
> Nov 24 10:44:00 localhost.HPNotebook dracut-initqueue[886]: inactive
'/dev/vgfedora/pool00' [929.75 GiB] inherit
> Nov 24 10:44:00 localhost.HPNotebook dracut-initqueue[886]: inactive
'/dev/vgfedora/fedora' [700.00 GiB] inherit
> Nov 24 10:44:00 localhost.HPNotebook dracut-initqueue[888]: modprobe: FATAL: Module
dm-thin-pool not found in directory /lib/modules/5.8.18-200.fc32.x86_64
> Nov 24 10:44:00 localhost.HPNotebook dracut-initqueue[888]: /usr/sbin/modprobe
failed: 1
> Nov 24 10:44:00 localhost.HPNotebook dracut-initqueue[888]: Can't process LV
vgfedora/fedora: thin target support missing from kernel?
>
> But I have no idea how to solve it.
>
> I have marked the logical volumes as active via :
>
>              lvchange -kn vgfedora/pool00
>              lvchange -kn vgfedora/fedora
>
> But no change.
>
> How will I add the dm-thin-pool module ?
>
> I cannot even boot.
>
>
> On Tue, Nov 24, 2020 at 2:33 PM Sreyan Chakravarty <sreyan32(a)gmail.com&gt; wrote:
>>
>> Hi,
>>
>> I have cloned my original root LVM, and then migrated to the newer thin LVMs
using partclone.
>>
>> Both backup and restore went off without causing any problems.
>>
>> But when I am now trying to boot into my system, it asks for the encryption LUKS
passphrase but after that I am getting:
>>
>>               dracut-initqueue[300]: Warning: dracut-initqueue timeout  - 
starting timeout scripts
>>
>> I have not changed anything. Even my logical volume name is the same as before.
>>
>> These are the backup and restore commands that I have used from the Fedora live
environment:
>>         (Creating the backup)
>>          #   partclone.ext4 -c -s /dev/vgfedora/fedora | gzip -c --best >
/run/media/liveuser/externalhdd/fedora.tar.gz
>>          (Restoring the backup)
>>         #  gzip -c -d /run/media/externalhdd/fedora.tar.gz | partclone.ext4 -r -o
/dev/vgfedora/fedora
>>
>> These commands were run as root from Fedora live environment and prior to taking
my backup I had set my logical volume to read-only via:
>>
>>                #  lvchange -pr vgfedora/fedora
>>
>> This is my current volume group and logical volumes:
>>
>> #  lvs
>>
>>  LV     VG       Attr       LSize   Pool   Origin Data%  Meta%  Move Log Cpy%Sync
Convert
>>   fedora vgfedora Vwi-a-tz-- 700.00g pool00        16.61
>>   pool00 vgfedora twi-aotz-- 929.75g               12.50  16.61
>>
>> #  vgs
>>
>>   VG       #PV #LV #SN Attr   VSize   VFree
>>   vgfedora   1   2   0 wz--n- 929.99g 8.00m
>>
>>
>> This is my /etc/fstab:
>>
>> /dev/mapper/vgfedora-fedora /                       ext4   
defaults,x-systemd.device-timeout=0 1 1
>> UUID=4d112d8e-080d-44f2-8962-9c13f20a7885 /boot                   ext4   
defaults        1 2
>> UUID=041A-E511          /boot/efi               vfat   
defaults,uid=0,gid=0,umask=077,shortname=winnt 0 2
>> /fedora.swap none swap defaults 0 2
>>
>> I have not changed the /etc/fstab after the migration since nothing has changed.
>> /dev/mapper/vgfedora-fedora still exists. The only difference is that it is a
thin LVM volume.
>>
>> This is my GRUB_CMDLINE:
>>
>> GRUB_CMDLINE_LINUX="rd.luks.uuid=luks-2ec7f1ae-6f9b-4896-a7b2-be7809e9d2f4
rd.lvm.lv=vgfedora/fedora resume=/dev/disk/by-uuid/03aef3ba-dca1-4cba-a3f5-36c5c0fe948e
resume_offset=1449984"
>>
>> The UUID of the LUKS encryption is still the same, as I can confirm from here :
>>
>>                  #  ll /dev/disk/by-uuid/
>>                  #  lrwxrwxrwx. 1 root root 10 Nov 24 03:38
2ec7f1ae-6f9b-4896-a7b2-be7809e9d2f4 -> ../../sda3
>>
>> I don't know what else I have to change, or what is going wrong. Grub is
asking for the password for LUKS and is setting up the encrypted partition, but after that
nothing happens.
>>
>> I guess it is not able to load the root logical volume.
>>
>> Do I have to re-generate my initramfs ? I didn't think that would be
necessary since the LVM thin tools were already installed.
>>
>> Please let me know if any other info is required from my side.
>>
>> Need some help as I am unable to use my laptop.
>> --
>> Regards,
>> Sreyan Chakravarty
>
>
>
> --
> Regards,
> Sreyan Chakravarty
> _______________________________________________
> users mailing list -- users(a)lists.fedoraproject.org
> To unsubscribe send an email to users-leave(a)lists.fedoraproject.org
> Fedora Code of Conduct:
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives:
https://lists.fedoraproject.org/archives/list/[e-mail]

[Sreyan Chakravarty]

On Tue, Nov 24, 2020 at 9:23 PM Roger Heflin <rogerheflin(a)gmail.com&gt; wrote:

>  If you have a "rescue" option in grub then boot it, the
> rescue install
>  should be built with hostonly=no and is expected to include all
>  drivers including that one.  I verified on one of my machines that one
>  is included in my rescue.   The normal initrd is hostonly=yes and only
>  includes the critical modules needed to boot the system, and thin was
>  not previously need and so not included.
>
>  Once booted rebuild the other initrd and that should include the thin
>  module now.
>
>  lsinitrd <initrdname> | grep -i thin    should show you if it gets put
>  into the initrd after the rebuild and can be used to confirm that it
>  is not in any initramfs except rescue.
>  
I went the dumb path.

I made a regular logical volume, used partclone to restore my back up and
then booted into the OS.

Once booted in I added the following to /etc/dracut.conf:

              add_drivers+=" dm-thin-pool "

and then rebuild by initramfs by

         dracut -f

and then booted into my other two fedora kernels and did the same.

After that, I created the thread pool and thin volumes again and restored
my backup again, and logged in.

-- 
Regards,
Sreyan Chakravarty