Background Start: On 11/01/2024 upgraded 40->41 on my old Jetta laptop and everything was fine. The Jetta does not have encrypted disks. On 11/02/2024 upgrade 40->41 failed on old supermicro X10Sat media server with some encrypted disks. After upgrade, everything seemed fine. Then, upon reboot, the luks encryption password failed to work with message: Warning: /dev/disk/by-uuid/d06....96 does not exist Generating "/run/initramfs/rdsosreport.txt" Entering emergency mode. Exit shell to continue. Type "journalctl" to view system logs. You might want to save "/run/initramfs/rdsosreport.txt" to a USB stick or /boot after mounting them and attach it to a bug report. Give root password for maintenance (of press Control-D to continue):
I tried to boot the machine a bunch of times.
Note Start: Also on 11/02/2024 tried to upgrade 2 DELL XPS (with encrypted disks) before the media server. They seemed to be OK. But, after the problem with media server after reboot, I rebooted each of the Dell laptops and both had the same problem, the encryption key no longer worked. But, note, before the reboot they ran F41 just fine - the encryption key is always enter prior to the upgrade. The laptops did not have much on them, so, I simple re-installed fedora actually, I tried installing: 11/2/2024 Fedora-KDE-Live-x86_64-41-1.4.iso and got message Operation System not found (message generate by supermicro board, and, yes it is "Operation"). 11/3/2024 Fedora-KDE-Live-x86_64-40-1.14.iso and got message Operation System not found 11/3/2024 Fedora-Xfce-Live_38.1.6 and got Operation System not found 11/3/2024 installed linuxmint-20.3-xfce-64bit.iso and it worked Success (I do not use linuxmint ... but at this point I got lucky) 11/3/2024 installed linuxmint-22-xfce-64bit.iso and got message Operation System not found 11/4/2024 re-installed linuxmint-20.3-xfce-64bit.iso Success 11/4/2024 installed Fedora-Xfce-Live_28 but did not reclaim space of the linuxmint /boot/efi directory. Success 11/4/2024 installed Fedora-Xfce-Live-x86_64-41-1.4.iso on top of linuxmint Fedora-Xfce-Live_28 Success I wondered how an install knew whether or not its was a bios or efi system. Since motherboards (I guessed) do not have an API an installer can call, the hint must be on the disk. Hence, my guess was it was the /boot/efi directory. Note End Background End:
From sometime in the past I had run (and saved the output) lsblk on the media server's sda drive:
$ lsblk NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINTS sda 8:0 0 223.6G 0 disk ├─sda1 8:1 0 250M 0 part /boot ├─sda2 8:2 0 105.5G 0 part │ └─luks-a2ebb2b0-527d-47f3-83ef-e5908805f31d │ 253:3 0 105.5G 0 crypt /ssd ├─sda3 8:3 0 97.7G 0 part │ └─luks-35719a97-5898-4420-9a56-1576ffdc6db3 │ 253:1 0 97.7G 0 crypt / ├─sda4 8:4 0 1K 0 part ├─sda5 8:5 0 9.8G 0 part │ └─luks-5ee2ed8e-4bdf-43e1-adb0-34a70610a77f │ 253:2 0 9.8G 0 crypt /tmp └─sda6 8:6 0 9.8G 0 part └─luks-03c06df8-f9b9-4f0d-847e-79a7ed527888 253:0 0 9.8G 0 crypt [SWAP]
So, some stuff is encrypted.
I notice that it is possible to remove the encryption on a disk: https://wiki.archlinux.org/title/Removing_system_encryption. But, in fedora maintenance mode, the binary cryptsetup is not available. Curiously, the libcryptsetup is available. So, my question, is it possible to remove the encryption on a disk in maintenance mode. If one can, then, maybe, I might be able to login into my media server.
Thanks Richard
On Nov 5, 2024, at 18:04, richard emberson emberson.rich@gmail.com wrote:
So, my question, is it possible to remove the encryption on a disk in maintenance mode. If one can, then, maybe, I might be able to login into my media server.
So, if I’m reading this right, you updated to Fedora 41 and now you can’t unlock the LUKS volume upon boot. Is that correct?
Then you did a bunch of stuff but described it so incomprehensible (and possibly repeatedly?) I have no idea what you did.
And now you are asking if you can remove the encryption.
First off, you would need to be able to decrypt the LUKS volume to change the encryption settings. If you can’t do that, you’re stuck.
But more importantly, have you tried booting older kernels in GRUB2 on boot? It’s possible (albeit unlikely) that the new kernel was built with broken cryptsetup binaries.
Another thing to mention, I see a lot of people who come to my work’s help desk because they suddenly can’t unlock their LUKS volume on their laptop. Most often it’s a typo, or they typed their login password instead of the LUKS passphrase Sometimes it’s a broken keyboard. Other times it’s because the keymap isn’t the same as before. Very rarely is it because some corruption broke it.
You should make sure you’ve tested all those possibilities. Also, if you can boot off a live image, you should be able to unlock the volume using cryptsetup or even a graphical tool like GNOME disks.
On Tue, 2024-11-05 at 18:44 -0500, Jonathan Billings wrote:
Sometimes it’s a broken keyboard.
Oh the fun (not) of trying to login with a crappy laptop keyboard. You can't see what characters are being typed, so you have zero clues about what's going wrong...
On older releases I could switch on the on-screen keyboard (that you use with the mouse, or a touch screen). Mate on Fedora 40 won't let me do that.
Early in Maintenance Mode I tested the keyboard and typed in the passphrase, twice, and it appeared on the screen correctly.
On 11/5/24 6:17 PM, Tim via users wrote:
On Tue, 2024-11-05 at 18:44 -0500, Jonathan Billings wrote:
Sometimes it’s a broken keyboard.
Oh the fun (not) of trying to login with a crappy laptop keyboard. You can't see what characters are being typed, so you have zero clues about what's going wrong...
On older releases I could switch on the on-screen keyboard (that you use with the mouse, or a touch screen). Mate on Fedora 40 won't let me do that.
One must enter the encryption passphrase before you get to select what kernel to load.
On 11/5/24 3:44 PM, Jonathan Billings wrote:
But more importantly, have you tried booting older kernels in GRUB2 on boot? It’s possible (albeit unlikely) that the new kernel was built with broken cryptsetup binaries.
I tried to upgrade 3 different machine from 40 to 41. All older that boot with /boot/efi. Two machines were laptops and one machine was a desktop media server. All three failed the same way. The disk decryption mechanism did not work. Same error message generated.
If I had to guess, I believe that fedora 41 update breaks for older boot machines that are encrypted. Older machines (/boot/efi) are testing corner cases. Older machines (/boot/efi) that are encrypted are corner cases of corner cases and, so, this was most likely not tested.
On 11/5/24 3:44 PM, Jonathan Billings wrote:
So, if I’m reading this right, you updated to Fedora 41 and now you can’t unlock the LUKS volume upon boot. Is that correct?
On Tue, 2024-11-05 at 18:25 -0800, richard emberson wrote:
Early in Maintenance Mode I tested the keyboard and typed in the passphrase, twice, and it appeared on the screen correctly.
Just to be thorough... On the graphical login screen see if there's an icon for choosing keyboard layout/language. It is possible for the command line (purely textual interface) to be different from the graphical interface.
You can't remove disc encryption without re-writing every bit of data stored on the disc. When using an encrypted disc the system is decrypting the data on the fly, it's always encrypted on the disc.
Are you sure you want to do that, or simply access the encrypted data?
Of course, when it comes to updating an old installation of something with a new version of something else, there is every chance that the old encryption scheme isn't supported any more, so you would have to re-encrypt the drive (or just decrypt it, if you don't want encrypted contents any more).
You should probably be booting from something else, and something that's compatible with the original encryption scheme, to do this. I wouldn't risk trying anything that was going to decrypt in-place when its running from itself, well not if the OS is encrypted. I'd be more confident about doing that if it's simply user data space that was encrypted.
Thank you for your suggestions.
Keyboard does not look like it is the issue.
I also tried to start the fedora rescue selection, but it also required the entry of the LUKS passphrase (Curiously, while a normal release starts with a different gui and stops asking for the passphrase after 4 or 5 attempts, the rescue selection just keeps asking for the passphrase ... maybe forever.)
So, I decided to see if I could get the DVD drive to be recognized by the motherboard. When I put a DVD into it and executed from Maintenance Mode: cat /dev/sr0 nothing is produced. The media server has been quietly sitting in a corner for years. Anyway, with a twice over with my small air compressor making sure I manually opened the DVD drive and clearing any dust, the system then recognize a DVD and I could do an installation: installed linuxmint-20.3-xfce-64bit.iso then installed Fedora-Xfce-Live-x86_64-41-1.4.iso over the linuxmint installation but kept the /boot/efi directory so that the installation did an efi installation (not bios). I assume this is a trick that is not known by many and as my original email said, It was just chance that I guessed it might work.
While there was some stuff in the original /home directory, all of the media was on other, non-encrypted drives which I've now been able to mount in this new installation. Thanks again.
On 11/5/24 8:05 PM, Tim via users wrote:
On Tue, 2024-11-05 at 18:25 -0800, richard emberson wrote:
Early in Maintenance Mode I tested the keyboard and typed in the passphrase, twice, and it appeared on the screen correctly.
Just to be thorough... On the graphical login screen see if there's an icon for choosing keyboard layout/language. It is possible for the command line (purely textual interface) to be different from the graphical interface.
You can't remove disc encryption without re-writing every bit of data stored on the disc. When using an encrypted disc the system is decrypting the data on the fly, it's always encrypted on the disc.
Are you sure you want to do that, or simply access the encrypted data?
Of course, when it comes to updating an old installation of something with a new version of something else, there is every chance that the old encryption scheme isn't supported any more, so you would have to re-encrypt the drive (or just decrypt it, if you don't want encrypted contents any more).
You should probably be booting from something else, and something that's compatible with the original encryption scheme, to do this. I wouldn't risk trying anything that was going to decrypt in-place when its running from itself, well not if the OS is encrypted. I'd be more confident about doing that if it's simply user data space that was encrypted.
On 5 Nov 2024, at 23:04, richard emberson emberson.rich@gmail.com wrote:
So, my question, is it possible to remove the encryption on a disk in maintenance mode. If one can, then, maybe, I might be able to login into my media server.
I suggest that you boot from a fedora install usb and then use cryptsetup to mount your encrypted disk.
In that environment it will be easy to check the passphrase is typed correctly.
Once you know you can get to your encrypted data you can consider if you want to unencrypt the partition.
For what it’s worth…. Both my systems with encrypted disks upgraded without issue with the encryption. One is a 5 year old supermicro motherboard system.
Barry
The encryption defaults changed sometime recently.
The defaults cryptsetup command I had in a script stopped mount my encrypted filesystem until I did a bunch of research and found out what parameters needed to be specified to match the prior default.
If you want to try what I found out reply and I can boot up the encrypted machine and see what parameters I needed to add.
On Wed, Nov 6, 2024 at 1:48 PM richard emberson emberson.rich@gmail.com wrote:
Thank you for your suggestions.
Keyboard does not look like it is the issue.
I also tried to start the fedora rescue selection, but it also required the entry of the LUKS passphrase (Curiously, while a normal release starts with a different gui and stops asking for the passphrase after 4 or 5 attempts, the rescue selection just keeps asking for the passphrase ... maybe forever.)
So, I decided to see if I could get the DVD drive to be recognized by the motherboard. When I put a DVD into it and executed from Maintenance Mode: cat /dev/sr0 nothing is produced. The media server has been quietly sitting in a corner for years. Anyway, with a twice over with my small air compressor making sure I manually opened the DVD drive and clearing any dust, the system then recognize a DVD and I could do an installation: installed linuxmint-20.3-xfce-64bit.iso then installed Fedora-Xfce-Live-x86_64-41-1.4.iso over the linuxmint installation but kept the /boot/efi directory so that the installation did an efi installation (not bios). I assume this is a trick that is not known by many and as my original email said, It was just chance that I guessed it might work.
While there was some stuff in the original /home directory, all of the media was on other, non-encrypted drives which I've now been able to mount in this new installation. Thanks again.
On 11/5/24 8:05 PM, Tim via users wrote:
On Tue, 2024-11-05 at 18:25 -0800, richard emberson wrote:
Early in Maintenance Mode I tested the keyboard and typed in the passphrase, twice, and it appeared on the screen correctly.
Just to be thorough... On the graphical login screen see if there's an icon for choosing keyboard layout/language. It is possible for the command line (purely textual interface) to be different from the graphical interface.
You can't remove disc encryption without re-writing every bit of data stored on the disc. When using an encrypted disc the system is decrypting the data on the fly, it's always encrypted on the disc.
Are you sure you want to do that, or simply access the encrypted data?
Of course, when it comes to updating an old installation of something with a new version of something else, there is every chance that the old encryption scheme isn't supported any more, so you would have to re-encrypt the drive (or just decrypt it, if you don't want encrypted contents any more).
You should probably be booting from something else, and something that's compatible with the original encryption scheme, to do this. I wouldn't risk trying anything that was going to decrypt in-place when its running from itself, well not if the OS is encrypted. I'd be more confident about doing that if it's simply user data space that was encrypted.
-- _______________________________________________ users mailing list -- users@lists.fedoraproject.org To unsubscribe send an email to users-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/users@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
On Thu, Nov 7, 2024 at 8:03 PM Roger Heflin rogerheflin@gmail.com wrote:
The encryption defaults changed sometime recently.
I don't see the change documented at https://fedoraproject.org/wiki/Releases/41/ChangeSet or https://docs.fedoraproject.org/en-US/fedora/latest/release-notes/.
That's unfortunate.
The defaults cryptsetup command I had in a script stopped mount my encrypted filesystem until I did a bunch of research and found out what parameters needed to be specified to match the prior default.
If you want to try what I found out reply and I can boot up the encrypted machine and see what parameters I needed to add.
Jeff
I am not sure what version mine last worked on. I would guess the default changed on 39 or 40.
What fixed it for me (type plain password from stdin) was adding --hash ripemd160 (they appear to have changed the default hash, BAD developer).
Guessing related to this: https://gitlab.com/cryptsetup/cryptsetup/-/issues/758
And given people could have a script/command to mount like this(and ask for a password), changing the default is not nice especially since it will break the password working with no indication of anything except the password/hash not decrypting the fs.
On Thu, Nov 7, 2024 at 8:35 PM Jeffrey Walton noloader@gmail.com wrote:
On Thu, Nov 7, 2024 at 8:03 PM Roger Heflin rogerheflin@gmail.com wrote:
The encryption defaults changed sometime recently.
I don't see the change documented at https://fedoraproject.org/wiki/Releases/41/ChangeSet or https://docs.fedoraproject.org/en-US/fedora/latest/release-notes/.
That's unfortunate.
The defaults cryptsetup command I had in a script stopped mount my encrypted filesystem until I did a bunch of research and found out what parameters needed to be specified to match the prior default.
If you want to try what I found out reply and I can boot up the encrypted machine and see what parameters I needed to add.
Jeff
users mailing list -- users@lists.fedoraproject.org To unsubscribe send an email to users-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/users@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Roger Heflin wrote:
I am not sure what version mine last worked on. I would guess the default changed on 39 or 40.
What fixed it for me (type plain password from stdin) was adding --hash ripemd160 (they appear to have changed the default hash, BAD developer).
Guessing related to this: https://gitlab.com/cryptsetup/cryptsetup/-/issues/758
And given people could have a script/command to mount like this(and ask for a password), changing the default is not nice especially since it will break the password working with no indication of anything except the password/hash not decrypting the fs.
This would have been in cryptsetup-2.7.0, so Fedora 40. Upstream does note this backward incompatible change in the 2.7.0 release notes¹ and it doesn't seem like they made it just to make it. Sure, it's annoying when it happens and catches one of us, but it is a necessary evil sometimes.
Relevant to the initial problem in this thread, does anything in Fedora use cryptsetup plain mode by default? Doesn't the installer use LUKS when encrypting a drive (which isn't affected by this change) and lack support encrypting /boot or /boot/efi? I thought that was something you could do, but you had to work it out on your own?
¹ https://gitlab.com/cryptsetup/cryptsetup/-/blob/main/docs/v2.7.0-ReleaseNote...
-
Barry -
Jeffrey Walton -
Jonathan Billings -
richard emberson -
Roger Heflin -
Tim -
Todd Zullinger