RFI/RFC: Fedora Linux graphical recovery environment

[Neal Gompa]

Hey all,

Earlier this week, the Fedora Workstation WG discussed a ticket
brought to us asking for a GUI-based rescue/recovery environment[1].
While we all agreed in principle that such a thing would be a very
good thing to have, we don't really know how to achieve such a thing.
Additionally, we're not really sure what the scope of things should be
provided in said recovery environment and what kind of things people
would expect to be able to fix in there.

So I come to y'all to ask about this and give us some feedback on the
idea, how to do it, and what kinds of things you expect people to need
a recovery environment for.

[1]: https://pagure.io/fedora-workstation/issue/288

--
真実はいつも一つ！/ Always, there's only one truth!

[Marius Schwarz]

Am 31.03.22 um 23:38 schrieb Neal Gompa:
>  Hey all,
>
>  Earlier this week, the Fedora Workstation WG discussed a ticket
>  brought to us asking for a GUI-based rescue/recovery environment[1].
>  While we all agreed in principle that such a thing would be a very
>  good thing to have, we don't really know how to achieve such a thing.
>  Additionally, we're not really sure what the scope of things should be
>  provided in said recovery environment and what kind of things people
>  would expect to be able to fix in there.
>
>  So I come to y'all to ask about this and give us some feedback on the
>  idea, how to do it, and what kinds of things you expect people to need
>  a recovery environment for. 
My suggestion, as i needed it last week:

Preinstall Qphotorec into a normal Fedora LiveDisk and put it in the 
autostart config.
The "do you want  to test Fedora or install it " requester could be 
removed, or extended for the third option "Recovery"

But, if you walk this path, you can add a fourth option rightaway, as 
requests will come in soon: "Do AVScan of System".

best regards,
Marius

[Fabio Valentini]

On Thu, Mar 31, 2022 at 11:39 PM Neal Gompa <ngompa13(a)gmail.com&gt; wrote:
>  Hey all,
>
>  Earlier this week, the Fedora Workstation WG discussed a ticket
>  brought to us asking for a GUI-based rescue/recovery environment[1].
>  While we all agreed in principle that such a thing would be a very
>  good thing to have, we don't really know how to achieve such a thing.
>  Additionally, we're not really sure what the scope of things should be
>  provided in said recovery environment and what kind of things people
>  would expect to be able to fix in there.
>
>  So I come to y'all to ask about this and give us some feedback on the
>  idea, how to do it, and what kinds of things you expect people to need
>  a recovery environment for. 
This sounds interesting.

The only situation in which I would really have needed a "recovery
environment" was when after upgrading my old Fedora installs to Fedora
33 or something (whenever we switched GRUB to use BLS snippets, I
think) some required GRUB modules that were split off into subpackages
didn't get pulled in (I think it was grub2-efi-x64 ?), leaving the
system in an unbootable state, which I was only able to fix by booting
from a Live USB and installing the missing GRUB modules.

If I didn't know what to do (or wouldn't have had a Live USB at hand),
that would have basically bricked my system (or locked me into booting
Windows with its own bootloader). Not sure if it would be possible
with the "recovery environment" you would have in mind, but a basic
"are all the components that are required to boot there" check, with
suggestions how to fix them if they're missing, would have saved me
tons of time recovering from the borked grub install.

Fabio

[Neal Becker]

There are several recovery disks available that could serve as examples.
https://www.system-rescue.org/

On Mon, Apr 4, 2022 at 7:12 AM Fabio Valentini <decathorpe(a)gmail.com&gt; wrote:

>  On Thu, Mar 31, 2022 at 11:39 PM Neal Gompa
> <ngompa13(a)gmail.com&gt; wrote:
>  >
>  > Hey all,
>  >
>  > Earlier this week, the Fedora Workstation WG discussed a ticket
>  > brought to us asking for a GUI-based rescue/recovery environment[1].
>  > While we all agreed in principle that such a thing would be a very
>  > good thing to have, we don't really know how to achieve such a thing.
>  > Additionally, we're not really sure what the scope of things should be
>  > provided in said recovery environment and what kind of things people
>  > would expect to be able to fix in there.
>  >
>  > So I come to y'all to ask about this and give us some feedback on the
>  > idea, how to do it, and what kinds of things you expect people to need
>  > a recovery environment for.
>
>  This sounds interesting.
>
>  The only situation in which I would really have needed a "recovery
>  environment" was when after upgrading my old Fedora installs to Fedora
>  33 or something (whenever we switched GRUB to use BLS snippets, I
>  think) some required GRUB modules that were split off into subpackages
>  didn't get pulled in (I think it was grub2-efi-x64 ?), leaving the
>  system in an unbootable state, which I was only able to fix by booting
>  from a Live USB and installing the missing GRUB modules.
>
>  If I didn't know what to do (or wouldn't have had a Live USB at hand),
>  that would have basically bricked my system (or locked me into booting
>  Windows with its own bootloader). Not sure if it would be possible
>  with the "recovery environment" you would have in mind, but a basic
>  "are all the components that are required to boot there" check, with
>  suggestions how to fix them if they're missing, would have saved me
>  tons of time recovering from the borked grub install.
>
>  Fabio
>  _______________________________________________
>  devel mailing list -- devel(a)lists.fedoraproject.org
>  To unsubscribe send an email to devel-leave(a)lists.fedoraproject.org
>  Fedora Code of Conduct:
>  https://docs.fedoraproject.org/en-US/project/code-of-conduct/
>  List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
>  List Archives:
>  https://lists.fedoraproject.org/archives/list/[e-mail]
>  Do not reply to spam on the list, report it:
>  https://pagure.io/fedora-infrastructure
>  

-- 
*Those who don't understand recursion are doomed to repeat it*

[Robert Marcano]

On 3/31/22 5:38 PM, Neal Gompa wrote:
>  Hey all,
>  
>  Earlier this week, the Fedora Workstation WG discussed a ticket
>  brought to us asking for a GUI-based rescue/recovery environment[1].
>  While we all agreed in principle that such a thing would be a very
>  good thing to have, we don't really know how to achieve such a thing.
>  Additionally, we're not really sure what the scope of things should be
>  provided in said recovery environment and what kind of things people
>  would expect to be able to fix in there.
>  
>  So I come to y'all to ask about this and give us some feedback on the
>  idea, how to do it, and what kinds of things you expect people to need
>  a recovery environment for.
>  
>  [1]: https://pagure.io/fedora-workstation/issue/288
>   
I am not sure about having a full graphics environment, but it would be 
nice to have the functionality of the server installers rescue mode, 
where the system try to create a system tree at /mnt/sysimage IIRC and 
allows you to run `chroot /mnt/sysimage` to enter it.

This is the most basic functionality I would like to have on a rescue 
image. This allow someone to try to fix a badly modified or updated 
file. Some people would like to have network connectivity from the CLI 
at least.

And at the same time to have an option to not have this rescue menu 
option on the boot menu, like if I remove something like the package 
fedora-rescue or something like that, it isn't present on the menu.

Of topic but related: I wish there was supported option to remove the 
current rescue kernel, and hopefully a way to automate old kernels 
removals when the latest kernel is successfully installed and confirmed 
working. On company managed workstations I have a timer that checks the 
install date of the latest installed kernel and after a few days it 
removes the old kernels (if the users haven't complained for days, we 
assume the kernel is right). This is to avoid users reverting to older 
kernels in order to use some privilege escalation vulnerability.

[Gregory Bartholomew]

On Mon, Apr 4, 2022 at 10:53 AM Robert Marcano via devel <
devel(a)lists.fedoraproject.org&gt; wrote:

>  I am not sure about having a full graphics environment, but it would
> be
>  nice to have the functionality of the server installers rescue mode,
>  where the system try to create a system tree at /mnt/sysimage IIRC and
>  allows you to run `chroot /mnt/sysimage` to enter it.
>  
Ditto from me. I wonder though, if the rescue environment could/should try
to source a few different options? If, for example, the environment were
stored as a tar or cpio archive of extra packages and functionality, then
maybe it could look for a full-rescue-env.tar somewhere on the rootfs. But
failing finding that, it could fall back to a smaller and simpler
textmode-rescue-env.tar stored on the ESP and the final fallback would be
the current dracut rescue shell. Then the user would have the option not to
install one or the other of the rescue environments if they so chose.


>  Of topic but related: I wish there was supported option to remove
> the
>  current rescue kernel, 

Is echo "dracut_rescue_image=no" > /etc/dracut.conf.d/rescue.conf not
sufficient?

[Kevin Kofler]

Robert Marcano via devel wrote:
>  I am not sure about having a full graphics environment, but it would
> be
>  nice to have the functionality of the server installers rescue mode,
>  where the system try to create a system tree at /mnt/sysimage IIRC and
>  allows you to run `chroot /mnt/sysimage` to enter it. 
I think more useful than a dedicated GUI rescue environment would be a 
"Rescue" button (desktop icon or whatever is used for "Install to Hard
Disk" 
on the desktop environment on the image) on every GUI live image (e.g., on 
the KDE Live) that does the above (within the live image's GUI, not as a 
separate boot option).

I have (ab)used a standard GUI live image as a rescue image more often than 
I have actually used the text-based rescue images, and getting the system 
mounted was always the most painful part of it. Which is particularly sad 
considering that the logic to auto-detect and mount the installed system 
exists, but only on the rescue image.

        Kevin Kofler

[Gordon Messmer]

On 3/31/22 14:38, Neal Gompa wrote:
>  Additionally, we're not really sure what the scope of things
> should be
>  provided in said recovery environment and what kind of things people
>  would expect to be able to fix in there. 

The ticket mentions Boot Repair, which is the first thing that comes to 
mind: https://help.ubuntu.com/community/Boot-Repair

Additionally, I think a one-click option to do some of these could be 
useful repair options:

* Fix owner, permission, and capabilities: rpm --restore *
* SELinux relabel, or schedule it for the next boot (if that's the best 
option available)
* Until Fedora has something like snapper to roll back the filesystem in 
the event of an interrupted rpm upgrade, it'd be very nice to have a 
tool that can attempt a fix.  In the past I've used this:

|#!/bin/bash exec > /var/log/fix_updates 2>&1 # Remove the "new"
package 
so that the old package can be upgraded again. package-cleanup 
--cleandupes --removenewestdupes -y # Upgrade everything that can be 
safely upgraded yum update --skip-broken -y # Collect the remaining 
duplicate packages for later reinstallation broken=$(package-cleanup 
--dupes | xargs rpm -q --qf '%{NAME}\n' | uniq) # Remove the "old" 
package, so that the new package can be reinstalled package-cleanup 
--cleandupes -y test -n "$broken" && yum reinstall -y $broken # cleanup

leftover transaction yum-complete-transaction --cleanup-only|

[Rajeesh K V]

>  So I come to y'all to ask about this and give us some feedback on
> the
>  idea, how to do it, and what kinds of things you expect people to need
>  a recovery environment for. 
As somebody used to often get a knock on the door past midnight in the
early 2000’s because some guy in the hostel borked the GRUB setup
(very often caused by a Windows dual boot issue); fiddling with
/mnt/sysimage and grub-reinstall was the most common rescue operation
I used to do (and occasionally still do, but EFI booting has improved
the landscape).
The challenges were/are, generally:
  1.  Finding the correct boot partition
  2.  Invoking the correct magical incantations to reinstall grub
  3.  With GPT & EFI, recently I faced a fresh (dual boot) install of
Fedora 35 didn't boot afterwards; had to force anaconda into inst.gpt
mode and install again.
  4.  A misconfiguration in /etc/fstab causes boot failure

Anything that could potentially ease these procedures: helpful
nudges/suggestions to the user, ready reference to 'magical
incantations' to make (such as pointing to the correct initrd.img...)
etc. would be very useful, I think.

[Marc Pervaz Boocha]

On Thu, 2022-03-31 at 17:38 -0400, Neal Gompa wrote:
>  Hey all,
>  
>  Earlier this week, the Fedora Workstation WG discussed a ticket
>  brought to us asking for a GUI-based rescue/recovery environment[1].
>  While we all agreed in principle that such a thing would be a very
>  good thing to have, we don't really know how to achieve such a thing.
>  Additionally, we're not really sure what the scope of things should
>  be
>  provided in said recovery environment and what kind of things people
>  would expect to be able to fix in there.
>  
>  So I come to y'all to ask about this and give us some feedback on the
>  idea, how to do it, and what kinds of things you expect people to
>  need
>  a recovery environment for. 
There are a couple of things.
One important but overlooked usecase is offline repartioning. Modifing
the disk partioning is imposiple on a live disk(maybe with lvm). Its
nice to to have a equivalent of gparted live available for such works.

Another interesting idea is factory reset where the disk image is reset
to default fedora spin installation while maintaining /home.

>  
>  [1]: https://pagure.io/fedora-workstation/issue/288
>  
>  --
>  真実はいつも一つ！/ Always, there's only one truth!
>  _______________________________________________
>  devel mailing list -- devel(a)lists.fedoraproject.org
>  To unsubscribe send an email to devel-leave(a)lists.fedoraproject.org
>  Fedora Code of Conduct:
>  https://docs.fedoraproject.org/en-US/project/code-of-conduct/
>  List Guidelines:
>  https://fedoraproject.org/wiki/Mailing_list_guidelines
>  List Archives:
>  https://lists.fedoraproject.org/archives/list/[e-mail]
>  Do not reply to spam on the list, report it:
>  https://pagure.io/fedora-infrastructure

[Richard Shaw]

On Tue, Apr 5, 2022 at 8:20 AM Marc Pervaz Boocha via devel <
devel(a)lists.fedoraproject.org&gt; wrote:

>  On Thu, 2022-03-31 at 17:38 -0400, Neal Gompa wrote:
>  > Hey all,
>  >
>  > Earlier this week, the Fedora Workstation WG discussed a ticket
>  > brought to us asking for a GUI-based rescue/recovery environment[1].
>  > While we all agreed in principle that such a thing would be a very
>  > good thing to have, we don't really know how to achieve such a thing.
>  > Additionally, we're not really sure what the scope of things should
>  > be
>  > provided in said recovery environment and what kind of things people
>  > would expect to be able to fix in there.
>  >
>  > So I come to y'all to ask about this and give us some feedback on the
>  > idea, how to do it, and what kinds of things you expect people to
>  > need
>  > a recovery environment for.
>
>  There are a couple of things.
>  One important but overlooked usecase is offline repartioning. Modifing
>  the disk partioning is imposiple on a live disk(maybe with lvm). Its
>  nice to to have a equivalent of gparted live available for such works.
>  
Just an FYI since people are responding on the ticket and here in the
mailing list, but I mentioned in the ticket that I use gparted on a regular
basis from System Rescue CD. If this was built into the live media (or
dedicated rescue media) then I wouldn't need to do that.

Thanks,
Richard

[stan]

On Mon, 4 Apr 2022 15:58:14 -0500
Gregory Bartholomew <gregory.lee.bartholomew(a)gmail.com&gt; wrote:

>  > Of topic but related: I wish there was supported option to
> remove
>  > the current rescue kernel,  
>  
>  Is echo "dracut_rescue_image=no" > /etc/dracut.conf.d/rescue.conf not
>  sufficient? 
That is an interesting option.  It isn't documented in man dracut.conf.
Is it new?  I just manually remove the rescue vmlinuz and initramfs and
then run 
/usr/lib/kernel/install.d/51-dracut-rescue.install add $(uname -r) ""
/lib/modules/$(uname -r)/vmlinuz
in the /boot directory to build a new rescue kernel from the currently
running kernel.  Is there an option to do that also?  i.e. I invoke
dracut from the command line and it automatically does all that if a
rescue_build.conf file is present in /etc/dracut.conf.d/

[Neal Gompa]

On Tue, Apr 5, 2022 at 10:47 AM stan via devel
<devel(a)lists.fedoraproject.org&gt; wrote:
>  On Mon, 4 Apr 2022 15:58:14 -0500
>  Gregory Bartholomew <gregory.lee.bartholomew(a)gmail.com&gt; wrote:
>
>  > > Of topic but related: I wish there was supported option to remove
>  > > the current rescue kernel,
>  >
>  > Is echo "dracut_rescue_image=no" > /etc/dracut.conf.d/rescue.conf not
>  > sufficient?
>
>  That is an interesting option.  It isn't documented in man dracut.conf.
>  Is it new?  I just manually remove the rescue vmlinuz and initramfs and
>  then run
>  /usr/lib/kernel/install.d/51-dracut-rescue.install add $(uname -r) ""
> /lib/modules/$(uname -r)/vmlinuz
>  in the /boot directory to build a new rescue kernel from the currently
>  running kernel.  Is there an option to do that also?  i.e. I invoke
>  dracut from the command line and it automatically does all that if a
>  rescue_build.conf file is present in /etc/dracut.conf.d/ 
You could also just remove the "dracut-config-rescue" package from your system.



-- 
真実はいつも一つ！/ Always, there's only one truth!

[Gregory Bartholomew]

On Tue, Apr 5, 2022 at 9:47 AM stan via devel <devel(a)lists.fedoraproject.org&gt;
wrote:

>  On Mon, 4 Apr 2022 15:58:14 -0500
>  Gregory Bartholomew <gregory.lee.bartholomew(a)gmail.com&gt; wrote:
>
>  > > Of topic but related: I wish there was supported option to remove
>  > > the current rescue kernel,
>  >
>  > Is echo "dracut_rescue_image=no" > /etc/dracut.conf.d/rescue.conf not
>  > sufficient?
>
>  That is an interesting option.  It isn't documented in man dracut.conf.
>  Is it new? 

It's definitely not new. I didn't realize it wasn't documented. I don't
remember where I found that info now. I may have gotten it from reading the
/usr/lib/kernel/install.d/51-dracut-rescue.install script directly.


>    I just manually remove the rescue vmlinuz and initramfs and
>  then run
>  /usr/lib/kernel/install.d/51-dracut-rescue.install add $(uname -r) ""
>  /lib/modules/$(uname -r)/vmlinuz
>  in the /boot directory to build a new rescue kernel from the currently
>  running kernel.  Is there an option to do that also?  i.e. I invoke 
dracut from the command line and it automatically does all that if a
>  rescue_build.conf file is present in /etc/dracut.conf.d/
>  
Well sort of. If you really want complete control over how the rescue
kernels are installed one option is to copy
the /usr/lib/kernel/install.d/51-dracut-rescue.install script
to /etc/kernel/install.d and tweak it to do whatever you want under
whatever conditions you want. As long as the script under
/etc/kernel/install.d has the same name as the script under
/usr/lib/kernel/install.d, it will override (i.e. run instead of) the one
under /usr/lib/kernel/install.d.

[Gregory Bartholomew]

On Mon, Apr 4, 2022 at 9:40 PM Gordon Messmer <gordon.messmer(a)gmail.com&gt;
wrote:

>  The ticket mentions Boot Repair, which is the first thing that comes
> to
>  mind: https://help.ubuntu.com/community/Boot-Repair 

Boot repair is obviously tricky because you have to have something bootable
to initiate the repair. Practically speaking, I think the best option is to
better support dual-drive installations. It should be possible to maintain
two EFI partitions in a, b style when the user selects to have a
mirrored-disk configuration. In fact I've been doing this on my Fedora
Linux systems and I've even written some scripts to support it here:
https://github.com/gregory-lee-bartholomew/bootsync

Disclaimer though -- I use systemd boot and zfs, not the default grub and
btrfs that most Fedora Linux systems are configured with.

[stan]

On Tue, 5 Apr 2022 11:26:35 -0500
Gregory Bartholomew <gregory.lee.bartholomew(a)gmail.com&gt; wrote:

>  On Tue, Apr 5, 2022 at 9:47 AM stan via devel
>  <devel(a)lists.fedoraproject.org&gt; wrote: 
>  >   I just manually remove the rescue vmlinuz and initramfs and
>  > then run
>  > /usr/lib/kernel/install.d/51-dracut-rescue.install add $(uname -r)
>  > "" /lib/modules/$(uname -r)/vmlinuz
>  > in the /boot directory to build a new rescue kernel from the
>  > currently running kernel.  Is there an option to do that also?
>  > i.e. I invoke  
> > 
> > dracut from the command line and it automatically does all that if a
>  > rescue_build.conf file is present in /etc/dracut.conf.d/
>  >  
>  
>  Well sort of. If you really want complete control over how the rescue
>  kernels are installed one option is to copy
>  the /usr/lib/kernel/install.d/51-dracut-rescue.install script
>  to /etc/kernel/install.d and tweak it to do whatever you want under
>  whatever conditions you want. As long as the script under
>  /etc/kernel/install.d has the same name as the script under
>  /usr/lib/kernel/install.d, it will override (i.e. run instead of) the
>  one under /usr/lib/kernel/install.d. 
Thanks for the information.  It will probably be a rainy day project,
as doing it manually is not onerous, since I only do it when the
production kernel version changes.  e.g. 5.17 to 5.18  So, about every
3 months?

I actually might take your suggestion about how to turn off rescue
kernels, since, like you, I have two installations of fedora, and use
one (previous version) as a rescue system.  On the other hand, belt and
suspenders.

[stan]

On Tue, 5 Apr 2022 11:35:16 -0500
Gregory Bartholomew <gregory.lee.bartholomew(a)gmail.com&gt; wrote:

>  On Mon, Apr 4, 2022 at 9:40 PM Gordon Messmer
>  <gordon.messmer(a)gmail.com&gt; wrote:
>  
>  > The ticket mentions Boot Repair, which is the first thing that
>  > comes to mind: https://help.ubuntu.com/community/Boot-Repair  
>  
>  
>  Boot repair is obviously tricky because you have to have something
>  bootable to initiate the repair. Practically speaking, I think the
>  best option is to better support dual-drive installations. It should
>  be possible to maintain two EFI partitions in a, b style when the
>  user selects to have a mirrored-disk configuration. In fact I've been
>  doing this on my Fedora Linux systems and I've even written some
>  scripts to support it here:
>  https://github.com/gregory-lee-bartholomew/bootsync
>  
>  Disclaimer though -- I use systemd boot and zfs, not the default grub
>  and btrfs that most Fedora Linux systems are configured with. 
This is possible using standard efi boot with the cooperation of the
bios.  Just use two efi partitions, and prioritize the default one in
the bios boot menu.  Not as sophisticated as what you do, but I am
usually not running updates on the rescue system. I install the latest
fedora rawhide once a year, usually, so the rescue system is out of
support. Fine for rescue.

[Robert Marcano]

On 4/5/22 11:11 AM, Neal Gompa wrote:
>  On Tue, Apr 5, 2022 at 10:47 AM stan via devel
>  <devel(a)lists.fedoraproject.org&gt; wrote:
> >
> > On Mon, 4 Apr 2022 15:58:14 -0500
> > Gregory Bartholomew <gregory.lee.bartholomew(a)gmail.com&gt; wrote:
> >
> >>> Of topic but related: I wish there was supported option to remove
> >>> the current rescue kernel,
> >>
> >> Is echo "dracut_rescue_image=no" > /etc/dracut.conf.d/rescue.conf
> not
> >> sufficient?
> >
> > That is an interesting option.  It isn't documented in man dracut.conf.
> > Is it new?  I just manually remove the rescue vmlinuz and initramfs and
> > then run
> > /usr/lib/kernel/install.d/51-dracut-rescue.install add $(uname -r) ""
> /lib/modules/$(uname -r)/vmlinuz
> > in the /boot directory to build a new rescue kernel from the currently
> > running kernel.  Is there an option to do that also?  i.e. I invoke
> > dracut from the command line and it automatically does all that if a
> > rescue_build.conf file is present in /etc/dracut.conf.d/
>  
>  You could also just remove the "dracut-config-rescue" package from your
> system.
>   
And this is the magic bit. Thanks.

[Chris Murphy]

On Mon, Apr 4, 2022 at 5:12 AM Fabio Valentini <decathorpe(a)gmail.com&gt; wrote:
>  On Thu, Mar 31, 2022 at 11:39 PM Neal Gompa <ngompa13(a)gmail.com&gt; wrote:
>  >
>  > Hey all,
>  >
>  > Earlier this week, the Fedora Workstation WG discussed a ticket
>  > brought to us asking for a GUI-based rescue/recovery environment[1].
>  > While we all agreed in principle that such a thing would be a very
>  > good thing to have, we don't really know how to achieve such a thing.
>  > Additionally, we're not really sure what the scope of things should be
>  > provided in said recovery environment and what kind of things people
>  > would expect to be able to fix in there.
>  >
>  > So I come to y'all to ask about this and give us some feedback on the
>  > idea, how to do it, and what kinds of things you expect people to need
>  > a recovery environment for.
>
>  This sounds interesting.
>
>  The only situation in which I would really have needed a "recovery
>  environment" was when after upgrading my old Fedora installs to Fedora
>  33 or something (whenever we switched GRUB to use BLS snippets, I
>  think) some required GRUB modules that were split off into subpackages
>  didn't get pulled in (I think it was grub2-efi-x64 ?), leaving the
>  system in an unbootable state, which I was only able to fix by booting
>  from a Live USB and installing the missing GRUB modules. 
On UEFI, the blscfg.mod that brough BLS support is built-in to
grub$arch.efi on the ESP, and on conventional RPM installations is
replaced whenever the, e.g. grub2-efi-x64 package is updated. So I'm
not sure what this problem could have been. Meanwhile on BIOS systems,
the embedded "stage 2" bootloader (a.k.a. core.img which goes in the
MBR gap, or in the BIOS boot partition on GPT) is never updated
automatically, even when doing full system upgrades. This embedded
core.img does become stale. We had a problem where old (circa Fedora
24?) GRUB core.img became confused with a new generation blscfg.mod
found in /boot/grub2, and the fix was to point GRUB command line to
load the old grub.cfg, thereby being able to boot, and then
grub2-install to get the latest core.img created and embedded.

So yeah this was sufficiently esoteric and low level enough that it
would have prevented booting a rescue environment, at least one that's
under discussion. I think it's more the domain of coreos/bootupd, to
perhaps support an A/B approach to updating the bootloader, and thus
there'd be a fallback bootloader.


>  If I didn't know what to do (or wouldn't have had a Live USB
> at hand),
>  that would have basically bricked my system (or locked me into booting
>  Windows with its own bootloader). Not sure if it would be possible
>  with the "recovery environment" you would have in mind, but a basic
>  "are all the components that are required to boot there" check, with
>  suggestions how to fix them if they're missing, would have saved me
>  tons of time recovering from the borked grub install. 
A fairly simple tool could check the bootloader path:
1. Does NVRAM contain one Fedora entry;
2. Does that NVRAM entry point to a file that exists;
3. Is that file shim, and does its hash match its RPM, and is it the
latest version;
4. Is grub$arch.efi present, in the correct location, does its hash match RPM;
5. does /boot/efi/fedora/grub.cfg exist;
6. does the fs-uuid contained in this grub.cfg exist per libblkid;
7. is that fs-uuid mountable, i.e. mount it;
8. does that mounted fs have grub2/ directory, and does grub.cfg exist;
9. does that grub.cfg do insmod blscfg;
10. does the mounted fs have loader/entries, and do snippets exist;
11. minimally parse/check those bls snippets, and do they pass/fail
basic sanity checking;


All of these are just pass/fail tests, not repairing anything. I think
we'd need a more sophisticated tool to handle repairs and making sure
those repairs are reversible when possible. But any kind of repair is
risky because it changes things. So I'd like to emphasize collecting
information and comparing it to what's expected, rather than become
invasive and make risky changes. There isn't much hand holding here,
but the user is on perhaps a sparsely populated island rather than a
deserted island.


-- 
Chris Murphy

[Chris Murphy]

On Mon, Apr 4, 2022 at 6:35 PM Kevin Kofler via devel
<devel(a)lists.fedoraproject.org&gt; wrote:
>  Robert Marcano via devel wrote:
>  > I am not sure about having a full graphics environment, but it would be
>  > nice to have the functionality of the server installers rescue mode,
>  > where the system try to create a system tree at /mnt/sysimage IIRC and
>  > allows you to run `chroot /mnt/sysimage` to enter it.
>
>  I think more useful than a dedicated GUI rescue environment would be a
>  "Rescue" button (desktop icon or whatever is used for "Install to Hard
> Disk"
>  on the desktop environment on the image) on every GUI live image (e.g., on
>  the KDE Live) that does the above (within the live image's GUI, not as a
>  separate boot option).
>
>  I have (ab)used a standard GUI live image as a rescue image more often than
>  I have actually used the text-based rescue images, and getting the system
>  mounted was always the most painful part of it. Which is particularly sad
>  considering that the logic to auto-detect and mount the installed system
>  exists, but only on the rescue image. 
I don't know why anaconda can't run the rescue mode when launched from
a Live OS environment.

There are some ideas to make install media with separate Live OS and
Install boot menu entries. There'd be a Live OS to use Fedora in a
volatile environment (perhaps one day make persistence an option out
of the box). But you'd reboot and choose a separate installation
option that would be a non-Live desktop, or kiosk mode, environment
for the installer to run in dedicated fashion. In this case the
installer's rescue option could be used, and run through the discovery
logic to assemble the installed system.



-- 
Chris Murphy

[Kevin Kofler]

Chris Murphy wrote:
>  I don't know why anaconda can't run the rescue mode when
> launched from
>  a Live OS environment. 
The main thing that is missing is an icon/button on the desktop, I guess. 
Though of course Anaconda needs to provide a CLI executable (or a flag to 
the liveinst one that is already shipped anyway) for it.

>  There are some ideas to make install media with separate Live OS and
>  Install boot menu entries. There'd be a Live OS to use Fedora in a
>  volatile environment (perhaps one day make persistence an option out
>  of the box). But you'd reboot and choose a separate installation
>  option that would be a non-Live desktop, or kiosk mode, environment
>  for the installer to run in dedicated fashion. In this case the
>  installer's rescue option could be used, and run through the discovery
>  logic to assemble the installed system. 
But that is exactly what I do not want. I want to be able to use the full 
live environment as my rescue environment, e.g., the KDE Partition Manager 
that is installed on the KDE Live image anyway.

        Kevin Kofler

[Chris Murphy]

On Tue, Apr 5, 2022 at 7:21 AM Marc Pervaz Boocha via devel
<devel(a)lists.fedoraproject.org&gt; wrote:

>  There are a couple of things.
>  One important but overlooked usecase is offline repartioning. Modifing
>  the disk partioning is imposiple on a live disk(maybe with lvm). Its
>  nice to to have a equivalent of gparted live available for such works. 
Yeah when the disk containing an active root file system has its
partition modified, the kernel will not update its view of that
partition map. It requires a reboot. It's a little annoying. The
traditional answer is, device-mapper. I'm not sure what it would take
to review the limitation still in place for conventional partitions.


>  Another interesting idea is factory reset where the disk image is
> reset
>  to default fedora spin installation while maintaining /home. 
This is logically straightforward if we use btrfs snapshots.

We could make a read-only snapshot from within the installer, once the
installation is done. e.g. the current default creates two btrfs
subvolumes at the top-level of the file system: root and home, mounted
at / and /home respectively. Once installation is complete, one of the
last steps could be:

btrfs subvolume snapshot -r root root.rescue

Later on, a fairly heavy weight rollback could be as simple as:

mv root root.old
btrfs subvolume snapshot root.rescue root
reboot

* Changing the name of a subvolume while it's in use is not a problem.
* Btrfs snapshots are cheap and fast, they take up no additional
space, initially; as root subvolume is updated, space is consumed
because the old binaries aren't freed due to the snapshot.
* The user could easily choose to delete root.rescue if they prefer to
have that space back, instead of graphical rescue. It's much easier to
opt out than with a dedicated partition.
* There are two roots following a reset. We'd need to consider garbage
collection. One idea is to immediately delete root.old/usr -  because
that's fairly safe but also space conserving. The root.old/var could
still be pretty huge though, and it's more controversial to just
obliterate it. Maybe some select subdirs could be pruned though like
for caches and debuginfos? Or use subvolumes to split those areas out
further?
* Yes there are more gotchas [1].

Every "what if" and decision point comes with a deeper rabbit hole
that needs to be explored to discover liabilities. And then we need to
consider the cost of those liabilities knowing it's less about right
vs wrong, and more about a bunch of trade offs. And the hard work is
finding all the tradeoffs and weighing them against each other.


[1]
* it rolls back everything system related including logs and VM's and
reverts everything the user has uninstalled or installed;
* yes we could split out a subvolume "varlog" mounted at /var/log, and
neither snapshot it nor roll it back so that the log always carries
forward in time, similar to "home" subvolume;
* also could have a subvolume "varlibvirtimages" mounted at
/var/lib/libvirt/images, so that VM's aren't snapshot or rolled back;
* and still another "varlibflatpaks" mounted at /var/lib/flatpaks, so
that flatpaks aren't snapshot or rolled back;
* this rescue environment essentially comes with root privileges,
unless we do something to constrain it;
* the rescue environment probably should be updated or it'll go stale
and likely become a security risk,
* the less customized the root.rescue, the more similar it is to a
clean install, when doing a "system reset", but;
* the less customized the root.rescue, the more cluttered it is as a
rescue environment, and
* the more customized (minimized) the root.rescue, it makes for a
simpler/cleaner rescue environment, but not as good for "system
reset"...

... So we'd need to think through how to handle this aspect, perhaps
with some kind of filtered view (i.e. all the preinstalled apps are
really there but aren't seen in the rescue boot, but are seen
following a reset). An alternative is a minimalist rescue and reset
snapshot, which keeps the footprint small. But then needs to be built
back up somehow, following the reboot from being reset, e.g. dnf group
install "Fedora Workstation" or "KDE Plasma Workspaces" etc.

-- 
Chris Murphy

[Chris Murphy]

Recently I've been messing with the following minimal modifications to
see what happens:

; mount the top level of the fs
mount /dev/sdXY /mnt
cd /mnt
; make a snapshot
btrfs sub snap -r root root.rescue
; modify the rescue boot entry
nano /boot/loader/entries/$(cat /etc/machine-id)-0-rescue.conf

rootflags=subvol=root.rescue fstab=no systemd.volatile=overlay

Now systemd will mount the root.rescue snapshot, and it will ignore
fstab so nothing else with your system is being mounted, and you'll
get a normal read-write behaving environment with volatile changes. If
you reboot, the changes are all lost. It's a pretty rudimentary
starting point because of all the gotchas I mentioned previously. But
it's a lot better than the dracut shell.

Also, today's rescue boot entry has variable behavior depending on the
age of the system. Initially, it'll boot straight to GDM on its own.
But once a 3rd kernel is installed, and the original kernel package is
removed, the matching /usr/lib/modules/$kernelversion also goes away.
Interestingly, due to:

* vfat is not baked into the kernel itself
* vfat is in the initramfs
* switch root happens before mount /boot/efi
* upon switch root the initramfs and thus vfat module is unavailable
* /usr/lib/modules/$kernelversion is gone
* /etc/fstab for /boot/efi doesn't include nofail or noauto

Now /boot/efi can't mount because vfat.ko isn't available, and
therefore we get dropped to a dracut shell instead of graphical boot.
So we actually could just try to figure out how to get the rescue
entry to get us to graphical boot. Maybe by adding nofail to
/boot/efi? Or another idea?

If the initial root -> root.rescue snapshot is created by the
installer, this will pin this initially installed kernel's kernel
modules. The package gets deleted from "root" subvolume but not the
"root.rescue" snapshot. So it ends up not having this problem. The
simplest fix might be good enough. But I admit that I like the idea of
root.rescue being used as a quick way of doing a clean install /
reset. And I also like the volatile overlay with a read-only root.



Chris Murphy

[Chris Murphy]

On Tue, Apr 12, 2022 at 9:42 AM Kevin Kofler via devel
<devel(a)lists.fedoraproject.org&gt; wrote:
>  Chris Murphy wrote:
>  > I don't know why anaconda can't run the rescue mode when launched from
>  > a Live OS environment.
>
>  The main thing that is missing is an icon/button on the desktop, I guess.
>  Though of course Anaconda needs to provide a CLI executable (or a flag to
>  the liveinst one that is already shipped anyway) for it.
>
>  > There are some ideas to make install media with separate Live OS and
>  > Install boot menu entries. There'd be a Live OS to use Fedora in a
>  > volatile environment (perhaps one day make persistence an option out
>  > of the box). But you'd reboot and choose a separate installation
>  > option that would be a non-Live desktop, or kiosk mode, environment
>  > for the installer to run in dedicated fashion. In this case the
>  > installer's rescue option could be used, and run through the discovery
>  > logic to assemble the installed system.
>
>  But that is exactly what I do not want. I want to be able to use the full
>  live environment as my rescue environment, e.g., the KDE Partition Manager
>  that is installed on the KDE Live image anyway. 
Yeah I agree it would be nice to have both. So maybe it just needs a
separate icon that runs Anaconda rescue mode.

-- 
Chris Murphy