Are there typical ways Fedora tends to break? part 2

[Chris Murphy]

OK thanks for the responses so far. I have followup questions for everyone, even if you
didn't previously respond.

Do you think a graphical rescue environment would be helpful in troubleshooting system
problems?

Do you think a graphical rescue environment using volatile storage, would be useful?  i.e.
similar to a Live boot, by default no changes to the system or Live user environment would
be written to persistent media; e.g. Firefox cache files and history, or even installing
software, would be entirely lost on reboot from this graphical rescue environment

Do you think a mechanism for system snapshots and rollbacks would be useful in
troubleshooting system problems?

Do you think a snapshot+rollback mechanism would be more or less useful than a graphical
rescue environment, for troubleshooting system problems?

Thanks!
--
Chris Murphy

[Tom Horsley]

On Tue, 26 Jul 2022 14:39:25 -0000
Chris Murphy wrote:

>  Do you think a snapshot+rollback mechanism would be more or less
> useful than a graphical rescue environment, for troubleshooting system problems? 
I think a snapshot system would only be helpful if I could utterly
disable it. (I eventually figured out how to do that on Windows).

You know what what be really useful? (but would take 10 or 15 years to implement):

Go over every error and information message in linux with a group of
volunteers who never heard of linux and fix the messages so no one says
"What on earth does that mean?" any longer.

Another really useful tool (which might even be possible to implement):
A specialized google search tied to the current version of all the
software currently installed which filters out the 31,101,471 hits
you otherwise get explaining things that were true on old versions
but are no longer true now :-).

Although I personally use a snapshot+rollback mechanism, I think for most users a rescue
environment is more useful, because from my own experience and what I read from last
question, many cases when the system end up cannot boot is from users playing with the
boot process (eg grub) and make some mistakes. However conventional snapshot mechanism
only backup the root/home partition, not boot and efi partition because of the filesystem
limitation. A rescue environment would be helpful to at least help the user restore a grub
install and provide a default kernel cmdline that can help the system boot. But of course,
if /boot can be snapshotted and rolled back, it would be a very good thing (perhaps change
boot partition to btrfs? But I don’t know if it is possible)

On Tue, Jul 26, 2022, at 10:39 PM, Chris Murphy wrote:
>  OK thanks for the responses so far. I have followup questions for
> everyone, even if you didn't previously respond.
>  
>  Do you think a graphical rescue environment would be helpful in troubleshooting system
> problems?
>  
>  Do you think a graphical rescue environment using volatile storage, would be useful? 
> i.e. similar to a Live boot, by default no changes to the system or Live user environment
> would be written to persistent media; e.g. Firefox cache files and history, or even
> installing software, would be entirely lost on reboot from this graphical rescue
> environment
>  
>  Do you think a mechanism for system snapshots and rollbacks would be useful in
> troubleshooting system problems?
>  
>  Do you think a snapshot+rollback mechanism would be more or less useful than a graphical
> rescue environment, for troubleshooting system problems?
>  
>  Thanks!
>  --
>  Chris Murphy
>  _______________________________________________
>  users mailing list -- users(a)lists.fedoraproject.org
>  To unsubscribe send an email to users-leave(a)lists.fedoraproject.org
>  Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
>  List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
>  List Archives:
> https://lists.fedoraproject.org/archives/list/[e-mail]
>  Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
>

[Doug Herr]

On Tue, Jul 26, 2022, at 7:39 AM, Chris Murphy wrote:
>  OK thanks for the responses so far. I have followup questions for 
>  everyone, even if you didn't previously respond.
>
>  Do you think a graphical rescue environment would be helpful in 
>  troubleshooting system problems? 
I like having a graphical environment available for rescue. The "live" DVDs
might cover this already but I personally just keep a "systemrescue" image on a
"Ventoy" bootable USB stick.

>  Do you think a mechanism for system snapshots and rollbacks would be
>
>  useful in troubleshooting system problems? 
Not sure I would be interested in this one, but I would keep reading about it if it were
implemented and my mind might be changed.

[Ralf Corsépius]

Am 26.07.22 um 16:39 schrieb Chris Murphy:
>  OK thanks for the responses so far. I have followup questions for
> everyone, even if you didn't previously respond.
>  
>  Do you think a graphical rescue environment would be helpful in troubleshooting system
> problems?
>  
>  Do you think a graphical rescue environment using volatile storage, would be useful? 
> i.e. similar to a Live boot, by default no changes to the system or Live user environment
> would be written to persistent media; e.g. Firefox cache files and history, or even
> installing software, would be entirely lost on reboot from this graphical rescue
> environment
>  
>  Do you think a mechanism for system snapshots and rollbacks would be useful in
> troubleshooting system problems?
>  
>  Do you think a snapshot+rollback mechanism would be more or less useful than a graphical
> rescue environment, for troubleshooting system problems? 
"No" to all.

In almost all cases, I have encountered in the last 10 years and more, 
either "switching back to an older kernel" (i.e. cases of kernel bugs) 
or landing in a "root shell" is sufficient.

Ralf

[Barry]

>  On 26 Jul 2022, at 15:39, Chris Murphy
> <lists(a)colorremedies.com&gt; wrote:
>  
>  OK thanks for the responses so far. I have followup questions for everyone, even if you
> didn't previously respond.
>  
>  Do you think a graphical rescue environment would be helpful in troubleshooting system
> problems?
>  
>  Do you think a graphical rescue environment using volatile storage, would be useful? 
> i.e. similar to a Live boot, by default no changes to the system or Live user environment
> would be written to persistent media; e.g. Firefox cache files and history, or even
> installing software, would be entirely lost on reboot from this graphical rescue
> environment
>  
>  Do you think a mechanism for system snapshots and rollbacks would be useful in
> troubleshooting system problems? 
So long as the snapshot does not include any user data. Maybe only on /usr?
/home must not rollback or /var which can have dbs with user data.
Rolling back /etc is an interest situation to ponder. Maybe if a diff of changes can be
generated before the rollback so that system change can be applied again, slowly.
>  
>  Do you think a snapshot+rollback mechanism would be more or less useful than a graphical
> rescue environment, for troubleshooting system problems? 
I built a fedora install on a USB ssd. It took some thinking about to figure out the dance
to do that.

It would be like to be able to do that install from fedora, rather the. Boot live cd and
install to USB device.

Barry
>  
>  Thanks!
>  --
>  Chris Murphy
>  _______________________________________________
>  users mailing list -- users(a)lists.fedoraproject.org
>  To unsubscribe send an email to users-leave(a)lists.fedoraproject.org
>  Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
>  List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
>  List Archives:
> https://lists.fedoraproject.org/archives/list/[e-mail]
>  Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure

[Roberto Ragusa]

On 7/26/22 16:39, Chris Murphy wrote:
>  OK thanks for the responses so far. I have followup questions for
> everyone, even if you didn't previously respond.
>  
>  Do you think a graphical rescue environment would be helpful in troubleshooting system
> problems? 
No, a graphical environment implies too many things have to be working (gfx drivers
etc.),
rescue should be doable in minimal environments (root shell, possibly run in an initrd
rescue image).

>  Do you think a graphical rescue environment using volatile storage,
> would be useful?  i.e. similar to a Live boot, by default no changes to the system or Live
> user environment would be written to persistent media; e.g. Firefox cache files and
> history, or even installing software, would be entirely lost on reboot from this graphical
> rescue environment 
What would this be useful for?
Sort of using another computer without touching the broken one.

>  Do you think a mechanism for system snapshots and rollbacks would be
> useful in troubleshooting system problems? 
No. We should stay away from all the bad ideas done elsewhere.

>  Do you think a snapshot+rollback mechanism would be more or less
> useful than a graphical rescue environment, for troubleshooting system problems? 
Both are minimally useful.

The best way to debug problems is booting to a minimal root shell,
with mounted filesystems and network access to get packages.

It used to be "1" or "init=/bin/bash" in grub.

Today it is complicated:
- grub hides itself and is difficult to interrupt
- UEFI gets in the middle
- secure boot is in the middle
- proper system operation requires many things: systemd, NetworkManager, ...
- anything graphical requires daemons left and right: dbus, gconf, key agents, ...

Despite all this, booting a USB rescue image is still the ultimate
way to fix things (especially with chroot).

Regards.
-- 
    Roberto Ragusa    mail at robertoragusa.it

[Tom Horsley]

On Tue, 26 Jul 2022 21:32:38 +0200
Roberto Ragusa wrote:

>  - grub hides itself and is difficult to interrupt 
Yea, I've got a "big hammer" script that runs automatically after
dnf to beat on the grub.cfg file in case it just got written by
the dnf update. It clobbers any place that sets timeout to zero
so I'll get a chance to choose. It also turns off any code
attempting to hide the menu.

I spent a long time looking for some way to fix that "correctly"
with parameters in /etc/default/grub and never found one, so I
adopted the big hammer (which I use for other things as well):

https://tomhorsley.com/game/Mjolnir.html

[Tim]

On Tue, 2022-07-26 at 14:39 +0000, Chris Murphy wrote:
>  Do you think a graphical rescue environment would be helpful in
>  troubleshooting system problems? 
For some situations/people, yes.  I think you need a bare-bones command
line to get back control of a borked system.  But an additional
graphical option could be useful for those who need to point and click
a few bad settings back into normality.

>  Do you think a graphical rescue environment using volatile storage,
>  would be useful? 
There's merits in being able to test things knowing that nothing you do
will make permanent changes, so you can go through your options to fix
something one after another until you find the right one, without
making things worse in the meantime.

There's also times when you need to boot into a simplified system and
make changes that will stick.

But if everything is non-volatile in your rescue environment, how would
you actually make any change that fixes a borked system?

>  Do you think a mechanism for system snapshots and rollbacks would be
>  useful in troubleshooting system problems? 
They never did any good on other OSs (for me, at least).  The undo you
needed to do was always in the middle of a set of changes.

There are so many interconnected things on modern OSs that it's very
difficult to remove a slab of things and not create a new problem.

System's borked, we'll roll back to how it was three days ago and still
working.  But what about all the work I've done since then?  Sorry,
that's all going to get trashed.  You can do it again.  No, I can't.

I'm not just talking about user data, documents you've written, etc.  A
person's work can be things that they were doing with the system.

When I look at backup management software, I give up in dispair.  It
works in that "go back several days" mentality.  It's hard to get back
the one thing you need.  Most system borks seem to be that you made a
goof (or it did) about three steps back, the fix is to work on that
goof, and not undo all the other things you did after it that were
perfectly fine.

It's exactly the same issue with wanting to undo one thing in the
middle of a word processor doc, or artwork in a graphic program.  The
undos/redos are all time-sequential, and not confineable to a specific
area of the data.

-- 
 
uname -rsvp
Linux 3.10.0-1160.71.1.el7.x86_64 #1 SMP Tue Jun 28 15:37:28 UTC 2022 x86_64
 
Boilerplate:  All unexpected mail to my mailbox is automatically deleted.
I will only get to see the messages that are posted to the mailing list.

[George N. White III]

On Tue, Jul 26, 2022 at 8:23 PM Tim via users <users(a)lists.fedoraproject.org&gt;
wrote:

>  On Tue, 2022-07-26 at 14:39 +0000, Chris Murphy wrote:
>  > Do you think a graphical rescue environment would be helpful in
>  > troubleshooting system problems?
>
>  For some situations/people, yes.  I think you need a bare-bones command
>  line to get back control of a borked system.  But an additional
>  graphical option could be useful for those who need to point and click
>  a few bad settings back into normality.
>
>  > Do you think a graphical rescue environment using volatile storage,
>  > would be useful?
>
>  There's merits in being able to test things knowing that nothing you do
>  will make permanent changes, so you can go through your options to fix
>  something one after another until you find the right one, without
>  making things worse in the meantime.
>
>  There's also times when you need to boot into a simplified system and
>  make changes that will stick.
>
>  But if everything is non-volatile in your rescue environment, how would
>  you actually make any change that fixes a borked system?
>  
A someone who is prepared to use bare-bones comand-line, the GUI should
provide the capabilities you get from a rescue kernel to mount
filesystems and use chroot.

Before all the Windows drives were encrypted, I have often booted a "dead"
Windows system with linux to copy a user's important files to an external
drive
before giving the Windows box back to IT, who just restored the corporate
standard
Windows image and did updates.


>  > Do you think a mechanism for system snapshots and rollbacks
> would be
>  > useful in troubleshooting system problems?
>
>  They never did any good on other OSs (for me, at least).  The undo you
>  needed to do was always in the middle of a set of changes.
>
>  There are so many interconnected things on modern OSs that it's very
>  difficult to remove a slab of things and not create a new problem.
>
>  System's borked, we'll roll back to how it was three days ago and still
>  working.  But what about all the work I've done since then?  Sorry,
>  that's all going to get trashed.  You can do it again.  No, I can't.
>  
I used to work in a group with many Apple systems.  Every user had a
USB disk set up for Time Machine and the cost was more than justified
by the number of times it was used to recover from accidental deletions
of work that would take days to reproduce.


>  I'm not just talking about user data, documents you've written, etc.  A
>  person's work can be things that they were doing with the system.
>
>  When I look at backup management software, I give up in dispair.  It
>  works in that "go back several days" mentality.  It's hard to get back
>  the one thing you need.  Most system borks seem to be that you made a
>  goof (or it did) about three steps back, the fix is to work on that
>  goof, and not undo all the other things you did after it that wereSome
>  very minimal data collection
>  perfectly fine.
>
>  It's exactly the same issue with wanting to undo one thing in the
>  middle of a word processor doc, or artwork in a graphic program.  The
>  undos/redos are all time-sequential, and not confineable to a specific
>  area of the data.
>  
There are applications that allow you to maintain a record of your workflow
and then replay it.

I use a remote-sensing GUI called SNAP from the European Space Agency
which has a parallel command-line tool.  You can use the GUI to develop
a workflow and then use a (GUI) tool to re-create the workflow as a graph
with parameters to apply the same workflow to many files in batch
mode.  It is very helpful if you break the workflow into small steps
and check the results as you go.  The batch processing can be moved to
a server, and takes advantage of high-end hardware (many CPU's
and GPU's).

The ability to convert a GUI workflow to a batch script is important
for "reproducible research", and very helpful when troubleshooting bugs.
NASA has adapted the ESA GUI to some of their remote-sensing
systems as "SeaDAS".

-- 
George N. White III

[Tim]

On Wed, 2022-07-27 at 09:04 -0300, George N. White III wrote:
>  A someone who is prepared to use bare-bones comand-line, the GUI
>  should provide the capabilities you get from a rescue kernel to mount
>  filesystems and use chroot. 
Yes, but what if graphical display is one of the things going bonkers?
Text-only is the least likely to screw up.

But often we need to get into a network to fix something, and the
system didn't come with a text interface to the network mangler pre-
installed, and getting the network up without it is a world of fun.  So
there's merits in having both available.
 
-- 
 
uname -rsvp
Linux 3.10.0-1160.71.1.el7.x86_64 #1 SMP Tue Jun 28 15:37:28 UTC 2022 x86_64
 
Boilerplate:  All unexpected mail to my mailbox is automatically deleted.
I will only get to see the messages that are posted to the mailing list.

[Tim]

Tim:
> > It's exactly the same issue with wanting to undo one thing in
> the
> > middle of a word processor doc, or artwork in a graphic program. 
> > The undos/redos are all time-sequential, and not confineable to a
> > specific area of the data. 
George N. White III:
>  There are applications that allow you to maintain a record of your
>  workflow and then replay it. 
I wish that were more common.  I'd hate to have to try and deal with an
undo inside the middle of a lengthy document, without losing everything
else you did afterwards.

I'm used to doing saves every now and then (never relying on auto-
saves).  To deal with the *retrieve something ten steps back* problem,
it's a save-as to duplicate the current thing, undo like crazy, fix the
mistake.  Then open the duplicate separately, copy and paste the new
stuff into the now-fixed thing.

Thankfully I don't have to do that often, but while that's do-able with
a lengthy document, there are other things that's less achievable.
 
-- 
 
uname -rsvp
Linux 3.10.0-1160.71.1.el7.x86_64 #1 SMP Tue Jun 28 15:37:28 UTC 2022 x86_64
 
Boilerplate:  All unexpected mail to my mailbox is automatically deleted.
I will only get to see the messages that are posted to the mailing list.