F38 proposal: Modernize Live Media (System-Wide Change proposal)

[Ben Cotton]

https://fedoraproject.org/wiki/Changes/ModernizeLiveMedia

This document represents a proposed Change. As part of the Changes
process, proposals are publicly announced in order to receive
community feedback. This proposal will only be implemented if approved
by the Fedora Engineering Steering Committee.


== Summary ==
Modernize the live media by switching to the "new" live environment
setup scripts provided by {{package|livesys-scripts}} and leverage new
functionality in {{package|dracut}} to enable support for
automatically enabling persistent overlays when flashed to USB sticks.

== Owner ==
* Name: [[User:Ngompa|Neal Gompa]], [[User:Mcoleman|Matt Coleman]]
* Email: ngompa13(a)gmail.com, matt(a)1eanda.com


== Detailed Description ==
As part of preparing to change our tooling around producing images, we
need to update the way we produce working live media. This has been
done in two parts: the live environment setup scripts have been
reworked to run properly through systemd and are packaged in
{{package|livesys-scripts}} and new functionality in
{{package|dracut}} has been added to enable support for automatically
enabling persistent overlays when flashed to USB sticks so that
<code>livecd-iso-to-disk.sh</code> can be retired.


== Benefit to Fedora ==
Since we introduced [[Releases/FeatureLiveCD|live media in Fedora
Linux 7]], the actual mechanism in which the live environment sets
itself up has been complex and intricately tied to the method in which
we produce the media (using Kickstarts). The nature of the
implementation of those scripts means that they are hard to understand
and debug, which has caused problems in the past whenever we've needed
to update them.

As we look forward to new and better tooling for producing images
(such as {{package|kiwi}} and {{package|osbuild}}), we cannot continue
to rely on kickstart-driven image builds that construct shell scripts
on the fly to embed in the image as we do now. With
{{package|livesys-scripts}}, those scripts have been simplified and
turned into systemd services that activate only in live environments.

This also gives us the opportunity to introduce new functionality for
live media. [https://github.com/dracutdevs/dracut/pull/1991 New
functionality was added to dracut] and
[https://src.fedoraproject.org/rpms/dracut/pull-request/26 backported
to Fedora] so that we can retire the remaining usage of
<code>livecd-iso-to-disk.sh</code> and provide a better experience
with our live media, particularly for portable backup and rescue
environments by introducing the ability to automatically setup
persistence on boot when unpartitioned space is detected on a USB
stick on boot.

== Scope ==
* Proposal owners:
** Modify [https://pagure.io/fedora-kickstarts fedora-kickstarts] to
drop embedded livesys setup scripts and use
{{package|livesys-scripts}}
** Add <code>livesys.service</code> and
<code>livesys-late.service</code> to the systemd presets in
{{package|fedora-release}}
** [https://src.fedoraproject.org/rpms/dracut/pull-request/26 Backport
<code>dmsquash-live-autooverlay</code> module] to {{package|dracut}}
** Modify [https://github.com/weldr/lorax/tree/master/share/templates.d/99-generic
lorax-generic-templates] to offer menu entries for configuring the
persistent overlay

* Other developers:
** Anaconda may need adaptations to correctly filter out more live
environment specific boot arguments when configuring the installed
environment.

* Release engineering: [https://pagure.io/releng/issue/11091 #11091]

* Policies and guidelines: N/A (not needed for this Change)

* Trademark approval: N/A (not needed for this Change)

* Alignment with Objectives: N/A

== Upgrade/compatibility impact ==
There should be no impact here, as this affects only the live environment.

== How To Test ==
Once the changes are merged, simply try to boot produced live media.

There should be new options for resetting the persistent overlay and
booting with no persistence. The default options should boot with
persistence and setup of persistence should work.

Testing persistence behavior is simple:

# Boot normally
# Create a file in the home directory
# Reboot into the environment
# See the file is there still

Testing the reset behavior would go similar to the regular test,
except you should expect the file to not show up in step 4. Same for
booting with no persistence.

Otherwise, the media should work like before, and things like live
installation should work as expected.

== User Experience ==
When booting live media, two new menu options will become available:
one to reset the persistent overlay while booting, and one to boot
without persistence. The default boot options will create a persistent
overlay if it doesn't exist and can be written (e.g. when booting from
a USB stick).

== Dependencies ==
* [https://pagure.io/fedora-kickstarts fedora-kickstarts]
* {{package|anaconda}}
* {{package|dracut}}
* {{package|livesys-scripts}}
* {{package|lorax}}

== Contingency Plan ==
* Contingency mechanism: Revert changes to fedora-kickstarts and lorax
to go back to previous behavior
* Contingency deadline: Final Freeze
* Blocks release? Yes


== Documentation ==
Information on the persistent overlay functionality is included in the
Dracut documentation.

== Release Notes ==
By default, Fedora Linux live environments flashed to writable USB
media with sufficient free space will maintain user changes to the
environment. This "persistence" can be reset at boot time through the
boot menu.


-- 
Ben Cotton
He / Him / His
Fedora Program Manager
Red Hat
TZ=America/Indiana/Indianapolis

[Kamil Paral]

On Tue, Oct 18, 2022 at 10:43 PM Ben Cotton <bcotton(a)redhat.com&gt; wrote:

>  There should be new options for resetting the persistent overlay and
>  booting with no persistence. The default options should boot with
>  persistence and setup of persistence should work.
>  
Neal, Matt, what is the rationale for enabling persistence for the default
boot option? I have mixed opinions about this. One of the benefits of a
Live image, as we use it today, is that it's always the same/fresh. If I
use it and then hand it over to a friend/colleague, I don't need to worry
that some personal files (like a browser history) were left behind. I also
don't need to worry that I (or the previous user) made some changes which
would negatively impact the installation process or my user environment
(like configuring a different keymap, or installing some updates). It's
always as the creators intended. With the proposed Change, suddenly I need
to care and need to worry.

My impression is that currently persistence use is basically non-existent.
Our well-advertised tools like Fedora Media Writer don't support it. Even
if we flip this to make it easily available (which is probably a good
thing), how many users do you estimate would actually want to make use of
it? Who would want to work from a Live image regularly? I'm sure there are
some use cases, but they seem so niche to me, that making it a non-default
boot option wouldn't be a problem at all.

I wonder if you've thought about this and why you decided to propose making
it enabled by default. Thanks.

[Neal Gompa]

On Wed, Oct 19, 2022 at 2:57 AM Kamil Paral <kparal(a)redhat.com&gt; wrote:
>  On Tue, Oct 18, 2022 at 10:43 PM Ben Cotton <bcotton(a)redhat.com&gt; wrote:
> >
> > There should be new options for resetting the persistent overlay and
> > booting with no persistence. The default options should boot with
> > persistence and setup of persistence should work.
>
>
>  Neal, Matt, what is the rationale for enabling persistence for the default boot option? I
> have mixed opinions about this. One of the benefits of a Live image, as we use it today,
> is that it's always the same/fresh. If I use it and then hand it over to a
> friend/colleague, I don't need to worry that some personal files (like a browser
> history) were left behind. I also don't need to worry that I (or the previous user)
> made some changes which would negatively impact the installation process or my user
> environment (like configuring a different keymap, or installing some updates). It's
> always as the creators intended. With the proposed Change, suddenly I need to care and
> need to worry.
>  
Hmm, that makes sense. The main reason I was thinking of it was to
make the live media useful in portable and rescue environments. Maybe
it would make sense to make the default boot option activate
persistence if it exists, but use a temporary overlay if it doesn't
(which is what we technically do today) and create menu options to
trigger overlay creation.

That said, SoaS would probably benefit from default persistence, since
I believe it's supposed to work from a USB stick.

>  My impression is that currently persistence use is basically
> non-existent. Our well-advertised tools like Fedora Media Writer don't support it.
> Even if we flip this to make it easily available (which is probably a good thing), how
> many users do you estimate would actually want to make use of it? Who would want to work
> from a Live image regularly? I'm sure there are some use cases, but they seem so niche
> to me, that making it a non-default boot option wouldn't be a problem at all.
>
>  I wonder if you've thought about this and why you decided to propose making it
> enabled by default. Thanks.
>  
I know that a big part of why persistence use is non-existent is that
support for integrating it into FMW has been deferred for years. Over
that time, the script has see-sawed from working to not working.
Having persistence means people can have Fedora environments they can
carry around, and if we advertised the capability, I think people
*would* use it.

Enabling persistence is pretty much a matter of adding a boot option
to the grub menu item. If we could modify the grub menu configuration
from FMW, then we could probably have a checkbox there to turn on/off
the capability.



-- 
真実はいつも一つ！/ Always, there's only one truth!

[Michael J Gruber]

>  On Tue, Oct 18, 2022 at 10:43 PM Ben Cotton
> <bcotton(a)redhat.com&gt; wrote:
>  
>  
>  Neal, Matt, what is the rationale for enabling persistence for the default
>  boot option? I have mixed opinions about this. One of the benefits of a
>  Live image, as we use it today, is that it's always the same/fresh. If I
>  use it and then hand it over to a friend/colleague, I don't need to worry
>  that some personal files (like a browser history) were left behind. I also
>  don't need to worry that I (or the previous user) made some changes which
>  would negatively impact the installation process or my user environment
>  (like configuring a different keymap, or installing some updates). It's
>  always as the creators intended. With the proposed Change, suddenly I need
>  to care and need to worry. 
Yes, default should be the pristine image, at least as long no persistent overlay exists.

>  My impression is that currently persistence use is basically
> non-existent.
>  Our well-advertised tools like Fedora Media Writer don't support it. Even 
And that is exactly why.

I used to use it quite a alot - and be it just selecting the right keyboard layout for
trying out an RC or such on various boxes which all happened to have the same keyboard
layout, being in the same country.

Another use case is as a handy rescue option, where - again - I might want to set up my
environment (home overlay), or even trying out selective updates between RCs, say for QA
purposes (system overlay).

[Vitaly Zaitsev]

On 18/10/2022 22:35, Ben Cotton wrote:
>  There should be new options for resetting the persistent overlay and
>  booting with no persistence. The default options should boot with
>  persistence and setup of persistence should work. 
I don't think that persistence by default is a good idea, because modern 
USB-flash drives are very unreliable and don't have a wear-cell 
balancer, so it will wear out very quickly for some frequently modified 
files.

Persistence is a good option, but should only be enabled at the user's 
choice.

-- 
Sincerely,
   Vitaly Zaitsev (vitaly(a)easycoding.org)

[Matthew Miller]

On Wed, Oct 19, 2022 at 09:22:23AM +0200, Vitaly Zaitsev via devel wrote:
>  I don't think that persistence by default is a good idea,
> because
>  modern USB-flash drives are very unreliable and don't have a
>  wear-cell balancer, so it will wear out very quickly for some
>  frequently modified files. 
Yeah — lots of very-low-quality flash drives out there. Ran into this just
trying to get decent ones for give-away swag.

I expect this to get worse and worse, as more and more people just use
dropbox and other cloud storage for file transfer and random storage. Once
again we're increasingly a niche case.

(It is, on the other hand, easier and easier to get really good MicroSD
cards for relatively cheap. SanDisk High Endurance for ten bucks. But not
quite cheap enough for giveways. Maybe the giveway item should be a branded
card reader...) 

-- 
Matthew Miller
<mattdm(a)fedoraproject.org&gt;
Fedora Project Leader

[Frank R Dana Jr.]

>  With
>  {{package|livesys-scripts}}, those scripts have been simplified and
>  turned into systemd services that activate only in live environments. 
Just to confirm, will live-media-only these systemd services all be contained within a
package (either livesys-scripts, or some other RPM), and will that package be kept
dependent-less so it can be uninstalled at any point if it's not needed?

I'm just picturing all of these new "systemd services that activate only in live
environments" possibly ending up on _installed_ systems. (Particularly since the
supported desktop installation method is to install from the live media.)

Not a problem if they can just be `dnf remove`-d, but less great if they clutter up
`systemctl list-unit-files` with a bunch of useless, perma-disabled units that can't
(easily) be removed because some core package either contains or Requires: them. systemd
_already_ comes with a **large** default collection of unit files.

[Colin Walters]

On Tue, Oct 18, 2022, at 4:35 PM, Ben Cotton wrote:
>  https://fedoraproject.org/wiki/Changes/ModernizeLiveMedia 
Just for reference, today Fedora CoreOS uses a different implementation of this:
https://github.com/coreos/fedora-coreos-config/tree/testing-devel/overlay...
See specifically
https://github.com/coreos/fedora-coreos-config/blob/testing-devel/overlay...