F41 Change Proposal: Enabling composefs by default for Atomic Desktops, CoreOS and IoT (Self-Contained)

[Aoife Moloney]

Wiki - https://fedoraproject.org/wiki/Changes/ComposefsAtomicCoreOSIoT
Discussion Thread -
https://discussion.fedoraproject.org/t/f41-change-proposal-enabling-composef...

This is a proposed Change for Fedora Linux.
This document represents a proposed Change. As part of the Changes
process, proposals are publicly announced in order to receive
community feedback. This proposal will only be implemented if approved
by the Fedora Engineering Steering Committee.

== Summary ==

We want to enable composefs by default for Fedora Atomic Desktops,
Fedora CoreOS and Fedora IoT. This makes the root mount of the system
(`/`) a truly read only filesystem, increasing the system integrity
and robustness. This is the first step toward a full ''at runtime''
verification of filesystem integrity.

This change will be enabled only for the Bootable Container images of
Fedora Atomic Desktops and not the classic ostree ones.

== Owner ==

* [[User:jbtrystram| Jean-Baptiste Trystram]], [e-mail]
* [[User:Siosm| Timothée Ravier]], [e-mail]
* [[User: pwhalen| Paul Whalen]], [e-mail]

== Detailed Description ==

Ostree based systems currently have `/usr` mounted as read-only and
managed by ostree/rpm-ostree. The integrity of the content of `/usr`
is only validated by ostree/rpm-ostree during updates and deployment
operations, but not at "runtime". If a file is corrupted on disk
(maliciously or not), it will only be detected if a full check is
performed using `ostree fsck`.

On those systems, the runtime root (`/`) of the system is currently
mounted as read-write but with the `immutable` bit set (`chattr +i /`)
to prevent accidental modifications.

[https://github.com/containers/composefs composefs] is a new project
that combines several existing filesystems (overlayfs, EROFS) to
provide a very flexible mechanism to support read-only mountable
filesystem trees, stacking on top of an underlying "lower" Linux
filesystem.

Using composefs, it will no longer be possible to mutate the
underlaying file content that is part of the system (`/usr`) nor the
layout of the root directory. It will result in I/O errors at the
kernel level.

The content is `/etc` and `/var` will remain writtable as it is today.

This change is part of the
[https://fedoraproject.org/wiki/Initiatives/Fedora_bootc Fedora
Bootable Containers Initiative]. The `bootc` container images already
enable composefs thus this change is to align existing variants to the
new Bootable Containers defaults.

It is tracked in:
* Fedora Atomic Desktops: https://gitlab.com/fedora/ostree/sig/-/issues/35
* Fedora CoreOS: https://github.com/coreos/fedora-coreos-tracker/issues/1718
* Fedora IoT: https://github.com/fedora-iot/iot-distro/issues/52

This is the first step toward a full boot chain integrity, that will
requiring signing the composefs metadata during composes and using
Unified Kernel Images (UKI). See:
https://gitlab.com/fedora/bootc/tracker/-/issues/14

As podman also use composefs to store containers layers, this enable
deduplication of files between containers and host. This will result
in less disk usage but also faster container startup and less memory
use. See https://github.com/containers/composefs/issues/125

== Feedback ==

Nothing specific so far.

We have the following "known issues":
* Conflicts with `ostree-grub2`, which impacts Dual Boot support:
** https://github.com/ostreedev/ostree/issues/3198
** We will remove ostree-grub2 from Fedora Atomic Desktops bootable
container images
** Related to: https://fedoraproject.org/wiki/Changes/FedoraSilverblueBootupd
** We can do this ''now'' for the container images as they are not
officially released for Fedora, and generally ''newer'', so it's more
likely that the bootloader on those systems are already BLS capable.
** We can not do this for "classic ostree" Atomic Desktops yet as we
need a transition period with bootupd enabled by default before
removing ostree-grub2.
** However with the recent Secure Boot issue
(https://github.com/fedora-silverblue/issue-tracker/issues/543)
forcing everybody to manually update their bootloader, we might be
able to shorten this transition period.
** See for Dual Boot:
https://github.com/fedora-silverblue/issue-tracker/issues/530
* No longer possible to create root level direcotries (`chattr -i` workaround):
** Requires derivation, thus the container flow
** https://github.com/coreos/rpm-ostree/issues/337
** Alternative: https://github.com/ostreedev/ostree/pull/3114
** Might impact Podman Desktop for Fedora CoreOS. They will likely
disable it until a solution is found.
* Issues with kdump:
** https://bugzilla.redhat.com/show_bug.cgi?id=2284097
** https://gitlab.com/fedora/bootc/tracker/-/issues/19

== Benefit to Fedora ==

This will increase the robustness of image based Fedora systems and
prepare them for future increased security guarantees.

This will align the existing image based variants of Fedora (Atomic
Desktops, CoreOS, IoT) to the work that is done as part of the
Bootable Containers Initiative.

== Scope ==

* Proposal owners:
** Enable composefs in Atomic Desktops (bootable containers only)
** Enable composefs in CoreOS
** Enable composefs in IoT

* Other developers:
** Applications doing disk-full checks on `/` will have to be updated
to look at other places as `/` will be small (a few MB) and full (100%
used).

* Release engineering: N/A (not needed for this Change)

* Policies and guidelines: N/A (not needed for this Change)

* Trademark approval: N/A (not needed for this Change)

* Alignment with the Fedora Strategy 2028:
** Aligns with the goal: "Immutable variants are the majority of
Fedora Linux in use"

== Upgrade/compatibility impact ==

To be fleshed out

== Early Testing (Optional) ==

Do you require 'QA Blueprint' support? N

== How To Test ==

* Make sure that you do not rely on Dual Boot support
* Make sure that you bootloader is recent enough to support BLS configs
** If you don't know, update it using the instructions from
https://github.com/fedora-silverblue/issue-tracker/issues/543#issuecomment-2...
first
* Remove `ostree-grub2` from the upcoming deployment: `rpm-ostree
override remove ostree-grub2`
* Enable composefs: `sudo ostree config set ex-integrity.composefs yes`
* Update your system to a new version: `rpm-ostree update`
** Or do a manual (re)deploy of the current version: `sudo ostree
admin deploy fedora/39/x86_64/silverblue`
* Reboot into the new deployment

== User Experience ==

The main visible change will be that the root filesystem (`/`) is now
small and full (a few MB, 100% used). The real root is mounted in
`/sysroot` and most of the data is stored in `/var`.

== Dependencies ==

For the Atomic Desktops, this change depends on:
* Bootupd support:
** https://gitlab.com/fedora/ostree/sig/-/issues/1
** https://fedoraproject.org/wiki/Changes/FedoraSilverblueBootupd

CoreOS and IoT already do not depends on `ostree-grub2`.

== Contingency Plan ==

* Contingency mechanism: (What to do?  Who will do it?) Undo the
change. It's a single line change in a configuration file.
* Contingency deadline: Beta Freeze / Release Freeze
* Blocks release? No

== Documentation ==

To be written.

== Release Notes ==

To be written once the change is accepted.