F36 Change: Silverblue and Kinoite will have /var on its own Btrfs subvolume (Self-Contained Change proposal)

[Ben Cotton]

https://fedoraproject.org/wiki/Changes/VarSubvol4SilverblueKinoite

== Summary ==
Silverblue and Kinoite: For new clean automatic (guided)
installations, create a "var" subvolume to be mounted at /var.

== Owner ==
* Name: [[User:chrismurphy| Chris Murphy]]
* Email: [e-mail]

== Detailed Description ==
Currently, Silverblue and Kinoite mimic other Fedora desktops. There
is a "root" subvolume mounted at `/` and a "home" subvolume mounted at
`/home`.

This proposal adds a "var" subvolume to be mounted at `/var`.

The "var" subvolume will be located at the top-level of the Btrfs file
system, along-side the "root" and "home" subvolumes. An entry in
`/etc/fstab` will mount it at `/var` during startup.

== Benefit to Fedora ==

Users who opt into Btrfs features like snapshots and rollbacks.

* By moving /var into its own subvolume, it will be excluded from
snapshots and rollbacks of the "root" subvolume, which contains `/etc`
and `/usr`.
* Users will find it straightforward to rollback "root" while not
rolling back "var": including logs, VMs, databases, flatpaks, etc.
* The ability to snapshot only "var" and use Btrfs send/receive to
replicate only "var" permits for an efficient way of backing up the
variable system data.
** A clean install can restore the "root", therefore it doesn't
strictly need to be backed up. Meanwhile "var" and "home" can be
restored using snapshot replication via send/receive.

== Scope ==
* Proposal owners:
** changes to lorax and anaconda as needed so that Silverblue and
Kinoite variants have their own installation kickstart, such that
automatic/guided installation automatically creates "var".
*** possible liability, determine whether the the addition of /var
mount point for Btrfs scheme results in /var mount point for other
schemes (and inhibit)

== Upgrade/compatibility impact ==
Change will not be applied to upgrades. But we can document steps to
apply the change to existing installations.

== How To Test ==

* Do a clean installation and check `df` and `/etc/fstab` for an
explicitly listed `/var` mount point.

== User Experience ==

* The change won't generally be noticeable to users
* Users will see an additional `/var` mount point in /etc/fstab, and `df`
* Some utilities, notably backup programs like borg backup, and rsync
with -x option, will treat Btrfs subvolumes as separate file systems
and may not descend (recursively) into them.

== Dependencies ==

* Anaconda/blivet, lorax, and possibly kickstarts

== Contingency Plan ==

* Contingency deadline: beta freeze
* Blocks release? No

== Documentation ==
No significant documentation is planned other than this change proposal.

== Release Notes ==

[Neal Gompa]

On Tue, Jan 18, 2022 at 3:44 PM Ben Cotton  wrote:
> https://fedoraproject.org/wiki/Changes/VarSubvol4SilverblueKinoite
>
> == Summary ==
> Silverblue and Kinoite: For new clean automatic (guided)
> installations, create a "var" subvolume to be mounted at /var.
>
> == Owner ==
> Name: [[User:chrismurphy| Chris Murphy]]
> Email: [e-mail]
>
> == Detailed Description ==
> Currently, Silverblue and Kinoite mimic other Fedora desktops. There
> is a "root" subvolume mounted at `/` and a "home" subvolume mounted at
> `/home`.
>
> This proposal adds a "var" subvolume to be mounted at `/var`.
>
> The "var" subvolume will be located at the top-level of the Btrfs file
> system, along-side the "root" and "home" subvolumes. An entry in
> `/etc/fstab` will mount it at `/var` during startup.
>
> == Benefit to Fedora ==
>
> Users who opt into Btrfs features like snapshots and rollbacks.
> By moving /var into its own subvolume, it will be excluded from
>
> snapshots and rollbacks of the "root" subvolume, which contains `/etc`
> and `/usr`.
> Users will find it straightforward to rollback "root" while not
>
> rolling back "var": including logs, VMs, databases, flatpaks, etc.
> The ability to snapshot only "var" and use Btrfs send/receive to
>
> replicate only "var" permits for an efficient way of backing up the
> variable system data.
> ** A clean install can restore the "root", therefore it doesn't
> strictly need to be backed up. Meanwhile "var" and "home" can be
> restored using snapshot replication via send/receive.
>
> == Scope ==
> Proposal owners:
>
> ** changes to lorax and anaconda as needed so that Silverblue and
> Kinoite variants have their own installation kickstart, such that
> automatic/guided installation automatically creates "var".
> *** possible liability, determine whether the the addition of /var
> mount point for Btrfs scheme results in /var mount point for other
> schemes (and inhibit)
>
> == Upgrade/compatibility impact ==
> Change will not be applied to upgrades. But we can document steps to
> apply the change to existing installations.
>
> == How To Test ==
> Do a clean installation and check `df` and `/etc/fstab` for an
>
> explicitly listed `/var` mount point.
>
> == User Experience ==
> The change won't generally be noticeable to users
> Users will see an additional `/var` mount point in /etc/fstab, and `df`
> Some utilities, notably backup programs like borg backup, and rsync
>
> with -x option, will treat Btrfs subvolumes as separate file systems
> and may not descend (recursively) into them.
>
> == Dependencies ==
> Anaconda/blivet, lorax, and possibly kickstarts
>
> == Contingency Plan ==
> Contingency deadline: beta freeze
> Blocks release? No
>
> == Documentation ==
> No significant documentation is planned other than this change proposal.
>
> == Release Notes ==

This is a nice enhancement and works fine because the rpmdb is already
on /usr with rpm-ostree. I'm not sure if Anaconda supports custom
storage configurations per type in the profiles used to define variant
configurations?

--
真実はいつも一つ！/ Always, there's only one truth!

[Casey Jao]

Doesn't rpm-ostree already provide transactional, image-based updates without the use of filesystem snapshots? In addition, roofs snapshots are only really useful if they are coordinated with bootloader management, which is already built into rpm-ostree but not yet written for a hypothetical btrfs-based atomic os updater.

[Neal Gompa]

On Wed, Jan 19, 2022 at 6:05 AM Casey Jao via devel
 wrote:
> Doesn't rpm-ostree already provide transactional, image-based updates without the use of filesystem snapshots? In addition, roofs snapshots are only really useful if they are coordinated with bootloader management, which is already built into rpm-ostree but not yet written for a hypothetical btrfs-based atomic os updater.

I think the bigger value would be around being able to use filesystem
snapshots with /var.

[Colin Walters]

On Wed, Jan 19, 2022, at 6:38 AM, Neal Gompa wrote:
> On Wed, Jan 19, 2022 at 6:05 AM Casey Jao via devel
>  wrote:
> > Doesn't rpm-ostree already provide transactional, image-based updates without the use of filesystem snapshots? In addition, roofs snapshots are only really useful if they are coordinated with bootloader management, which is already built into rpm-ostree but not yet written for a hypothetical btrfs-based atomic os updater.
>
> I think the bigger value would be around being able to use filesystem
> snapshots with /var.

Right.  In the ostree model the idea is all user data (except config files in /etc) are in /var.  So if you want to back up all machine-local state (including your home directories, container storage, libvirt VMs, etc.), you just need one filesystem to save.

It could make sense to save a snapshot of /var before performing major OS upgrades.  But they're not strictly related.  (As an aside: personally, I think snapshots have mostly just obscure use cases versus backup systems)

I would agree with Casey though in that the opposite case of trying to snapshot /usr (and then boot it) outside of ostree's control is likely to confuse ostree at best at least today.

[Chris Murphy]

On Wed, Jan 19, 2022 at 6:08 AM Colin Walters  wrote:
> On Wed, Jan 19, 2022, at 6:38 AM, Neal Gompa wrote:
> > On Wed, Jan 19, 2022 at 6:05 AM Casey Jao via devel
> >  wrote:
> > > Doesn't rpm-ostree already provide transactional, image-based updates without the use of filesystem snapshots? In addition, roofs snapshots are only really useful if they are coordinated with bootloader management, which is already built into rpm-ostree but not yet written for a hypothetical btrfs-based atomic os updater.
> >
> > I think the bigger value would be around being able to use filesystem
> > snapshots with /var.
>
> Right.  In the ostree model the idea is all user data (except config files in /etc) are in /var.  So if you want to back up all machine-local state (including your home directories, container storage, libvirt VMs, etc.), you just need one filesystem to save.
>
> It could make sense to save a snapshot of /var before performing major OS upgrades.  But they're not strictly related.  (As an aside: personally, I think snapshots have mostly just obscure use cases versus backup systems)
>
> I would agree with Casey though in that the opposite case of trying to snapshot /usr (and then boot it) outside of ostree's control is likely to confuse ostree at best at least today.

Yep. I've made an adjustment to emphasize the benefit of backing up
"var" rather than rolling back "root" - because while that can work,
it requires specialized knowledge to get the bootloader to do the
right thing. And as it's possible /boot contains only new
vmlinuz+initramfs compared to a potentially much older "root"
snapshot.

The rpm-ostree model could dispense with the "home" subvolume. But we
could cross this bridge later with systemd-homed, whether to store
that material in a "home" subvolume or "var" might become more clear.
The sd-homed btrfs and fscrypt backends will create per user
subvolumes when on btrfs. That moots the "var" and "home subvolume
distinction entirely. But the distinction remains for the luks backend
which is "filesystem on luks on loop" backing files, which subvolume
those user backing files are ideally located on.

My instinct is this matters less for installation and daily usage, and
more as a logical puzzle for disaster recovery. What items need to be
restored and how, in order to make a bad day less bad.

Chris Murphy