F37 Change: Enable read only /sysroot for Fedora Silverblue & Kinoite (Self-Contained Change proposal)

[Ben Cotton]

https://fedoraproject.org/wiki/Changes/Silverblue_Kinoite_readonly_sysroot

== Summary ==

This change is about enabling an opt-in ostree feature that re-mounts
`/sysroot` as read only to avoid accidental changes.

Users and administrators are not expected to directly interact with
the content available there and should instead use the interface
offered by rpm-ostree, GNOME Software or (soon) Plasma Discover to
manage their system.

== Owner ==

* Name: [[User:Siosm| Timothée Ravier]], [[User:Tpopela| Tomáš
Popela]], [[User:jkonecny| Jiří Konečný]]
* Email: [e-mail], [e-mail], [e-mail]
* FESCo shepherd: [[User:Ngompa| Neal Gompa]] [e-mail]

== Detailed Description ==

On rpm-ostree based systems, the real root (the root directory of the
root partition on the disk) is mounted under the `/sysroot` path. By
default it contains the state of the system (the content of `var` and
`etc`) as well as the system versions themselves (each versioned copy
of `/usr`) in the ostree repository (`/ostree/repo`).

This change is about enabling an opt-in ostree feature that re-mounts
`/sysroot` as read only to avoid accidental changes.

Users and administrators are not expected to directly interact with
the content available there and should instead use the interface
offered by rpm-ostree, GNOME Software or (soon) Plasma Discover to
manage their system.

Example of issue: https://github.com/fedora-silverblue/issue-tracker/issues/232

This change replicates for Fedora Silverblue/Kinoite what has been
done in Fedora CoreOS in a previous release.

== Feedback ==

None so far.

== Benefit to Fedora ==

This will make Fedora Silverblue/Kinoite more robust to accidental
damage from users.

== Scope ==
* Proposal owners:
** Work on the changes requires for new installations (potentially
Anaconda configuration changes) and support for in place updates for
existing installations (requires a two step process).
* Other developers:
** Potential Anaconda changes required.
* Release engineering: N/A
* Policies and guidelines: N/A (not needed for this Change)
* Trademark approval: N/A (not needed for this Change)
* Alignment with Objectives: N/A

== Upgrade/compatibility impact ==

We will create a systemd unit that perform the updates in place for
existing systems. This will require a two step process (changing the
existing kernel arguments, and then enabling the ostree feature). Once
the feature is enabled, user won't be able to rollback to previous
deployments where the kernel argument is not set. We will have to
clearly document that in the documentation for easier troubleshooting.

== How To Test ==

Only try the following if you are confortable debugging an un-bootable
system and have made backups!

`$ sudo rpm-ostree kargs --append-if-missing=rw`

`$ sudo ostree config --repo=/sysroot/ostree/repo set "sysroot.readonly" "true"`

`$ sudo systemctl reboot`

Note that you can not "rollback" to the previous deployment to undo
this change. You will have to boot into a Live ISO and edit the config
file in the ostree repo to remove this config option.

== User Experience ==

There should be no visible change in user experience.

== Dependencies ==

Requires changes in Anaconda (maybe just config?) to set default kargs
and property on ostree repo for new installations.

== Contingency Plan ==

Revert the change before the release.

== Documentation ==

N/A (not a System Wide Change)

== Release Notes ==

TODO

[Stephen Snow]

On Wed, 2022-02-16 at 12:12 -0500, Ben Cotton wrote:
> https://fedoraproject.org/wiki/Changes/Silverblue_Kinoite_readonly_sysroot
>
> == Summary ==
>
> This change is about enabling an opt-in ostree feature that re-mounts
> `/sysroot` as read only to avoid accidental changes.
>
> Users and administrators are not expected to directly interact with
> the content available there and should instead use the interface
> offered by rpm-ostree, GNOME Software or (soon) Plasma Discover to
> manage their system.

I use Silverblue. How does this affect my ability to modify /etc in the
opt-in scenario? Does rpm-ostree offer a method to modify /etc in that
case? What if I want a mutable /var, like I currently have, does this
change under this proposal? What is the value of this for the normal
Fedora Linux user?
> == Owner ==
> Name: [[User:Siosm| Timothée Ravier]], [[User:Tpopela| Tomáš
>
> Popela]], [[User:jkonecny| Jiří Konečný]]
> Email: [e-mail], [e-mail],
>
> [e-mail]
> FESCo shepherd: [[User:Ngompa| Neal Gompa]]
>
> [e-mail]
>
> == Detailed Description ==
>
> On rpm-ostree based systems, the real root (the root directory of the
> root partition on the disk) is mounted under the `/sysroot` path. By
> default it contains the state of the system (the content of `var` and
> `etc`) as well as the system versions themselves (each versioned copy
> of `/usr`) in the ostree repository (`/ostree/repo`).
>
> This change is about enabling an opt-in ostree feature that re-mounts
> `/sysroot` as read only to avoid accidental changes.
>
> Users and administrators are not expected to directly interact with
> the content available there and should instead use the interface
> offered by rpm-ostree, GNOME Software or (soon) Plasma Discover to
> manage their system.
>
> Example of issue:
> https://github.com/fedora-silverblue/issue-tracker/issues/232
>
> This change replicates for Fedora Silverblue/Kinoite what has been
> done in Fedora CoreOS in a previous release.
>
> == Feedback ==
>
> None so far.
>
> == Benefit to Fedora ==
>
> This will make Fedora Silverblue/Kinoite more robust to accidental
> damage from users.
>
> == Scope ==
> Proposal owners:
>
> ** Work on the changes requires for new installations (potentially
> Anaconda configuration changes) and support for in place updates for
> existing installations (requires a two step process).
> Other developers:
>
> ** Potential Anaconda changes required.
> Release engineering: N/A
> Policies and guidelines: N/A (not needed for this Change)
> Trademark approval: N/A (not needed for this Change)
> Alignment with Objectives: N/A
>
> == Upgrade/compatibility impact ==
>
> We will create a systemd unit that perform the updates in place for
> existing systems. This will require a two step process (changing the
> existing kernel arguments, and then enabling the ostree feature).
> Once
> the feature is enabled, user won't be able to rollback to previous
> deployments where the kernel argument is not set. We will have to
> clearly document that in the documentation for easier
> troubleshooting.

See, that's an unwelcome thing IMO.
> == How To Test ==
>
> Only try the following if you are confortable debugging an un-
> bootable
> system and have made backups!
>
> `$ sudo rpm-ostree kargs --append-if-missing=rw`
>
> `$ sudo ostree config --repo=/sysroot/ostree/repo set
> "sysroot.readonly" "true"`
>
> `$ sudo systemctl reboot`
>
> Note that you can not "rollback" to the previous deployment to undo
> this change. You will have to boot into a Live ISO and edit the
> config
> file in the ostree repo to remove this config option.
>
> == User Experience ==
>
> There should be no visible change in user experience.

I don't know. I think not being able to boot into my previous
deployments a visible change to my user experience.
> == Dependencies ==
>
> Requires changes in Anaconda (maybe just config?) to set default
> kargs
> and property on ostree repo for new installations.
>
> == Contingency Plan ==
>
> Revert the change before the release.
>
> == Documentation ==
>
> N/A (not a System Wide Change)
>
> == Release Notes ==
>
> TODO

Seems like there is lot's more todo.
>

Stephen
> -- 
> Ben Cotton
> He / Him / His
> Fedora Program Manager
> Red Hat
> TZ=America/Indiana/Indianapolis
> _______________________________________________
> devel mailing list -- [e-mail]
> To unsubscribe send an email to [e-mail]
> Fedora Code of Conduct:
> https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> List Guidelines:
> https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives:
> https://lists.fedoraproject.org/archives/list/[e-mail]
> Do not reply to spam on the list, report it:
> https://pagure.io/fedora-infrastructure

[Colin Walters]

On Wed, Feb 16, 2022, at 12:48 PM, Stephen Snow wrote:
> On Wed, 2022-02-16 at 12:12 -0500, Ben Cotton wrote:
> > https://fedoraproject.org/wiki/Changes/Silverblue_Kinoite_readonly_sysroot
> >
> > == Summary ==
> >
> > This change is about enabling an opt-in ostree feature that re-mounts
> > `/sysroot` as read only to avoid accidental changes.
> >
> > Users and administrators are not expected to directly interact with
> > the content available there and should instead use the interface
> > offered by rpm-ostree, GNOME Software or (soon) Plasma Discover to
> > manage their system.
>
> I use Silverblue. How does this affect my ability to modify /etc in the
> opt-in scenario?

It doesn't; `/etc` is mounted writable.
> Does rpm-ostree offer a method to modify /etc in that
> case? What if I want a mutable /var, like I currently have, does this
> change under this proposal?

`/var` does not change either.
> What is the value of this for the normal Fedora Linux user?

The basic idea is that `/sysroot` is actually an ostree implementation detail, and really nothing else should be writing to it.

Fedora CoreOS has worked this way for a long time; we just didn't make the change in ostree by default out of conservatism.
> See, that's an unwelcome thing IMO.

I think actually the migration service could inject `rw` into all bootloader entries actually.
> I don't know. I think not being able to boot into my previous
> deployments a visible change to my user experience.

The service adjusting all bootloader entries is the easy fix for this.

Or, TL;DR: Don't panic, no power is being removed and it's very likely that no one will notice.

[Stephen Snow]

Thanks for your clarification Colin.

Stephen
On Wed, 2022-02-16 at 13:16 -0500, Colin Walters wrote:
> On Wed, Feb 16, 2022, at 12:48 PM, Stephen Snow wrote:
> > On Wed, 2022-02-16 at 12:12 -0500, Ben Cotton wrote:
> > > https://fedoraproject.org/wiki/Changes/Silverblue_Kinoite_readonly_sysroot
> > >
> > > == Summary ==
> > >
> > > This change is about enabling an opt-in ostree feature that re-
> > > mounts
> > > `/sysroot` as read only to avoid accidental changes.
> > >
> > > Users and administrators are not expected to directly interact
> > > with
> > > the content available there and should instead use the interface
> > > offered by rpm-ostree, GNOME Software or (soon) Plasma Discover
> > > to
> > > manage their system.
> >
> > I use Silverblue. How does this affect my ability to modify /etc in
> > the
> > opt-in scenario?
>
> It doesn't; `/etc` is mounted writable.
> > Does rpm-ostree offer a method to modify /etc in that
> > case? What if I want a mutable /var, like I currently have, does
> > this
> > change under this proposal?
>
> `/var` does not change either.
> > What is the value of this for the normal Fedora Linux user?
>
> The basic idea is that `/sysroot` is actually an ostree
> implementation detail, and really nothing else should be writing to
> it.
>
> Fedora CoreOS has worked this way for a long time; we just didn't
> make the change in ostree by default out of conservatism.
> > See, that's an unwelcome thing IMO.
>
> I think actually the migration service could inject `rw` into all
> bootloader entries actually.
> > I don't know. I think not being able to boot into my previous
> > deployments a visible change to my user experience.
>
> The service adjusting all bootloader entries is the easy fix for
> this.
>
> Or, TL;DR: Don't panic, no power is being removed and it's very
> likely that no one will notice.
> _______________________________________________
> devel mailing list -- [e-mail]
> To unsubscribe send an email to [e-mail]
> Fedora Code of Conduct:
> https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> List Guidelines:
> https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives:
> https://lists.fedoraproject.org/archives/list/[e-mail]
> Do not reply to spam on the list, report it:
> https://pagure.io/fedora-infrastructure

[Timothée Ravier]

> I use Silverblue. How does this affect my ability to modify /etc in the
> opt-in scenario? Does rpm-ostree offer a method to modify /etc in that
> case? What if I want a mutable /var, like I currently have, does this
> change under this proposal? What is the value of this for the normal
> Fedora Linux user?

Note that this is only about `/sysroot`, not `/` (root). Thus `/etc` and `var` are still writable in all cases.
> I don't know. I think not being able to boot into my previous
> deployments a visible change to my user experience.

Once the new option has been set, you will have to append `rw` to the kernel command line to previous deployments that were created before.

We could also probably make this even more conservative and have it only enabled once all your deployments have the new kernel option. This should avoid the case where you can not bee an older deployment anymore.

The idea is to make the update happen in two steps:
- First add the `rw` kernel arguments,
- Then set up the `sysroot.readonly` option once all previous deployments have the rw option.

--
Timothée Ravier
CoreOS engineer & Fedora Kinoite maintainer

[Stephen Snow]

Very good,
Thank you for further clarification.

Stephen

On Wed, 2022-02-16 at 18:18 +0000, Timothée Ravier wrote:
> > I use Silverblue. How does this affect my ability to modify /etc in
> > the
> > opt-in scenario? Does rpm-ostree offer a method to modify /etc in
> > that
> > case? What if I want a mutable /var, like I currently have, does
> > this
> > change under this proposal? What is the value of this for the
> > normal
> > Fedora Linux user?
>
> Note that this is only about `/sysroot`, not `/` (root). Thus `/etc`
> and `var` are still writable in all cases.
> > I don't know. I think not being able to boot into my previous
> > deployments a visible change to my user experience.
>
> Once the new option has been set, you will have to append `rw` to the
> kernel command line to previous deployments that were created before.
>
> We could also probably make this even more conservative and have it
> only enabled once all your deployments have the new kernel option.
> This should avoid the case where you can not bee an older deployment
> anymore.
>
> The idea is to make the update happen in two steps:
> First add the `rw` kernel arguments,
> Then set up the `sysroot.readonly` option once all previous
>
> deployments have the rw option.
>
> --
> Timothée Ravier
> CoreOS engineer & Fedora Kinoite maintainer
> _______________________________________________
> devel mailing list -- [e-mail]
> To unsubscribe send an email to [e-mail]
> Fedora Code of Conduct:
> https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> List Guidelines:
> https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives:
> https://lists.fedoraproject.org/archives/list/[e-mail]
> Do not reply to spam on the list, report it:
> https://pagure.io/fedora-infrastructure

[Kuba Sierzega]

> We could also probably make this even more conservative and have it only enabled once all
> your deployments have the new kernel option. This should avoid the case where you can not
> bee an older deployment anymore.

Hello

I would like to ask how it will work if I have pinned deployments for older version? It is very common to pin last deployment of Fedora 36 before an upgrade to Fedora 37 so the deployment will not be automatically removed. So, it is possible that the system will never be in state that all deployments is updated to use new kernel option. So, what about this?

[Jiri Konecny]

Thanks a lot for creating this. It wasn't nice experience after running 
`restorecon -Rv /` -- yeah, now I know you shouldn't do that on SB.

Jirka

Dne 16. 02. 22 v 18:12 Ben Cotton napsal(a):
> https://fedoraproject.org/wiki/Changes/Silverblue_Kinoite_readonly_sysroot
>
> == Summary ==
>
> This change is about enabling an opt-in ostree feature that re-mounts
> `/sysroot` as read only to avoid accidental changes.
>
> Users and administrators are not expected to directly interact with
> the content available there and should instead use the interface
> offered by rpm-ostree, GNOME Software or (soon) Plasma Discover to
> manage their system.
>
> == Owner ==
> Name: [[User:Siosm| Timothée Ravier]], [[User:Tpopela| Tomáš
>
> Popela]], [[User:jkonecny| Jiří Konečný]]
> Email: [e-mail], [e-mail], [e-mail]
> FESCo shepherd: [[User:Ngompa| Neal Gompa]] [e-mail]
>
> == Detailed Description ==
>
> On rpm-ostree based systems, the real root (the root directory of the
> root partition on the disk) is mounted under the `/sysroot` path. By
> default it contains the state of the system (the content of `var` and
> `etc`) as well as the system versions themselves (each versioned copy
> of `/usr`) in the ostree repository (`/ostree/repo`).
>
> This change is about enabling an opt-in ostree feature that re-mounts
> `/sysroot` as read only to avoid accidental changes.
>
> Users and administrators are not expected to directly interact with
> the content available there and should instead use the interface
> offered by rpm-ostree, GNOME Software or (soon) Plasma Discover to
> manage their system.
>
> Example of issue: https://github.com/fedora-silverblue/issue-tracker/issues/232
>
> This change replicates for Fedora Silverblue/Kinoite what has been
> done in Fedora CoreOS in a previous release.
>
> == Feedback ==
>
> None so far.
>
> == Benefit to Fedora ==
>
> This will make Fedora Silverblue/Kinoite more robust to accidental
> damage from users.
>
> == Scope ==
> Proposal owners:
>
> ** Work on the changes requires for new installations (potentially
> Anaconda configuration changes) and support for in place updates for
> existing installations (requires a two step process).
> Other developers:
>
> ** Potential Anaconda changes required.
> Release engineering: N/A
> Policies and guidelines: N/A (not needed for this Change)
> Trademark approval: N/A (not needed for this Change)
> Alignment with Objectives: N/A
>
> == Upgrade/compatibility impact ==
>
> We will create a systemd unit that perform the updates in place for
> existing systems. This will require a two step process (changing the
> existing kernel arguments, and then enabling the ostree feature). Once
> the feature is enabled, user won't be able to rollback to previous
> deployments where the kernel argument is not set. We will have to
> clearly document that in the documentation for easier troubleshooting.
>
> == How To Test ==
>
> Only try the following if you are confortable debugging an un-bootable
> system and have made backups!
>
> `$ sudo rpm-ostree kargs --append-if-missing=rw`
>
> `$ sudo ostree config --repo=/sysroot/ostree/repo set "sysroot.readonly" "true"`
>
> `$ sudo systemctl reboot`
>
> Note that you can not "rollback" to the previous deployment to undo
> this change. You will have to boot into a Live ISO and edit the config
> file in the ostree repo to remove this config option.
>
> == User Experience ==
>
> There should be no visible change in user experience.
>
> == Dependencies ==
>
> Requires changes in Anaconda (maybe just config?) to set default kargs
> and property on ostree repo for new installations.
>
> == Contingency Plan ==
>
> Revert the change before the release.
>
> == Documentation ==
>
> N/A (not a System Wide Change)
>
> == Release Notes ==
>
> TODO

[Zbigniew Jędrzejewski-Szmek]

On Wed, Feb 16, 2022 at 12:12:59PM -0500, Ben Cotton wrote:
> == Summary ==
>
> This change is about enabling an opt-in ostree feature that re-mounts
> `/sysroot` as read only to avoid accidental changes.
>
> Users and administrators are not expected to directly interact with
> the content available there and should instead use the interface
> offered by rpm-ostree, GNOME Software or (soon) Plasma Discover to
> manage their system.
> This change is about enabling an opt-in ostree feature that re-mounts
> `/sysroot` as read only to avoid accidental changes.
>
> Users and administrators are not expected to directly interact with
> the content available there and should instead use the interface
> offered by rpm-ostree, GNOME Software or (soon) Plasma Discover to
> manage their system.

Not a big thing, but please remove the duplicated text.

Zbyszek