booting successfully with read-only file system

[Zbigniew Jędrzejewski-Szmek]

Hi,

this is partially an outgrowth of the discussion about btrfs as
default, but makes sense independently too...

It would be great if we could fairly reliably boot with a read-only
root file system, all the way to the graphical environment. Obviously,
such a machine will not be fully functional, but for users, debugging a
disk problem when they have the normal environment with windows,
tabbed terminals, graphical editors, and internet is vastly easier.

It also creates an image of robustness. Imagine that instead of being
rudely dropped to a terminal prompt, the user is instead able to log in
as usual and see a popup like
>  Your home directory is read-only. Do this and that. See https://...
Is the goal to have *everything* working? No. Some services will and
should fail. If I have a database or anything else which only makes
sense with permanent storage, failing early and loudly is appropriate.
But services which need writable storage only tangentially or not at
all should be robust and not fail. Journald behaves in a fashion where
it stores logs to /run during early boot and them flushes them to /var/log
when that becomes available. If /var/log never become available, we
have a functional logs, with journalctl showing previous and current boot
just fine. The only caveat is that logs for current boot will be lost
upon reboot. Such graceful failure should be the norm.

systemd upstream recently gained a cool feature [1] which allows all
block devices to be toggled read-only as soon as they are detected by
udev. The primary use case is forensics and recovery, but it is also
great for testing read-only boot ;)

It turns out that systemd itself is not very good in this situation.
For example, any unit with PrivateTmp=yes would fail to start :(
But it turns out that this is fairly easy to solve. I opened two
PRs today [2, 3], and with that, systemd boots to a working
multi-user.target with no services bundled with systemd failing!

But I was not able to go all the way to a gnome session :(
I have been opening bugs as I went along: dnf [4], python [5], sssd [6],
gssproxy [7], gdm [8], btrfs [9], xfs[10]. The good new is that the
first is almost solved, the second is already solved, the next two
seem fairly easy, and the btrfs one is being handled. The one for gdm
unfortunately looks tougher :(

I hope we can all cooperate to make read-only boots nicely robust and
functional. Please play with this and report bugs. I'll try to solve any
that relate to systemd. The systemd version with udev.blockdev-read-only
is not released yet, but is available from koji ci builds [11].

[1] https://github.com/systemd/systemd/commit/95ac523030
[2] https://github.com/systemd/systemd/pull/16340
[3] https://github.com/systemd/systemd/pull/16344
[4] https://bugzilla.redhat.com/show_bug.cgi?id=1852365
[5] https://bugzilla.redhat.com/show_bug.cgi?id=1852941
[6] https://bugzilla.redhat.com/show_bug.cgi?id=1853261
[7] https://bugzilla.redhat.com/show_bug.cgi?id=1853293
[8] https://bugzilla.redhat.com/show_bug.cgi?id=1853409
[9] https://bugzilla.redhat.com/show_bug.cgi?id=1851608
[10] https://bugzilla.redhat.com/show_bug.cgi?id=1829792
[11] https://src.fedoraproject.org/rpms/systemd/pull-request/29

Zbyszek

[Langdon White]

On Thu, Jul 2, 2020 at 11:54 AM Zbigniew Jędrzejewski-Szmek <
zbyszek(a)in.waw.pl&gt; wrote:

>  Hi,
>
>  this is partially an outgrowth of the discussion about btrfs as
>  default, but makes sense independently too...
>
>  It would be great if we could fairly reliably boot with a read-only
>  root file system, all the way to the graphical environment. Obviously,
>  such a machine will not be fully functional, but for users, debugging a
>  disk problem when they have the normal environment with windows,
>  tabbed terminals, graphical editors, and internet is vastly easier.
>
>  It also creates an image of robustness. Imagine that instead of being
>  rudely dropped to a terminal prompt, the user is instead able to log in
>  as usual and see a popup like
>  > Your home directory is read-only. Do this and that. See https://...
>
>  Is the goal to have *everything* working? No. Some services will and
>  should fail. If I have a database or anything else which only makes
>  sense with permanent storage, failing early and loudly is appropriate.
>  But services which need writable storage only tangentially or not at
>  all should be robust and not fail. Journald behaves in a fashion where
>  it stores logs to /run during early boot and them flushes them to /var/log
>  when that becomes available. If /var/log never become available, we
>  have a functional logs, with journalctl showing previous and current boot
>  just fine. The only caveat is that logs for current boot will be lost
>  upon reboot. Such graceful failure should be the norm.
>
>  systemd upstream recently gained a cool feature [1] which allows all
>  block devices to be toggled read-only as soon as they are detected by
>  udev. The primary use case is forensics and recovery, but it is also
>  great for testing read-only boot ;)
>
>  It turns out that systemd itself is not very good in this situation.
>  For example, any unit with PrivateTmp=yes would fail to start :(
>  But it turns out that this is fairly easy to solve. I opened two
>  PRs today [2, 3], and with that, systemd boots to a working
>  multi-user.target with no services bundled with systemd failing!
>
>  But I was not able to go all the way to a gnome session :(
>  I have been opening bugs as I went along: dnf [4], python [5], sssd [6],
>  gssproxy [7], gdm [8], btrfs [9], xfs[10]. The good new is that the
>  first is almost solved, the second is already solved, the next two
>  seem fairly easy, and the btrfs one is being handled. The one for gdm
>  unfortunately looks tougher :(
>
>  I hope we can all cooperate to make read-only boots nicely robust and
>  functional. Please play with this and report bugs. I'll try to solve any
>  that relate to systemd. The systemd version with udev.blockdev-read-only
>  is not released yet, but is available from koji ci builds [11].
>
>  [1] https://github.com/systemd/systemd/commit/95ac523030
>  [2] https://github.com/systemd/systemd/pull/16340
>  [3] https://github.com/systemd/systemd/pull/16344
>  [4] https://bugzilla.redhat.com/show_bug.cgi?id=1852365
>  [5] https://bugzilla.redhat.com/show_bug.cgi?id=1852941
>  [6] https://bugzilla.redhat.com/show_bug.cgi?id=1853261
>  [7] https://bugzilla.redhat.com/show_bug.cgi?id=1853293
>  [8] https://bugzilla.redhat.com/show_bug.cgi?id=1853409
>  [9] https://bugzilla.redhat.com/show_bug.cgi?id=1851608
>  [10] https://bugzilla.redhat.com/show_bug.cgi?id=1829792
>  [11] https://src.fedoraproject.org/rpms/systemd/pull-request/29
>
>  Zbyszek
>  
+about a billion or so! Particularly fun if you forget to set the root
password after install and can't even get to the prompt.

langdon



>  _______________________________________________
>  devel mailing list -- devel(a)lists.fedoraproject.org
>  To unsubscribe send an email to devel-leave(a)lists.fedoraproject.org
>  Fedora Code of Conduct:
>  https://docs.fedoraproject.org/en-US/project/code-of-conduct/
>  List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
>  List Archives:
>  https://lists.fedoraproject.org/archives/list/[e-mail]
>

[Daniel P. Berrangé]

On Thu, Jul 02, 2020 at 03:53:26PM +0000, Zbigniew Jędrzejewski-Szmek wrote:
>  Hi,
>  
>  this is partially an outgrowth of the discussion about btrfs as
>  default, but makes sense independently too...
>  
>  It would be great if we could fairly reliably boot with a read-only
>  root file system, all the way to the graphical environment. Obviously,
>  such a machine will not be fully functional, but for users, debugging a
>  disk problem when they have the normal environment with windows,
>  tabbed terminals, graphical editors, and internet is vastly easier.
>  
>  It also creates an image of robustness. Imagine that instead of being
>  rudely dropped to a terminal prompt, the user is instead able to log in
>  as usual and see a popup like
>  > Your home directory is read-only. Do this and that. See https://...
>  
>  Is the goal to have *everything* working? No. Some services will and
>  should fail. If I have a database or anything else which only makes
>  sense with permanent storage, failing early and loudly is appropriate.
>  But services which need writable storage only tangentially or not at
>  all should be robust and not fail. Journald behaves in a fashion where
>  it stores logs to /run during early boot and them flushes them to /var/log
>  when that becomes available. If /var/log never become available, we
>  have a functional logs, with journalctl showing previous and current boot
>  just fine. The only caveat is that logs for current boot will be lost
>  upon reboot. Such graceful failure should be the norm. 
I presume you're referring to regular Fedora here, but this description
feels like it is approx asking for what Fedora Silverblue has delivered,
only with the writable area for apps being just a ram disk with no
persistence.

  https://docs.fedoraproject.org/en-US/fedora-silverblue/

  "Silverblue is a variant of Fedora Workstation. It looks,
   feels and behaves like a regular desktop operating system,
   and the experience is similar to what you find with using 
   a standard Fedora Workstation.

   However, unlike other operating systems, Silverblue is 
   immutable. This means that every installation is identical
   to every other installation of the same version. The operating
   system that is on disk is exactly the same from one machine to
   the next, and it never changes as it is used.

   Silverblue’s immutable design is intended to make it more 
   stable, less prone to bugs, and easier to test and develop. 
   Finally, Silverblue’s immutable design also makes it an 
   excellent platform for containerized applications as well 
   as container-based software development. In each case, 
   applications (apps) and containers are kept separate from 
   the host system, improving stability and reliability."


Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

[Neal Gompa]

On Thu, Jul 2, 2020 at 12:05 PM Daniel P. Berrangé <berrange(a)redhat.com&gt; wrote:
>  On Thu, Jul 02, 2020 at 03:53:26PM +0000, Zbigniew Jędrzejewski-Szmek wrote:
>  > Hi,
>  >
>  > this is partially an outgrowth of the discussion about btrfs as
>  > default, but makes sense independently too...
>  >
>  > It would be great if we could fairly reliably boot with a read-only
>  > root file system, all the way to the graphical environment. Obviously,
>  > such a machine will not be fully functional, but for users, debugging a
>  > disk problem when they have the normal environment with windows,
>  > tabbed terminals, graphical editors, and internet is vastly easier.
>  >
>  > It also creates an image of robustness. Imagine that instead of being
>  > rudely dropped to a terminal prompt, the user is instead able to log in
>  > as usual and see a popup like
>  > > Your home directory is read-only. Do this and that. See https://...
>  >
>  > Is the goal to have *everything* working? No. Some services will and
>  > should fail. If I have a database or anything else which only makes
>  > sense with permanent storage, failing early and loudly is appropriate.
>  > But services which need writable storage only tangentially or not at
>  > all should be robust and not fail. Journald behaves in a fashion where
>  > it stores logs to /run during early boot and them flushes them to /var/log
>  > when that becomes available. If /var/log never become available, we
>  > have a functional logs, with journalctl showing previous and current boot
>  > just fine. The only caveat is that logs for current boot will be lost
>  > upon reboot. Such graceful failure should be the norm.
>
>  I presume you're referring to regular Fedora here, but this description
>  feels like it is approx asking for what Fedora Silverblue has delivered,
>  only with the writable area for apps being just a ram disk with no
>  persistence.
>  
Silverblue fails when the disk can't mount as read-write either. So
this benefits Silverblue by making it possible for those failure modes
to work properly.




--
真実はいつも一つ！/ Always, there's only one truth!

[Zbigniew Jędrzejewski-Szmek]

On Thu, Jul 02, 2020 at 05:05:09PM +0100, Daniel P. Berrangé wrote:
>  On Thu, Jul 02, 2020 at 03:53:26PM +0000, Zbigniew Jędrzejewski-Szmek
> wrote:
>  > Hi,
>  > 
>  > this is partially an outgrowth of the discussion about btrfs as
>  > default, but makes sense independently too...
>  > 
>  > It would be great if we could fairly reliably boot with a read-only
>  > root file system, all the way to the graphical environment. Obviously,
>  > such a machine will not be fully functional, but for users, debugging a
>  > disk problem when they have the normal environment with windows,
>  > tabbed terminals, graphical editors, and internet is vastly easier.
>  > 
>  > It also creates an image of robustness. Imagine that instead of being
>  > rudely dropped to a terminal prompt, the user is instead able to log in
>  > as usual and see a popup like
>  > > Your home directory is read-only. Do this and that. See https://...
>  > 
>  > Is the goal to have *everything* working? No. Some services will and
>  > should fail. If I have a database or anything else which only makes
>  > sense with permanent storage, failing early and loudly is appropriate.
>  > But services which need writable storage only tangentially or not at
>  > all should be robust and not fail. Journald behaves in a fashion where
>  > it stores logs to /run during early boot and them flushes them to /var/log
>  > when that becomes available. If /var/log never become available, we
>  > have a functional logs, with journalctl showing previous and current boot
>  > just fine. The only caveat is that logs for current boot will be lost
>  > upon reboot. Such graceful failure should be the norm.
>  
>  I presume you're referring to regular Fedora here, but this description
>  feels like it is approx asking for what Fedora Silverblue has delivered,
>  only with the writable area for apps being just a ram disk with no
>  persistence. 
No, quite the opposite. I am talking about making the best out of a failure
situation on a normal Fedora installation. I don't want to boot routinely
in such a mode. I want to boot ro, look for help on the internet and in man
pages, do some fsck operations on the read-only disk, and finally remount
the disk rw and carry on with my day.

Zbyszek

>   https://docs.fedoraproject.org/en-US/fedora-silverblue/
> 
>   "Silverblue is a variant of Fedora Workstation. It looks,
>    feels and behaves like a regular desktop operating system,
>    and the experience is similar to what you find with using 
>    a standard Fedora Workstation.
> 
>    However, unlike other operating systems, Silverblue is 
>    immutable. This means that every installation is identical
>    to every other installation of the same version. The operating
>    system that is on disk is exactly the same from one machine to
>    the next, and it never changes as it is used.
> 
>    Silverblue’s immutable design is intended to make it more 
>    stable, less prone to bugs, and easier to test and develop. 
>    Finally, Silverblue’s immutable design also makes it an 
>    excellent platform for containerized applications as well 
>    as container-based software development. In each case, 
>    applications (apps) and containers are kept separate from 
>    the host system, improving stability and reliability."
> 
> 
> Regards,
> Daniel
> -- 
> |: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
> |: https://libvirt.org         -o-            https://fstop138.berrange.com :|
> |: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|
> _______________________________________________
> devel mailing list -- devel(a)lists.fedoraproject.org
> To unsubscribe send an email to devel-leave(a)lists.fedoraproject.org
> Fedora Code of Conduct:
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives:
https://lists.fedoraproject.org/archives/list/[e-mail]

[Colin Walters]

On Thu, Jul 2, 2020, at 12:05 PM, Daniel P. Berrangé wrote:

>  I presume you're referring to regular Fedora here, but this
> description
>  feels like it is approx asking for what Fedora Silverblue has delivered,
>  only with the writable area for apps being just a ram disk with no
>  persistence. 
No no no!  That's *not at all* what it is.  I am actually kind of at war with the
"immutable" terminology because it's more isleading than it is accurate.

https://blog.verbum.org/2019/12/23/starting-from-open-and-foss/
is a bit related to this.

/etc and /var are fully persistent!  It's just an "image based" update
system, and /usr is read-only by default, but you can still layer/override packages, and
in the future I'd like to make it easier to use ostree to persist non-rpm changes.

[Vitaly Zaitsev]

On 02.07.2020 17:53, Zbigniew Jędrzejewski-Szmek wrote:
>  It would be great if we could fairly reliably boot with a read-only
>  root file system, all the way to the graphical environment. 
Already implemented - Silverblue.

-- 
Sincerely,
  Vitaly Zaitsev (vitaly(a)easycoding.org)

[Zbigniew Jędrzejewski-Szmek]

On Thu, Jul 02, 2020 at 06:27:44PM +0200, Vitaly Zaitsev via devel wrote:
>  On 02.07.2020 17:53, Zbigniew Jędrzejewski-Szmek wrote:
>  > It would be great if we could fairly reliably boot with a read-only
>  > root file system, all the way to the graphical environment.
>  
>  Already implemented - Silverblue. 
If you read past the first sentence, you'd notice that I'm talking
about a system in failure conditions.

If you looked in at earlier replies to my mail, you'd also notice that
your idea was already discussed upthread.

But instead of just replying to the subject, you actually got as far
as the first sentence, so ... there is hope ;)

Zbyszek

PS. I wrote quite seriously about a real problem. I don't need quick
replies that don't answer anything.

[Christopher Engelhard]

On 02.07.20 17:53, Zbigniew Jędrzejewski-Szmek wrote:
>  It would be great if we could fairly reliably boot with a read-only
>  root file system, all the way to the graphical environment. Obviously,
>  such a machine will not be fully functional, but for users, debugging a
>  disk problem when they have the normal environment with windows,
>  tabbed terminals, graphical editors, and internet is vastly easier.
>  
>  It also creates an image of robustness. Imagine that instead of being
>  rudely dropped to a terminal prompt, the user is instead able to log in
>  as usual and see a popup like
> > Your home directory is read-only. Do this and that. See https://... 
That would be fantastic, and would be miles ahead from any UX I had on
any computer, ever.

>  I hope we can all cooperate to make read-only boots nicely robust
> and
>  functional. Please play with this and report bugs. I'll try to solve any
>  that relate to systemd. The systemd version with udev.blockdev-read-only
>  is not released yet, but is available from koji ci builds [11]. 
Thanks for working on this, I will definitely give it a try myself.

Christopher

[Brandon Nielsen]

On 7/2/20 3:10 PM, Christopher Engelhard wrote:
>  On 02.07.20 17:53, Zbigniew Jędrzejewski-Szmek wrote:
> > It would be great if we could fairly reliably boot with a read-only
> > root file system, all the way to the graphical environment. Obviously,
> > such a machine will not be fully functional, but for users, debugging a
> > disk problem when they have the normal environment with windows,
> > tabbed terminals, graphical editors, and internet is vastly easier.
> >
> > It also creates an image of robustness. Imagine that instead of being
> > rudely dropped to a terminal prompt, the user is instead able to log in
> > as usual and see a popup like
> >> Your home directory is read-only. Do this and that. See https://...
>  
>  That would be fantastic, and would be miles ahead from any UX I had on
>  any computer, ever.
>  
> > I hope we can all cooperate to make read-only boots nicely robust and
> > functional. Please play with this and report bugs. I'll try to solve any
> > that relate to systemd. The systemd version with udev.blockdev-read-only
> > is not released yet, but is available from koji ci builds [11].
>  
>  Thanks for working on this, I will definitely give it a try myself.
>  
>  Christopher 
This sounds excellent!

Could we somehow provide a list of links that may be helpful to recover 
from certain common failures? Maybe in a MOTD?

[Zbigniew Jędrzejewski-Szmek]

On Thu, Jul 02, 2020 at 03:17:58PM -0500, Brandon Nielsen wrote:
>  On 7/2/20 3:10 PM, Christopher Engelhard wrote:
>  >On 02.07.20 17:53, Zbigniew Jędrzejewski-Szmek wrote:
>  >>It would be great if we could fairly reliably boot with a read-only
>  >>root file system, all the way to the graphical environment. Obviously,
>  >>such a machine will not be fully functional, but for users, debugging a
>  >>disk problem when they have the normal environment with windows,
>  >>tabbed terminals, graphical editors, and internet is vastly easier.
>  >>
>  >>It also creates an image of robustness. Imagine that instead of being
>  >>rudely dropped to a terminal prompt, the user is instead able to log in
>  >>as usual and see a popup like
>  >>>Your home directory is read-only. Do this and that. See https://...
>  >
>  >That would be fantastic, and would be miles ahead from any UX I had on
>  >any computer, ever.
>  >
>  >>I hope we can all cooperate to make read-only boots nicely robust and
>  >>functional. Please play with this and report bugs. I'll try to solve any
>  >>that relate to systemd. The systemd version with udev.blockdev-read-only
>  >>is not released yet, but is available from koji ci builds [11].
>  >
>  >Thanks for working on this, I will definitely give it a try myself.
>  >
>  >Christopher
>  
>  This sounds excellent!
>  
>  Could we somehow provide a list of links that may be helpful to
>  recover from certain common failures? Maybe in a MOTD? 
Ubuntu's MOTD are well known and people seem to like them a
lot. Fedora hasn't been making that much use of them. But I think we
should in general.

If we switch to btrfs by default, we'll certainly need to beef up the
docs about diagnosis and recovery. If MOTD poped up a link to them if
there are any fs-related issues in the initial boot would be pretty
nifty.

Zbyszek

[Roberto Ragusa]

On 2020-07-03 08:07, Zbigniew Jędrzejewski-Szmek wrote:

>  Ubuntu's MOTD are well known and people seem to like them a
>  lot. Fedora hasn't been making that much use of them. But I think we
>  should in general. 
Do not follow Ubuntu on this too much.
I've had a case of a machine with many short-lived incoming ssh connections
where the performance bottleneck was the constant regeneration of
the motd on each login. Sabotaging that stuff brutally was the only way
to get back to good performance.

Regards.

-- 
    Roberto Ragusa    mail at robertoragusa.it

[Neal Gompa]

On Fri, Jul 3, 2020 at 6:50 AM Roberto Ragusa <mail(a)robertoragusa.it&gt; wrote:
>  On 2020-07-03 08:07, Zbigniew Jędrzejewski-Szmek wrote:
>
>  > Ubuntu's MOTD are well known and people seem to like them a
>  > lot. Fedora hasn't been making that much use of them. But I think we
>  > should in general.
>
>  Do not follow Ubuntu on this too much.
>  I've had a case of a machine with many short-lived incoming ssh connections
>  where the performance bottleneck was the constant regeneration of
>  the motd on each login. Sabotaging that stuff brutally was the only way
>  to get back to good performance.
>  
Ubuntu's MOTD requires network connectivity to function, I think ours
would not. That would eliminate the network resource contention issues
you were seeing.



-- 
真実はいつも一つ！/ Always, there's only one truth!

[Vitaly Zaitsev]

On 03.07.2020 13:42, Neal Gompa wrote:
>  Ubuntu's MOTD requires network connectivity to function, I think
> ours
>  would not. That would eliminate the network resource contention issues
>  you were seeing. 
The good MOTD is an empty MOTD.

-- 
Sincerely,
  Vitaly Zaitsev (vitaly(a)easycoding.org)

[Vitaly Zaitsev]

On 03.07.2020 08:07, Zbigniew Jędrzejewski-Szmek wrote:
>  Ubuntu's MOTD are well known and people seem to like them a
>  lot. Fedora hasn't been making that much use of them. But I think we
>  should in general. 
Ubuntu MOTD contains ads. Most of Ubuntu users completely disable it
right after the installation.

-- 
Sincerely,
  Vitaly Zaitsev (vitaly(a)easycoding.org)

[Colin Walters]

On Thu, Jul 2, 2020, at 11:53 AM, Zbigniew Jędrzejewski-Szmek wrote:

>  It would be great if we could fairly reliably boot with a read-only
>  root file system, 
Eh, just mount a tmpfs for /var, and an overlayfs for /etc (backed by a tmpfs).

That's what we do for Fedora CoreOS based live images, see
https://github.com/coreos/fedora-coreos-config/blob/testing-devel/overlay...
It works.

(Which we recently switched to involve a loopback-mounted xfs on tmpfs because SELinux,
but that is mostly only necessary because we want to support Ignition which does system
provisioning in the initramfs, which is not true on non-Ignition based systems)

[Zbigniew Jędrzejewski-Szmek]

On Fri, Jul 03, 2020 at 09:18:42AM -0400, Colin Walters wrote:
>  On Thu, Jul 2, 2020, at 11:53 AM, Zbigniew Jędrzejewski-Szmek wrote:
>  
>  > It would be great if we could fairly reliably boot with a read-only
>  > root file system,
>  
>  Eh, just mount a tmpfs for /var, and an overlayfs for /etc (backed by a tmpfs). 
I see that this thread is one massive communication failure on my part :(

I wrote about "booting successfully with a read-only file system", but I
see that I didn't say "... when the disk cannot be mounted rw because of
file system errors". I thought it'd be clear from the context, but it's
clearly not. Anyway, while I'm a big fan of coreos and read-only-on-purpose,
I was writing about traditional systems in a read-only-by-accident scenario,
i.e. about the system behaving gracefully when the disk is ***unexpectedly***
read-only.

Zbyszek

PS. OK, I know I wrote about making it read-only on purpose using a
kernel commandline option, so really we're just pretending it was
unexpected for testing purposes, but you get what I mean I hope.

[Peter Pentchev]

On Fri, Jul 03, 2020 at 01:32:12PM +0000, Zbigniew Jędrzejewski-Szmek wrote:
>  On Fri, Jul 03, 2020 at 09:18:42AM -0400, Colin Walters wrote:
>  > On Thu, Jul 2, 2020, at 11:53 AM, Zbigniew Jędrzejewski-Szmek wrote:
>  > 
>  > > It would be great if we could fairly reliably boot with a read-only
>  > > root file system,
>  > 
>  > Eh, just mount a tmpfs for /var, and an overlayfs for /etc (backed by a tmpfs).
>  
>  I see that this thread is one massive communication failure on my part :(
>  
>  I wrote about "booting successfully with a read-only file system", but I
>  see that I didn't say "... when the disk cannot be mounted rw because of
>  file system errors". 
...but Colin did not say that. Neither tmpfs nor overlayfs (backed by
tmpfs) require any existing disk filesystems to be mounted read-write.
TBH, that was my first thought when I read your original e-mail: "why
would units fail if they only want to write non-persistent stuff to
an area that may perfectly well be mounted as tmpfs, just as /run
usually is?"

>  I thought it'd be clear from the context, but it's
>  clearly not. Anyway, while I'm a big fan of coreos and read-only-on-purpose,
>  I was writing about traditional systems in a read-only-by-accident scenario,
>  i.e. about the system behaving gracefully when the disk is ***unexpectedly***
>  read-only.
>  
>  Zbyszek
>  
>  PS. OK, I know I wrote about making it read-only on purpose using a
>  kernel commandline option, so really we're just pretending it was
>  unexpected for testing purposes, but you get what I mean I hope. 
G'luck,
Peter

-- 
Peter Pentchev  roam(a)ringlet.net roam(a)debian.org pp(a)storpool.com
PGP key:        http://people.FreeBSD.org/~roam/roam.key.asc
Key fingerprint 2EE7 A7A5 17FC 124C F115  C354 651E EFB0 2527 DF13

[Colin Walters]

On Fri, Jul 3, 2020, at 9:32 AM, Zbigniew Jędrzejewski-Szmek wrote:
>  On Fri, Jul 03, 2020 at 09:18:42AM -0400, Colin Walters wrote:
>  > On Thu, Jul 2, 2020, at 11:53 AM, Zbigniew Jędrzejewski-Szmek wrote:
>  > 
>  > > It would be great if we could fairly reliably boot with a read-only
>  > > root file system,
>  > 
>  > Eh, just mount a tmpfs for /var, and an overlayfs for /etc (backed by a tmpfs).
>  
>  I see that this thread is one massive communication failure on my part :(
>  
>  I wrote about "booting successfully with a read-only file system", but I
>  see that I didn't say "... when the disk cannot be mounted rw because of
>  file system errors". I thought it'd be clear from the context, but it's
>  clearly not. Anyway, while I'm a big fan of coreos and read-only-on-purpose, 
It's really unfortunate how much confusion there is on the "read only"
topic...
I know there's a fair amount of subtley here but I would hope at least
a few more people in the Fedora community take the time to actually
dive in to the ostree model and understand things.

What I was pointing at is the Fedora CoreOS *LIVE* ISO, which is definitely
fully read only (or phased more usefully), does not support persistence
at all because physical ISOs don't - same as any other "Live" system
from Anaconda to all the others.

But that's a totally distinct thing from merely having /usr mounted ro
by default, while still supporting persistence for /etc and /var (i.e. the ostree model).

>  I was writing about traditional systems in a read-only-by-accident
> scenario,
>  i.e. about the system behaving gracefully when the disk is ***unexpectedly***
>  read-only. 
That is an important detail indeed =)

To be clear I agree with the effort!  I think it's going to get really messy
to take it very far...and that's what I was getting at in arguing for
using overlayfs backed by tmpfs basically.

Or maybe it should be more like a "recovery shell" - rather than trying
to log in as your regular user and watch e.g. Firefox fail because it can't
write to /home and wonder just how much of the next years of your
life is going to involve patching software to make this work ;)
get enough to get the a desktop launched for a separate ephemeral
"live" user with sudo rights or so.

[Neal Gompa]

On Fri, Jul 3, 2020 at 9:56 AM Colin Walters <walters(a)verbum.org&gt; wrote:
>  What I was pointing at is the Fedora CoreOS *LIVE* ISO, which is definitely
>  fully read only (or phased more usefully), does not support persistence
>  at all because physical ISOs don't - same as any other "Live" system
>  from Anaconda to all the others.
>  
This is actually not true. All Fedora live ISOs are supposed to
support persistence, and can be configured that way on a USB stick
using livecd-iso-to-disk from the livecd-iso-to-mediums package.
However, this is not yet exposed in Fedora Media Writer.



-- 
真実はいつも一つ！/ Always, there's only one truth!

[Zbigniew Jędrzejewski-Szmek]

On Fri, Jul 03, 2020 at 09:56:03AM -0400, Colin Walters wrote:
>  
>  
>  On Fri, Jul 3, 2020, at 9:32 AM, Zbigniew Jędrzejewski-Szmek wrote:
>  > On Fri, Jul 03, 2020 at 09:18:42AM -0400, Colin Walters wrote:
>  > > On Thu, Jul 2, 2020, at 11:53 AM, Zbigniew Jędrzejewski-Szmek wrote:
>  > > 
>  > > > It would be great if we could fairly reliably boot with a read-only
>  > > > root file system,
>  > > 
>  > > Eh, just mount a tmpfs for /var, and an overlayfs for /etc (backed by a
> tmpfs).
>  > 
>  > I see that this thread is one massive communication failure on my part :(
>  > 
>  > I wrote about "booting successfully with a read-only file system", but I
>  > see that I didn't say "... when the disk cannot be mounted rw because of
>  > file system errors". I thought it'd be clear from the context, but
> it's
>  > clearly not. Anyway, while I'm a big fan of coreos and read-only-on-purpose,
>  
>  It's really unfortunate how much confusion there is on the "read only"
> topic... 
Yeah ;(

>  I know there's a fair amount of subtley here but I would hope at
> least
>  a few more people in the Fedora community take the time to actually
>  dive in to the ostree model and understand things.
>  
>  What I was pointing at is the Fedora CoreOS *LIVE* ISO, which is definitely
>  fully read only (or phased more usefully), does not support persistence
>  at all because physical ISOs don't - same as any other "Live" system
>  from Anaconda to all the others.
>  
>  But that's a totally distinct thing from merely having /usr mounted ro
>  by default, while still supporting persistence for /etc and /var (i.e. the ostree model).
Right, so IIUC, if /var is read-only, ostree will have the same problems.

>  > I was writing about traditional systems in a
> read-only-by-accident scenario,
>  > i.e. about the system behaving gracefully when the disk is ***unexpectedly***
>  > read-only.
>  
>  That is an important detail indeed =)
>  
>  To be clear I agree with the effort!  I think it's going to get really messy
>  to take it very far...and that's what I was getting at in arguing for
>  using overlayfs backed by tmpfs basically. 
This would be another approach. E.g. we systemd could detect that root
cannot be remounted rw and insert the overlay. But I don't think this
as useful as it sounds, because it would create a fake impression that
the storage is rw. The patch for systemd to deal with ro /var/tmp for
PrivateTmp=yes is careful to set things up so that the service actually
gets a real EROFS error from the kernel when it writes to /var/tmp. It
also gets the real EROFS when it uses StateDirectory=foo and /var/foo is ro.
I think such real errors are good — they allow services which can do
a graceful fallback to do it, and services which should fail to fail.
So systemd-update-utmp.service may just write to utmp and warn about
wtmp being inaccessible, but postgresql.service should fail to start and
not accept any transactions that would then be lost upon reboot.

An overlay would also make it hard to go back to normal operation.
Maybe that's not so important since one can always reboot, but I think
the current property of being able to fix the fs and remount it rw is
quite nice. (Also, less risky then trying to reboot and realizing that
there are more errors which have caused the fs to be mounted ro again.)

>  Or maybe it should be more like a "recovery shell" - rather
> than trying
>  to log in as your regular user and watch e.g. Firefox fail because it can't
>  write to /home and wonder just how much of the next years of your
>  life is going to involve patching software to make this work ;)
>  get enough to get the a desktop launched for a separate ephemeral
>  "live" user with sudo rights or so. 
I'm not sure if this is such a remote goal. Programs *should* already
be prepared for reasonable read-only operation, because this can
already happen for a few different reasons: apart from the fs being
mounted read-only, the disk may be full or the user may exceed quota.
For an application the effect is similar. And those conditions can
happen at any point at runtime. Disks may also be remounted ro at
runtime if errors are detected. So any applications which falls flat
on its face if it cannot write to the disk is problematic already.

Zbyszek

[Steven Whitehouse]

Hi,

On 03/07/2020 14:18, Colin Walters wrote:
>  On Thu, Jul 2, 2020, at 11:53 AM, Zbigniew Jędrzejewski-Szmek wrote:
>
> > It would be great if we could fairly reliably boot with a read-only
> > root file system,
>  Eh, just mount a tmpfs for /var, and an overlayfs for /etc (backed by a tmpfs).
>
>  That's what we do for Fedora CoreOS based live images, see
>
> https://github.com/coreos/fedora-coreos-config/blob/testing-devel/overlay...
>  It works.
>
>  (Which we recently switched to involve a loopback-mounted xfs on tmpfs because SELinux,
> but that is mostly only necessary because we want to support Ignition which does system
> provisioning in the initramfs, which is not true on non-Ignition based systems) 
If there is additional support required in overlayfs, then please do 
file a bug and request it,

Steve.