7:15 a.m.
I'm trying to pin down what exposed this bug:
https://bugzilla.redhat.com/show_bug.cgi?id=1569970
The immediate trigger seems to be that all shutdowns on my system leave
the XFS root file system in an unclean state, so that GRUB cannot read
recently written files under /boot (assuming that /boot is on the same
file system).
Since GRUB does not deal with the XFS journal, the effect is that the
system is unbootable until you mount / and trigger journal recovery,
after which everything is fine again (because the files in /boot are
only written to rarely).
Any ideas what could cause this, and how to debug this further? It's
clearly GRUB's fault that it cannot read the XFS journal, but the
consistently unclean shutdowns probably have been caused by a kernel or
systemd change, or maybe something else entirely.
Thanks,
Florian
1:20 p.m.
On Fri, Apr 20, 2018 at 6:15 AM, Florian Weimer <fweimer(a)redhat.com> wrote:
I'm trying to pin down what exposed this bug:
https://bugzilla.redhat.com/show_bug.cgi?id=1569970
The immediate trigger seems to be that all shutdowns on my system leave the
XFS root file system in an unclean state, so that GRUB cannot read recently
written files under /boot (assuming that /boot is on the same file system).
Since GRUB does not deal with the XFS journal, the effect is that the system
is unbootable until you mount / and trigger journal recovery, after which
everything is fine again (because the files in /boot are only written to
rarely).
Any ideas what could cause this, and how to debug this further? It's
clearly GRUB's fault that it cannot read the XFS journal, but the
consistently unclean shutdowns probably have been caused by a kernel or
systemd change, or maybe something else entirely.
It's sabotage. The characteristics leading up to this sabotage:
a. User and installer, innocently ignorant of the sabotage, put the
bootloader configuration on rootfs (i.e. /boot is not a separate fs).
b. grubby, grub-mkconfig do not guarantee the modified bootloader
configuration is fully committed to disk (flushed from the log)
c. GRUB doesn't support log replay, which is why it is obligated under
b. to flush it's shit correctly
d. systemd uses the honor system to allow services to exempt
themselves from being killed, plymouth exempts itself in this commit
https://cgit.freedesktop.org/plymouth/commit/?id=9e5a276f322cfce46b5b2ed2...
That leads to systemd neither cleanly umounting rootfs nor remounting
read-only, which means the log is not flushed when it forcibly reboots
anyway.
Result: the grub.cfg can't be read by the bootloader and now the user
is betrayed.
If any of those four things are false, the problem doesn't happen. The
unscientific sampling suggests XFS is more prone to the problem than
ext4, but switching file systems or a file system specific ioctl is in
my view inadequate. Btrs seems immune, but in theory it's possible
you'd see the bootloader find a stale bootloader configuration which
boots the previous kernel, log replays, and at next reboot the
bootloader sees new bootloader config and kernel.
I'm honestly mystified why the plymouth commit hasn't been reverted in
the interim. But I'm also mystified why the bootloader folks don't
give a shit to commit their configuration files to disk when they know
they can't do journal replay and have known that for 20 years. But
then I'm also mystified why systemd developers won't fallback to
freeze/thaw if rootfs remount-ro fails three times, instead of just
giving up and forcing a reboot, they did discuss doing this a year ago
and then poof, no action.
--
Chris Murphy
2:23 p.m.
Once upon a time, Chris Murphy <lists(a)colorremedies.com> said:
On Fri, Apr 20, 2018 at 6:15 AM, Florian Weimer
<fweimer(a)redhat.com> wrote:
> I'm trying to pin down what exposed this bug:
>
> https://bugzilla.redhat.com/show_bug.cgi?id=1569970
>
> The immediate trigger seems to be that all shutdowns on my system leave the
> XFS root file system in an unclean state, so that GRUB cannot read recently
> written files under /boot (assuming that /boot is on the same file system).
Just to note: I believe I ran into this with a separate /boot. I was
hitting what turned out to be a Mesa bug that caused my Intel GPU to
lock up; I tried updating to a newer kernel to see if that would help,
but happened to hit the lockup right after "dnf update" finished, so I
hit the power button.
Poof, grub2 found no config on boot. Oddly, I could still view the
grub.cfg with the grub CLI, so (eventually) I was able to manually boot,
write out a new grub.cfg, and recover.
So while systemd could (and should) do better, IMHO the problem is up
the line with some combination of GRUB not reading the journal and
grubby not forcing a sync.
Another solution might be to have a dnf plugin that forces a journal
flush (not a bad idea in general after loading updates), but that would
be kind of a band-aid over this problem.
--
Chris Adams <linux(a)cmadams.net>
4:32 p.m.
On Fri, Apr 20, 2018 at 1:23 PM, Chris Adams <linux(a)cmadams.net> wrote:
Once upon a time, Chris Murphy <lists(a)colorremedies.com> said:
> On Fri, Apr 20, 2018 at 6:15 AM, Florian Weimer <fweimer(a)redhat.com> wrote:
> > I'm trying to pin down what exposed this bug:
> >
> > https://bugzilla.redhat.com/show_bug.cgi?id=1569970
> >
> > The immediate trigger seems to be that all shutdowns on my system leave the
> > XFS root file system in an unclean state, so that GRUB cannot read recently
> > written files under /boot (assuming that /boot is on the same file system).
Just to note: I believe I ran into this with a separate /boot. I was
hitting what turned out to be a Mesa bug that caused my Intel GPU to
lock up; I tried updating to a newer kernel to see if that would help,
but happened to hit the lockup right after "dnf update" finished, so I
hit the power button.
Poof, grub2 found no config on boot. Oddly, I could still view the
grub.cfg with the grub CLI, so (eventually) I was able to manually boot,
write out a new grub.cfg, and recover.
So while systemd could (and should) do better, IMHO the problem is up
the line with some combination of GRUB not reading the journal and
grubby not forcing a sync.
Another solution might be to have a dnf plugin that forces a journal
flush (not a bad idea in general after loading updates), but that would
be kind of a band-aid over this problem.
It's a good anecdote showing that separate /boot isn't a sure solution
for this problem.
The central problem is bad design, wrong assumptions, by grubby and
GRUB (grub-mkconfig). They both assume something else will commit the
changes to /boot that add kernel, initramfs, and bootloader
configuration. And sync() is only sufficient on non-journaled file
systems, on journaled file systems sync() only ensures metadata is in
the log, it's considered acceptable to depend on log replay to make
the file system consistent, but of course the bootloader can't do
that. And that's why whatever modifies /boot last is most responsible
for making sure the changes are fully committed, and the only way to
do that on a journaled file system is freeze/thaw.
Hence
https://github.com/rhboot/grubby/issues/25
https://savannah.gnu.org/bugs/index.php?52657
--
Chris Murphy
6:49 p.m.
Once upon a time, Chris Murphy <lists(a)colorremedies.com> said:
And sync() is only sufficient on non-journaled file
systems, on journaled file systems sync() only ensures metadata is in
the log, it's considered acceptable to depend on log replay to make
the file system consistent, but of course the bootloader can't do
that.
So, to me, this is the bug. GRUB claims to read XFS, but it only reads
XFS in certain states (and doesn't throw an error if the filesystem is
in any other state, just fails). If GRUB can't read XFS in the common
reasonable states, then it shouldn't claim XFS support, and Fedora
shouldn't be using XFS for any filesystem GRUB needs to read.
That IMHO would mean one of:
- fix GRUB to be able to handle the XFS journal in some form
- change Fedora so that if there's a /boot, it gets ext2
- change Fedora so that otherwise, don't use XFS for /
I understand not handling significant filesystem errors, but "file is in
the journal" shouldn't be one of them.
--
Chris Adams <linux(a)cmadams.net>
3:23 p.m.
On Fri, Apr 20, 2018 at 5:49 PM, Chris Adams <linux(a)cmadams.net> wrote:
Once upon a time, Chris Murphy <lists(a)colorremedies.com> said:
> And sync() is only sufficient on non-journaled file
> systems, on journaled file systems sync() only ensures metadata is in
> the log, it's considered acceptable to depend on log replay to make
> the file system consistent, but of course the bootloader can't do
> that.
So, to me, this is the bug. GRUB claims to read XFS, but it only reads
XFS in certain states (and doesn't throw an error if the filesystem is
in any other state, just fails). If GRUB can't read XFS in the common
reasonable states, then it shouldn't claim XFS support, and Fedora
shouldn't be using XFS for any filesystem GRUB needs to read.
That IMHO would mean one of:
- fix GRUB to be able to handle the XFS journal in some form
Discussed on XFS devel, and considered overly complicated to deal with
the memory requirements and effectively two different code bases that
would result.
- change Fedora so that if there's a /boot, it gets ext2
- change Fedora so that otherwise, don't use XFS for /
It's valid to use this to exemplify the absurdity of the problem. But
it's as impractical as throwing the baby out with the bathwater.
Nothing is actually being fixed with these changes, and worse it
ignores the fundamental problem with /boot modifying programs.
I understand not handling significant filesystem errors, but
"file is in
the journal" shouldn't be one of them.
And there's a prescribed solution by fs devs for this: FIFREEZE/FITHAW
Only recently did grubby start doing an fsync after grub.cfg is
renamed, and grub-mkconfig doesn't use sync of any variety at all
ever. Seems to me they were designed with the assumption /boot would
be cleanly unmounted and I don't think that's reasonable.
--
Chris Murphy
5:27 p.m.
On Fri, Apr 20, 2018 at 5:32 PM, Chris Murphy <lists(a)colorremedies.com> wrote:
On Fri, Apr 20, 2018 at 1:23 PM, Chris Adams
<linux(a)cmadams.net> wrote:
> Another solution might be to have a dnf plugin that forces a
journal
> flush (not a bad idea in general after loading updates), but that would
> be kind of a band-aid over this problem.
It's a good anecdote showing that separate /boot isn't a sure solution
for this problem.
The central problem is bad design, wrong assumptions, by grubby and
GRUB (grub-mkconfig). They both assume something else will commit the
changes to /boot that add kernel, initramfs, and bootloader
configuration. And sync() is only sufficient on non-journaled file
systems, on journaled file systems sync() only ensures metadata is in
the log, it's considered acceptable to depend on log replay to make
the file system consistent, but of course the bootloader can't do
that. And that's why whatever modifies /boot last is most responsible
for making sure the changes are fully committed, and the only way to
do that on a journaled file system is freeze/thaw.
The command line tool "sync" or something like it used to be a
critical step, even before Linux, in shutting down operating systems.
In the meantime, does the "sync" command line tool do a full
"freeze/thaw"? Is there such a tool right now, one that can be
incorporated as an optional procedure for people using XFS and
installing new kernels, until this can be resolved? I can well
understand that the "sync" command line tool can cause a system lockup
if there is media that is mounted but unresponsive, such as network
drives or disabled storage media. It can also cause a real performance
hit to run "sync" on a very, very busy system.
end an email to devel-leave(a)lists.fedoraproject.org
4:32 p.m.
On Fr, 20.04.18 12:20, Chris Murphy (lists(a)colorremedies.com) wrote:
I'm honestly mystified why the plymouth commit hasn't been
reverted in
the interim. But I'm also mystified why the bootloader folks don't
give a shit to commit their configuration files to disk when they know
they can't do journal replay and have known that for 20 years. But
then I'm also mystified why systemd developers won't fallback to
freeze/thaw if rootfs remount-ro fails three times, instead of just
giving up and forcing a reboot, they did discuss doing this a year ago
and then poof, no action.
Quite frankly, if you want to put the blame somewhere, I'd probably
place it with the xfs folks? I mean, there's a well-defined API on
linux for syncing a file system to disk so that it is in a clean
state, it's called sync(). Turns out that doesn't work though, it
doesn't actually do that.
systemd calls that API during shutdown if it is unable to unmount some
file system because the kernel refuses it. sysvinit did it like
that. Upstart did it that way. *Everybody* else does it that way, too.
There's FIFREEZE/FITHAW in xfs. It has a very different purpose from
syncing. It does considerably more: it also stops all further I/O to
the file system. That is seriously awful, if the process invoking it
actually runs from that very file system. But sure, we can pretend
this wasn't an issue and call mlockall() first (which we do), and
immediately FITHAW after FIFREEZE. Besides obviously being brittle and
ugly, will that be sufficient? I am not sure. Let's not forget the
actual trigger of the issue: plymouth is running but its binary was
updated while it alreadywas. The old file is now deleted but remains
pinned as long as plymouth is running. Basically as long as plymouth
is running the file system will have operations pending, hence it is
likely to get dirtied pretty soon again after that FIFREEZE/FITHAW
dance...
But then there's the other thing: in order to call FIFREEZE/FITHAW we
need to open() an fd on it. Which means actually doing disk accesses
(including possibly enqueing write accesses due to atime) actually,
and that's something we currently try hard to avoid, because doing
that on file systems that aren't healthy anymore means deadlocks. In
particular for network backed file systems this actually matters: for
them we want to issue an umount() or remount syscall, and only that,
we never want to actually access the file system, because the network
is very likely already down or otherwise unavailable. Now you might
say "but xfs is not a network file system!". That's not true
unfortunately, iscsi, nbd and all that other stacked jumble means
everything is a network file system these days.
The sync() syscall doesn't suffer by this issue. Yes, it will of
course trigger disk I/O too, that's it's whole point after
all. However, it will only do so on file systems that known dirty, and
won't generate new I/O on its own.
There was a thread about this a while back on systemd-devel:
https://lists.freedesktop.org/archives/systemd-devel/2017-April/038615.html
(and around there). Back than the open() issue wasn't clear to me,
this only came up recently when we worked on some NFS-related umount
work (specifically: current systemd will now implement in userspace a
time-out around mount() and sync() because of the general flakiness of
that interface).
Hence I am pretty sure FIFREEZE is really not useful. It has a
different purpose, and by using it we might make things nicer for some
but much worse for many others. In that thread I indicated I'd merge a
patch that adds it. At this point I am convinced that that would just
be a game of whack-a-mole, and we'd just make things worse on
networking fs...
Hence, I am pretty sure that xfs should fix their implementation of
sync(). I'd also be fine with calling some other API for this if they
really don't want to fix sync() — as long as it is generic, and not
some xfs specific hack. The key really here is that the API should
actually do what is needed here, i.e. no pausing of IO or so. And it
should not require us to open an fd on the file system in question.
That all said: I figure plymouth should be changed to start in the
initrd and then stick around for good, and never be updated/replaced
by any binary from the host system. That way it can't and won't keep
an files pinned from the host fs, if it's updated, as it will be
purely backed by the initrd file system. We generally require this
from storage tools, and plymouth should do the same.
Lennart
5:05 p.m.
On Fri, Apr 20, 2018 at 3:32 PM, Lennart Poettering
<mzerqung(a)0pointer.de> wrote:
On Fr, 20.04.18 12:20, Chris Murphy (lists(a)colorremedies.com) wrote:
> I'm honestly mystified why the plymouth commit hasn't been reverted in
> the interim. But I'm also mystified why the bootloader folks don't
> give a shit to commit their configuration files to disk when they know
> they can't do journal replay and have known that for 20 years. But
> then I'm also mystified why systemd developers won't fallback to
> freeze/thaw if rootfs remount-ro fails three times, instead of just
> giving up and forcing a reboot, they did discuss doing this a year ago
> and then poof, no action.
Quite frankly, if you want to put the blame somewhere, I'd probably
place it with the xfs folks? I mean, there's a well-defined API on
linux for syncing a file system to disk so that it is in a clean
state, it's called sync(). Turns out that doesn't work though, it
doesn't actually do that.
No we've already been over this a year ago. That well defined API
predated journaled file systems, it's only a guarantee on
non-journaled file systems. Requiring this for journaled file systems
would break them by making them insanely slow for every fsync. It
would totally obviate the whole point of journaling.
systemd calls that API during shutdown if it is unable to unmount
some
file system because the kernel refuses it. sysvinit did it like
that. Upstart did it that way. *Everybody* else does it that way, too.
The systemd sync() is sufficient for pretty much everything except the
bootloader stuff. Log replay during boot will make everything
consistent again. So while I appreciate the idea of systemd doing a
freeze/thaw and I'm not totally opposed to the idea, I think the
burden here is really on grubby and GRUB first and foremost because
they are the ones who can't depend on either fsck or log replay for
their changes to be properly on disk.
The alternative, as Eric Sandeed suggests in the bug report, is a
compulsory /boot on ext2 (or maybe ext4 with journal disabled) which
then makes sync() fully flush to stable media, as there is no log.
*shrug* We do this weird thing with persistently mounted /boot and
this exposes that file system to unclean shutdowns too, so now there's
an obligatory fsck which the bootloader can't do either. I'm not
completely certain that a no log file system on /boot is really the
way forward - not least of which it amounts to "we need you to
reinstall your system to fix this problem, which by the way also
requires a new partitioning layout from what you've been using"
There's FIFREEZE/FITHAW in xfs. It has a very different purpose
from
syncing. It does considerably more: it also stops all further I/O to
the file system. That is seriously awful, if the process invoking it
actually runs from that very file system. But sure, we can pretend
this wasn't an issue and call mlockall() first (which we do), and
immediately FITHAW after FIFREEZE. Besides obviously being brittle and
ugly, will that be sufficient?
Nope, because a crash, hang, user forces power off, can happen well
before the presumably last second freeze thaw that systemd would
issue.
I am not sure. Let's not forget the
actual trigger of the issue: plymouth is running but its binary was
updated while it alreadywas. The old file is now deleted but remains
pinned as long as plymouth is running. Basically as long as plymouth
is running the file system will have operations pending, hence it is
likely to get dirtied pretty soon again after that FIFREEZE/FITHAW
dance...
Fair point. But I also note that it's exempting itself from being
clobbered by systemd contrary to:
Again: if your code is being run from the root file system, then this
logic suggested above is NOT for you. Sorry. Talk to us, we can
probably help you to find a different solution to your problem.
https://www.freedesktop.org/wiki/Software/systemd/RootStorageDaemons/
And while that is an instigator in this problem, I still don't think
logically this absolves grubby and GRUB from fully committing their
changes before reboot is even called.
But then there's the other thing: in order to call
FIFREEZE/FITHAW we
need to open() an fd on it. Which means actually doing disk accesses
(including possibly enqueing write accesses due to atime) actually,
and that's something we currently try hard to avoid, because doing
that on file systems that aren't healthy anymore means deadlocks. In
particular for network backed file systems this actually matters: for
them we want to issue an umount() or remount syscall, and only that,
we never want to actually access the file system, because the network
is very likely already down or otherwise unavailable. Now you might
say "but xfs is not a network file system!". That's not true
unfortunately, iscsi, nbd and all that other stacked jumble means
everything is a network file system these days.
The sync() syscall doesn't suffer by this issue. Yes, it will of
course trigger disk I/O too, that's it's whole point after
all. However, it will only do so on file systems that known dirty, and
won't generate new I/O on its own.
There was a thread about this a while back on systemd-devel:
https://lists.freedesktop.org/archives/systemd-devel/2017-April/038615.html
(and around there). Back than the open() issue wasn't clear to me,
this only came up recently when we worked on some NFS-related umount
work (specifically: current systemd will now implement in userspace a
time-out around mount() and sync() because of the general flakiness of
that interface).
Hence I am pretty sure FIFREEZE is really not useful. It has a
different purpose, and by using it we might make things nicer for some
but much worse for many others. In that thread I indicated I'd merge a
patch that adds it. At this point I am convinced that that would just
be a game of whack-a-mole, and we'd just make things worse on
networking fs...
OK.
Hence, I am pretty sure that xfs should fix their implementation of
sync(). I'd also be fine with calling some other API for this if they
really don't want to fix sync() — as long as it is generic, and not
some xfs specific hack. The key really here is that the API should
actually do what is needed here, i.e. no pausing of IO or so. And it
should not require us to open an fd on the file system in question.
From my conversations with fs devs, this is not an XFS problem. It can
happen on ext4 as well, it just seems to flush its log a lot faster
than XFS. And what I recall about Btrfs is that it manifests
differently, the bootloader ends up loading a stale bootloader
configuration since everything is copy on write, and thus it boots the
previous kernel, and during boot the log tree updates the fs metadata,
and the second boot uses the new bootloader config and the new kernel
is booted. I have no idea how f2fs behaves.
Anyway, I remain unconvinced this is an XFS specific problem, it's
just a lot easier to reproduce on XFS.
That all said: I figure plymouth should be changed to start in the
initrd and then stick around for good, and never be updated/replaced
by any binary from the host system. That way it can't and won't keep
an files pinned from the host fs, if it's updated, as it will be
purely backed by the initrd file system. We generally require this
from storage tools, and plymouth should do the same.
That would fix the problem in the broad case, because it would allow
systemd remount-ro to succeed, which then fully flushes /boot changes
to disk rather than just to the log. But in the narrower case of
crash, hang, or forced poweroff in between /boot modification and
successful remount-ro it actually doesn't help. The user is screwed.
So I still think that the last thing to modify /boot is duty bound to
commit changes to disk - even if it has to hang for 1 minute, which
would allow the file system to commit changes on its own.
I have no idea if what you've learned about freeze thaw limitations
apply to grubby and grub-mkconfig and rpm-ostree doing it? Right now a
part of grubby, new-kernel-pkg calls freeze thaw but only on PPC64LE.
So extending this to happen on all archs whenever /boot is on XFS,
ext4, and Btrfs seems a lot easier than most any other single thing
I've come across so far.
--
Chris Murphy
12:36 p.m.
On Fri, Apr 20, 2018 at 3:32 PM, Lennart Poettering
<mzerqung(a)0pointer.de> wrote:
On Fr, 20.04.18 12:20, Chris Murphy (lists(a)colorremedies.com) wrote:
> I'm honestly mystified why the plymouth commit hasn't been reverted in
> the interim. But I'm also mystified why the bootloader folks don't
> give a shit to commit their configuration files to disk when they know
> they can't do journal replay and have known that for 20 years. But
> then I'm also mystified why systemd developers won't fallback to
> freeze/thaw if rootfs remount-ro fails three times, instead of just
> giving up and forcing a reboot, they did discuss doing this a year ago
> and then poof, no action.
Quite frankly, if you want to put the blame somewhere, I'd probably
place it with the xfs folks? I mean, there's a well-defined API on
linux for syncing a file system to disk so that it is in a clean
state, it's called sync(). Turns out that doesn't work though, it
doesn't actually do that.
OK after a bunch of additional testing, one thing is certain; grubby
and grub-mkconfig are not even remotely crash safe. They do not
fsync() or sync() at all. That's not good no matter the file system.
The best near term change that I think we should make is putting a
sync() somewhere in grubby. I've tested modifying new-kernel-pkg to
add a sync right before the last line, and
# dnf update -y /path/to/kernelrpms && echo b > /proc/sysrq-trigger
With very limited testing (dozens of reboots in multiple fs
configurations of a VM only, using unsafe caching), all configurations
including XFS are better off with sync() added. It a highly
non-deterministic mess to sort out without sync().
But with sync() and with /boot on XFS: the kernel, initramfs, or
grub.cfg - there is an unknown (let's say 50/50) probability the
bootloader can't find or read those files despite the sync(). The most
common outcome I ran into was a partially modified grub.cfg containing
an incomplete default entry for the new kernel which is missing the
grubby insertion for the initrd. So it can't boot. But if the grub
menu appears at all, the previous kernel and rescue entries are still
there, and they work. Log replay during boot fixes up everything and
now the next boot works fine by default without user interaction. So
even though sync() is still not good enough for XFS always, it's way
better than no sync().
Meanwhile FAT, ext2, ext4, and Btrfs all appear to flush to disk with
sync() sufficient for the bootloader to have no problems.
Note that fsync() is actually insufficient at least some of the time
on all journaled file systems. e.g. dracut -f only does fsync() on the
copied over initramfs to /boot and that new initramfs is often not
seen by the bootloader for journaled file systems. Instead, the
bootloader sees the old initramfs that had been deleted, which during
boot (if successful) replays the log and completes the delete of old
initramfs, and now the new one appears and isused at next boot. Other
combinations are possible.
Minor in comparison, I see grubby doing a metric ton of fdatasync() on
/var/log/grubby - 80 fdatasync() 's for 17 lines and 1.2KiB of
changes for a single kernel installation. Does that make sense to
anyone? How about one fdatasync() for all of those changes? Or just
fsync() the file?
I think any plan for doing freeze/thaw anywhere can be postponed
pending further discussion and testing.
In the meantime I'll come up with a patch for new-kernel-pkg to do a
sync() unless someone else beats me to it.
--
Chris Murphy
1:12 p.m.
On Di, 24.04.18 11:36, Chris Murphy (lists(a)colorremedies.com) wrote:
Note that fsync() is actually insufficient at least some of the time
on all journaled file systems. e.g. dracut -f only does fsync() on the
copied over initramfs to /boot and that new initramfs is often not
seen by the bootloader for journaled file systems. Instead, the
bootloader sees the old initramfs that had been deleted, which during
boot (if successful) replays the log and completes the delete of old
initramfs, and now the new one appears and isused at next boot. Other
combinations are possible.
BTW, just to mention this: by default, if you let systemd mount the
EFI ESP for you then it will do that as automount point, meaning after
a few seconds of not accessing the ESP it will be unmounted
again. This means during normal operation the ESP won't be mounted at
all, and it will only be mounted (and thus potentially be in a dirty
state) for a very short time window around actual accesses, even
though it appears available all the time due to the automount logic.
Quite frankly, I think it would be a good idea, to just drop the
seperate /boot stuff, and simply place the kernel+initrd directly in
the ESP, and let systemd do its automount magic on it. That has the
benefit that the firmware groks the file system as well as our kernel
does, and that we do our best to keep the fs in the cleanest state
possible. In addition it's a much simpler approach, as we just reuse
the stuff already there and don't invent anything complex new.
Lennart
--
Lennart Poettering, Red Hat
2:08 p.m.
On Fri, Apr 20, 2018 at 3:32 PM, Lennart Poettering
<mzerqung(a)0pointer.de> wrote:
OK after a bunch of additional testing, one thing is certain; grubby
and grub-mkconfig are not even remotely crash safe. They do not
fsync() or sync() at all. That's not good no matter the file system.
May I point you to https://github.com/rhboot/grubby/pull/24 which *should* fix most of
that.
https://github.com/rhboot/grubby/pull/28 will fix *all* of that when /boot is a separate
FS.
The best near term change that I think we should make is putting a
sync() somewhere in grubby. I've tested modifying new-kernel-pkg to
add a sync right before the last line, and
# dnf update -y /path/to/kernelrpms && echo b > /proc/sysrq-trigger
With very limited testing (dozens of reboots in multiple fs
configurations of a VM only, using unsafe caching), all configurations
including XFS are better off with sync() added. It a highly
non-deterministic mess to sort out without sync().
But with sync() and with /boot on XFS: the kernel, initramfs, or
grub.cfg - there is an unknown (let's say 50/50) probability the
bootloader can't find or read those files despite the sync(). The most
common outcome I ran into was a partially modified grub.cfg containing
an incomplete default entry for the new kernel which is missing the
grubby insertion for the initrd. So it can't boot. But if the grub
menu appears at all, the previous kernel and rescue entries are still
there, and they work. Log replay during boot fixes up everything and
now the next boot works fine by default without user interaction. So
even though sync() is still not good enough for XFS always, it's way
better than no sync().
Meanwhile FAT, ext2, ext4, and Btrfs all appear to flush to disk with
sync() sufficient for the bootloader to have no problems.
Note that fsync() is actually insufficient at least some of the time
on all journaled file systems. e.g. dracut -f only does fsync() on the
copied over initramfs to /boot and that new initramfs is often not
seen by the bootloader for journaled file systems.
Not after
https://github.com/dracutdevs/dracut/commit/58e3971b920fbb60e5a90edfd30aa...
Instead, the
bootloader sees the old initramfs that had been deleted, which during
boot (if successful) replays the log and completes the delete of old
initramfs, and now the new one appears and isused at next boot. Other
combinations are possible.
Minor in comparison, I see grubby doing a metric ton of fdatasync() on
/var/log/grubby - 80 fdatasync() 's for 17 lines and 1.2KiB of
changes for a single kernel installation. Does that make sense to
anyone? How about one fdatasync() for all of those changes? Or just
fsync() the file?
I think any plan for doing freeze/thaw anywhere can be postponed
pending further discussion and testing.
In the meantime I'll come up with a patch for new-kernel-pkg to do a
sync() unless someone else beats me to it.
9:07 p.m.
On Tue, Apr 24, 2018 at 1:08 PM, François Cami <fdc(a)fcami.net> wrote:
> On Fri, Apr 20, 2018 at 3:32 PM, Lennart Poettering
> <mzerqung(a)0pointer.de> wrote:
>
> OK after a bunch of additional testing, one thing is certain; grubby
> and grub-mkconfig are not even remotely crash safe. They do not
> fsync() or sync() at all. That's not good no matter the file system.
May I point you to https://github.com/rhboot/grubby/pull/24 which *should* fix most of
that.
Merged upstream almost a year ago, meanwhile Fedora is based on
something 3 years old. I have no idea what's going on with that. I
know for sure fsync() alone does not fully commit everything, most
changes are in the journal. So while it is crash safe as far as kernel
log replay code is concerned, it's not bootloader crash safe for ext4,
XFS, or Btrfs. I've tested that.
https://github.com/rhboot/grubby/pull/28 will fix *all* of that when /boot is a separate
FS.
It doesn't.
1. The biggest problem is when /boot is on / and when / can't remount-ro.
2. It still doesn't sync() when /boot is not separate.
3. I could be mistaken but, pretty sure on non-journaled file systems
you need to fsync() the file and fsync() the containing directory for
it to fully commit; or you need to use sync().
But I am confused how this is merged almost a year ago and Fedora's
grubby appears to last be rebased three years ago, so it doesn't have
this fix.
> The best near term change that I think we should make is putting a
> sync() somewhere in grubby. I've tested modifying new-kernel-pkg to
> add a sync right before the last line, and
>
> # dnf update -y /path/to/kernelrpms && echo b > /proc/sysrq-trigger
>
> With very limited testing (dozens of reboots in multiple fs
> configurations of a VM only, using unsafe caching), all configurations
> including XFS are better off with sync() added. It a highly
> non-deterministic mess to sort out without sync().
>
> But with sync() and with /boot on XFS: the kernel, initramfs, or
> grub.cfg - there is an unknown (let's say 50/50) probability the
> bootloader can't find or read those files despite the sync(). The most
> common outcome I ran into was a partially modified grub.cfg containing
> an incomplete default entry for the new kernel which is missing the
> grubby insertion for the initrd. So it can't boot. But if the grub
> menu appears at all, the previous kernel and rescue entries are still
> there, and they work. Log replay during boot fixes up everything and
> now the next boot works fine by default without user interaction. So
> even though sync() is still not good enough for XFS always, it's way
> better than no sync().
>
> Meanwhile FAT, ext2, ext4, and Btrfs all appear to flush to disk with
> sync() sufficient for the bootloader to have no problems.
>
> Note that fsync() is actually insufficient at least some of the time
> on all journaled file systems. e.g. dracut -f only does fsync() on the
> copied over initramfs to /boot and that new initramfs is often not
> seen by the bootloader for journaled file systems.
Not after
https://github.com/dracutdevs/dracut/commit/58e3971b920fbb60e5a90edfd30aa...
OK but still doesn't work if the containing directory is not a mount
point, so /boot directory on / is exempt *and* there is no sync() so
even on ext4 and Btrfs it's not bootloader crash safe with the earlier
fsync() alone.
--
Chris Murphy
2193
days inactive
2198
days old
12 comments
6 participants
participants (6)
-
Chris Adams -
Chris Murphy -
Florian Weimer -
François Cami -
Lennart Poettering -
Nico Kadel-Garcia