9:44 a.m.
I'd like to brainstorm a bit about how to make kernel updates more
resilient to running out of disk space. I've talked to a few people on
IRC about this but I'm not at the point where I know enough to file good
bugs and figured that a larger discussion might help.
Summary of my questions:
Why is the grub configuration modified to make a newly installed kernel bootable
before the initramfs has been generated?
Why do errors from generating the initramfs not have any effect on the
kernel installation process?
Is there anything (other than perhaps fixing the above two weirdnesses)
that we can reasonably do to prevent systems from ending up in the state
mine were in? It's easy (but slightly messy) to make the kernel package
require some free space in /var/tmp; is it reasonable to do that?
Longer discussion:
The other day I found a couple of VMs wouldn't reboot because the
initramfs files were missing. (I could still boot an older kernel,
fortunately.) In turn, that was due to the filesystem holding /var/tmp
being quite full.
I did some digging and discovered the following:
The kernel package is set up properly to ensure that the RPM will fail
to install if there isn't space for the generated initramfs file in
/boot. (That file is marked %ghost, but the kernel spec creates a 20MB
dummy file so that RPM will ensure the space is available.)
After the kernel package is installed, its %posttrans scriptlet is run
which just calls /bin/kernel-install. A failure here is recorded in the
dnf history and logged in /var/log/dnf.rpm.log:
2018-09-02T08:23:44Z ERROR Non-fatal POSTTRANS scriptlet failure in rpm package
kernel-core
The kernel-install script is part of the systemd-udev package. It just
runs scripts in order from /{etc,usr/lib}/kernel/install.d. It keeps
track of errors from these scripts but does not act on them, except for
the magic return code of 77, which causes it to stop processing scripts.
If one or more of the scripts fails, install-kernel will exit with the
sum of their error codes, but a failure here can't do anything like
uninstall the problematic package.
On my systems, /etc/kernel/install.d/20-grubby.install and
90-loaderentry.install are empty. I believe this means that they mask
the scripts with the same names in /usr/lib/kernel/install.d. This
means the following scripts are run on kernel install, in this order:
* 20-grub.install
* 50-depmod.install
* 50-dracut.install
* 51-dracut-rescue.install
And running dracut is another big bunch of scripts and plugins; I didn't
dig down that far. However, it does appear that dracut will compose the
initramfs in /var/tmp, compress it and move it into place. This means
that it requires something on the order of twice the initramfs size in
order to run.
So, my questions:
* Why is the grub configuration changed first, before dracut gets to
run? If dracut fails, the system is probably not going to be able to
boot that kernel, so setting it up as the default as the first action
taken doesn't seem like a great idea.
* Why are errors in these scripts ignored by kernel-install? Even if
grub were moved later in the process, it would still be run even if an
earlier script failed.
* If those two were fixed, the end result of a dracut failure would be
that the kernel packages are installed but don't show up in the grub
menu. Is there anything we can reasonably do about that?
* The kernel could "pre-allocate" space in /var/tmp (by %ghosting a
dummy nonzero-length file there) so that the package simply won't
install if there isn't space sufficient to generate the initramfs.
Would that be something we could reasonably do?
This would result in the kernel package "owning" a file in /var/tmp
that doesn't exist. The name doesn't actually matter and the file
wouldn't actually be in the package. RPM would just ensure that
enough space would exist in /var/tmp before allowing the installation
to begin.
Of course, this requires the kernel package to guess as the size of
the initramfs, and to begin to care about how much space dracut needs
and where it stores its temporary files. It already guesses about the
size of the initramfs (20MB, which is just larger than the smallest I
can get).
- J<
4:05 p.m.
FWIW, rpm-ostree fixes all of this today - the entire update is really transactional,
and fully handles hitting ENOSPC at any point.
9:42 p.m.
On Thu, Sep 6, 2018 at 8:44 AM, Jason L Tibbitts III <tibbs(a)math.uh.edu> wrote:
I'd like to brainstorm a bit about how to make kernel updates
more
resilient to running out of disk space. I've talked to a few people on
IRC about this but I'm not at the point where I know enough to file good
bugs and figured that a larger discussion might help.
A few releases ago, Anaconda now defaults to a 1GiB boot partition
which should prevent the problem you're describing for anyone using
the default 3 retained kernels, as well as the extra kdump initramfs
if you're using kdump (not enabled by default on Fedora).
So realistically people need to do clean installs eventually, or
otherwise increase the size of /boot, because changing this at upgrade
time is more complex than I think anyone wants to take on.
Summary of my questions:
Why is the grub configuration modified to make a newly installed kernel bootable
before the initramfs has been generated?
I think it's in new-kernel-pkg where grubby ends up being called
twice, and why you see this. I don't know what modifies grubenv to
change the default kernel to the latest version; that should only
happen after kernel and initramfs and the last grub.cfg modification
happens, so that any failure means grubenv still points to the working
kernel. And it's possible grubby does that too and just does it before
everything else is finished.
Anyway the grubby grubby is going away. And modifying grub.cfg is
going away. It was supposed to happen for Fedora 29, but is being
pushed to Fedora 30. This is the relevant feature doing that:
https://fedoraproject.org/wiki/Changes/BootLoaderSpecByDefault
So you could just give up and focus on the new thing being made robust
if you want.
--
Chris Murphy
11:27 a.m.
>>>> "CM" == Chris Murphy
<lists(a)colorremedies.com> writes:
CM> A few releases ago, Anaconda now defaults to a 1GiB boot partition
CM> which should prevent the problem you're describing for anyone using
CM> the default 3 retained kernels, as well as the extra kdump initramfs
CM> if you're using kdump (not enabled by default on Fedora).
That is not true; the issue is not about free space in /boot. The
kernel package is properly constructed to preserve 20MB in boot and RPM
itself will not allow the transaction to proceed if that space isn't
available. (I would argue that 20MB is cutting it a bit close, but in
the specific case I had (with a 19MB initramfs file) it was sufficient.
The specific issue that caused the initramfs to fail to generate was the
lack free space in /var/tmp for dracut's temporary files.
CM> I think it's in new-kernel-pkg where grubby ends up being called
CM> twice, and why you see this.
OK, so 20-grub.install will call new-kernel-package. But the problem
remains the same: initramfs creation fails, but grub is still updated.
So maybe the issue isn't in the ordering of the kernel-install scripts
but in new-kernel-package.
CM> And modifying grub.cfg is going away. It was supposed to happen for
CM> Fedora 29, but is being pushed to Fedora 30.
OK, that's good. It will still be important that whatever configuration
file actually sets up the bootloader for the new kernel is not enabled
until after the initramfs has been generated.
CM> So you could just give up and focus on the new thing being made
CM> robust if you want.
Seems unfortunate to just ignore the problem. Maybe it's simply rare
enough, but to be fair I have seen an unpleasant number of desktops I
manage not able to reboot without human intervention due to missing
initramfs files. I would wager that they're down to the same base issue
(bootloader config done regardless of whether the initramfs was
generated) but they aren't all due to lack of free space in /var/tmp.
- J<
1:15 p.m.
On Fri, Sep 7, 2018 at 10:27 AM, Jason L Tibbitts III <tibbs(a)math.uh.edu> wrote:
>>>>> "CM" == Chris Murphy
<lists(a)colorremedies.com> writes:
CM> A few releases ago, Anaconda now defaults to a 1GiB boot partition
CM> which should prevent the problem you're describing for anyone using
CM> the default 3 retained kernels, as well as the extra kdump initramfs
CM> if you're using kdump (not enabled by default on Fedora).
That is not true; the issue is not about free space in /boot.
OK so the enospc is happening on / not on /boot - I must've blown by
that detail somehow.
Why is /var/tmp running out of space? Is this on a separate /var or /
volume that's just too small from the start, or is /var/tmp
accumulating too much? From 'man systemd-tmpfiles' I can't tell
exactly what "age parameter configured" for the --clean option means.
But it sounds like /var/tmp isn't cleaned up more aggressively when
free space is low, it only does it based on some aging of files.
The
kernel package is properly constructed to preserve 20MB in boot and RPM
itself will not allow the transaction to proceed if that space isn't
available. (I would argue that 20MB is cutting it a bit close, but in
the specific case I had (with a 19MB initramfs file) it was sufficient.
All of my hostonly initramfs files are 22+MB, and the nohostonly is 67+MB.
The specific issue that caused the initramfs to fail to generate was the
lack free space in /var/tmp for dracut's temporary files.
Gotcha. So why is there a lack of free space there? Sounds like the
volume is too small from the outset, or something is running away and
filling up that volume.
Seems to me the install script should check for /var/tmp free space
before starting. Between kernel and the non-compressed initramfs
creation, it's gonna need maybe upwards of 100M or maybe 150M if these
are nohostonly initramfs's? So test for a minimum of 200M free space,
and if not true, fail?
CM> I think it's in new-kernel-pkg where grubby ends up being called
CM> twice, and why you see this.
OK, so 20-grub.install will call new-kernel-package. But the problem
remains the same: initramfs creation fails, but grub is still updated.
So maybe the issue isn't in the ordering of the kernel-install scripts
but in new-kernel-package.
For sure grubby is called twice. And for sure the first time it's
called, the grub.cfg is modified such that a new entry is created
without an initramfs. And to me it's a suboptimal design but appears
to be happening by design. I don't know if it's a grubby limitation
that it works like this, or other.
CM> And modifying grub.cfg is going away. It was supposed to happen for
CM> Fedora 29, but is being pushed to Fedora 30.
OK, that's good. It will still be important that whatever configuration
file actually sets up the bootloader for the new kernel is not enabled
until after the initramfs has been generated.
I'm not fully understanding the whole sequence of the new method; but
I think the base bootloader configuration script is part of the kernel
package, and is copied from /lib/modules/<kernel>/bls.conf and might
get minimally modified by some kernel script, maybe new-kernel-pkg I'm
not sure, and then dropped into /boot/loader/entries where GRUB later
picks it up. So I agree, the way it ought to work is to not drop that
kernel's conf file into /boot/loader/entries until everything else is
done. The order matters so hopefully it's all happening serially and
not in parallel.
CM> So you could just give up and focus on the new thing being made
CM> robust if you want.
Seems unfortunate to just ignore the problem. Maybe it's simply rare
enough, but to be fair I have seen an unpleasant number of desktops I
manage not able to reboot without human intervention due to missing
initramfs files. I would wager that they're down to the same base issue
(bootloader config done regardless of whether the initramfs was
generated) but they aren't all due to lack of free space in /var/tmp.
There's more than one bug here, arguably. Top two bugs in my view
would be not testing for enough free space right from the start. And
then whatever is modifying grubenv before the initramfs and grub.cfg
are for sure committed to stable media.
--
Chris Murphy
3:32 p.m.
>>>> "CM" == Chris Murphy
<lists(a)colorremedies.com> writes:
CM> Why is /var/tmp running out of space?
Is that really important? In this case it was due to things simply
being too small (they are minimal VMs, after all) and dnf caching
packages there. The fact that dnf downloads a significant but quite
variable amount of data do that directory immediately before the install
starts (which can be removed immediately after the install completes)
doesn't make it any easier. In this case things worked out just
"perfectly". The package download brought the available space down to
the point that there was sufficient space for RPM's space checks to
pass, but not enough space for dracut's temporaries.
CM> Gotcha. So why is there a lack of free space there? Sounds like the
CM> volume is too small from the outset, or something is running away
CM> and filling up that volume.
Why does that matter? Sure, you could tell me "you really should make
sure there's plenty of free space there" and I wouldn't disagree, but
the punishment should be the inability to install an updated kernel,
some logged errors or maybe even a kernel package installed with no
bootloader entry. It certainly should not be a bootloader entry which
is both broken _and set as default_.
CM> Seems to me the install script should check for /var/tmp free space
CM> before starting. Between kernel and the non-compressed initramfs
CM> creation, it's gonna need maybe upwards of 100M or maybe 150M if
CM> these are nohostonly initramfs's? So test for a minimum of 200M free
CM> space, and if not true, fail?
Yes, or the kernel package just %ghosts a dummy file in /var/tmp to
reserve some space there. Then RPM itself won't even begin to install
the package. If a more messy failure is OK, it could also check for
space in %pre (or %pretrans, but that's especially problematic).
Anything done later will result in the kernel package remaining
installed and then the best you can do is make sure that either a
bootloader entry isn't created or at least it isn't made the default.
One problem with ghosting a file like that is that it's a terrible
layering violation. The kernel package shouldn't really need to know
how dracut does its job internally, how much space it needs for
temporaries, etc. The best it could do is guess, which I guess is what
it's already doing with the 20MB it reserves in boot.
CM> For sure grubby is called twice. And for sure the first time it's
CM> called, the grub.cfg is modified such that a new entry is created
CM> without an initramfs. And to me it's a suboptimal design but appears
CM> to be happening by design. I don't know if it's a grubby limitation
CM> that it works like this, or other.
I guess if the end goal is to remove grubby from the process entirely
then it's not worth digging too deeply why it's doing this. But maybe a
quick fix would be obvious to someone who knows how that works.
- J<
3:42 p.m.
On Fri, Sep 7, 2018 at 2:32 PM, Jason L Tibbitts III <tibbs(a)math.uh.edu> wrote:
I guess if the end goal is to remove grubby from the process entirely
then it's not worth digging too deeply why it's doing this. But maybe a
quick fix would be obvious to someone who knows how that works.
Well, I can't really parse this very well, but the default kernel
looks like it is changed by grubby, but I can't figure out the
ordering. It's seems plausible, perhaps likely, that it starts off
dracut and the grub2-editenv at the same time, and of course editenv
completes pretty much instantly compared to dracut.
https://github.com/rhboot/grubby/blob/master/grubby.c
--
Chris Murphy
4:58 p.m.
>>>> "CM" == Chris Murphy
<lists(a)colorremedies.com> writes:
CM> Well, I can't really parse this very well, but the default kernel
CM> looks like it is changed by grubby, but I can't figure out the
CM> ordering. It's seems plausible, perhaps likely, that it starts off
CM> dracut and the grub2-editenv at the same time, and of course editenv
CM> completes pretty much instantly compared to dracut.
I don't know; I don't see anything being done asynchronously. I think
it's not complicated at all, actually, and isn't grubby's fauilt.
Instead it's simply 20-grub.install doing things in the wrong order. It
calls new-kernel-package times (with the pre-usrmove path):
/sbin/new-kernel-pkg --package "kernel${flavor}" --install
"$KERNEL_VERSION" || exit $?
/sbin/new-kernel-pkg --package "kernel${flavor}" --mkinitrd --dracut
--depmod --update "$KERNEL_VERSION" || exit $?
/sbin/new-kernel-pkg --package "kernel${flavor}" --rpmposttrans
"$KERNEL_VERSION" || exit $?
So it calls grubby first and creates the initrd afterwards. The third
call just ends up running
/etc/kernel/postinst.d/51-dracut-rescue-postinst.sh. Which calls
new-kernel-pkg again (WTF?):
new-kernel-pkg --install "$KERNEL_VERSION" --kernel-image
"$NEW_KERNEL_IMAGE" --initrdfile "$INITRDFILE" --banner "$NAME
$VERSION_ID Rescue $MACHINE_ID"
I've never seen anything other than the rescue image put on the system
at initial install so I'm not sure at all what that's about. And for
extra fun, the /usr/lib/kernel/install.d/51-dracut-rescue.install script
will run after all of this, though I haven't figured out what that
actually does at this point since it appears to mostly work with the new
bootloader entry system.
The timeline of Primer is less confusing than this.
- J<
5:38 p.m.
On Fri, Sep 7, 2018 at 3:58 PM, Jason L Tibbitts III <tibbs(a)math.uh.edu> wrote:
>>>>> "CM" == Chris Murphy
<lists(a)colorremedies.com> writes:
CM> Well, I can't really parse this very well, but the default kernel
CM> looks like it is changed by grubby, but I can't figure out the
CM> ordering. It's seems plausible, perhaps likely, that it starts off
CM> dracut and the grub2-editenv at the same time, and of course editenv
CM> completes pretty much instantly compared to dracut.
I don't know; I don't see anything being done asynchronously. I think
it's not complicated at all, actually, and isn't grubby's fauilt.
Instead it's simply 20-grub.install doing things in the wrong order. It
calls new-kernel-package times (with the pre-usrmove path):
/sbin/new-kernel-pkg --package "kernel${flavor}" --install
"$KERNEL_VERSION" || exit $?
/sbin/new-kernel-pkg --package "kernel${flavor}" --mkinitrd --dracut
--depmod --update "$KERNEL_VERSION" || exit $?
/sbin/new-kernel-pkg --package "kernel${flavor}" --rpmposttrans
"$KERNEL_VERSION" || exit $?
So it calls grubby first and creates the initrd afterwards. The third
call just ends up running
/etc/kernel/postinst.d/51-dracut-rescue-postinst.sh. Which calls
new-kernel-pkg again (WTF?):
But also the first new-kernel-pkg with --install calls grubby with
makedefault, which is what I think is setting grubenv with the new
kernel variable before the initramfs and final grub.cfg modification
are done.
new-kernel-pkg --install "$KERNEL_VERSION" --kernel-image
"$NEW_KERNEL_IMAGE" --initrdfile "$INITRDFILE" --banner "$NAME
$VERSION_ID Rescue $MACHINE_ID"
I've never seen anything other than the rescue image put on the system
at initial install so I'm not sure at all what that's about. And for
extra fun, the /usr/lib/kernel/install.d/51-dracut-rescue.install script
will run after all of this, though I haven't figured out what that
actually does at this point since it appears to mostly work with the new
bootloader entry system.
The timeline of Primer is less confusing than this.
Well I love how you start out with it's not complicated at all, but
then end with, it's a Rube Goldberg contraption. :-P
I definitely think it's a Rube Goldberg contraption.
I can't tell you stracing this helps figuring out the actual order
because at least with -ff -o I end up with hundreds of files. So
that's pretty tedious to sift through. But if you do, also use -D, or
the stracing will prevent FITHAW from happening, and your system will
be locked up for good, unless you already have a shell handy to
manually issue fsfreeze -u
--
Chris Murphy
6:09 p.m.
>>>> "CM" == Chris Murphy
<lists(a)colorremedies.com> writes:
CM> But also the first new-kernel-pkg with --install calls grubby with
CM> makedefault, which is what I think is setting grubenv with the new
CM> kernel variable before the initramfs and final grub.cfg modification
CM> are done.
Right, which is why I wonder if switching the two lines wouldn't work.
Given the complexity I assume it's being done this way for a reason; I
just don't know what the reason is.
- J<
6:43 p.m.
On Fri, Sep 7, 2018 at 5:09 PM, Jason L Tibbitts III <tibbs(a)math.uh.edu> wrote:
>>>>> "CM" == Chris Murphy
<lists(a)colorremedies.com> writes:
CM> But also the first new-kernel-pkg with --install calls grubby with
CM> makedefault, which is what I think is setting grubenv with the new
CM> kernel variable before the initramfs and final grub.cfg modification
CM> are done.
Right, which is why I wonder if switching the two lines wouldn't work.
Given the complexity I assume it's being done this way for a reason; I
just don't know what the reason is.
As I think about it, new-kernel-pkg is part of grubby. And some time
ago I heard new-kernel-pkg was either going away or being refactored.
I don't know the status of that, but if grubby is going away, then
new-kernel-pkg is either going away or being moved. My understanding
of the BLS by default change, is that grubby don't be installed. While
I see it in the upstream grubby project...
$ sudo dnf provides /usr/sbin/new-kernel-pkg
[sudo] password for chris:
Last metadata expiration check: 2:43:04 ago on Fri 07 Sep 2018 02:59:45 PM MDT.
Error: No Matches found
*shrug*
--
Chris Murphy
6:57 p.m.
>>>> "CM" == Chris Murphy
<lists(a)colorremedies.com> writes:
CM> As I think about it, new-kernel-pkg is part of grubby.
Yes, I assume it will go away and we'll be left with only the
install-kernel infrastructure. Which would strip at least three layers
from this onion. It also conveniently forces the issue, since it's the
order of the new-kernel-pkg calls which appears to cause the problem.
So I guess the answer here is "live with it for now and try to make sure
F30 is better".
- J<
12:03 a.m.
On 6.9.2018 16:44, Jason L Tibbitts III wrote:
* The kernel could "pre-allocate" space in /var/tmp (by
%ghosting a
dummy nonzero-length file there) so that the package simply won't
install if there isn't space sufficient to generate the initramfs.
Would that be something we could reasonably do?
This would result in the kernel package "owning" a file in /var/tmp
that doesn't exist. The name doesn't actually matter and the file
wouldn't actually be in the package. RPM would just ensure that
enough space would exist in /var/tmp before allowing the installation
to begin.
FWIW, the name does matter. You'd have to ensure the name is as unlikely
to collide with whatever anyone puts there as possible. Otherwise you
could get a user file removed when the kernel is removed, right?
Ondřej Lysoněk
2058
days inactive
2059
days old
12 comments
4 participants
participants (4)
-
Chris Murphy -
Colin Walters -
Jason L Tibbitts III -
Ondřej Lysoněk