On my 3 systems, F34, F34, and CentOS7, they are 1, 2, and 6 years old respectively.
Are old rescue kernels still useful? (6 years?)
Are there automated or manual procedures to update a rescue kernel?
Are there best practices for rescue kernel update? If there are, I've missed them.
On Thu, Jun 03, 2021 at 02:20:33PM -0400, Jon LaBadie wrote:
On my 3 systems, F34, F34, and CentOS7, they are 1, 2, and 6 years old respectively.
Are old rescue kernels still useful? (6 years?)
Yes -- they will let you boot into the system. The rescue initrd includes all available drivers and so can boot even if the drive is in a very different system from the one it was installed on.
Are there automated or manual procedures to update a rescue kernel?
There's generally no reason to. But you can with /etc/kernel/postinst.d/51-dracut-rescue-postinst.sh $(uname -r) /boot/vmlinuz-$(uname -r)
Are there best practices for rescue kernel update? If there are, I've missed them.
That's because the best practice is generally to not worry about it.
On 6/3/21 12:20 PM, Jon LaBadie wrote:
Are old rescue kernels still useful? (6 years?)
They're still just as useful as they were when they were installed. Of course, that means that any function that was added later isn't there, but that doesn't matter because you're only going to use it in emergencies to troubleshoot a broken system.
On 6/3/21 2:33 PM, Matthew Miller wrote:
On Thu, Jun 03, 2021 at 02:20:33PM -0400, Jon LaBadie wrote:
On my 3 systems, F34, F34, and CentOS7, they are 1, 2, and 6 years old respectively.
Are old rescue kernels still useful? (6 years?)
Yes -- they will let you boot into the system. The rescue initrd includes all available drivers and so can boot even if the drive is in a very different system from the one it was installed on.
Are there automated or manual procedures to update a rescue kernel?
There's generally no reason to. But you can with /etc/kernel/postinst.d/51-dracut-rescue-postinst.sh $(uname -r) /boot/vmlinuz-$(uname -r)
I am on F33, but I don't have such a file. What creates it, or where does it come from?
Are there best practices for rescue kernel update? If there are, I've missed them.
That's because the best practice is generally to not worry about it.
On 6/3/21 11:20 AM, Jon LaBadie wrote:
If you delete the rescue kernel and initrd, they will be recreated the next time you install a kernel.
[KlugscheissMode on] what is only valide when: - no one has changed the default "yes" to "no" in /usr/lib/dracut/dracut.conf.d/02-rescue.conf - a new dracut was installed/updated after the above change to "no", cause dracut obvious doesn't *respect* user settings, Hrmph, :-( [KlugscheissMode off]
On Thu, 2021-06-03 at 12:51 -0600, Joe Zeff wrote:
On 6/3/21 12:20 PM, Jon LaBadie wrote:
Are old rescue kernels still useful? (6 years?)
They're still just as useful as they were when they were installed. Of course, that means that any function that was added later isn't there, but that doesn't matter because you're only going to use it in emergencies to troubleshoot a broken system.
Surely an old rescue kernel may not be able to mount a BTRFS filesystem?
poc
On Thu, 2021-06-03 at 15:48 -0600, Joe Zeff wrote:
On 6/3/21 3:18 PM, Patrick O'Callaghan wrote:
Surely an old rescue kernel may not be able to mount a BTRFS filesystem?
BTRFS has been a fully stable part of the kernel since 2013. How old is your rescue kernel?
It was a theoretical question. My rescue kernel is recent.
poc
On Thu, Jun 3, 2021 at 12:20 PM Jon LaBadie jonfu@jgcomp.com wrote:
On my 3 systems, F34, F34, and CentOS7, they are 1, 2, and 6 years old respectively.
Are old rescue kernels still useful? (6 years?)
They might be useful to a sysadmin, I think they are useless. The rescue kernel is really just a "no host-only" initramfs that contains a bunch of extra dracut and kernel modules that the host only initramfs doesn't. The difficulty is the rescue initramfs can't do a full graphical boot once /usr/lib/modules/ dir for that kernel has been removed, which was likely in its first 4 weeks following installation.
Since you won't get graphical boot anyway, I'm not sure you have a good chance of dracut building a new host only initramfs that contains the driver needed for whatever new hardware you've added or changed to. It's pretty esoteric landing in a dracut shell even for experienced users. So I am not a fan.
What I would like to see is (a) an initramfs that can boot a graphical stack (b) contains the Live OS dracut modules (c) and overlayfs, and wire it up so that the rescue boot entry does a read-only sysroot boot + writable overlay like a LiveOS. So now folks can use a web browser normally, get on irc or whatever, and get some help with why they can't boot without having to resort to mobile or a 2nd computer they may not have handy.
A side plus for Btrfs cases, it has a unique ro,rescue=all mount option that tolerates file system problems. Plausibly we can still boot read-only in situations where other file systems would face plant until they get an fsck. Whereas on Btrfs we really want to steer folks towards freshening backups before they attempt a repair, if they end up in a disaster situation. But nevertheless, such an effort would be generically beneficial no matter the file system.
A variation on that might be a read-only "rescue" or "recovery" snapshot that would be immutable, paired with the same LiveOS+overlayfs concept. That way a boot is possible in a variety of other more likely user error or update related scenarios, i.e. file system isn't damaged, it's the installation that's messed up, and what you need is a live boot but don't have a USB stick handy, so just bake it into a small snapshot. *shrug* maybe. We could make it completely self contained including the kernel, initramfs, and the whole graphics stack. But, near term such a thing would be btrfs only unless we dedicate a literal partition and stick a Live OS ISO image on it (effectively).
On Thu, 2021-06-03 at 17:23 -0600, Chris Murphy wrote:
What I would like to see is (a) an initramfs that can boot a graphical stack (b) contains the Live OS dracut modules (c) and overlayfs, and wire it up so that the rescue boot entry does a read- only sysroot boot + writable overlay like a LiveOS. So now folks can use a web browser normally, get on irc or whatever, and get some help with why they can't boot without having to resort to mobile or a 2nd computer they may not have handy.
I really don't know how anybody manages to fix a middling computer problem without having a second computer. You nearly always need to research something, or yank a drive and copy files over in one direction or the other. Trying to do that on a banjaxed computer is really difficult.
On Thu, Jun 03, 2021 at 03:48:29PM -0600, Joe Zeff wrote:
On 6/3/21 3:18 PM, Patrick O'Callaghan wrote:
Surely an old rescue kernel may not be able to mount a BTRFS filesystem?
BTRFS has been a fully stable part of the kernel since 2013. How old is your rescue kernel?
Close, one system's rescue kernel is early 2015, but others could have even older rescue kernels.
But I believe POC's point was not about BTRFS specifically, but that something could have changed over the years to make an old rescue kernel unusable.
If you accept that premise, shouldn't the rescue kernel be periodically updated?
On Thu, Jun 03, 2021 at 05:23:20PM -0600, Chris Murphy wrote:
On Thu, Jun 3, 2021 at 12:20 PM Jon LaBadie jonfu@jgcomp.com wrote:
On my 3 systems, F34, F34, and CentOS7, they are 1, 2, and 6 years old respectively.
Are old rescue kernels still useful? (6 years?)
They might be useful to a sysadmin, I think they are useless. The rescue kernel is really just a "no host-only" initramfs that contains a bunch of extra dracut and kernel modules that the host only initramfs doesn't. The difficulty is the rescue initramfs can't do a full graphical boot once /usr/lib/modules/ dir for that kernel has been removed, which was likely in its first 4 weeks following installation.
Since you won't get graphical boot anyway, I'm not sure you have a good chance of dracut building a new host only initramfs that contains the driver needed for whatever new hardware you've added or changed to. It's pretty esoteric landing in a dracut shell even for experienced users. So I am not a fan.
Might it be useful for /usr/lib/modules/rescue... (and maybe also /usr/src/kernels/rescue...) to be created/preserved for the rescue kernel?
On Thu, Jun 3, 2021 at 3:19 PM Patrick O'Callaghan pocallaghan@gmail.com wrote:
On Thu, 2021-06-03 at 12:51 -0600, Joe Zeff wrote:
On 6/3/21 12:20 PM, Jon LaBadie wrote:
Are old rescue kernels still useful? (6 years?)
They're still just as useful as they were when they were installed. Of course, that means that any function that was added later isn't there, but that doesn't matter because you're only going to use it in emergencies to troubleshoot a broken system.
Surely an old rescue kernel may not be able to mount a BTRFS filesystem?
Not only Btrfs but any file system. A new mkfs may set options that an old kernel doesn't support. There's quite a bit of that in ext4 and XFS land.
If you mkfs.btrfs with today's progs and use defaults, an ancient kernel will mount it. But there are features that old kernels don't support, like for example zstd compression which arrived in kernel 4.14, thus not mountable (incompatible) with older kernels. Free space tree v2 exists since kernel 4.5, but its flag permits ro mount with old kernels.
On Fri, 2021-06-04 at 14:46 +0930, Tim via users wrote:
On Thu, 2021-06-03 at 17:23 -0600, Chris Murphy wrote:
What I would like to see is (a) an initramfs that can boot a graphical stack (b) contains the Live OS dracut modules (c) and overlayfs, and wire it up so that the rescue boot entry does a read- only sysroot boot + writable overlay like a LiveOS. So now folks can use a web browser normally, get on irc or whatever, and get some help with why they can't boot without having to resort to mobile or a 2nd computer they may not have handy.
I really don't know how anybody manages to fix a middling computer problem without having a second computer. You nearly always need to research something, or yank a drive and copy files over in one direction or the other. Trying to do that on a banjaxed computer is really difficult.
Yep. You can sometimes make progress with a Live Image but I keep an ancient laptop (circa 2008) around just for this.
poc
On Fri, 4 Jun 2021 at 02:51, Jon LaBadie jonfu@jgcomp.com wrote:
On Thu, Jun 03, 2021 at 03:48:29PM -0600, Joe Zeff wrote:
But I believe POC's point was not about BTRFS specifically, but that something could have changed over the years to make an old rescue kernel unusable.
If you accept that premise, shouldn't the rescue kernel be periodically updated?
Drivers come and go and come back, so older hardware may
not be supported in newer kernels. The rescue kernel should work as well as it did for the original installation until some major hardware change like a new wifi card.
I have a couple old (>10 years) systems. For one, wifi was broken for many years (so I used a USB dongle) but does work in recent kernels. For the other, the ethernet driver was dropped by the kernel for a year or two (and has now come back).
The worst case would be a hardware failure that requires adding a new device (e.g., USB wifi or ethernet when the original device dies) that isn't supported by the old rescue kernel.
On Thu, Jun 3, 2021, at 11:20 AM, Jon LaBadie wrote:
On my 3 systems, F34, F34, and CentOS7, they are 1, 2, and 6 years old respectively.
Are old rescue kernels still useful? (6 years?)
Are there automated or manual procedures to update a rescue kernel?
Are there best practices for rescue kernel update? If there are, I've missed them.
I don't think anybody gave an example of procedure to update...
1. Turn off auto updates to dnf. 2. Daily update: sudo dnf upgrade 2a. If there is not kernel update hit "Yes" and stop here. 2b. If there is a kernel update hit "No" 3. Delete the "rescue" stuff in /boot and /boot/loader/entries/ 4. sudo dnf upgrade -y 5. Reenable auto updates if you wish.
This will make sure you get your rescue built right away without having to push the build yourself. This just lets the system do it for you. You could just do the deletes and then wait, but it might be days before a new kernel comes in to push the rebuild.
On Fri, Jun 04, 2021 at 07:35:54AM -0700, Doug H. wrote:
On Thu, Jun 3, 2021, at 11:20 AM, Jon LaBadie wrote:
On my 3 systems, F34, F34, and CentOS7, they are 1, 2, and 6 years old respectively.
Are old rescue kernels still useful? (6 years?)
Are there automated or manual procedures to update a rescue kernel?
Are there best practices for rescue kernel update? If there are, I've missed them.
I don't think anybody gave an example of procedure to update...
- Turn off auto updates to dnf.
- Daily update: sudo dnf upgrade
2a. If there is not kernel update hit "Yes" and stop here. 2b. If there is a kernel update hit "No" 3. Delete the "rescue" stuff in /boot and /boot/loader/entries/ 4. sudo dnf upgrade -y 5. Reenable auto updates if you wish.
This will make sure you get your rescue built right away without having to push the build yourself. This just lets the system do it for you. You could just do the deletes and then wait, but it might be days before a new kernel comes in to push the rebuild.
Thanks for the stepwise instructions Doug.
One thought, should one be certain that they are happy with the most recent kernel before embarking on replacing the rescue kernel. I don't reboot every time a kernel is installed. So I may not have yet run the most recent. Murphy's law says that will be the one that breaks something and I wouldn't want it as my rescue kernel ;)
Slightly off topic, my really old rescue kernel (2015) is on a CentOS system and has no /boot/loader directory. Was there an alternative before /boot/loader was introduced?
Doug H. writes:
- Delete the "rescue" stuff in /boot and /boot/loader/entries/
We should have a canned wrapper for this. We already have
dnf config-manager --set-enabled repo
to simply flip a single character in a .conf file.
It will be helpful in avoiding typos to have something similar, for removing the rescue kernel.
-
Chris Murphy -
Doug H. -
George N. White III -
Joe Zeff -
Jon LaBadie -
Lester M Petrie -
Matthew Miller -
old sixpack13 -
Patrick O'Callaghan -
Sam Varshavchik -
Samuel Sieb -
Tim