Fw: Latest rawhide kernel, kernel-5.19.0-0.rc6.20220714git4a57a8400075.49.fc37.src.rpm, builds but won't boot
Forgot to copy the list.
I think I am going to downgrade dracut to an older version that doesn't have problems. I have the latest release, installed today, and it exhibited the problem.
Begin forwarded message:
Date: Tue, 19 Jul 2022 14:56:56 -0700 From: stan upaitag@zoho.com To: Justin Forbes jmforbes@linuxtx.org Subject: Re: Latest rawhide kernel, kernel-5.19.0-0.rc6.20220714git4a57a8400075.49.fc37.src.rpm, builds but won't boot
On Tue, 19 Jul 2022 16:01:55 -0500 Justin Forbes jmforbes@linuxtx.org wrote:
It sounds like you are not so far off. This doesn't seem to be a kernel issue at all though. To verify, take a known good kernel, save a copy of the initramfs for that one somewhere, and then regenerate the initramfs and try to boot it. I am guessing it will fail in the same way with a new initramfs.
You are right. I took an older kernel, regenerated the initramfs, and it failed. The size of the initramfs was drastically smaller (~ 1/2) than the original, as it is for the latest rc7 kernels that fail as well (~ 3/4 of older 5.19 kernels).
What is my next step? Open a bugzilla against dracut? Or is there a setting I can pass to dracut that will fix this?
On Tue, 19 Jul 2022 15:06:38 -0700 stan via kernel kernel@lists.fedoraproject.org wrote:
You are right. I took an older kernel, regenerated the initramfs, and it failed. The size of the initramfs was drastically smaller (~ 1/2) than the original, as it is for the latest rc7 kernels that fail as well (~ 3/4 of older 5.19 kernels).
What is my next step? Open a bugzilla against dracut? Or is there a setting I can pass to dracut that will fix this?
I downgraded dracut to a version (56) before the problem, and rebuilt the initramfs for the local rc7 build. It still failed to boot with the same symptom. So, I think that lets dracut off the hook. Something else has changed that is causing the problem. A configuration setting? A library it is using?
Hundreds of packages have changed in the last few weeks.
On Tue, 19 Jul 2022 15:36:09 -0700 stan via kernel kernel@lists.fedoraproject.org wrote:
I downgraded dracut to a version (56) before the problem, and rebuilt the initramfs for the local rc7 build. It still failed to boot with the same symptom. So, I think that lets dracut off the hook. Something else has changed that is causing the problem. A configuration setting? A library it is using?
Hundreds of packages have changed in the last few weeks.
I unpacked the rc4 initramfs that works, and the rc7 initramfs that doesn't work, and there are a few differences in permissions. As you can see from this sample, the group and world readable permissions have been removed from several items in the newer dracut. These are from dracut version 57, but they were also there when using the dracut version 56 to create a new initramfs. It looks like some configuration options have changed. I think I will be opening a bugzilla against dracut, just to get their input.
d 1c1 < Image: /boot/initramfs-5.19.0-0.rc4.20220701gita175eca0f3d7.36.20220703.fc37.x86_64.img: 48M ---
Image: /boot/initramfs-5.19.0-0.rc7.53.20220719.fc37.x86_64.img: 32M
3c3 < Version: dracut-056-2.fc37 ---
Version: dracut-057-1.fc37
37,57c37,57 < drwxr-xr-x 12 root root 0 Apr 19 16:17 . < crw-r--r-- 1 root root 5, 1 Apr 19 16:17 dev/console < crw-r--r-- 1 root root 1, 11 Apr 19 16:17 dev/kmsg < crw-r--r-- 1 root root 1, 3 Apr 19 16:17 dev/null < crw-r--r-- 1 root root 1, 8 Apr 19 16:17 dev/random < crw-r--r-- 1 root root 1, 9 Apr 19 16:17 dev/urandom < lrwxrwxrwx 1 root root 7 Apr 19 16:17 bin -> usr/bin < drwxr-xr-x 2 root root 0 Apr 19 16:17 dev < drwxr-xr-x 13 root root 0 Apr 19 16:17 etc < drwxr-xr-x 2 root root 0 Apr 19 16:17 etc/authselect < -rw-r--r-- 1 root root 703 Apr 19 16:17 etc/authselect/nsswitch.conf < drwxr-xr-x 2 root root 0 Apr 19 16:17 etc/cmdline.d < drwxr-xr-x 2 root root 0 Apr 19 16:17 etc/conf.d < -rw-r--r-- 1 root root 124 Apr 19 16:17 etc/conf.d/systemd.conf < drwxr-xr-x 7 root root 0 Apr 19 16:17 etc/dbus-1 < drwxr-xr-x 2 root root 0 Apr 19 16:17 etc/dbus-1/interfaces < drwxr-xr-x 2 root root 0 Apr 19 16:17 etc/dbus-1/services < -rw-r--r-- 1 root root 838 Mar 10 02:47 etc/dbus-1/session.conf < drwxr-xr-x 2 root root 0 Apr 19 16:17 etc/dbus-1/session.d < -rw-r--r-- 1 root root 833 Mar 10 02:47 etc/dbus-1/system.conf < drwxr-xr-x 2 root root 0 Apr 19 16:17 etc/dbus-1/system.d ---
drwxr-xr-x 12 root root 0 Jul 18 04:21 . crw------- 1 root root 5, 1 Jul 18 04:21 dev/console crw------- 1 root root 1, 11 Jul 18 04:21 dev/kmsg crw------- 1 root root 1, 3 Jul 18 04:21 dev/null crw------- 1 root root 1, 8 Jul 18 04:21 dev/random crw------- 1 root root 1, 9 Jul 18 04:21 dev/urandom lrwxrwxrwx 1 root root 7 Jul 18 04:21 bin -> usr/bin drwxr-xr-x 2 root root 0 Jul 18 04:21 dev drwxr-xr-x 13 root root 0 Jul 18 04:21 etc drwxr-xr-x 2 root root 0 Jul 18 04:21 etc/authselect -rw-r--r-- 1 root root 703 Jul 18 04:21 etc/authselect/nsswitch.conf drwx------ 2 root root 0 Jul 18 04:21 etc/cmdline.d drwx------ 2 root root 0 Jul 18 04:21 etc/conf.d -rw------- 1 root root 124 Jul 18 04:21 etc/conf.d/systemd.conf drwxr-xr-x 7 root root 0 Jul 18 04:21 etc/dbus-1 drwxr-xr-x 2 root root 0 Jul 18 04:21 etc/dbus-1/interfaces drwxr-xr-x 2 root root 0 Jul 18 04:21 etc/dbus-1/services -rw-r--r-- 1 root root 838 Jul 12 09:59 etc/dbus-1/session.conf drwxr-xr-x 2 root root 0 Jul 18 04:21 etc/dbus-1/session.d -rw-r--r-- 1 root root 833 Jul 12 09:59 etc/dbus-1/system.conf drwxr-xr-x 2 root root 0 Jul 18 04:21 etc/dbus-1/system.d
On Wed, 20 Jul 2022 08:19:17 -0700 stan via kernel kernel@lists.fedoraproject.org wrote:
configuration options have changed. I think I will be opening a bugzilla against dracut, just to get their input.
On Wed, 20 Jul 2022 09:00:06 -0700 stan upaitag@zoho.com wrote:
On Wed, 20 Jul 2022 08:19:17 -0700 stan via kernel kernel@lists.fedoraproject.org wrote:
configuration options have changed. I think I will be opening a bugzilla against dracut, just to get their input.
The mass rebuild is finished, and ~4500 packages updated on my rawhide system. I rebuilt the kernel, and reinstalled it, and the initramfs is still significantly smaller than the working versions, and it still doesn't boot with an immediate hang after trying to read the disk.
I downgraded both dracut and systemd to versions that worked, and they still didn't build a valid initramfs. I think that lets them off the hook. It has to be a configuration change they are responding to. But I have no idea where to begin looking for such a change.
On Tue, 26 Jul 2022 15:27:40 -0700 stan upaitag@zoho.com wrote:
I updated this with my latest update on resolving the problem.
But I have no idea where to begin looking for such a change.
The latest stock fedora kernel for rawhide, 5.19.0-65.fc37.x86_64, exhibits the same problem as I reported above, except now the library that can't be found is libsystemd-core-251-3-2.fc37.so. $ ls -nZ /usr/lib/systemd/libsystemd-core-251.3-2.fc37.so -rwxr-xr-x. 1 0 0 system_u:object_r:lib_t:s0 2126024 Jul 23 03:09 /usr/lib/systemd/libsystemd-core-251.3-2.fc37.so And the system has a panic and hangs.
I have tried all sorts of ways to rebuild the initramfs, that haven't worked, mentioned in the bugzilla. Same result, a brief read of the disk and then a hang with a blank screen.
I've also been trying to create a kernel that doesn't require an initramfs to boot, but the Fedora kernels have the initramfs hard coded. I moved the initramfs, thinking it would try the kernel if there was no initramfs, but it turns out that the kernel builds in a rudimentary backup initramfs, and it is invoked instead of just directly booting the kernel.
Meanwhile the kernel I built and installed on 20220703 continues to boot and run flawlessly, thank goodness.
All of the above with the latest updates in rawhide applied. There are still some issues with erlang* and wx*, but I don't think they would impact this problem.
kernel@lists.fedoraproject.org
-
stan