I can't make sense of this output, can you?
# rpm -q kernel-core kernel-core-6.9.9-100.fc39.x86_64 kernel-core-6.9.12-200.fc40.x86_64 kernel-core-6.10.3-200.fc40.x86_64
# rpm -V kernel-core-6.10.3-200.fc40.x86_64
# dnf -y reinstall kernel-core-6.10.3-200.fc40.x86_64 Last metadata expiration check: 2:37:23 ago on Sun 11 Aug 2024 07:37:49 AM PDT. Dependencies resolved. ================================================================================ Package Architecture Version Repository Size ================================================================================ Reinstalling: kernel-core x86_64 6.10.3-200.fc40 updates 17 M Transaction Summary ================================================================================ Total download size: 17 M Installed size: 69 M Downloading Packages: kernel-core-6.10.3-200.fc40.x86_64.rpm 6.4 MB/s | 17 MB 00:02 -------------------------------------------------------------------------------- Total 5.5 MB/s | 17 MB 00:03 Running transaction check Transaction check succeeded. Running transaction test Transaction test succeeded. Running transaction Preparing : 1/1 Reinstalling : kernel-core-6.10.3-200.fc40.x86_64 1/2 Running scriptlet: kernel-core-6.10.3-200.fc40.x86_64 1/2 Running scriptlet: kernel-core-6.10.3-200.fc40.x86_64 2/2 Cleanup : kernel-core-6.10.3-200.fc40.x86_64 2/2 Running scriptlet: kernel-core-6.10.3-200.fc40.x86_64 2/2 dkms: running auto installation service for kernel 6.10.3-200.fc40.x86_64 dkms: autoinstall for kernel 6.10.3-200.fc40.x86_64 Done. Reinstalled: kernel-core-6.10.3-200.fc40.x86_64 Complete!
# rpm -V kernel-core-6.10.3-200.fc40.x86_64
# rpm -ql kernel-core-6.10.3-200.fc40.x86_64 /boot/.vmlinuz-6.10.3-200.fc40.x86_64.hmac /boot/System.map-6.10.3-200.fc40.x86_64 /boot/config-6.10.3-200.fc40.x86_64 /boot/initramfs-6.10.3-200.fc40.x86_64.img /boot/symvers-6.10.3-200.fc40.x86_64.xz /boot/vmlinuz-6.10.3-200.fc40.x86_64 /lib/modules /lib/modules/6.10.3-200.fc40.x86_64 /lib/modules/6.10.3-200.fc40.x86_64/.vmlinuz.hmac /lib/modules/6.10.3-200.fc40.x86_64/System.map /lib/modules/6.10.3-200.fc40.x86_64/config /lib/modules/6.10.3-200.fc40.x86_64/modules.builtin /lib/modules/6.10.3-200.fc40.x86_64/modules.builtin.modinfo /lib/modules/6.10.3-200.fc40.x86_64/symvers.xz /lib/modules/6.10.3-200.fc40.x86_64/vmlinuz /usr/share/licenses/kernel-core /usr/share/licenses/kernel-core/COPYING-6.10.3-200.fc40
# for x in $(rpm -ql kernel-core-6.10.3-200.fc40.x86_64); do ls -d $x; done /usr/bin/ls: cannot access '/boot/.vmlinuz-6.10.3-200.fc40.x86_64.hmac': No such file or directory /usr/bin/ls: cannot access '/boot/System.map-6.10.3-200.fc40.x86_64': No such file or directory /usr/bin/ls: cannot access '/boot/config-6.10.3-200.fc40.x86_64': No such file or directory /usr/bin/ls: cannot access '/boot/initramfs-6.10.3-200.fc40.x86_64.img': No such file or directory /boot/symvers-6.10.3-200.fc40.x86_64.xz /usr/bin/ls: cannot access '/boot/vmlinuz-6.10.3-200.fc40.x86_64': No such file or directory /lib/modules /lib/modules/6.10.3-200.fc40.x86_64 /lib/modules/6.10.3-200.fc40.x86_64/.vmlinuz.hmac /lib/modules/6.10.3-200.fc40.x86_64/System.map /lib/modules/6.10.3-200.fc40.x86_64/config /lib/modules/6.10.3-200.fc40.x86_64/modules.builtin /lib/modules/6.10.3-200.fc40.x86_64/modules.builtin.modinfo /lib/modules/6.10.3-200.fc40.x86_64/symvers.xz /lib/modules/6.10.3-200.fc40.x86_64/vmlinuz /usr/share/licenses/kernel-core /usr/share/licenses/kernel-core/COPYING-6.10.3-200.fc40
So, what happened to kernel 6.10.3-200? RPM -V says the package is complete but the files are not there!
On 11/08/2024 18:30, Dave Close wrote:
I can't make sense of this output, can you?
# rpm -q kernel-core kernel-core-6.9.9-100.fc39.x86_64 kernel-core-6.9.12-200.fc40.x86_64 kernel-core-6.10.3-200.fc40.x86_64
That kernel runs for me, but I have romm for only two.
How big is /boot ?
Hi.
On Sun, 11 Aug 2024 10:30:14 -0700 Dave Close wrote:
# dnf -y reinstall kernel-core-6.10.3-200.fc40.x86_64
So, what happened to kernel 6.10.3-200? RPM -V says the package is complete but the files are not there!
The missing files are ghost files (for RPM, see: rpm -Vv kernel-core-6.10.3-200.fc40.x86_64 ) and are normally build in posttrans.
It seems that a reinstall do not apply the posstrans script.
Can you run it manually:
/bin/kernel-install add 6.10.3-200.fc40.x86_64 \ /lib/modules/6.10.3-200.fc40.x86_64/vmlinuz
John Pilkington wrote:
On 11/08/2024 18:30, Dave Close wrote:
I can't make sense of this output, can you?
# rpm -q kernel-core kernel-core-6.9.9-100.fc39.x86_64 kernel-core-6.9.12-200.fc40.x86_64 kernel-core-6.10.3-200.fc40.x86_64
That kernel runs for me, but I have romm for only two.
And it runs for me, on other machines.
How big is /boot ?
# df -h /boot Filesystem Size Used Avail Use% Mounted on /dev/sda6 974M 549M 358M 61% /boot
It is held at least three kernels in the past.
Francis.Montagnac@inria.fr wrote:
On Sun, 11 Aug 2024 10:30:14 -0700 Dave Close wrote:
# dnf -y reinstall kernel-core-6.10.3-200.fc40.x86_64
So, what happened to kernel 6.10.3-200? RPM -V says the package is complete but the files are not there!
The missing files are ghost files (for RPM, see: rpm -Vv kernel-core-6.10.3-200.fc40.x86_64 ) and are normally build in posttrans.
Thanks. I had never tried -Vv before. But rpm -qf (on another machine) says the kernel is part of this package so I was surprised that the package is installed but the file is not present.
It seems that a reinstall do not apply the posstrans script.
Can you run it manually:
/bin/kernel-install add 6.10.3-200.fc40.x86_64 \ /lib/modules/6.10.3-200.fc40.x86_64/vmlinuz
Yes, but the file is still not present. Of course, I could copy it from another machine but I'd really like to understand the problem.
# /bin/kernel-install add 6.10.3-200.fc40.x86_64 \ /lib/modules/6.10.3-200.fc40.x86_64/vmlinuz dkms: running auto installation service for kernel 6.10.3-200.fc40.x86_64 dkms: autoinstall for kernel 6.10.3-200.fc40.x86_64 Done.
# rpm -Vv kernel-core-6.10.3-200.fc40.x86_64 missing g /boot/.vmlinuz-6.10.3-200.fc40.x86_64.hmac missing g /boot/System.map-6.10.3-200.fc40.x86_64 missing g /boot/config-6.10.3-200.fc40.x86_64 missing g /boot/initramfs-6.10.3-200.fc40.x86_64.img ......... g /boot/symvers-6.10.3-200.fc40.x86_64.xz missing g /boot/vmlinuz-6.10.3-200.fc40.x86_64 ......... /lib/modules ......... /lib/modules/6.10.3-200.fc40.x86_64 ......... /lib/modules/6.10.3-200.fc40.x86_64/.vmlinuz.hmac ......... /lib/modules/6.10.3-200.fc40.x86_64/System.map ......... /lib/modules/6.10.3-200.fc40.x86_64/config ......... /lib/modules/6.10.3-200.fc40.x86_64/modules.builtin ......... /lib/modules/6.10.3-200.fc40.x86_64/modules.builtin.modinfo ......... /lib/modules/6.10.3-200.fc40.x86_64/symvers.xz ......... /lib/modules/6.10.3-200.fc40.x86_64/vmlinuz ......... /usr/share/licenses/kernel-core ......... l /usr/share/licenses/kernel-core/COPYING-6.10.3-200.fc40
On 12 Aug 2024, at 03:40, Dave Close dave@compata.com wrote:
# df -h /boot Filesystem Size Used Avail Use% Mounted on /dev/sda6 974M 549M 358M 61% /boot
It is held at least three kernels in the past.
That does seem to have a lot of space used.
Here is mine that has 3 kernels.
$ df -h /boot Filesystem Size Used Avail Use% Mounted on /dev/sda3 974M 265M 642M 30% /boot
What does `du -sh /boot` show?
Here is mine:
$ du -sh /boot/* 272K /boot/config-6.10.3-200.fc40.x86_64 268K /boot/config-6.9.11-200.fc40.x86_64 268K /boot/config-6.9.12-200.fc40.x86_64 7.5M /boot/efi 2.4M /boot/grub2 80M /boot/initramfs-0-rescue-23861aed63d748da85011d84ee28e601.img 34M /boot/initramfs-6.10.3-200.fc40.x86_64.img 33M /boot/initramfs-6.9.11-200.fc40.x86_64.img 33M /boot/initramfs-6.9.12-200.fc40.x86_64.img 24K /boot/loader 16K /boot/lost+found 164K /boot/symvers-6.10.3-200.fc40.x86_64.xz 160K /boot/symvers-6.9.11-200.fc40.x86_64.xz 160K /boot/symvers-6.9.12-200.fc40.x86_64.xz 8.9M /boot/System.map-6.10.3-200.fc40.x86_64 8.9M /boot/System.map-6.9.11-200.fc40.x86_64 8.9M /boot/System.map-6.9.12-200.fc40.x86_64 11M /boot/vmlinuz-0-rescue-23861aed63d748da85011d84ee28e601 16M /boot/vmlinuz-6.10.3-200.fc40.x86_64 16M /boot/vmlinuz-6.9.11-200.fc40.x86_64 16M /boot/vmlinuz-6.9.12-200.fc40.x86_64
dkms: running auto installation service for kernel 6.10.3-200.fc40.x86_64 dkms: autoinstall for kernel 6.10.3-200.fc40.x86_64 Done.
What are you using dkms for? I wonder if that is causing your issue?
I see you did the dnf reinstall and that also had the dkms messages.
Barry
Barry Scott wrote:
On 12 Aug 2024, at 03:40, Dave Close dave@compata.com wrote:
# df -h /boot Filesystem Size Used Avail Use% Mounted on /dev/sda6 974M 549M 358M 61% /boot
It is held at least three kernels in the past.
That does seem to have a lot of space used.
Here is mine that has 3 kernels.
$ df -h /boot Filesystem Size Used Avail Use% Mounted on /dev/sda3 974M 265M 642M 30% /boot
What does `du -sh /boot` show?
# du -sh /boot 549M /boot
# du -sh /boot/* 268K /boot/config-6.9.9-100.fc39.x86_64 307M /boot/efi 1.1M /boot/extlinux 16K /boot/flask 11M /boot/grub2 150M /boot/initramfs-0-rescue-7cf63543075b47d48d09f1649641c3a1.img 41M /boot/initramfs-6.9.9-100.fc39.x86_64.img 20K /boot/loader 16K /boot/lost+found 148K /boot/memtest86+x64.bin 164K /boot/symvers-6.10.3-200.fc40.x86_64.xz 160K /boot/symvers-6.9.12-200.fc40.x86_64.xz 160K /boot/symvers-6.9.9-100.fc39.x86_64.xz 8.7M /boot/System.map-6.9.9-100.fc39.x86_64 15M /boot/vmlinuz-0-rescue-7cf63543075b47d48d09f1649641c3a1 16M /boot/vmlinuz-6.9.9-100.fc39.x86_64 4.0K /boot/xen-4.18.2.config 1.2M /boot/xen-4.18.2.gz
Comparing your list to mine, I was struck by the size of /boot/efi. The size of mine is inflated by one directory of 288M, /boot/efi/7cf63543075b47d48d09f1649641c3a1. I don't know what that is or why it's there. But the contents look suggestive:
# ls -lR /boot/efi/7cf63543075b47d48d09f1649641c3a1 /boot/efi/7cf63543075b47d48d09f1649641c3a1: total 16 drwxr-xr-x 2 root root 4096 2024-07-20 19:03 0-rescue drwxr-xr-x 2 root root 4096 2024-08-13 09:42 6.10.3-200.fc40.x86_64 drwxr-xr-x 2 root root 4096 2024-08-01 04:04 6.9.12-200.fc40.x86_64 drwxr-xr-x 2 root root 4096 2024-07-20 19:03 7.00
/boot/efi/7cf63543075b47d48d09f1649641c3a1/0-rescue: total 188184 -rw------- 1 root root 176793402 2024-07-20 19:03 initrd -rwxr-xr-x 1 root root 15898984 2024-07-20 19:02 linux
/boot/efi/7cf63543075b47d48d09f1649641c3a1/6.10.3-200.fc40.x86_64: total 53844 -rw-r--r-- 1 root root 39122311 2024-08-13 09:42 initrd -rw-r--r-- 1 root root 16009576 2024-08-13 09:42 linux
/boot/efi/7cf63543075b47d48d09f1649641c3a1/6.9.12-200.fc40.x86_64: total 52776 -rw-r--r-- 1 root root 38149613 2024-08-01 04:04 initrd -rw-r--r-- 1 root root 15890792 2024-08-01 04:04 linux
/boot/efi/7cf63543075b47d48d09f1649641c3a1/7.00: total 0
dkms: running auto installation service for kernel 6.10.3-200.fc40.x86_64 dkms: autoinstall for kernel 6.10.3-200.fc40.x86_64 Done.
What are you using dkms for? I wonder if that is causing your issue?
I see you did the dnf reinstall and that also had the dkms messages.
I don't know why dkms is there. Must have been part of the initial installation or an update. However, "dnf remove dkms" and then running "dnf reinstall" did not fix the problem.
It certainly seems as though the installation of the last two kernels got put into the efi directory. How could that happen?
On 13 Aug 2024, at 17:58, Dave Close dave@compata.com wrote:
Comparing your list to mine, I was struck by the size of /boot/efi. The size of mine is inflated by one directory of 288M, /boot/efi/7cf63543075b47d48d09f1649641c3a1. I don't know what that is or why it's there. But the contents look suggestive:
My /boot/efi only has EFI in it that is the mount point UEFI boot files.
I think you can delete the /boot/efi/7cf63543075b47d48d09f1649641c3a1 safely.
Barry
Dave Close wrote:
Comparing your list to mine, I was struck by the size of /boot/efi. The size of mine is inflated by one directory of 288M, /boot/efi/7cf63543075b47d48d09f1649641c3a1. I don't know what that is or why it's there. But the contents look suggestive: # ls -lR /boot/efi/7cf63543075b47d48d09f1649641c3a1 /boot/efi/7cf63543075b47d48d09f1649641c3a1: total 16 drwxr-xr-x 2 root root 4096 2024-07-20 19:03 0-rescue drwxr-xr-x 2 root root 4096 2024-08-13 09:42 6.10.3-200.fc40.x86_64 drwxr-xr-x 2 root root 4096 2024-08-01 04:04 6.9.12-200.fc40.x86_64 drwxr-xr-x 2 root root 4096 2024-07-20 19:03 7.00 /boot/efi/7cf63543075b47d48d09f1649641c3a1/0-rescue: total 188184 -rw------- 1 root root 176793402 2024-07-20 19:03 initrd -rwxr-xr-x 1 root root 15898984 2024-07-20 19:02 linux /boot/efi/7cf63543075b47d48d09f1649641c3a1/6.10.3-200.fc40.x86_64: total 53844 -rw-r--r-- 1 root root 39122311 2024-08-13 09:42 initrd -rw-r--r-- 1 root root 16009576 2024-08-13 09:42 linux /boot/efi/7cf63543075b47d48d09f1649641c3a1/6.9.12-200.fc40.x86_64: total 52776 -rw-r--r-- 1 root root 38149613 2024-08-01 04:04 initrd -rw-r--r-- 1 root root 15890792 2024-08-01 04:04 linux /boot/efi/7cf63543075b47d48d09f1649641c3a1/7.00: total 0 It certainly seems as though the installation of the last two kernels got put into the efi directory. How could that happen?
I suspect I know why this happened.
Do you have the "sdubby" or "systemd-boot-unsigned" package installed (which brings in sdubby)?
The sdubby package installs an /etc/kernel/install.conf that tells the kernel-install script that runs in the kernel-core %post to install kernels and initrds in /boot/efi/$MACHINE_ID/. (Where $MACHINE_ID is from the contents of /etc/machine-id).
This is something that systemd-boot uses, and I'm not sure why it happens, but some dependency seems to pull that package in for some people.
Backing it out simply means you need to uninstall the sdubby package and re-run the kernel-install command, or more simply, reinstall the kernel-core package. Once that's done, it will do the normal GRUB2 boot method of putting files into /boot/ and blscfg entries in /boot/loader/entries/.
"Jonathan Billings" wrote:
I suspect I know why this happened.
Do you have the "sdubby" or "systemd-boot-unsigned" package installed (which brings in sdubby)?
The sdubby package installs an /etc/kernel/install.conf that tells the kernel-install script that runs in the kernel-core %post to install kernels and initrds in /boot/efi/$MACHINE_ID/. (Where $MACHINE_ID is from the contents of /etc/machine-id).
This is something that systemd-boot uses, and I'm not sure why it happens, but some dependency seems to pull that package in for some people.
Backing it out simply means you need to uninstall the sdubby package and re-run the kernel-install command, or more simply, reinstall the kernel-core package. Once that's done, it will do the normal GRUB2 boot method of putting filesinto /boot/ and blscfg entries in /boot/loader/entries/.
That worked! Thanks!
Now I'll try to go back through my logs and try to see when sdubby got installed and why.
Dave Close wrote:
Now I'll try to go back through my logs and try to see when sdubby got installed and why.
Once you've verified it boots into new kernels, feel free to delete the /boot/efi/$MACHINE_ID and /boot/efi/entries directory, so you aren't wasting space on the EFI volume or preventing any future attempts at booting from those files.
I wrote:
Now I'll try to go back through my logs and try to see when sdubby got installed and why.
Jonathan Billings wrote:
Once you've verified it boots into new kernels, feel free to delete the /boot/efi/$MACHINE_ID and /boot/efi/entries directory, so you aren't wasting space on the EFI volume or preventing any future attempts at booting from those files.
I can't tell when sdubby got installed. My logs go back about six months but the only references to sdubby are "upgraded".
I also found that, even reinstalling kernels did not completely fix the contents of /boot until after I had deleted /boot/efi entirely. (Well, actually, just moved it elsewhere. To be deleted eventually.)
-
Barry Scott -
Dave Close -
Francis.Montagnac＠inria.fr -
John Pilkington -
Jonathan Billings