kernel leftovers in /boot/efi/EFI/Linux/

[Germano Massullo]

Hello, during a dnf update I noticed that /boot/efi was full and the 
system was not able to update.
So I found out in /boot/efi/EFI/Linux/ there are many files belonging to 
kernel RPMs no longer installed on the system.
I am wondering why there are these leftovers. Some of the kernel with 
.fc39.1 in their name are from RPMs I created to test kernel patches, 
but they don't have anything special compared to regular Fedora kernel 
RPMs, since I just rebuilt the package after having added a patch.

root@machine:/home/user# eza -lah /boot/efi/EFI/Linux/
Permissions Size User Date Modified Name
.rwx------@  61M root 11 dic  2023 
3fca3f6e08d1481a961d4954bff61ddf-6.6.4-100.fc39.1.x86_64+debug.efi
.rwx------@  44M root 11 dic  2023 
3fca3f6e08d1481a961d4954bff61ddf-6.6.4-100.fc39.1.x86_64.efi
.rwx------@  61M root 11 dic  2023 
3fca3f6e08d1481a961d4954bff61ddf-6.6.4-200.fc39.x86_64+debug.efi
.rwx------@  44M root 11 dic  2023 
3fca3f6e08d1481a961d4954bff61ddf-6.6.4-200.fc39.x86_64.efi
.rwx------@  62M root 13 dic  2023 
3fca3f6e08d1481a961d4954bff61ddf-6.6.6-200.fc39.1.x86_64+debug.efi
.rwx------@  44M root 13 dic  2023 
3fca3f6e08d1481a961d4954bff61ddf-6.6.6-200.fc39.1.x86_64.efi
.rwx------@  61M root 20 dic  2023 
3fca3f6e08d1481a961d4954bff61ddf-6.6.6-200.fc39.x86_64+debug.efi
.rwx------@  62M root 17 dic  2023 
3fca3f6e08d1481a961d4954bff61ddf-6.6.6-201.fc39.1.x86_64+debug.efi
.rwx------@  44M root 17 dic  2023 
3fca3f6e08d1481a961d4954bff61ddf-6.6.6-201.fc39.1.x86_64.efi
.rwx------@ 1,2M root 20 dic  2023 
3fca3f6e08d1481a961d4954bff61ddf-6.6.7-200.fc39.x86_64+debug.efi
.rwx------@    0 root 31 dic  2023 
3fca3f6e08d1481a961d4954bff61ddf-6.6.8-200.fc39.x86_64+debug.efi
.rwx------@    0 root  7 gen 00:31 
3fca3f6e08d1481a961d4954bff61ddf-6.6.9-200.fc39.x86_64+debug.efi
.rwx------@    0 root 16 gen 11:04 
3fca3f6e08d1481a961d4954bff61ddf-6.6.11-200.fc39.x86_64+debug.efi
.rwx------@    0 root 24 gen 20:44 
3fca3f6e08d1481a961d4954bff61ddf-6.6.12-200.fc39.x86_64+debug.efi
.rwx------@    0 root 26 gen 23:15 
3fca3f6e08d1481a961d4954bff61ddf-6.6.13-200.fc39.x86_64+debug.efi
.rwx------@    0 root  4 feb 17:50 
3fca3f6e08d1481a961d4954bff61ddf-6.6.14-200.fc39.x86_64+debug.efi

root@machine:/home/user# rpm -q kernel
kernel-6.6.12-200.fc39.x86_64
kernel-6.6.13-200.fc39.x86_64
kernel-6.6.14-200.fc39.x86_64

[Gerd Hoffmann]

On Sun, Feb 18, 2024 at 10:48:32AM +0100, Germano Massullo wrote:
> Hello, during a dnf update I noticed that /boot/efi was full and the system
> was not able to update.
> So I found out in /boot/efi/EFI/Linux/ there are many files belonging to
> kernel RPMs no longer installed on the system.
> I am wondering why there are these leftovers. Some of the kernel with
> .fc39.1 in their name are from RPMs I created to test kernel patches, but
> they don't have anything special compared to regular Fedora kernel RPMs,
> since I just rebuilt the package after having added a patch.

The kernel-install plugins /usr/lib/kernel/install.d/90-uki-copy.install
and (in case uki-direct is installed)
/usr/lib/kernel/install.d/99-uki-uefi-setup.install should handle both
installing and removing the kernel images.  Apparently there is a bug on
one of these two scripts ...

Try 'rpm -q --scripts kernel-uki-virt' and run the 'kernel-install add'
and 'kernel-install remove' commands manually, with '-v' option added
for verbose output.  That should give a clue where things are failing.

take care,
  Gerd

[Germano Massullo]

Thank you for your support.
Perhaps I am missing something in the kernel-install commands

root@machine:/home/user# rpm -q --scripts kernel-uki-virt
package kernel-uki-virt not installed

root@machine:/home/user# kernel-install -v add
Loading /usr/lib/kernel/install.conf…
Loaded /usr/lib/kernel/install.conf.
MACHINE_ID=xxx set via /etc/machine-id.
Found cgroup2 on /sys/fs/cgroup/, full unified hierarchy
Found container virtualization none.
/dev/nvme0n1p1: Partition has wrong PART_ENTRY_TYPE=xxx for XBOOTLDR 
partition.
Couldn't find an XBOOTLDR partition.
Failed to check file system type of "/efi": No such file or directory
File system "/boot" is not a FAT EFI System Partition (ESP) file system.
Using EFI System Partition at /boot/efi as $BOOT_ROOT.
Using entry token: xxx
Too few arguments.

root@machine:/home/user# kernel-install -v remove
Loading /usr/lib/kernel/install.conf…
Loaded /usr/lib/kernel/install.conf.
MACHINE_ID=xxx set via /etc/machine-id.
Found cgroup2 on /sys/fs/cgroup/, full unified hierarchy
Found container virtualization none.
/dev/nvme0n1p1: Partition has wrong PART_ENTRY_TYPE=xxx for XBOOTLDR 
partition.
Couldn't find an XBOOTLDR partition.
Failed to check file system type of "/efi": No such file or directory
File system "/boot" is not a FAT EFI System Partition (ESP) file system.
Using EFI System Partition at /boot/efi as $BOOT_ROOT.
Using entry token: xxx
Too few arguments.

[Gerd Hoffmann]

On Wed, Feb 21, 2024 at 07:10:11PM +0100, Germano Massullo wrote:
> Thank you for your support.
> Perhaps I am missing something in the kernel-install commands
>
> root@machine:/home/user# rpm -q --scripts kernel-uki-virt
> package kernel-uki-virt not installed

Hmm.  I'm really wondering how you end up having anything
in /boot/efi/EFI/Linux if the kernel-uki-virt sub-rpm is
not installed.  Non-UKI kernels should never ever be copied
to that location.
> root@machine:/home/user# kernel-install -v add

The command is longer than that, with kernel version and image,
best cut+paste the complete line from the rpm install scripts,
then add '-v' for verbose output.

If kernel-uki-virt is not installed try
'rpm -q --scripts kernel-core'

take care,
  Gerd

[Germano Massullo]

Hello Gerd, by following your instruction, I searched in my build folder 
one of the kernels that have not been successfully erased, and then I 
retrieved the scripts of the kernel-core package, (uploaded at 
https://paste.centos.org/view/ac440ce5 )
Should I proceed with just the command
/bin/kernel-install remove 6.6.4-100.fc39.1.x86_64 || exit $?
or with the complete block from 11 to 15 line?

============
/bin/kernel-install remove 6.6.4-100.fc39.1.x86_64 || exit $?
if [ -x /usr/sbin/weak-modules ]
then
     /usr/sbin/weak-modules --remove-kernel 6.6.4-100.fc39.1.x86_64 || 
exit $?
fi

[Gerd Hoffmann]

On Wed, Mar 13, 2024 at 10:00:45AM +0100, Germano Massullo wrote:
> Hello Gerd, by following your instruction, I searched in my build folder one
> of the kernels that have not been successfully erased, and then I retrieved
> the scripts of the kernel-core package, (uploaded at
> https://paste.centos.org/view/ac440ce5 )
> Should I proceed with just the command
> /bin/kernel-install remove 6.6.4-100.fc39.1.x86_64 || exit $?
> or with the complete block from 11 to 15 line?

Just the kernel-install line, with '-v' or '--verbose' added for verbose
output, i.e. ...

/bin/kernel-install --verbose add 6.6.4-100.fc39.1.x86_64 /lib/modules/6.6.4-100.fc39.1.x86_64/vmlinuz

/bin/kernel-install --verbose remove 6.6.4-100.fc39.1.x86_64

... for install and uninstall

take care,
  Gerd

[Germano Massullo]

I was not able to remove such kernel entries from /boot/efi/EFI/Linux/ 
by using the command.
Output here
https://germano.fedorapeople.org/bugreport/kernel_leftovers/kernel_leftovers...

I also tried to enter the full name of the files, but I had no success :-(

[Gerd Hoffmann]

On Sun, Mar 17, 2024 at 01:20:06PM +0100, Germano Massullo wrote:
> I was not able to remove such kernel entries from /boot/efi/EFI/Linux/ by
> using the command.
> Output here
> https://germano.fedorapeople.org/bugreport/kernel_leftovers/kernel_leftovers...
>
> I also tried to enter the full name of the files, but I had no success :-(

Think I've nailed it:

https://github.com/kraxel/systemd/commit/6eed67a291bf896befb4d407dc57f9a7fca...

Can you try apply the patch to
/usr/lib/kernel/install.d/90-uki-copy.install and see if that helps?

thanks,
  Gerd

[Dan Horák]

On Mon, 18 Mar 2024 17:24:06 +0100
Gerd Hoffmann  wrote:
> On Sun, Mar 17, 2024 at 01:20:06PM +0100, Germano Massullo wrote:
> > I was not able to remove such kernel entries from /boot/efi/EFI/Linux/ by
> > using the command.
> > Output here
> > https://germano.fedorapeople.org/bugreport/kernel_leftovers/kernel_leftovers...
> >
> > I also tried to enter the full name of the files, but I had no success :-(
>
> Think I've nailed it:
>
> https://github.com/kraxel/systemd/commit/6eed67a291bf896befb4d407dc57f9a7fca...
>
> Can you try apply the patch to
> /usr/lib/kernel/install.d/90-uki-copy.install and see if that helps?

can't comment on the function, but there is a typo in the commit
message IMO

and *not* exist early -> and *not* exit early

Dan

[Gerd Hoffmann]

> > https://github.com/kraxel/systemd/commit/6eed67a291bf896befb4d407dc57f9a7fca...
>
> can't comment on the function, but there is a typo in the commit
> message IMO
>
> and *not* exist early -> and *not* exit early

Thanks, fixed.

take care,
  Gerd

[Germano Massullo]

I created a Copr repository with the patch
https://copr.fedorainfracloud.org/coprs/germano/systemd_fix_uki-copy_deinsta...
but I am experiencing some problems with the installation of the new 
file (see errors below).
I increased the release version with command
rpmdev-bumpspec -D systemd.spec
I don't know if this can have triggered the upgrade problem. I am not 
very familiar with the %autorelease macro I saw in the spec file.
Can you please help me?
Thank you

=======================
root@machine:/boot/efi/EFI/Linux# dnf update systemd* -y
Ultima verifica della scadenza dei metadati: 1:05:05 fa il mar 19 mar 
2024, 00:34:18.
Dipendenze risolte.

Problema: systemd-libs-254.10-1.fc39.i686 from @System  does not 
belong to a distupgrade repository
   - cannot install both systemd-libs-254.10-1.fc39.1.x86_64 from 
copr:copr.fedorainfracloud.org:germano:systemd_fix_uki-copy_deinstall_patch 
and systemd-libs-254.10-1.fc39.x86_64 from @System
   - cannot install both systemd-libs-254.10-1.fc39.x86_64 from updates 
and systemd-libs-254.10-1.fc39.1.x86_64 from 
copr:copr.fedorainfracloud.org:germano:systemd_fix_uki-copy_deinstall_patch
   - cannot install the best update candidate for package 
systemd-libs-254.10-1.fc39.i686
   - cannot install the best update candidate for package 
systemd-libs-254.10-1.fc39.x86_64
=============================================================================================================================================
  Package           Arch        Version Repository Size
=============================================================================================================================================
Esclusione dei pacchetti con conflitti:
(aggiungere '--best --allowerasing' alla linea di comando per forzarne 
l'aggiornamento):
  systemd-libs      x86_64      254.10-1.fc39.1 
copr:copr.fedorainfracloud.org:germano:systemd_fix_uki-copy_deinstall_patch 
687 k

Riepilogo della transazione
=============================================================================================================================================
Ignorati  1 pacchetto

Nessuna operazione da compiere.
Fatto!

[Gerd Hoffmann]

On Tue, Mar 19, 2024 at 01:46:54AM +0100, Germano Massullo wrote:
> I created a Copr repository with the patch
> https://copr.fedorainfracloud.org/coprs/germano/systemd_fix_uki-copy_deinsta...

Oh, I was thinking about just patching the file directly on the
installed system (90-uki-copy.install is simply copied on install,
so the patch should apply just fine) instead of doing a full
systemd package update ...

take care,
  Gerd

[Germano Massullo]

the patch worked, thank you
https://germano.fedorapeople.org/bugreport/kernel_leftovers/kernel_leftovers...

[Gerd Hoffmann]

On Tue, Mar 19, 2024 at 11:56:20AM +0100, Germano Massullo wrote:
> the patch worked, thank you
> https://germano.fedorapeople.org/bugreport/kernel_leftovers/kernel_leftovers...

Landed upstream:
https://github.com/systemd/systemd/commit/3037616d8ed68f3263746e3c6399d4a052...

take care,
  Gerd

[Germano Massullo]

For documentation purposes, I attach the patched file

# stat /usr/lib/kernel/install.d/90-uki-copy.install
   File: /usr/lib/kernel/install.d/90-uki-copy.install
   Dim.: 3142            Blocchi: 8          Blocco di IO: 4096 file 
regolare
Device: 253,1   Inode: 35127375    Links: 1
Accesso: (0755/-rwxr-xr-x)  Uid: (    0/    root)   Gid: (    0/ root)
Contesto: system_u:object_r:lib_t:s0
Accesso  : 2024-04-04 23:17:37.822927125 +0200
Modifica : 2024-03-19 11:31:38.454277675 +0100
Cambio   : 2024-03-19 11:31:38.475277569 +0100
Creazione: 2024-03-19 11:31:38.454277675 +0100