grub2 always defaults to old kernel despite correct configuration

[Not Random]

Hello,

I've had a confusing problem with grub2 for some time now.  I'm now running Fedora
31, but ever since upgrading to to Fedora 30 grub has always defaulted to selecting the
3rd entry.  This means that unless I stop it, Grub will always boot my oldest installed
kernel.

* My system is dual boot with Windows 8.1, which grub detects perfectly.  (This means
there are 5 entries; 3 kernel, 1 memtest and 1 Windows)
* I've tried moving all the grub configuration files aside and reinstalling the
package, but nothing changes.  Tried grub2-mkconfig etc but nothing changes.
* I've had multiple kernel updates during this time but nothing changes.
* Grubby seems to thing the default kernel is selected...but grub never boots it:
  # grubby --grub2 --default-kernel
  /boot/vmlinuz-5.3.7-301.fc31.x86_64
  # rpm -q kernel
  kernel-5.2.18-200.fc30.x86_64
  kernel-5.3.7-200.fc30.x86_64
  kernel-5.3.7-301.fc31.x86_64

I've tried searching for solutions, but cannot find anything like this problem - most
articles address choosing a permanent kernel entry - I want Fedora to go back to selecting
the newest kernel by default, like it did in Fedora 29 and previous versions.

Can anyone tell me where to investigate please?

Thanks!
Frustrated

[sixpack13]

...
>  * Grubby seems to thing the default kernel is selected...but grub
> never boots it:
>    # grubby --grub2 --default-kernel 
man grubby
=> no parameter "--grub2"

>    /boot/vmlinuz-5.3.7-301.fc31.x86_64
>    # rpm -q kernel
>    kernel-5.2.18-200.fc30.x86_64
>    kernel-5.3.7-200.fc30.x86_64
>    kernel-5.3.7-301.fc31.x86_64
>   
you could remove fc30 one's:

rpm -qa | grep kernel-core
sudo dnf remove kernel-core-<fill in the correct version numbers>


>  Can anyone tell me where to investigate please?
>   
cat /etc/default/grub

GRUB_DEFAULT=???

"0" or "saved"
I guess - my config is changed - "saved" is the fedora default

"0" should do what you want: the lastest installed kernel is the top most in the
grub-boot-menu.

- if I pick it right - "saved" needs to select the lastest kernel in grub-/boot
menu manually only ONCE and that selection should become the default forever (but only
until the next kernel update)  


if you change /etc/default/grub you need to run grub2-mkconfig:

[ -d /sys/firmware/efi/efivars ] && sudo grub2-mkconfig -o
/boot/efi/EFI/fedora/grub.cfg || sudo grub2-mkconfig -o /boot/grub2/grub.cfg

the last line is only ONE line from "[" to "g" ! 
it's a short form of an "if" - "then" - "else" 
where:
"&&" means "then"
"||"    means "else"

[Not Random]

Thank you!

I wrote that post from memory, so please excuse the mistake with the grub2 switch, without
that switch the correct kernel is returned as the default boot kernel.

While I could remove the Fedora 30 kernel(s), that doesn't fix the problem.  As soon
as I end up with 3 bootable kernels again it always selects the wrong one!

/etc/default/grub contains:
GRUB_DEFAULT=saved
which as you say is the default for Fedora.

There's no benefit in changing /etc/default/grub as it is already set correctly.

I need to understand how Fedora/grub determines the newest kernel as that logic is somehow
broken.

[Not Random]

I just tested removing one of the old kernels again.  It is still stubbornly booting the
3rd entry, which is now the Recovery option!

[Not Random]

I bet it's something to do with the change to BootLoaderSpec-style configuration files
in fedora 30! https://fedoraproject.org/wiki/Changes/BootLoaderSpecByDefault

[Samuel Sieb]

On 11/7/19 10:20 PM, Not Random wrote:
>  While I could remove the Fedora 30 kernel(s), that doesn't fix
> the problem.  As soon as I end up with 3 bootable kernels again it always selects the
> wrong one!
>  
>  /etc/default/grub contains:
>  GRUB_DEFAULT=saved
>  which as you say is the default for Fedora. 
What is the output of "grub2-editenv list"?

[Not Random]

Thanks!

# grub2-editenv list
saved_entry=2dc3a3c6f8494c8fb93c27f99dc3a246-5.3.8-300.fc31.x86_64
boot_success=1
boot_indeterminate=0
kernelopts=root=UUID=ac6336b1-7fd2-43d7-af44-51b592f3ab8f ro rhgb quiet

For reference, I now have 2 Fedora 31 kernels:

# rpm -q kernel
kernel-5.3.7-301.fc31.x86_64
kernel-5.3.8-300.fc31.x86_64

[Not Random]

>  What is the output of "grub2-editenv list"? 
As mentioned:

# grub2-editenv list
saved_entry=2dc3a3c6f8494c8fb93c27f99dc3a246-5.3.8-300.fc31.x86_64
boot_success=1
boot_indeterminate=0
kernelopts=root=UUID=ac6336b1-7fd2-43d7-af44-51b592f3ab8f ro rhgb quiet

The saved_entry field matches what grubby shows - From everything I can see it appears
correctly configured.  As I mentioned in my original post I've even moved all the
config files aside and reinstalled grub so that it would return to defaults.

I've now been reading https://systemd.io/BOOT_LOADER_SPECIFICATION but cannot find
anything related to this problem.  Everything looks OK.

[Tom Horsley]

On Fri, 08 Nov 2019 06:20:30 -0000
Not Random wrote:

>  I need to understand how Fedora/grub determines the newest kernel as
> that logic is somehow broken. 
I have no idea how it determines the newest, but the saved
kernel is saved in the grubenv file which you can edit
with the grub2-editenv command (but only after getting
hundreds of errors because the command line syntax
for editenv is very obscure and easy to scre up :-).

[Not Random]

>  I have no idea how it determines the newest, but the saved
>  kernel is saved in the grubenv file which you can edit
>  with the grub2-editenv command (but only after getting
>  hundreds of errors because the command line syntax
>  for editenv is very obscure and easy to scre up :-). 
Thank you.

The confusing thing is that the output of 'grub2-editenv list' and grubby both
show that currently the correct (newest) kernel is the saved kernel.  This kernel
isn't what is actually selected at boot though!

This thread has been helpful to confirm that I'm not crazy!

[Tom Horsley]

On Sat, 09 Nov 2019 04:59:50 -0000
Not Random wrote:

>  The confusing thing is that the output of 'grub2-editenv
> list' and grubby both show that currently the correct (newest) kernel is the saved
> kernel.  This kernel isn't what is actually selected at boot though! 
That sounds as if grub may actually be installed in two places
and the one you think your are booting isn't the one the
computer boots.

[Tim]

On Fri, 2019-11-08 at 06:20 +0000, Not Random wrote:
>  /etc/default/grub contains:
>  GRUB_DEFAULT=saved
>  which as you say is the default for Fedora.
>  
>  There's no benefit in changing /etc/default/grub as it is already set
>  correctly. 
I think the problem is that just tells Fedora to use the previously
saved entry as its default, and there's other things (perhaps more than
one) that determine what will actually be the default.

In the old grub (which was easier to follow the instructions) if you
wanted a particular boot entry to be remembered as the default, you had
to add a save default instruction to that boot stanza.  As that stanza
was executed, it set itself as default and booted that kernel.  Any
other entry that didn't have that instruction, wasn't remembered.  I'm
not sure if grub2 behaves that way, too.

There's a boot once option, which means that you make a decision to
boot from some other kernel, without changing the saved default.  You
might boot from a rescue option to fix something, and not want that to
be the default, you'll like your next normal boot to do the usual
kernel.

There's a boot next option, which allows you to set which entry to boot
next time, after doing that the subsequent boot will be the default. 
That can be used for things like making the PC reboot into the UEFI
config mode.  Once you exit the config, it'll reboot and boot up to
your usual kernel.

Doing a bit of googling, /boot/grub2/grubenv file cannot be manually
edited.  Use the following command instead: 

[root@host ~]# grub2-set-default 0
[root@host ~]# grub2-editenv list
saved_entry=0

That 0 should mean the most recently installed kernel.

-- 
 
uname -rsvp
Linux 3.10.0-1062.4.1.el7.x86_64 #1 SMP Fri Oct 18 17:15:30 UTC 2019 x86_64
 
Boilerplate:  All unexpected mail to my mailbox is automatically deleted.
I will only get to see the messages that are posted to the mailing list.

[Not Random]

>  I think the problem is that just tells Fedora to use the previously
>  saved entry as its default, and there's other things (perhaps more than
>  one) that determine what will actually be the default.
>  
>  In the old grub (which was easier to follow the instructions) if you
>  wanted a particular boot entry to be remembered as the default, you had
>  to add a save default instruction to that boot stanza.  As that stanza
>  was executed, it set itself as default and booted that kernel.  Any
>  other entry that didn't have that instruction, wasn't remembered.  I'm
>  not sure if grub2 behaves that way, too.
>  
>  There's a boot once option, which means that you make a decision to
>  boot from some other kernel, without changing the saved default.  You
>  might boot from a rescue option to fix something, and not want that to
>  be the default, you'll like your next normal boot to do the usual
>  kernel.
>  
>  There's a boot next option, which allows you to set which entry to boot
>  next time, after doing that the subsequent boot will be the default. 
>  That can be used for things like making the PC reboot into the UEFI
>  config mode.  Once you exit the config, it'll reboot and boot up to
>  your usual kernel.
>  
>  Doing a bit of googling, /boot/grub2/grubenv file cannot be manually
>  edited.  Use the following command instead: 
>  
>  [root@host ~]# grub2-set-default 0
>  [root@host ~]# grub2-editenv list
>  saved_entry=0
>  
>  That 0 should mean the most recently installed kernel. 
Thank you so much Tim!  When I ran 'grub2-set-default 0' again and ran
grub2-mkconfig it seems to have worked!  I've rebooted twice and the system seems to
work.

I'm not sure what the root cause was as I've never consciously use the boot next
option and I'm not aware of a failed boot in ...years.

[Tim]

Tim:
> > Doing a bit of googling, /boot/grub2/grubenv file cannot be
> manually
> > edited.  Use the following command instead: 
> >
> > [root@host ~]# grub2-set-default 0
> > [root@host ~]# grub2-editenv list
> > saved_entry=0
> >
> > That 0 should mean the most recently installed kernel. 
(I should add that the set-default command sets it, the subsequent list
command line just confirms the change.)


Not Random:
>  Thank you so much Tim!  When I ran 'grub2-set-default 0'
> again and
>  ran grub2-mkconfig it seems to have worked!  I've rebooted twice and
>  the system seems to work. 
Interesting.  The page I found the info on didn't mention using
"grub2-mkconfig", so I wonder if it's necessary.  You could try setting
the default to 1, then rebooting without using the mkconfig thing, and
see if the set default command works by itself.

(I'm not in a position to try this myself, just at the moment.)
 
-- 
 
uname -rsvp
Linux 3.10.0-1062.4.1.el7.x86_64 #1 SMP Fri Oct 18 17:15:30 UTC 2019 x86_64
 
Boilerplate:  All unexpected mail to my mailbox is automatically deleted.
I will only get to see the messages that are posted to the mailing list.