The well known BIOSboot partition issue - Fedora does not install flawlessly on HDs larger 2 TB in Raid 1

[Peter Boy]

I am doing some test installations to contribute some of our internal documentation for Fedora Server documentation. Thereby a problem with the BiOSboot partition occurred again.

To set up the system with RAID 1, manual partitioning must be performed during installation. I have created in the Anaconda GUI:
- BIOSboot Partition
- /boot as default partition and changed to raid 1
- /ROOT as LVM changed to raid 1 as well (and some additional LVs in that VG)

The installation aborted with the error message:

"The following error occurred while installing the boot loader. The system will not be bootable. Do you want to ignore this and continue with the installation?
Boot loader install failed"

Anaconda creates BIOSboot only on the first hard disk, but then tries to install grub also on the 2nd hard disk and fails because of the missing BIOSboot partition there.

This was already discussed a long time ago and considered to create a BIOSboot partition automatically when creating a GPT even without user action. However, concerns were raised about this and the proposal was obviously not implemented.

I have not found a way in the current Anaconda manual partition GUI to install BIOSboot on the 2nd disk as well. My simple workaround was to boot a live system, set up GPT and BIOSboot, shutdown, and then boot the Fedora Server installation image.

The installation then went without a fuss. But is this the intended way? I suspect it isn’t. I probably missed something.

Any hint appreciated.

Peter

[Peter Robinson]

On Thu, Dec 31, 2020 at 3:12 PM Peter Boy  wrote:
> I am doing some test installations to contribute some of our internal documentation for Fedora Server documentation. Thereby a problem with the BiOSboot partition occurred again.
>
> To set up the system with RAID 1, manual partitioning must be performed during installation. I have created in the Anaconda GUI:
> BIOSboot Partition
> /boot as default partition and changed to raid 1
> /ROOT as LVM changed to raid 1 as well (and some additional LVs in that VG)
>
> The installation aborted with the error message:
>
> "The following error occurred while installing the boot loader. The system will not be bootable. Do you want to ignore this and continue with the installation?
> Boot loader install failed"
>
> Anaconda creates BIOSboot only on the first hard disk, but then tries to install grub also on the 2nd hard disk and fails because of the missing BIOSboot partition there.
>
> This was already discussed a long time ago and considered to create a BIOSboot partition automatically when creating a GPT even without user action. However, concerns were raised about this and the proposal was obviously not implemented.
>
> I have not found a way in the current Anaconda manual partition GUI to install BIOSboot on the 2nd disk as well. My simple workaround was to boot a live system, set up GPT and BIOSboot, shutdown, and then boot the Fedora Server installation image.
>
> The installation then went without a fuss. But is this the intended way? I suspect it isn’t. I probably missed something.
>
> Any hint appreciated.

What is the system? Does it support being switches to booting from
UEFI and using GPT partitioning and then you won't need the BIOSboot
partition.

[Peter Boy]

> Am 31.12.2020 um 19:24 schrieb Peter Robinson :
> > I have not found a way in the current Anaconda manual partition GUI to install BIOSboot on the 2nd disk as well. My simple workaround was to boot a live system, set up GPT and BIOSboot, shutdown, and then boot the Fedora Server installation image.
> >
> > The installation then went without a fuss. But is this the intended way? I suspect it isn’t. I probably missed something.
> >
> > Any hint appreciated.
>
> What is the system? Does it support being switches to booting from
> UEFI and using GPT partitioning and then you won't need the BIOSboot
> partition.

It’s a rented dedicated Server (by Hetzner, a big European business) so I can’t adjust anything in the firmware. And i guess there are others who might have the same issue. So I would like to write a reliable and proper documentation / how to.

[Chris Murphy]

On Thu, Dec 31, 2020 at 11:24 AM Peter Robinson  wrote:
> What is the system? Does it support being switches to booting from
> UEFI and using GPT partitioning and then you won't need the BIOSboot
> partition.

It's even worse on UEFI. There it should create an ESP on each drive.
But this is not allowed because there can be only one ESP for
/boot/efi. The very idea of /boot/efi is deeply unfortunate.
Nevertheless, the installer won't create an ESP on each drive. And
even if you manually create an ESP on each drive, the installer only
mounts one to /boot/efi and thus only that one gets the bootloader.
And only one is added as a boot entry in NVRAM.

The area of multiple boot devices is unsolved. It might be solvable by
bootupd. Decouple /boot/efi and /boot from RPM entirely. Have a single
canonical reference for what their states should be, and use bootupd
to enforce sync on one or more volumes that apply to the installation.

[Peter Boy]

> Am 31.12.2020 um 23:33 schrieb Chris Murphy :
>
> On Thu, Dec 31, 2020 at 11:24 AM Peter Robinson  wrote:
> > What is the system? Does it support being switches to booting from
> > UEFI and using GPT partitioning and then you won't need the BIOSboot
> > partition.
>
> It's even worse on UEFI. There it should create an ESP on each drive.
> But this is not allowed because there can be only one ESP for
> /boot/efi.

I have to admit, I haven't used UEFI for Fedora yet. Does this really mean that you can't use a software raid with UEFI resp. that if the "main disk" fails, the system might continue to run but can no longer be booted? What a step backwards that would be.

[Chris Murphy]

On Fri, Jan 8, 2021 at 2:31 AM Peter Boy  wrote:
> > Am 31.12.2020 um 23:33 schrieb Chris Murphy :
> >
> > On Thu, Dec 31, 2020 at 11:24 AM Peter Robinson  wrote:
> > > What is the system? Does it support being switches to booting from
> > > UEFI and using GPT partitioning and then you won't need the BIOSboot
> > > partition.
> >
> > It's even worse on UEFI. There it should create an ESP on each drive.
> > But this is not allowed because there can be only one ESP for
> > /boot/efi.
>
> I have to admit, I haven't used UEFI for Fedora yet. Does this really mean that you can't use a software raid with UEFI resp. that if the "main disk" fails, the system might continue to run but can no longer be booted? What a step backwards that would be.

I haven't tried recently. I'm pretty sure it was only available in
CentOS and thus RHEL. It works but it's invalid, and practically
speaking it's asking for trouble because the firmware can write to the
ESP. And other possible confusion. The long term approach is probably
coreos/bootupd to sync multiple ESPs from a common source in /usr/ .

Since moving to BLS, the grub.cfg on the ESP is static. So changes are
only going to /boot which can be raid1. The near term approach is to
manually sync /boot/efi to the secondary ESPs. And then add an entry
in NVRAM so it's an explicit fallback.

[Chris Murphy]

On Thu, Dec 31, 2020 at 8:12 AM Peter Boy  wrote:
> I am doing some test installations to contribute some of our internal documentation for Fedora Server documentation. Thereby a problem with the BiOSboot partition occurred again.
>
> To set up the system with RAID 1, manual partitioning must be performed during installation. I have created in the Anaconda GUI:
> BIOSboot Partition
> /boot as default partition and changed to raid 1
> /ROOT as LVM changed to raid 1 as well (and some additional LVs in that VG)
>
> The installation aborted with the error message:
>
> "The following error occurred while installing the boot loader. The system will not be bootable. Do you want to ignore this and continue with the installation?
> Boot loader install failed"
>
> Anaconda creates BIOSboot only on the first hard disk, but then tries to install grub also on the 2nd hard disk and fails because of the missing BIOSboot partition there.
>
> This was already discussed a long time ago and considered to create a BIOSboot partition automatically when creating a GPT even without user action. However, concerns were raised about this and the proposal was obviously not implemented.
>
> I have not found a way in the current Anaconda manual partition GUI to install BIOSboot on the 2nd disk as well. My simple workaround was to boot a live system, set up GPT and BIOSboot, shutdown, and then boot the Fedora Server installation image.
>
> The installation then went without a fuss. But is this the intended way? I suspect it isn’t. I probably missed something.

I tried both 2-disk Automatic, and 2-disk Custom-Automatic
(Custom/manual partitioning, but click the blue text to create
partitions automatically). They both succeed but only one disk has
BIOS Boot

As I fiddle with Custom UI, manually creating BIOS boot, the
Device(s)>Modify dialog claims this partition is on both devices. But
the installation summary only shows it on one. And the installation
succeeds without failing.

storage.log shows the command: grub2-install --no-floppy /dev/vda

While /dev/vdb exists, it is not specified here. Now, there is a
separate dialog for bootloaders accessed in the Installation
Destination panel at the bottom, there's some blue text "full disk
summary and bootloader" which is just another way to get into trouble,
but is layered upon anything the user chooses in Custom that relates
to bootloader stuff.

I'd characterize this primarily as a bug. But the most central problem
is that esoteric bootloader related things should not exist in a GUI
installer. The installer should just do the right thing and present no
options at all whatsoever to the user. I'd go so far as to say this
should apply to $BOOT as well. All such things could be controlled by
kickstart, I leave that question up to the Anaconda and bootloader
teams. But their single design and recommendation for each use case
they want to support should not be user overridable in the GUI. Hands
down. It's currently a giant sand trap.

--
Chris Murphy

[Peter Boy]

Sorry, I had to repair my test systems which took some time.
> Am 31.12.2020 um 23:26 schrieb Chris Murphy :
>
> On Thu, Dec 31, 2020 at 8:12 AM Peter Boy  wrote:
> > I am doing some test installations to contribute some of our internal documentation for Fedora Server documentation. Thereby a problem with the BiOSboot partition occurred again.
> >
> > To set up the system with RAID 1, manual partitioning must be performed during installation. I have created in the Anaconda GUI:
> > BIOSboot Partition
> > /boot as default partition and changed to raid 1
> > /ROOT as LVM changed to raid 1 as well (and some additional LVs in that VG)
> >
> > The installation aborted with the error message:
> >
> > "The following error occurred while installing the boot loader. The system will not be bootable. Do you want to ignore this and continue with the installation?
> > Boot loader install failed"
> >
> > Anaconda creates BIOSboot only on the first hard disk, but then tries to install grub also on the 2nd hard disk and fails because of the missing BIOSboot partition there.
> >
> > This was already discussed a long time ago and considered to create a BIOSboot partition automatically when creating a GPT even without user action. However, concerns were raised about this and the proposal was obviously not implemented.
> >
> > I have not found a way in the current Anaconda manual partition GUI to install BIOSboot on the 2nd disk as well. My simple workaround was to boot a live system, set up GPT and BIOSboot, shutdown, and then boot the Fedora Server installation image.
> >
> > The installation then went without a fuss. But is this the intended way? I suspect it isn’t. I probably missed something.
>
> ...
>
> As I fiddle with Custom UI, manually creating BIOS boot, the
> Device(s)>Modify dialog claims this partition is on both devices. But
> the installation summary only shows it on one.

Ir’s even worse. If you open the dialog box the first time, both disks are selected (if you select two disks at the initial Installationdestination form. As soon as you click on one of the disks it changes to non-selected and it is no longer possible to select both at the same time.
> And the installation
> succeeds without failing.
>
> storage.log shows the command: grub2-install --no-floppy /dev/vda

I tried several time. As soon as you install a /boot partition on RAID the installation issues a warning, the system would not be bootable if you proceed.

If you ignore the warning, the system boots up. In storage.log you find:
> INFO:program:Running in chroot '/mnt/sysroot'... grub2-install --no-floppy /dev/sda
> INFO:program:Installing for i386-pc platform.
> INFO:program:Installation finished. No error reported.
> ...
> INFO:program:Running in chroot '/mnt/sysroot'... grub2-install --no-floppy /dev/sdb
> INFO:program:Installing for i386-pc platform.
> INFO:program:grub2-install: warning: ../grub-core/partmap/gpt.c:190:this GPT partition label contains no BIOS Boot 
> Partition; embedding won't be possible.
> INFO:program:grub2-install: error: embedding is not possible, but this is required for RAID and LVM install.
> DEBUG:program:Return code: 1
> ...

Anaconda analyses the problem correctly, but unfortunately does not provide a fix
> I'd characterize this primarily as a bug. But the most central problem
> is that esoteric bootloader related things should not exist in a GUI
> installer. The installer should just do the right thing and present no
> options at all whatsoever to the user.

A Fedora Server installation will probably run disks in RAID mode quite often. In this respect, this bug is decidedly "uncomely“. HD larger than 2 TB are becoming more and more affordable and are likely to become more common even in small installations.

I tried the „Advanced Custom (Bildet-GUI“. Here, it is possible to install a BIOSboot on multiple disks and if you do so the installation of RAID 1 system succeeds without failing. But it is much more complex and time-consuming (and error-prone).

Question is, how are we going to write an installation guide as a Fedora project. Do we want a "simple" way (ie. Booting a live system before installation and preparing the disks on the command line) or a "Fedora Tools" way (cumbersome Blivet GUI)?

[Chris Murphy]

On Fri, Jan 8, 2021 at 2:23 AM Peter Boy  wrote:
> Question is, how are we going to write an installation guide as a Fedora project. Do we want a "simple" way (ie. Booting a live system before installation and preparing the disks on the command line) or a "Fedora Tools" way (cumbersome Blivet GUI)?

I can't figure it out at all
https://bugzilla.redhat.com/show_bug.cgi?id=1914607

As for CLI, I tested this method and it works:
1. MBR skip this step. GPT create BIOS boot on each drive.
2. do an installation
3. post-install run grub2-install on both drives

--
Chris Murphy