GPT Partitioning

[Peter Boy]

Last meeting we discussed whether we should change to GPT as default partitioning. Now
some of our members have submitted a change proposal to make GPT the default partitioning
for new installations. I think this is actually long overdue, yet better communication
among us would actually be nice as well. 

https://fedoraproject.org/wiki/Changes/GPTforBIOSbyDefault 

I don’t know if there is a potential problem. Some time ago, if I remember correctly it
was Neil who wrote in a discussion about storage that a software RAID is no longer
possible, because - so my memory - the biosboot partition is not replicable over multiple
disks. So if the previous boot disk fails, you can't just boot from another disk.  

I never checked it and didn’t read the technical specification. And all our server have
hardware raid. So, does anybody knows about it in detail? 


Peter

[Carlos Vidal]

It is true that the BIOS boot partition is limited to a single physical
disk. What I normally do is to replicate it to the other disks with "dd".
As it remains unchanged upon updates, it should be fairly easy to add this
feature to Anaconda, just after the BIOS boot installation.

Of course, after a disk failure, you have to remember to make a copy again
to the new disk.

Another way to automate this would be to have a "checkBIOS" service upon
boot that verifies if all physical disks have the same BIOS-Grub MD5 and if
not, it suggests to replicate or to continue. It can even save the initial
MD5 somewhere under /etc to compare further checks against it.

Carlos


On Sat, May 14, 2022 at 7:52 PM Peter Boy <pboy(a)uni-bremen.de&gt; wrote:

>  Last meeting we discussed whether we should change to GPT as default
>  partitioning. Now some of our members have submitted a change proposal to
>  make GPT the default partitioning for new installations. I think this is
>  actually long overdue, yet better communication among us would actually be
>  nice as well.
>
>  https://fedoraproject.org/wiki/Changes/GPTforBIOSbyDefault
>
>  I don’t know if there is a potential problem. Some time ago, if I remember
>  correctly it was Neil who wrote in a discussion about storage that a
>  software RAID is no longer possible, because - so my memory - the biosboot
>  partition is not replicable over multiple disks. So if the previous boot
>  disk fails, you can't just boot from another disk.
>
>  I never checked it and didn’t read the technical specification. And all
>  our server have hardware raid. So, does anybody knows about it in detail?
>
>
>  Peter
>
>  _______________________________________________
>  server mailing list -- server(a)lists.fedoraproject.org
>  To unsubscribe send an email to server-leave(a)lists.fedoraproject.org
>  Fedora Code of Conduct:
>  https://docs.fedoraproject.org/en-US/project/code-of-conduct/
>  List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
>  List Archives:
>  https://lists.fedoraproject.org/archives/list/[e-mail]
>  Do not reply to spam on the list, report it:
>  https://pagure.io/fedora-infrastructure
>

[Neal Gompa]

On Sun, May 15, 2022 at 8:45 AM Carlos Vidal <carlos(a)tarkus.se&gt; wrote:
>  It is true that the BIOS boot partition is limited to a single physical disk. What I
> normally do is to replicate it to the other disks with "dd". As it remains
> unchanged upon updates, it should be fairly easy to add this feature to Anaconda, just
> after the BIOS boot installation.
>  
It may already do this. I think it already does for the ESP on EFI installs.

>  Of course, after a disk failure, you have to remember to make a copy
> again to the new disk.
>
>  Another way to automate this would be to have a "checkBIOS" service upon boot
> that verifies if all physical disks have the same BIOS-Grub MD5 and if not, it suggests to
> replicate or to continue. It can even save the initial MD5 somewhere under /etc to compare
> further checks against it.
>  
Yeah, there are a number of ways to skin this.


-- 
真実はいつも一つ！/ Always, there's only one truth!

[Peter Boy]

>  Am 15.05.2022 um 14:49 schrieb Neal Gompa
> <ngompa13(a)gmail.com&gt;:
>  
>  On Sun, May 15, 2022 at 8:45 AM Carlos Vidal <carlos(a)tarkus.se&gt; wrote:
> > 
> > It is true that the BIOS boot partition is limited to a single physical disk. What I
> normally do is to replicate it to the other disks with "dd". As it remains
> unchanged upon updates, it should be fairly easy to add this feature to Anaconda, just
> after the BIOS boot installation.
> > 
>  
>  It may already do this. I think it already does for the ESP on EFI installs. 
It does actually not, unfortunately. At least not until the f35 release cycle. I didn’t
check F36.

By the way, the issue is described in the Server user documentation.

If you have a GPT partition table on a bios boot system and choose the custom partitioning
option, Anaconda creates a biosboot partition on the „first“ disk it actually boots from.
On any other disks it doesn’t care about biosboot. It not even reserves the space.

Using a mbr partition table, if Anaconda detects a /boot partition in a raid, in installs
the mbr on any disk that is part of the raid. 

So, Anaconda has to be enabled to do the equivalent to biosboot on a GPT partition schema
(provided the technically specification allows it).

If you choose the ‚advanced custom option‘ (Blivet-GUI) you can manually create the
biosboot partition, but just on one disk. But you can create the partitions on the other
disks to start at the same space as on the „first“ disk, so reserve the space. And if I
remember correctly, I managed it one time to create a biosboot partition in that space
later, but I’m not sure. But I remember it was a mess.

As a consequence, we can only make GPT the default for biosboot machines when Anaconda can
manage the partitioning accordingly and after it has been *thoroughly tested* that
everything actually works in case of a boot disk failure.  

I think it is unrealistic that this is feasible for F37. 

On an UEFI system GPT is default any way. But remains the issue to setup a software raid.

[Chris Murphy]

On Sat, May 14, 2022 at 1:52 PM Peter Boy <pboy(a)uni-bremen.de&gt; wrote:
>  Last meeting we discussed whether we should change to GPT as default partitioning. Now
> some of our members have submitted a change proposal to make GPT the default partitioning
> for new installations. I think this is actually long overdue, yet better communication
> among us would actually be nice as well.
>
>  https://fedoraproject.org/wiki/Changes/GPTforBIOSbyDefault
>
>  I don’t know if there is a potential problem. Some time ago, if I remember correctly it
> was Neil who wrote in a discussion about storage that a software RAID is no longer
> possible, because - so my memory - the biosboot partition is not replicable over multiple
> disks. So if the previous boot disk fails, you can't just boot from another disk.
Last time I checked this particular configuration of /boot on multiple
drives in e.g. a raid1,10,5,6 - on BIOS firmware only, Anaconda issues
the grub2-install command pointing at all the member drives making up
/boot. Whether MBR or GPT, the proper GRUB core.img is installed on
each. So any of them will get you to at least a grub rescue prompt. Of
course it requires the minimum number of disks for GRUB's md driver to
assemble the member drives so it can read /boot to find additional
grub modules and grub.cfg and the kernel and initramfs.

The problem still remains unsolved though for UEFI. Upstream GRUB
suggest sequentially mounting each ESP on each drive, and issuing
grub-install. We don't use grub-install (or grub2-install in Fedora
land) for UEFI at all though, because the grubx64.efi is built in the
Fedora build system and signed for UEFI Secure Boot. Even if the
installer were to mount each ESP in turn, copying the bootloader
configuration files to each one, and then adding the proper entries in
NVRAM for each one to act as fallbacks - the problem is they all get
out of sync as kernels and bootloader updates happen. There is an idea
how to handle this in Fedora CoreOS, a project called bootupd. But
this particular work isn't finished yet, specifically for the
raid/multiple boot disks use case.

https://github.com/coreos/bootupd


-- 
Chris Murphy

[Chris Murphy]

On Mon, May 16, 2022 at 12:26 PM Chris Murphy <lists(a)colorremedies.com&gt; wrote:
>  On Sat, May 14, 2022 at 1:52 PM Peter Boy <pboy(a)uni-bremen.de&gt; wrote:
>  >
>  > Last meeting we discussed whether we should change to GPT as default partitioning.
> Now some of our members have submitted a change proposal to make GPT the default
> partitioning for new installations. I think this is actually long overdue, yet better
> communication among us would actually be nice as well.
>  >
>  > https://fedoraproject.org/wiki/Changes/GPTforBIOSbyDefault
>  >
>  > I don’t know if there is a potential problem. Some time ago, if I remember correctly
> it was Neil who wrote in a discussion about storage that a software RAID is no longer
> possible, because - so my memory - the biosboot partition is not replicable over multiple
> disks. So if the previous boot disk fails, you can't just boot from another disk.
>
>  Last time I checked this particular configuration of /boot on multiple
>  drives in e.g. a raid1,10,5,6 - on BIOS firmware only, Anaconda issues
>  the grub2-install command pointing at all the member drives making up
>  /boot. Whether MBR or GPT, the proper GRUB core.img is installed on
>  each. So any of them will get you to at least a grub rescue prompt. Of
>  course it requires the minimum number of disks for GRUB's md driver to
>  assemble the member drives so it can read /boot to find additional
>  grub modules and grub.cfg and the kernel and initramfs.
>
>  The problem still remains unsolved though for UEFI. Upstream GRUB
>  suggest sequentially mounting each ESP on each drive, and issuing
>  grub-install. We don't use grub-install (or grub2-install in Fedora
>  land) for UEFI at all though, because the grubx64.efi is built in the
>  Fedora build system and signed for UEFI Secure Boot. Even if the
>  installer were to mount each ESP in turn, copying the bootloader
>  configuration files to each one, and then adding the proper entries in
>  NVRAM for each one to act as fallbacks - the problem is they all get
>  out of sync as kernels and bootloader updates happen. There is an idea
>  how to handle this in Fedora CoreOS, a project called bootupd. But
>  this particular work isn't finished yet, specifically for the
>  raid/multiple boot disks use case.
>
>  https://github.com/coreos/bootupd 

Just to expand on this.

I personally prefer the idea of proscribing anything other than fwupd
and bootupd from writing to either /boot or /boot/efi. i.e. no more
dracut or rpm or dnf writing to /boot or /boo/efi. I also support
getting rid of persistently mounting /boot and /boot/efi. Those
volumes are only needed during boot by the bootloader, they aren't
even needed during startup (once the kernel is running and systemd is
starting up services). And when their contents are being updated.
Therefore the thing(s) responsible for updating them, should have the
sole ability to mount them, and an obligation to unmount them when the
updating is finished. That's a clean and secure way of managing /boot
(and by extension /boot/efi).

And it'd be neat if bootctl (part of systemd) were extended to act as
the user interface to bootupd where status, repair and customizations
are needed.

Further, I'd like to see the canonical location for /boot and
/boot/efi to exist in /usr, such that bootupd has the ability to just
sync files from the canonical /usr location to the ESP (or multiple
ESPs). That way the ESP is completely repairable in a simple way: fsck
the ESP, if that fails mkdosfs, mount it, repopulate it,
verify+restore the NVRAM entries.


-- 
Chris Murphy

[Peter Boy]

>  Am 16.05.2022 um 18:35 schrieb Chris Murphy
> <lists(a)colorremedies.com&gt;:
>  
>  Further, I'd like to see the canonical location for /boot and
>  /boot/efi to exist in /usr,... 
This would be the next deviation from the FHS, which Fedora at least pretends to adhere
to. 

(As sensible as that would probably be)

[Daniel Schier]

Hi all,

I am mostly lurking the list, since I don’t have much time to spend. Anyway, the whole
situation is quite interesting for me (personally and professionally).

If one can describe different scenarios that may occur (BIOSboot + GPT, UEFIboot + GPT,
with/without LVM, RAID configurations, etc.) and some test cases, I can have a look during
my worktime and offer the results.

I can also try to make the setups reproducible for automatic testing (Kickstart, Ansible,
whatnot), if this helps.

In case nom of the above sounds helpful, please feel free to ignore the message :)

Best regards,
Daniel

---
### Contact

dschier(a)while-true-do.io
dschier @libera.chat
linkedin.com/u/dschier
twitter.com/dschier_wtd
github.com/dschier-wtd/

### Projects

while-true-do.io
style-cheat.io

### Disclaimer

If not stated otherwise all of my work is licensed under a Creative Commons Attribution
4.0 International License (creativecommons.org/licenses/by/4.0/).

>  Am 17.05.2022 um 08:42 schrieb Peter Boy <pboy(a)uni-bremen.de&gt;:
>  
>  
>  
> > Am 16.05.2022 um 18:35 schrieb Chris Murphy <lists(a)colorremedies.com&gt;:
> > 
> > Further, I'd like to see the canonical location for /boot and
> > /boot/efi to exist in /usr,...
>  
>  This would be the next deviation from the FHS, which Fedora at least pretends to adhere
> to. 
>  
>  (As sensible as that would probably be) 
>  _______________________________________________
>  server mailing list -- server(a)lists.fedoraproject.org
>  To unsubscribe send an email to server-leave(a)lists.fedoraproject.org
>  Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
>  List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
>  List Archives:
> https://lists.fedoraproject.org/archives/list/[e-mail]
>  Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure

[Peter Boy]

>  Am 16.05.2022 um 18:26 schrieb Chris Murphy
> <lists(a)colorremedies.com&gt;:
>  
>  On Sat, May 14, 2022 at 1:52 PM Peter Boy <pboy(a)uni-bremen.de&gt; wrote:
> > ...
> > I don’t know if there is a potential problem. Some time ago, if I remember correctly
> it was Neil who wrote in a discussion about storage that a software RAID is no longer
> possible, because - so my memory - the biosboot partition is not replicable over multiple
> disks. So if the previous boot disk fails, you can't just boot from another disk.
>  
>  Last time I checked this particular configuration of /boot on multiple
>  drives in e.g. a raid1,10,5,6 - on BIOS firmware only, Anaconda issues
>  the grub2-install command pointing at all the member drives making up
>  /boot. Whether MBR or GPT, the proper GRUB core.img is installed on
>  each. So any of them will get you to at least a grub rescue prompt. 
Did you test that? If I remember correctly, without the biosboot partition in place, a
bios system doesn’t boot at all from a GPT disk. So you get a black screen.

[Daniel Schier]

That last mail I sent was looking somewhat weird in my mail thread.
Maybe related to the iOS client or so. If it was displayed the same way
for you I will send the same response via Geary:

---

Hi all,

I am mostly lurking the list, since I don’t have much time to spend. 
Anyway, the whole situation is quite interesting for me (personally and 
professionally).

If one can describe different scenarios that may occur (BIOSboot + GPT, 
UEFIboot + GPT, with/without LVM, RAID configurations, etc.) and some 
test cases, I can have a look during my worktime and offer the results.

I can also try to make the setups reproducible for automatic testing 
(Kickstart, Ansible, whatnot), if this helps.

In case none of the above sounds helpful, please feel free to ignore 
the message :)

Best regards,
Daniel
---
### Contact

dschier(a)while-true-do.io
dschier @libera.chat
linkedin.com/u/dschier
twitter.com/dschier_wtd
norden.social/@dschier
github.com/dschier-wtd/

### Projects

while-true-do.io
style-cheat.io

### Disclaimer

If not stated otherwise all of my work is licensed under a Creative 
Commons Attribution 4.0 International License 
(https://creativecommons.org/licenses/by/4.0/).

On Di, Mai 17 2022 at 08:43:34 +0200, Peter Boy <pboy(a)uni-bremen.de&gt; 
wrote:
>  
>  
> >  Am 16.05.2022 um 18:26 schrieb Chris Murphy 
> > <lists(a)colorremedies.com <mailto:[e-mail]>>:
> > 
> >  On Sat, May 14, 2022 at 1:52 PM Peter Boy <pboy(a)uni-bremen.de 
> > <mailto:[e-mail]>> wrote:
> >>  ...
> >>  I don’t know if there is a potential problem. Some time ago, if 
> >> I remember correctly it was Neil who wrote in a discussion about 
> >> storage that a software RAID is no longer possible, because - so my 
> >> memory - the biosboot partition is not replicable over multiple 
> >> disks. So if the previous boot disk fails, you can't just boot from 
> >> another disk.
> > 
> >  Last time I checked this particular configuration of /boot on 
> > multiple
> >  drives in e.g. a raid1,10,5,6 - on BIOS firmware only, Anaconda 
> > issues
> >  the grub2-install command pointing at all the member drives making 
> > up
> >  /boot. Whether MBR or GPT, the proper GRUB core.img is installed on
> >  each. So any of them will get you to at least a grub rescue prompt.
>  
>  Did you test that? If I remember correctly, without the biosboot 
>  partition in place, a bios system doesn’t boot at all from a GPT 
>  disk. So you get a black screen.
>  
>  
>  _______________________________________________
>  server mailing list -- server(a)lists.fedoraproject.org 
>  <mailto:[e-mail]>
>  To unsubscribe send an email to server-leave(a)lists.fedoraproject.org 
>  <mailto:[e-mail]>
>  Fedora Code of Conduct: 
>  <https://docs.fedoraproject.org/en-US/project/code-of-conduct/>
>  List Guidelines: 
>  <https://fedoraproject.org/wiki/Mailing_list_guidelines>
>  List Archives: 
>  <https://lists.fedoraproject.org/archives/list/[e-mail]....
>  Do not reply to spam on the list, report it: 
>  <https://pagure.io/fedora-infrastructure>

[Peter Boy]

Hi Daniel, 

>  Am 17.05.2022 um 17:49 schrieb Daniel Schier
> <dschier(a)while-true-do.io&gt;:
>  
>  That last mail I sent was looking somewhat weird in my mail thread.
>  Maybe related to the iOS client or so. If it was displayed the same way
>  for you I will send the same response via Geary: 
Here, everything looks fine. But, as the saying goes, twice is better.



>  I am mostly lurking the list, since I don´t have much time to spend.
> Anyway, the whole situation is quite interesting for me (personally and professionally).
You are cordially invited to just lurk. Just in case you lurk on our IRC meetings, too, it
would be really helpful to say shortly hello.


>  If one can describe different scenarios that may occur (BIOSboot +
> GPT, UEFIboot + GPT, with/without LVM, RAID configurations, etc.) and some test cases, I
> can have a look during my worktime and offer the results. 
That would be very very helpful. Thanks a lot. I’ll prepare a description of the
installation process and what to expect. And the test would be to follow the description
and check the installation results. 

>  I can also try to make the setups reproducible for automatic testing
> (Kickstart, Ansible, whatnot), if this helps. 
Oh, that would help a lot. That way we could make it part of our regular release testing.


>  In case none of the above sounds helpful, please feel free to ignore
> the message :) 
As I said, it’s very helpful  and I will come back. Probably next week.


Thanks
Peter

[Daniel Schier]

Hi,

Thank you for the response and invitation.
I‘ll try to join the next meeting and say Hello to everybody.

Let’s see what I can support you with.

>  Am 18.05.2022 um 16:51 schrieb Peter Boy <pboy(a)uni-bremen.de&gt;:
>  
>  Hi Daniel, 
>  
> > Am 17.05.2022 um 17:49 schrieb Daniel Schier <dschier(a)while-true-do.io&gt;:
> > 
> > That last mail I sent was looking somewhat weird in my mail thread.
> > Maybe related to the iOS client or so. If it was displayed the same way
> > for you I will send the same response via Geary:
>  
>  Here, everything looks fine. But, as the saying goes, twice is better.
>  
>  
>  
> > I am mostly lurking the list, since I don´t have much time to spend. Anyway, the
> whole situation is quite interesting for me (personally and professionally).
>  
>  You are cordially invited to just lurk. Just in case you lurk on our IRC meetings, too,
> it would be really helpful to say shortly hello.
>  
>  
> > If one can describe different scenarios that may occur (BIOSboot + GPT, UEFIboot +
> GPT, with/without LVM, RAID configurations, etc.) and some test cases, I can have a look
> during my worktime and offer the results.
>  
>  That would be very very helpful. Thanks a lot. I’ll prepare a description of the
> installation process and what to expect. And the test would be to follow the description
> and check the installation results. 
>  
> > I can also try to make the setups reproducible for automatic testing (Kickstart,
> Ansible, whatnot), if this helps.
>  
>  Oh, that would help a lot. That way we could make it part of our regular release
> testing.
>  
>  
> > In case none of the above sounds helpful, please feel free to ignore the message :)
>  
>  As I said, it’s very helpful  and I will come back. Probably next week.
>  
>  
>  Thanks
>  Peter
>  
>  
>  
>  _______________________________________________
>  server mailing list -- server(a)lists.fedoraproject.org
>  To unsubscribe send an email to server-leave(a)lists.fedoraproject.org
>  Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
>  List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
>  List Archives:
> https://lists.fedoraproject.org/archives/list/[e-mail]
>  Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure

[Peter Boy]

As discussed on our lastIRC meeting I created an issue about software raid installation on
biosboot machine and GPT and described 2 test cases:

https://pagure.io/fedora-server/issue/87

@Daniel: would be great if you could perform those test and document the results in a
comment. And it would be even better if you could make these tests run automatically or
semi-automatically so that we can make them a routine part of our release testing. There
will be some changes in Anaconda, so we may need to test it intensively for more than one
release. 

And maybe, you put yourself in CC of bug 2088113
(https://bugzilla.redhat.com/show_bug.cgi?id=2088113)?

[Adam Williamson]

On Sun, 2022-05-29 at 11:02 +0200, Peter Boy wrote:
>  As discussed on our lastIRC meeting I created an issue about software
> raid installation on biosboot machine and GPT and described 2 test cases:
>  
>  https://pagure.io/fedora-server/issue/87
>  
>  @Daniel: would be great if you could perform those test and document
>  the results in a comment. And it would be even better if you could
>  make these tests run automatically or semi-automatically so that we
>  can make them a routine part of our release testing. There will be
>  some changes in Anaconda, so we may need to test it intensively for
>  more than one release.  
openQA already does automated installation testing. That would be the
natural place to add this. If someone would file a ticket at
https://pagure.io/fedora-qa/os-autoinst-distri-fedora/issues , we can
do it. Thanks.
-- 
Adam Williamson
Fedora QA
IRC: adamw | Twitter: adamw_ha
https://www.happyassassin.net

[Peter Boy]

>  Am 03.06.2022 um 18:44 schrieb Adam Williamson
> <adamwill(a)fedoraproject.org&gt;:
>  
>  On Sun, 2022-05-29 at 11:02 +0200, Peter Boy wrote:
> > As discussed on our lastIRC meeting I created an issue about software raid
> installation on biosboot machine and GPT and described 2 test cases:
> > 
> > https://pagure.io/fedora-server/issue/87
> > 
> > @Daniel: would be great if you could perform those test and document
> > the results in a comment. And it would be even better if you could
> > make these tests run automatically or semi-automatically so that we
> > can make them a routine part of our release testing. There will be
> > some changes in Anaconda, so we may need to test it intensively for
> > more than one release. 
>  
>  openQA already does automated installation testing. That would be the
>  natural place to add this. If someone would file a ticket at
>  https://pagure.io/fedora-qa/os-autoinst-distri-fedora/issues , we can
>  do it. Thanks. 

Thanks, I just opened an issue (#262)


Peter

[Chris Murphy]

On Tue, May 17, 2022, 2:43 AM Peter Boy <pboy(a)uni-bremen.de&gt; wrote:

>  > Am 16.05.2022 um 18:26 schrieb Chris Murphy <lists(a)colorremedies.com&gt;:
>  >
>  > On Sat, May 14, 2022 at 1:52 PM Peter Boy <pboy(a)uni-bremen.de&gt; wrote:
>  >> ...
>  >> I don’t know if there is a potential problem. Some time ago, if I
>  remember correctly it was Neil who wrote in a discussion about storage that
>  a software RAID is no longer possible, because - so my memory - the
>  biosboot partition is not replicable over multiple disks. So if the
>  previous boot disk fails, you can't just boot from another disk.
>  >
>  > Last time I checked this particular configuration of /boot on multiple
>  > drives in e.g. a raid1,10,5,6 - on BIOS firmware only, Anaconda issues
>  > the grub2-install command pointing at all the member drives making up
>  > /boot. Whether MBR or GPT, the proper GRUB core.img is installed on
>  > each. So any of them will get you to at least a grub rescue prompt.
>
>  Did you test that? 

That's what i just described.

If I remember correctly, without the biosboot partition in place, a bios
>  system doesn’t boot at all from a GPT disk. So you get a black
> screen.
>  
Anaconda automatic partitioning always creates the required BIOS Boot
partition. And in Custom, it complains if you don't. So yes, i created them.

The reality on UEFI is degraded boot isn't possible unless you're willing
to setup ill advised janky nonsense. Like using mdadm raid1 n-copies for
n-drives for the ESP.

--
Chris Murphy

[Neal Gompa]

On Wed, May 18, 2022 at 7:24 AM Chris Murphy <lists(a)colorremedies.com&gt; wrote:
>  On Tue, May 17, 2022, 2:43 AM Peter Boy <pboy(a)uni-bremen.de&gt; wrote:
> >
> >
> >
> > > Am 16.05.2022 um 18:26 schrieb Chris Murphy <lists(a)colorremedies.com&gt;:
> > >
> > > On Sat, May 14, 2022 at 1:52 PM Peter Boy <pboy(a)uni-bremen.de&gt; wrote:
> > >> ...
> > >> I don’t know if there is a potential problem. Some time ago, if I remember
> correctly it was Neil who wrote in a discussion about storage that a software RAID is no
> longer possible, because - so my memory - the biosboot partition is not replicable over
> multiple disks. So if the previous boot disk fails, you can't just boot from another
> disk.
> > >
> > > Last time I checked this particular configuration of /boot on multiple
> > > drives in e.g. a raid1,10,5,6 - on BIOS firmware only, Anaconda issues
> > > the grub2-install command pointing at all the member drives making up
> > > /boot. Whether MBR or GPT, the proper GRUB core.img is installed on
> > > each. So any of them will get you to at least a grub rescue prompt.
> >
> > Did you test that?
>
>
>  That's what i just described.
>
> > If I remember correctly, without the biosboot partition in place, a bios system
> doesn’t boot at all from a GPT disk. So you get a black screen.
>
>
>  Anaconda automatic partitioning always creates the required BIOS Boot partition. And in
> Custom, it complains if you don't. So yes, i created them.
>
>  The reality on UEFI is degraded boot isn't possible unless you're willing to
> setup ill advised janky nonsense. Like using mdadm raid1 n-copies for n-drives for the
> ESP.
>  
To be fair, the same is true for legacy BIOS boot. You need the MBR
code installed on all bootable disks, you need the partition with the
boot code at the beginning of the disk on all disks, and they have to
be marked as bootable.



-- 
真実はいつも一つ！/ Always, there's only one truth!

[Peter Boy]

>  Am 18.05.2022 um 13:33 schrieb Neal Gompa
> <ngompa13(a)gmail.com&gt;:
>  
>  
>  To be fair, the same is true for legacy BIOS boot. You need the MBR
>  code installed on all bootable disks, you need the partition with the
>  boot code at the beginning of the disk on all disks, and they have to
>  be marked as bootable. 

With F34 it is different: If you define a partition as raid member and then install and
format /boot on it, Anaconda automatically installs the mbr on all disks - according to my
testing when I wrote the Server install documentation. So, as a sysadmin installing Server
you don’t need to fiddle with all of this. Just use custom and you are up and running.

But again, before we can make GPT the default, we must be sure it works smoothly with
software raid.

[Chris Murphy]

On Wed, May 18, 2022 at 7:34 AM Neal Gompa <ngompa13(a)gmail.com&gt; wrote:
>  On Wed, May 18, 2022 at 7:24 AM Chris Murphy <lists(a)colorremedies.com&gt; wrote:
>  >
>  >
>  >
>  > On Tue, May 17, 2022, 2:43 AM Peter Boy <pboy(a)uni-bremen.de&gt; wrote:
>  >>
>  >>
>  >>
>  >> > Am 16.05.2022 um 18:26 schrieb Chris Murphy
> <lists(a)colorremedies.com&gt;:
>  >> >
>  >> > On Sat, May 14, 2022 at 1:52 PM Peter Boy <pboy(a)uni-bremen.de&gt;
> wrote:
>  >> >> ...
>  >> >> I don’t know if there is a potential problem. Some time ago, if I
> remember correctly it was Neil who wrote in a discussion about storage that a software
> RAID is no longer possible, because - so my memory - the biosboot partition is not
> replicable over multiple disks. So if the previous boot disk fails, you can't just
> boot from another disk.
>  >> >
>  >> > Last time I checked this particular configuration of /boot on multiple
>  >> > drives in e.g. a raid1,10,5,6 - on BIOS firmware only, Anaconda issues
>  >> > the grub2-install command pointing at all the member drives making up
>  >> > /boot. Whether MBR or GPT, the proper GRUB core.img is installed on
>  >> > each. So any of them will get you to at least a grub rescue prompt.
>  >>
>  >> Did you test that?
>  >
>  >
>  > That's what i just described.
>  >
>  >> If I remember correctly, without the biosboot partition in place, a bios system
> doesn’t boot at all from a GPT disk. So you get a black screen.
>  >
>  >
>  > Anaconda automatic partitioning always creates the required BIOS Boot partition. And
> in Custom, it complains if you don't. So yes, i created them.
>  >
>  > The reality on UEFI is degraded boot isn't possible unless you're willing to
> setup ill advised janky nonsense. Like using mdadm raid1 n-copies for n-drives for the
> ESP.
>  >
>
>  To be fair, the same is true for legacy BIOS boot. You need the MBR
>  code installed on all bootable disks, you need the partition with the
>  boot code at the beginning of the disk on all disks, and they have to
>  be marked as bootable. 
OK the installer isn't doing what I'm expecting it to do based on
years ago testing. So the behavior has changed.

Currently, when I select 2 drives for installation:

Automatic partitioning results in a grub2-install command to only one
the drives. Whether MBR or GPT. And if GPT only one drive gets BIOS
Boot.

Custom partitioning has at least one bug. It shows BIOS Boot partition
with two devices selected in the manual partitioning UI, suggesting
that each drive will get a BIOS Boot partition. However, the
bootloader dialog accessed at the bottom of the Installation
Destination page no longer allows both drives to be chosen. And the
summary shows only one BIOS Boot will be created. And the final
installation likewise only has one BIOS Boot, but there are separate
grub2-install commands to each of the drives selected for installation
destination. The second command fails because the drive doesn't have a
BIOS Boot partition.

https://bugzilla.redhat.com/show_bug.cgi?id=2088113

In any case, if I create BIOSBoot on the drives, and if I make /boot
raid1 on all the drives, it issues grub2-install on each drive,
populating each BIOS Boot correctly, and it can find grub.cfg and
normal.mod on even a degraded raid1 /boot. There really isn't an
equivalent on UEFI right now. Some installers might still use mdadm
metadata format 1.0 to put the mdadm super at the end of the
partition, and create a raid1 for the ESPs. This looks like individual
FAT 32 volumes from the firmware's perspective. But a raid1 from
Linux's perspective. The problem is that any writes by the firmware to
any of the ESPs results in corruption of the array and it's ambiguous
to md which drive is correct - so it's all just trash unless you're
experienced enough to know how to manually deal with it.

So the long term solution for this use case needs to be something like
bootupd, which has exclusive ownership of the $efi and $boot
partitions, or even combine them.





>  --
>  真実はいつも一つ！/ Always, there's only one truth!
>  _______________________________________________
>  server mailing list -- server(a)lists.fedoraproject.org
>  To unsubscribe send an email to server-leave(a)lists.fedoraproject.org
>  Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
>  List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
>  List Archives:
> https://lists.fedoraproject.org/archives/list/[e-mail]
>  Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure


--
Chris Murphy

[Peter Boy]

>  Am 18.05.2022 um 13:23 schrieb Chris Murphy
> <lists(a)colorremedies.com&gt;:
>  
>  If I remember correctly, without the biosboot partition in place, a bios system doesn’t
> boot at all from a GPT disk. So you get a black screen.
>  
>  Anaconda automatic partitioning always creates the required BIOS Boot partition. And in
> Custom, it complains if you don't. So yes, i created them.
>   
Maybe, we should test is with F36 again. To my experiences Anaconda does create the
biosboot partition with custom boot automatically and  always on the boot disk only and
definitely not on any other disk. 

At least with F34 software raid doesn’t work smoothly with GPT. And before we make GPT
default we must ensure, it works with software raid.