F39 proposal: BiggerESP (Self-Contained Change proposal)

[Ben Cotton]

https://fedoraproject.org/wiki/Changes/BiggerESP

This document represents a proposed Change. As part of the Changes
process, proposals are publicly announced in order to receive
community feedback. This proposal will only be implemented if approved
by the Fedora Engineering Steering Committee.

== Summary ==

The Fedora installer includes an EFI System Partition of between 200MB
and 600MB by default, of which the lower size is much too small for
firmware updates on modern hardware and also for future bootloader
features like UKI.
This change will increase the minimum size of the ESP to be 500MB,
which is also the same value used by Microsoft for Windows 10 and
newer.

== Owner ==
* Name: [[User:rhughes| Richard Hughes]]
* Email: richard(a)hughsie.com


== Detailed Description ==

Modern hardware has UEFI firmware updates that are more than 64MB in
size. The OEMs recommend a ESP free space of double the flash size
plus 20MB and fwupd now enforces this requirement to ensure flash
success. As the ESP is often shared between Windows and Linux, and
also used for firmware updates, and soon to be used by UKIs it's not
enough to just allocate a few hundreds of megabytes. Windows 10 and 11
allocates an ESP of at least 500MiB. Arch also specifies a minimum of
512 MiB.

== Feedback ==

There is no alternative -- the ESP has to scale up if we want firmware
updates to continue to work and to support UKIs for next-generation
bootloaders.

== Benefit to Fedora ==

Firmware updates will work on future hardware, and we can boot the
kernel using UKIs using next-generation bootloaders.

== Scope ==
* Proposal owners:

We need to change a number in Anaconda:
https://github.com/rhinstaller/anaconda/pull/4711

== Upgrade/compatibility impact ==

We can't grow the ESP in size, and so this change will only affect new
installs. This is fine, as this will affect new hardware more than old
hardware.

== How To Test ==

Install Fedora and observe that /boot/efi has at least 276MB free
space, even when installed alongside Windows.

== Dependencies ==

Anaconda would need to be modified, and Fedora would have a / or /home
partition that's ~300MB smaller by default than it is now.

== Contingency Plan ==

* Contingency mechanism: (What to do?  Who will do it?) N/A (not a
System Wide Change)
* Contingency deadline: N/A (not a System Wide Change)
* Blocks release? N/A (not a System Wide Change), No

== Documentation ==

N/A (not a System Wide Change)

== Release Notes ==

Fedora now defaults to a larger EFI System Partition which allows
firmware updates to work on newer hardware, and allows future
bootloader and kernel modernizations.


-- 
Ben Cotton
He / Him / His
Fedora Program Manager
Red Hat
TZ=America/Indiana/Indianapolis

[Daniel P. Berrangé]

On Mon, Apr 24, 2023 at 12:15:12PM -0400, Ben Cotton wrote:
>  https://fedoraproject.org/wiki/Changes/BiggerESP
>  
>  This document represents a proposed Change. As part of the Changes
>  process, proposals are publicly announced in order to receive
>  community feedback. This proposal will only be implemented if approved
>  by the Fedora Engineering Steering Committee.
>  
>  == Summary ==
>  
>  The Fedora installer includes an EFI System Partition of between 200MB
>  and 600MB by default, of which the lower size is much too small for
>  firmware updates on modern hardware and also for future bootloader
>  features like UKI.
>  This change will increase the minimum size of the ESP to be 500MB,
>  which is also the same value used by Microsoft for Windows 10 and
>  newer. 
This refers to the minimum size being changed, but later it mentions
the default size being changed. Are the default & minimum sizes
effectively the same in this case ?

nitpick - the github change linked is 512 MiB rather than 500 MB.

>  == Owner ==
>  * Name: [[User:rhughes| Richard Hughes]]
>  * Email: richard(a)hughsie.com
>  
>  
>  == Detailed Description ==
>  
>  Modern hardware has UEFI firmware updates that are more than 64MB in
>  size. The OEMs recommend a ESP free space of double the flash size
>  plus 20MB and fwupd now enforces this requirement to ensure flash
>  success. As the ESP is often shared between Windows and Linux, and
>  also used for firmware updates, and soon to be used by UKIs it's not
>  enough to just allocate a few hundreds of megabytes. Windows 10 and 11
>  allocates an ESP of at least 500MiB. Arch also specifies a minimum of
>  512 MiB. 
My only thought is whether 512 MiB is sufficiently future proofed if we
start to make more use of UKIs, given that /boot by comparison is already
at 1 GiB by default IIUC ?

>  == Feedback ==
>  
>  There is no alternative -- the ESP has to scale up if we want firmware
>  updates to continue to work and to support UKIs for next-generation
>  bootloaders.
>  
>  == Benefit to Fedora ==
>  
>  Firmware updates will work on future hardware, and we can boot the
>  kernel using UKIs using next-generation bootloaders.
>  
>  == Scope ==
>  * Proposal owners:
>  
>  We need to change a number in Anaconda:
>  https://github.com/rhinstaller/anaconda/pull/4711
>  
>  == Upgrade/compatibility impact ==
>  
>  We can't grow the ESP in size, and so this change will only affect new
>  installs. This is fine, as this will affect new hardware more than old
>  hardware.
>  
>  == How To Test ==
>  
>  Install Fedora and observe that /boot/efi has at least 276MB free
>  space, even when installed alongside Windows.
>  
>  == Dependencies ==
>  
>  Anaconda would need to be modified, and Fedora would have a / or /home
>  partition that's ~300MB smaller by default than it is now. 
For any install which does end up using UKIs on the ESP, the /boot would
no longer need to be as large as it is today as it would not have kernel
images. In fact /boot could potentially not need to exist at all in any
EFI installs using UKIs.

IOW, the increased size for the ESP could potentially be won back by
permitting /boot to be smaller, or eliminating /boot. I'm not suggesting
this needs to be a pre-requisite of this change proposal though, just a
thought for the future. Could be something that is optimized in any cloud
image kickstarts that end up using UKIs.


With regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

[Richard Hughes]

On Mon, 24 Apr 2023 at 18:28, Daniel P. Berrangé <berrange(a)redhat.com&gt; wrote:
>  This refers to the minimum size being changed, but later it mentions
>  the default size being changed. Are the default & minimum sizes
>  effectively the same in this case ? 
I believe so.

>  nitpick - the github change linked is 512 MiB rather than 500 MB.
I don't actually know whether Microsoft told the OEMs MiB or MB. Will find out.

>  My only thought is whether 512 MiB is sufficiently future proofed if
> we
>  start to make more use of UKIs, given that /boot by comparison is already
>  at 1 GiB by default IIUC ? 
It's a tradeoff; on some devices 1GB is going to be a significant
chunk of the eMMC space used. I'll certainly bump up the maximum size
tho -- I'll continue that discussion on the anaconda PR.

>  For any install which does end up using UKIs on the ESP, the /boot
> would
>  no longer need to be as large as it is today as it would not have kernel
>  images. In fact /boot could potentially not need to exist at all in any
>  EFI installs using UKIs. 
Right, I don't disagree. I just think the switch to ESPs and
potentially nuking /boot is a different ChangeProposal :) As part of
that we could delete /boot and enlarge /boot/efi but that's not the
problem we're solving here. Let's do the little uncontroversial change
first so firmware updates keep working, then we can work on the bigger
changes that might be more controversial.

Richard.

[Peter Robinson]

On Mon, Apr 24, 2023 at 5:15 PM Ben Cotton <bcotton(a)redhat.com&gt; wrote:
>  https://fedoraproject.org/wiki/Changes/BiggerESP
>
>  This document represents a proposed Change. As part of the Changes
>  process, proposals are publicly announced in order to receive
>  community feedback. This proposal will only be implemented if approved
>  by the Fedora Engineering Steering Committee.
>
>  == Summary ==
>
>  The Fedora installer includes an EFI System Partition of between 200MB
>  and 600MB by default, of which the lower size is much too small for
>  firmware updates on modern hardware and also for future bootloader
>  features like UKI.
>  This change will increase the minimum size of the ESP to be 500MB,
>  which is also the same value used by Microsoft for Windows 10 and
>  newer. 
While I believe this to be low impact I do believe it should be a
system wide change as it impacts all aarch64 and basically all the
x86_64 we actively care about.

> == Owner ==
> * Name: [[User:rhughes| Richard Hughes]]
> * Email: richard(a)hughsie.com
>
>
> == Detailed Description ==
>
> Modern hardware has UEFI firmware updates that are more than 64MB in
> size. The OEMs recommend a ESP free space of double the flash size
> plus 20MB and fwupd now enforces this requirement to ensure flash
> success. As the ESP is often shared between Windows and Linux, and
> also used for firmware updates, and soon to be used by UKIs it's not
> enough to just allocate a few hundreds of megabytes. Windows 10 and 11
> allocates an ESP of at least 500MiB. Arch also specifies a minimum of
> 512 MiB.
>
> == Feedback ==
>
> There is no alternative -- the ESP has to scale up if we want firmware
> updates to continue to work and to support UKIs for next-generation
> bootloaders.
>
> == Benefit to Fedora ==
>
> Firmware updates will work on future hardware, and we can boot the
> kernel using UKIs using next-generation bootloaders.
>
> == Scope ==
> * Proposal owners:
>
> We need to change a number in Anaconda:
> https://github.com/rhinstaller/anaconda/pull/4711
>
> == Upgrade/compatibility impact ==
>
> We can't grow the ESP in size, and so this change will only affect new
> installs. This is fine, as this will affect new hardware more than old
> hardware.
>
> == How To Test ==
>
> Install Fedora and observe that /boot/efi has at least 276MB free
> space, even when installed alongside Windows.
>
> == Dependencies ==
>
> Anaconda would need to be modified, and Fedora would have a / or /home
> partition that's ~300MB smaller by default than it is now.
>
> == Contingency Plan ==
>
> * Contingency mechanism: (What to do?  Who will do it?) N/A (not a
> System Wide Change)
> * Contingency deadline: N/A (not a System Wide Change)
> * Blocks release? N/A (not a System Wide Change), No
>
> == Documentation ==
>
> N/A (not a System Wide Change)
>
> == Release Notes ==
>
> Fedora now defaults to a larger EFI System Partition which allows
> firmware updates to work on newer hardware, and allows future
> bootloader and kernel modernizations.
>
>
> --
> Ben Cotton
> He / Him / His
> Fedora Program Manager
> Red Hat
> TZ=America/Indiana/Indianapolis
> _______________________________________________
> devel mailing list -- devel(a)lists.fedoraproject.org
> To unsubscribe send an email to devel-leave(a)lists.fedoraproject.org
> Fedora Code of Conduct:
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives:
https://lists.fedoraproject.org/archives/list/[e-mail]
> Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue

[Richard Hughes]

On Tue, 25 Apr 2023 at 07:05, Peter Robinson <pbrobinson(a)gmail.com&gt; wrote:
>  While I believe this to be low impact I do believe it should be a
>  system wide change as it impacts all aarch64 and basically all the
>  x86_64 we actively care about. 
I guess you could argue it both ways -- I figured it was self
contained as it's really only a few constants changed in just one
package -- it's not like we need to rebuild the distro like you would
for a glibc update. If those that vote say it's a system wide change
then I'll happily "upgrade" the wiki page, but I don't want to cause
paperwork for no reason.

Richard.

[Zbigniew Jędrzejewski-Szmek]

Hi!

Sorry for the late reply, esp. that I need to grumble a bit. I missed
the thread and only came here via the fesco ticket.

On Mon, Apr 24, 2023 at 12:15:12PM -0400, Ben Cotton wrote:
>  https://fedoraproject.org/wiki/Changes/BiggerESP 
>  == Feedback ==
>  
>  There is no alternative -- the ESP has to scale up if we want firmware
>  updates to continue to work and to support UKIs for next-generation
>  bootloaders. 
Nitpick: this isn't actually true. UKIs can (and should) be loaded just
fine from the "other boot partition", i.e. /boot a.k.a. XBOOTLDR [1].

[1]
https://uapi-group.org/specifications/specs/boot_loader_specification/#ty...

>  == Summary ==
>  
>  This change will increase the minimum size of the ESP to be 500MB,
>  which is also the same value used by Microsoft for Windows 10 and
>  newer. 
This is both too much and not enough. Essentially, this grows the ESP,
but also leaves the XBOOTLDR partition large. Overall, the users pays
twice, and on some systems 1.5GB is not insignificant. OTOH, 500 or
512 MB seems not enough: three big UKIs and a rescue kernel and and
some Windows blobs and a firmware update would likely overflow.

If we want to change the default here, let's do some proper cleanup:
1. the split between ESP and XBOOTLDR is only useful in the case where
   ESP already existed and was small. If the installer is *creating*
   an ESP, it should just make it large enough.
2. having a second partition with a second (different) file system
   implementation just increases the footprint and attack surface for
   no gain. If we create XBOOTLDR, make it like the ESP (i.e. VFAT
   in almost all realistic scenarios).
3. if there are bootloaders that don't read one or the other partition
   as they should, fix them.

Then we can make the ESP 1 GB *and* save space compared to current
defaults.

(Point 2. is not really *necessary* for the size changes, but it'd be
nice to get rid of this anachronism if this area is being touched.)

Zbyszek

[Gerd Hoffmann]

  Hi,

>  If we want to change the default here, let's do some proper
> cleanup:
>  1. the split between ESP and XBOOTLDR is only useful in the case where
>     ESP already existed and was small. If the installer is *creating*
>     an ESP, it should just make it large enough. 
And install kernels to /boot/efi in case /boot is not a XBOOTLDR
filesystem?

>  2. having a second partition with a second (different) file system
>     implementation just increases the footprint and attack surface for
>     no gain. If we create XBOOTLDR, make it like the ESP (i.e. VFAT
>     in almost all realistic scenarios). 
While being at it also give the XBOOTLDR the correct type uuid according
to the discoverable partitions spec.

take care,
  Gerd

[Zbigniew Jędrzejewski-Szmek]

On Tue, May 09, 2023 at 12:02:26PM +0200, Gerd Hoffmann wrote:
>    Hi,
>  
>  > If we want to change the default here, let's do some proper cleanup:
>  > 1. the split between ESP and XBOOTLDR is only useful in the case where
>  >    ESP already existed and was small. If the installer is *creating*
>  >    an ESP, it should just make it large enough.
>  
>  And install kernels to /boot/efi in case /boot is not a XBOOTLDR
>  filesystem? 
If /boot is not a XBOOTLDR, then we only have one file system which is
the ESP. It could be mounted on /boot or on /efi or maybe even /boot/efi (*).
The kernels would then go to /boot/EFI/Linux, /efi/EFI/Linux, or /boot/efi/EFI/Linux,
respectively. (When you write /boot/efi, it's not clear what exactly you
mean. The duplication of "efi" and "EFI" on on case-insensitive
system
is confusing.)

(*) This is actually something that'd need to be figure out.
/boot/efi is the worst choice; either /boot or /efi would be OK,
but something needs to be chosen.

>  > 2. having a second partition with a second (different) file
> system
>  >    implementation just increases the footprint and attack surface for
>  >    no gain. If we create XBOOTLDR, make it like the ESP (i.e. VFAT
>  >    in almost all realistic scenarios).
>  
>  While being at it also give the XBOOTLDR the correct type uuid according
>  to the discoverable partitions spec. 
Of course ;-]

Zbyszek

[Neal Gompa]

On Tue, May 9, 2023 at 8:09 AM Zbigniew Jędrzejewski-Szmek
<zbyszek(a)in.waw.pl&gt; wrote:
>  On Tue, May 09, 2023 at 12:02:26PM +0200, Gerd Hoffmann wrote:
>  >   Hi,
>  >
>  > > If we want to change the default here, let's do some proper cleanup:
>  > > 1. the split between ESP and XBOOTLDR is only useful in the case where
>  > >    ESP already existed and was small. If the installer is *creating*
>  > >    an ESP, it should just make it large enough.
>  >
>  > And install kernels to /boot/efi in case /boot is not a XBOOTLDR
>  > filesystem?
>
>  If /boot is not a XBOOTLDR, then we only have one file system which is
>  the ESP. It could be mounted on /boot or on /efi or maybe even /boot/efi (*).
>  The kernels would then go to /boot/EFI/Linux, /efi/EFI/Linux, or /boot/efi/EFI/Linux,
>  respectively. (When you write /boot/efi, it's not clear what exactly you
>  mean. The duplication of "efi" and "EFI" on on case-insensitive
> system
>  is confusing.)
>
>  (*) This is actually something that'd need to be figure out.
>  /boot/efi is the worst choice; either /boot or /efi would be OK,
>  but something needs to be chosen.
>
>  > > 2. having a second partition with a second (different) file system
>  > >    implementation just increases the footprint and attack surface for
>  > >    no gain. If we create XBOOTLDR, make it like the ESP (i.e. VFAT
>  > >    in almost all realistic scenarios).
>  >
>  > While being at it also give the XBOOTLDR the correct type uuid according
>  > to the discoverable partitions spec.
>
>  Of course ;-]
>  
I've been asked to consider converting /boot to a Btrfs subvolume so
that it no longer has a fixed space allocation to deal with the ever
increasing amount of firmware required for NVIDIA GPUs[1]. This is
currently incompatible with how systemd views the world, because the
"discoverable partition spec" is wired to partitions, and there is no
equivalent spec for subvolumes of a volume. And I imagine that
XBOOTLDR (whatever that is) also would have a problem with this.

Also, as an aside, there is now a "from-scratch" Btrfs EFI driver in
development[2] (and for your personal horror, an NTFS one too[3]).

[1]: https://pagure.io/fedora-btrfs/project/issue/7#comment-855321
[2]: https://github.com/maharmstone/btrfs-efi
[3]: https://github.com/maharmstone/ntfs-efi



-- 
真実はいつも一つ！/ Always, there's only one truth!

[Lennart Poettering]

On Di, 09.05.23 08:22, Neal Gompa (ngompa13(a)gmail.com) wrote:

>  I've been asked to consider converting /boot to a Btrfs subvolume
> so
>  that it no longer has a fixed space allocation to deal with the ever
>  increasing amount of firmware required for NVIDIA GPUs[1]. This is
>  currently incompatible with how systemd views the world, because the
>  "discoverable partition spec" is wired to partitions, and there is no
>  equivalent spec for subvolumes of a volume. And I imagine that
>  XBOOTLDR (whatever that is) also would have a problem with this. 
This makese no sense. If you want /boot to just be a subvolume of the
rootfs btrfs, then this would imply it's also covered by the same
security choices, i.e. encryption. We want to bind that sooner or
later to things like TPM2, FIDO2, PKCS11. And that's simply not
feasible from a boot loader environment.

Hence, the place the kernel is loaded from (regardless if you call it
/efi or /boot or /boot/efi, and regardless what fs it is) must be
accessible from the boot loader easily, without requiring
implementation of TPM2/FIDO2/PKCS11 hookup in the boot loader.

Hence: btrfs subvols won't work for this

A simple vfat partition however will.

>  Also, as an aside, there is now a "from-scratch" Btrfs EFI
> driver in
>  development[2] (and for your personal horror, an NTFS one too[3]). 
Not sufficient. You'd also have to implement a LUKS EFI driver, and a
TPM2 EFI driver, and a FIDO2 EFI driver, and so on and so
on. Basically, you have to reimplement a good chunk of the Linux
kernel, of Linux userspace, systemd and so on in EFI mode.

Good luck with that.

Lennart

--
Lennart Poettering, Berlin

[Neal Gompa]

On Tue, May 9, 2023 at 12:31 PM Lennart Poettering <mzerqung(a)0pointer.de&gt; wrote:
>  On Di, 09.05.23 08:22, Neal Gompa (ngompa13(a)gmail.com) wrote:
>
>  > I've been asked to consider converting /boot to a Btrfs subvolume so
>  > that it no longer has a fixed space allocation to deal with the ever
>  > increasing amount of firmware required for NVIDIA GPUs[1]. This is
>  > currently incompatible with how systemd views the world, because the
>  > "discoverable partition spec" is wired to partitions, and there is no
>  > equivalent spec for subvolumes of a volume. And I imagine that
>  > XBOOTLDR (whatever that is) also would have a problem with this.
>
>  This makese no sense. If you want /boot to just be a subvolume of the
>  rootfs btrfs, then this would imply it's also covered by the same
>  security choices, i.e. encryption. We want to bind that sooner or
>  later to things like TPM2, FIDO2, PKCS11. And that's simply not
>  feasible from a boot loader environment.
>
>  Hence, the place the kernel is loaded from (regardless if you call it
>  /efi or /boot or /boot/efi, and regardless what fs it is) must be
>  accessible from the boot loader easily, without requiring
>  implementation of TPM2/FIDO2/PKCS11 hookup in the boot loader.
>
>  Hence: btrfs subvols won't work for this 
If we're not using LUKS for encryption, then this is not a problem.
We're generally looking toward encrypting subvolumes individually
using the upcoming Btrfs native encryption capability rather than
using LUKS. That allows us to

1. Pick which subvolumes are encrypted
2. Pick the security binding method per subvolume

For example, the root and home subvolumes would use TPM or some other
non-interactive binding. The user subvolume in home would decrypt with
user login.

-- 
真実はいつも一つ！/ Always, there's only one truth!

[Owen Taylor]

On Tue, May 9, 2023 at 12:39 PM Neal Gompa <ngompa13(a)gmail.com&gt; wrote:

>  On Tue, May 9, 2023 at 12:31 PM Lennart Poettering
> <mzerqung(a)0pointer.de&gt;
>  wrote:
>  >
>  > On Di, 09.05.23 08:22, Neal Gompa (ngompa13(a)gmail.com) wrote:
>  >
>  > > I've been asked to consider converting /boot to a Btrfs subvolume so
>  > > that it no longer has a fixed space allocation to deal with the ever
>  > > increasing amount of firmware required for NVIDIA GPUs[1]. This is
>  > > currently incompatible with how systemd views the world, because the
>  > > "discoverable partition spec" is wired to partitions, and there is
> no
>  > > equivalent spec for subvolumes of a volume. And I imagine that
>  > > XBOOTLDR (whatever that is) also would have a problem with this.
>  >
>  > This makese no sense. If you want /boot to just be a subvolume of the
>  > rootfs btrfs, then this would imply it's also covered by the same
>  > security choices, i.e. encryption. We want to bind that sooner or
>  > later to things like TPM2, FIDO2, PKCS11. And that's simply not
>  > feasible from a boot loader environment.
>  >
>  > Hence, the place the kernel is loaded from (regardless if you call it
>  > /efi or /boot or /boot/efi, and regardless what fs it is) must be
>  > accessible from the boot loader easily, without requiring
>  > implementation of TPM2/FIDO2/PKCS11 hookup in the boot loader.
>  >
>  > Hence: btrfs subvols won't work for this
>
>  If we're not using LUKS for encryption, then this is not a problem.
>  We're generally looking toward encrypting subvolumes individually
>  using the upcoming Btrfs native encryption capability rather than
>  using LUKS. That allows us to
>
>  1. Pick which subvolumes are encrypted
>  2. Pick the security binding method per subvolume
>
>  For example, the root and home subvolumes would use TPM or some other
>  non-interactive binding. The user subvolume in home would decrypt with
>  user login.
>
>  While this is true, and it would be nice if we could just make "size of
/boot" go away, if we can separate out "future of encryption" from
"future
of /boot", we're going to make our lives easier.

And even if the preferred path for encryption for Workstation ends up being
btrfs+fscrypt, that won't be the *only* path for Fedora and derivatives;
another reason to try and sort this out independently.

For the giant firmware problem, we have several ways to attack it:
 - Moderately increase boot/efi partition size as discussed here
 - Share firmware between multiple UKI's using system extensions (don't
quite see how this works, but knowledgeable people think it should)
 - Use efifb at boot time (eliminates need for giant firmware, some
possible regression of complicated screen scenarios)
 - Stop prompting for passphrases from the initrd (future of encryption,
makes those regressions more palatable)

Avoiding giant UKI's will likely also be a win from a performance point of
view.

- Owen

[Lennart Poettering]

On Di, 09.05.23 12:37, Neal Gompa (ngompa13(a)gmail.com) wrote:

>  On Tue, May 9, 2023 at 12:31 PM Lennart Poettering
> <mzerqung(a)0pointer.de&gt; wrote:
>  >
>  > On Di, 09.05.23 08:22, Neal Gompa (ngompa13(a)gmail.com) wrote:
>  >
>  > > I've been asked to consider converting /boot to a Btrfs subvolume so
>  > > that it no longer has a fixed space allocation to deal with the ever
>  > > increasing amount of firmware required for NVIDIA GPUs[1]. This is
>  > > currently incompatible with how systemd views the world, because the
>  > > "discoverable partition spec" is wired to partitions, and there is
> no
>  > > equivalent spec for subvolumes of a volume. And I imagine that
>  > > XBOOTLDR (whatever that is) also would have a problem with this.
>  >
>  > This makese no sense. If you want /boot to just be a subvolume of the
>  > rootfs btrfs, then this would imply it's also covered by the same
>  > security choices, i.e. encryption. We want to bind that sooner or
>  > later to things like TPM2, FIDO2, PKCS11. And that's simply not
>  > feasible from a boot loader environment.
>  >
>  > Hence, the place the kernel is loaded from (regardless if you call it
>  > /efi or /boot or /boot/efi, and regardless what fs it is) must be
>  > accessible from the boot loader easily, without requiring
>  > implementation of TPM2/FIDO2/PKCS11 hookup in the boot loader.
>  >
>  > Hence: btrfs subvols won't work for this
>
>  If we're not using LUKS for encryption, then this is not a problem. 
So you basically want to rebuild Fedora so that FDE is not available?

>  We're generally looking toward encrypting subvolumes
> individually
>  using the upcoming Btrfs native encryption capability rather than
>  using LUKS. That allows us to 
How do you establish trust in the underlying file system? The thing
that kernel fs maintainers made very clear is that they do not
consider Linux file systems safe regarding rogue offline
modification. Hence you must establish trust somehow *before* you
mount the fs, which pretty much means LUKS.

Linux fs maintainers also made very clear that they generally consider
alternative implementations of their file systems as unsupported, and
a problem. The big relevant Linux file systems consider only the
implementation in the Linux kernel as defining the format. Which means
that anything like an alternative implementation of btrfs or xfs or
ext4 in things like grub or EFI is expressly against the wishes of the
people who maintain the file systems.

Or in other words: what you are proposing appears like a very bad
idea, and in fact even upstream Grub wants to get away from
maintaining thei own fs drivers for Linux fs as I hear, because it's
so untenable to them, too.

Seriously, bury this idea.

Lennart

--
Lennart Poettering, Berlin

[Lennart Poettering]

On Mi, 10.05.23 15:13, Lennart Poettering (mzerqung(a)0pointer.de) wrote:

>  > We're generally looking toward encrypting subvolumes
> individually
>  > using the upcoming Btrfs native encryption capability rather than
>  > using LUKS. That allows us to
>
>  How do you establish trust in the underlying file system? The thing
>  that kernel fs maintainers made very clear is that they do not
>  consider Linux file systems safe regarding rogue offline
>  modification. Hence you must establish trust somehow *before* you
>  mount the fs, which pretty much means LUKS.
>
>  Linux fs maintainers also made very clear that they generally consider
>  alternative implementations of their file systems as unsupported, and
>  a problem. The big relevant Linux file systems consider only the
>  implementation in the Linux kernel as defining the format. Which means
>  that anything like an alternative implementation of btrfs or xfs or
>  ext4 in things like grub or EFI is expressly against the wishes of the
>  people who maintain the file systems.
>
>  Or in other words: what you are proposing appears like a very bad
>  idea, and in fact even upstream Grub wants to get away from
>  maintaining thei own fs drivers for Linux fs as I hear, because it's
>  so untenable to them, too.
>
>  Seriously, bury this idea. 
So to add to this. I happen to be at LFSMMBPF at the moment, the Linux
File System summit (among other things) where all the Linux FS people
meet. I spoke to a couple of FS maintainers here, and well, let me
make this very clear: using any of the major Linux file systems with
drivers that are not the ones in the Linux kernel is a very bad idea,
and expressly not supported by them. [They actually used much harsher
words, that I'll not repeat here – this is the "friendly" version of
their take on your idea.]

So, unless you want to go against what the people who actually
maintain the file systems expressly say please just get this idea out
of your head that porting Linux file systems into EFI fs drivers was a
good, supportable idea.

And Neal, Chris, if you don't believe the above, then hey, I am happy
to open a thread with them in CC where they can tell you in person how
bad an idea that is.

Lennart

--
Lennart Poettering, Berlin

[Chris Murphy]

On Wed, May 10, 2023, at 5:21 PM, Lennart Poettering wrote:
>  So to add to this. I happen to be at LFSMMBPF at the moment, the
> Linux
>  File System summit (among other things) where all the Linux FS people
>  meet. I spoke to a couple of FS maintainers here, and well, let me
>  make this very clear: using any of the major Linux file systems with
>  drivers that are not the ones in the Linux kernel is a very bad idea,
>  and expressly not supported by them. [They actually used much harsher
>  words, that I'll not repeat here – this is the "friendly" version of
>  their take on your idea.]
>
>  So, unless you want to go against what the people who actually
>  maintain the file systems expressly say please just get this idea out
>  of your head that porting Linux file systems into EFI fs drivers was a
>  good, supportable idea.
>
>  And Neal, Chris, if you don't believe the above, then hey, I am happy
>  to open a thread with them in CC where they can tell you in person how
>  bad an idea that is. 
I don't know what question you asked them. Any modifications (writes) performed
outside kernel code is not supported, since forever.

Read-only drivers, which are the only drivers under discussion here, aren't a per se
problem because they can't modify the file system. So they have no complaints about
that.



-- 
Chris Murphy

[Chris Adams]

Once upon a time, Chris Murphy <lists(a)colorremedies.com&gt; said:
>  Read-only drivers, which are the only drivers under discussion here,
> aren't a per se problem because they can't modify the file system. So they have no
> complaints about that. 
But those read-only drivers are incomplete and problematic, especially
as filesystems get more complex.  I've been bitten before by an
ill-timed unclean shutdown, where an update was still in the /boot ext4
journal but not comitted, so the system would not boot, because the
GRUB2 ext4 driver doesn't read the journal.

There should be _less_ put into GRUB2 filesystem support IMHO, not more.
No more complex filesystems; keep /boot something simple like ext2 that
GRUB2 can reasonably be expected to handle basically 100%, possibly
mounted read-only during normal operation, mount with "sync", and with
all updates as atomic as practical.

-- 
Chris Adams <linux(a)cmadams.net&gt;

[Chris Murphy]

On Wed, May 10, 2023, at 7:11 PM, Chris Adams wrote:
>  Once upon a time, Chris Murphy <lists(a)colorremedies.com&gt; said:
> > Read-only drivers, which are the only drivers under discussion here, aren't a per
> se problem because they can't modify the file system. So they have no complaints about
> that.
>
>  But those read-only drivers are incomplete and problematic, especially
>  as filesystems get more complex.  I've been bitten before by an
>  ill-timed unclean shutdown, where an update was still in the /boot ext4
>  journal but not comitted, so the system would not boot, because the
>  GRUB2 ext4 driver doesn't read the journal. 
Right.

But is the program updating /boot doing it correctly? Given the decision to use a
journaled file system but a bootloader that doesn't do journal replay, it means the
program needs to use a write order and sync policy to ensure there's no expectation of
journal replay. Otherwise inevitably it's going to break. So it's a series of
mistakes.


>  There should be _less_ put into GRUB2 filesystem support IMHO, not
> more.
>  No more complex filesystems; keep /boot something simple like ext2 that
>  GRUB2 can reasonably be expected to handle basically 100%, possibly
>  mounted read-only during normal operation, mount with "sync", and with
>  all updates as atomic as practical. 
It still needs the program that modifies /boot to do the updates in the proper sequence.
That's not happening right now so it's a risk no matter the file system. But if
simpler is better, and ext2 is acceptable, then FAT should also be acceptable. It has the
added benefit of all firmware supporting it. 


-- 
Chris Murphy

[Zbigniew Jędrzejewski-Szmek]

On Wed, May 10, 2023 at 05:54:45PM -0400, Chris Murphy wrote:
>  I don't know what question you asked them. Any modifications
>  (writes) performed outside kernel code is not supported, since
>  forever. 
>  Read-only drivers, which are the only drivers under discussion here,
>  aren't a per se problem because they can't modify the file
>  system. So they have no complaints about that. 
Just read-only is not enough: a user must be able to configure things in
the boot loader: the default boot entry, or screen resolution, etc.
Also we want boot counting, which means writing the number of boot
attempts somewhere. A solution which makes those things impossible
is not very attractive.

Zbyszek

[Neal Gompa]

On Thu, May 11, 2023 at 3:33 AM Zbigniew Jędrzejewski-Szmek
<zbyszek(a)in.waw.pl&gt; wrote:
>  On Wed, May 10, 2023 at 05:54:45PM -0400, Chris Murphy wrote:
>  > I don't know what question you asked them. Any modifications
>  > (writes) performed outside kernel code is not supported, since
>  > forever.
>
>  > Read-only drivers, which are the only drivers under discussion here,
>  > aren't a per se problem because they can't modify the file
>  > system. So they have no complaints about that.
>
>  Just read-only is not enough: a user must be able to configure things in
>  the boot loader: the default boot entry, or screen resolution, etc.
>  Also we want boot counting, which means writing the number of boot
>  attempts somewhere. A solution which makes those things impossible
>  is not very attractive.
>  
We already don't do boot counting from the bootloader side. That
happens in the operating system.


-- 
真実はいつも一つ！/ Always, there's only one truth!

[Luca Boccassi]

>  On Thu, May 11, 2023 at 3:33 AM Zbigniew Jędrzejewski-Szmek
>  <zbyszek(a)in.waw.pl&gt; wrote:
>  
>  We already don't do boot counting from the bootloader side. That
>  happens in the operating system. 
boot counting in the bootloader is a good thing to have, as that's the only place to
catch a few significant failures, and also it's the place where the decision is made.
If you are worried about ESP storage space usage, NV storage (and wear&tear) should be
even more concerning.

[Neal Gompa]

On Thu, May 11, 2023 at 6:26 AM Luca Boccassi <bluca(a)debian.org&gt; wrote:
>  > On Thu, May 11, 2023 at 3:33 AM Zbigniew Jędrzejewski-Szmek
>  > <zbyszek(a)in.waw.pl&gt; wrote:
>  >
>  > We already don't do boot counting from the bootloader side. That
>  > happens in the operating system.
>
>  boot counting in the bootloader is a good thing to have, as that's the only place to
> catch a few significant failures, and also it's the place where the decision is made.
> If you are worried about ESP storage space usage, NV storage (and wear&tear) should be
> even more concerning. 
I definitely am, considering how often there's no space and how easy
it is to brick many systems through it.



-- 
真実はいつも一つ！/ Always, there's only one truth!

[Luca Boccassi]

>  On Thu, May 11, 2023 at 6:26 AM Luca Boccassi
> <bluca(a)debian.org&gt; wrote:
>  
>  I definitely am, considering how often there's no space and how easy
>  it is to brick many systems through it. 
Right, and that's the only other way to convey this information to the bootloader. And
that's one of the reasons why in sd-boot we use the filename of the UKI to do boot
counting, rather than an NV var. There's really nothing else that can be used on an
average UEFI system - either NV storage or ESP.

[Zbigniew Jędrzejewski-Szmek]

On Thu, May 11, 2023 at 03:59:33AM -0400, Neal Gompa wrote:
>  On Thu, May 11, 2023 at 3:33 AM Zbigniew Jędrzejewski-Szmek
>  <zbyszek(a)in.waw.pl&gt; wrote:
>  >
>  > On Wed, May 10, 2023 at 05:54:45PM -0400, Chris Murphy wrote:
>  > > I don't know what question you asked them. Any modifications
>  > > (writes) performed outside kernel code is not supported, since
>  > > forever.
>  >
>  > > Read-only drivers, which are the only drivers under discussion here,
>  > > aren't a per se problem because they can't modify the file
>  > > system. So they have no complaints about that.
>  >
>  > Just read-only is not enough: a user must be able to configure things in
>  > the boot loader: the default boot entry, or screen resolution, etc.
>  > Also we want boot counting, which means writing the number of boot
>  > attempts somewhere. A solution which makes those things impossible
>  > is not very attractive.
>  >
>  
>  We already don't do boot counting from the bootloader side. That
>  happens in the operating system. 
The machine may fail before the userspace is established enough to be
able to write back to somewhere the boot loader will then read, and
such a boot *must* be counted as bad. And the only way to do this is
to actually do boot counting in the boot loader.

It is the OS that marks the boot as "good", but it's the boot loader
which decrements the boot counter before each boot, i.e. it actually
implements the counting in boot counting. Both systemd-boot and grub2
do this.

Zbyszek

[Lennart Poettering]

On Do, 11.05.23 03:59, Neal Gompa (ngompa13(a)gmail.com) wrote:

>  On Thu, May 11, 2023 at 3:33 AM Zbigniew Jędrzejewski-Szmek
>  <zbyszek(a)in.waw.pl&gt; wrote:
>  >
>  > On Wed, May 10, 2023 at 05:54:45PM -0400, Chris Murphy wrote:
>  > > I don't know what question you asked them. Any modifications
>  > > (writes) performed outside kernel code is not supported, since
>  > > forever.
>  >
>  > > Read-only drivers, which are the only drivers under discussion here,
>  > > aren't a per se problem because they can't modify the file
>  > > system. So they have no complaints about that.
>  >
>  > Just read-only is not enough: a user must be able to configure things in
>  > the boot loader: the default boot entry, or screen resolution, etc.
>  > Also we want boot counting, which means writing the number of boot
>  > attempts somewhere. A solution which makes those things impossible
>  > is not very attractive.
>  >
>
>  We already don't do boot counting from the bootloader side. That
>  happens in the operating system. 
If that was the case you could only count good boots, i.e. the ones
where you actually boot far enough into the OS. But that's useless
information. What you really care about for the purpose of automatic
fallback on failure is to count *bad* boots, and that means you have
to start counting *before* you hand off control to the OS, i.e. as
early as you can and then see if for that count you actually manage to
reach the OS.

Hence, if you care about realibility, automatic fallback and such
things, you need a writable boot partition. And that doesn't leave
many options, but VFAT.

Lennart

--
Lennart Poettering, Berlin

[Lennart Poettering]

On Mi, 10.05.23 17:54, Chris Murphy (lists(a)colorremedies.com) wrote:

>  On Wed, May 10, 2023, at 5:21 PM, Lennart Poettering wrote:
>  > So to add to this. I happen to be at LFSMMBPF at the moment, the Linux
>  > File System summit (among other things) where all the Linux FS people
>  > meet. I spoke to a couple of FS maintainers here, and well, let me
>  > make this very clear: using any of the major Linux file systems with
>  > drivers that are not the ones in the Linux kernel is a very bad idea,
>  > and expressly not supported by them. [They actually used much harsher
>  > words, that I'll not repeat here – this is the "friendly" version of
>  > their take on your idea.]
>  >
>  > So, unless you want to go against what the people who actually
>  > maintain the file systems expressly say please just get this idea out
>  > of your head that porting Linux file systems into EFI fs drivers was a
>  > good, supportable idea.
>  >
>  > And Neal, Chris, if you don't believe the above, then hey, I am happy
>  > to open a thread with them in CC where they can tell you in person how
>  > bad an idea that is.
>
>  I don't know what question you asked them. Any modifications
>  (writes) performed outside kernel code is not supported, since
>  forever. 
No, reading isn't fine either. You might remember the issue with a
sync()ed XFS not being properly readable by grub because linux xfs
semantics meant that sync() returns after data hit the journal but
grub never checked the journal? this mess even intermittendly landed
on my doorstep because people wanted us to freeze/thaw the disk during
shutdown to escape this mess...

The Linux FS people simply dont want to be bound by fucked up
semantics of alternative implementations that do not how to deal with
the situation on disk the same way as Linux.

Hence: get this idea out of your head please. btrfs, xfs, ext4 are not
a sensible option to read from EFI or grub, it's not sustainable.

(Moreover, read-only access doesn't cut it. If you want boot counting
you want write access.)

>  Read-only drivers, which are the only drivers under discussion here,
>  aren't a per se problem because they can't modify the file
>  system. So they have no complaints about that. 
No. Not true.

Lennart

--
Lennart Poettering, Berlin

[Gary Buhrmaster]

On Thu, May 11, 2023 at 3:10 PM Lennart Poettering <mzerqung(a)0pointer.de&gt; wrote:
>  On Mi, 10.05.23 17:54, Chris Murphy (lists(a)colorremedies.com) wrote:
>  >
>  > Read-only drivers, which are the only drivers under discussion here,
>  > aren't a per se problem because they can't modify the file
>  > system. So they have no complaints about that.
>
>  No. Not true. 
There could (in theory) exist a read-only
implementation of a journaling file system that
applied the journal as part of its processing in
memory (or some other transitory copy method).

To say that would be extremely fragile is an
understatement (it is, in essence, a full
implementation of the filesystem code).

I can't say I am a great fan of the fat filesystem,
but it is ubiquitous, and part of the machine
boot standards.  Proposing an alternative may
be an interesting intellectual exercise, but
one needs to work with the various industry
consortium's to make it a viable reality and
then wait 10+ years before it get implemented
everywhere, and 20+ years before someone
will not complain you are obsoleting some
perfectly usable hardware.

Quite honestly, I think that if the end game
is likely UKI on the ESP even 500MB might
be a little small, as one needs not only space
for the kernel images, but firmware updates
(and some firmware update images are
getting rather large), and any alternative OS
files, but it is much better than 200MB.

[Gregory Bartholomew]

On Thu, May 11, 2023 at 10:10 AM Lennart Poettering <mzerqung(a)0pointer.de&gt;
wrote:

>  (Moreover, read-only access doesn't cut it. If you want boot counting
>  you want write access.)
>
>  
Just interjecting a quick thought -- would it be possible to use FAT's
reserved sectors for the boot counting? (You can find a description of them
under the -R option in the mkfs.vfat man page.) I still think it would be
nice to be able to mirror the ESP with mdraid. But if the FAT filesystem is
writable by the bootloader, then that won't work. The reserved sectors are
specifically reserved for bootloader code by design and if write access by
the bootloader/firmware is limited to those sectors then there should be no
danger of filesystem corruption if software mirroring is used.

[Simo Sorce]

On Thu, 2023-05-11 at 12:52 -0500, Gregory Bartholomew wrote:
>  On Thu, May 11, 2023 at 10:10 AM Lennart Poettering
> <mzerqung(a)0pointer.de&gt;
>  wrote:
>  
>  > 
>  > (Moreover, read-only access doesn't cut it. If you want boot counting
>  > you want write access.)
>  > 
>  > 
>  Just interjecting a quick thought -- would it be possible to use FAT's
>  reserved sectors for the boot counting? (You can find a description of them
>  under the -R option in the mkfs.vfat man page.) I still think it would be
>  nice to be able to mirror the ESP with mdraid. But if the FAT filesystem is
>  writable by the bootloader, then that won't work. The reserved sectors are
>  specifically reserved for bootloader code by design and if write access by
>  the bootloader/firmware is limited to those sectors then there should be no
>  danger of filesystem corruption if software mirroring is used. 
Given changes to the ESP should be rare, it may make more sense to just
teach the system to make a copy of the contents to a separate partition
whenever changes are made, rather then relying on RAID 1 in this case.
It would certainly be more robust.
And could even be used as a "recovery" partition if you update the
contents of the second partition only after successful reboot after
update of the first...

Simo.

-- 
Simo Sorce
RHEL Crypto Team
Red Hat, Inc

[Simo Sorce]

On Tue, 2023-05-09 at 12:37 -0400, Neal Gompa wrote:
>  On Tue, May 9, 2023 at 12:31 PM Lennart Poettering
> <mzerqung(a)0pointer.de&gt; wrote:
>  > 
>  > On Di, 09.05.23 08:22, Neal Gompa (ngompa13(a)gmail.com) wrote:
>  > 
>  > > I've been asked to consider converting /boot to a Btrfs subvolume so
>  > > that it no longer has a fixed space allocation to deal with the ever
>  > > increasing amount of firmware required for NVIDIA GPUs[1]. This is
>  > > currently incompatible with how systemd views the world, because the
>  > > "discoverable partition spec" is wired to partitions, and there is
> no
>  > > equivalent spec for subvolumes of a volume. And I imagine that
>  > > XBOOTLDR (whatever that is) also would have a problem with this.
>  > 
>  > This makese no sense. If you want /boot to just be a subvolume of the
>  > rootfs btrfs, then this would imply it's also covered by the same
>  > security choices, i.e. encryption. We want to bind that sooner or
>  > later to things like TPM2, FIDO2, PKCS11. And that's simply not
>  > feasible from a boot loader environment.
>  > 
>  > Hence, the place the kernel is loaded from (regardless if you call it
>  > /efi or /boot or /boot/efi, and regardless what fs it is) must be
>  > accessible from the boot loader easily, without requiring
>  > implementation of TPM2/FIDO2/PKCS11 hookup in the boot loader.
>  > 
>  > Hence: btrfs subvols won't work for this
>  
>  If we're not using LUKS for encryption, then this is not a problem.
>  We're generally looking toward encrypting subvolumes individually
>  using the upcoming Btrfs native encryption capability rather than
>  using LUKS. That allows us to
>  
>  1. Pick which subvolumes are encrypted
>  2. Pick the security binding method per subvolume
>  
>  For example, the root and home subvolumes would use TPM or some other
>  non-interactive binding. The user subvolume in home would decrypt with
>  user login. 
Neal,
I think you are barking up the wrong tree here.

The design of the boot loader *nedds* to be simple.

Simple means, VFAT and *no trust* in the filesystem, individual files
are signed and verified.
Only the bare minimum *necessary* is in the boot partition, which means
kernel images and the bare minimum init image needed to unlock and
mount the root partition.

There is no point in building a more complex system than that and load
tons of garbage drivers in the EFI.

Booting is a staged system, and should be kept as simple as possible to
avoid duplication (which means subtle bugs and a ton of maintenance
overhead).

Simo.

-- 
Simo Sorce
RHEL Crypto Team
Red Hat, Inc

[Leslie Satenstein]

Leslie Satenstein    

Some end-user feedback.

I believe in the KISS approach (Keep It Simple Sxxxx).
 I would consider /boot as a btrfs volume, and not a sub-volume, but why bother? 
For me, it being a btrfs partition is definitely not a priority or urgency, as I use rsync
for backup/restore of the ext4 partition.
For my desktop setup, I have several disks with independent btrfs partitions.These are not
sub-volumes, again, they are fully independent.
I manually add them to my /etc/fstab after I install the vanilla Fedora Linux distribution

I make use of them as I independent volumes. They definitely are not sub-volumes.
Some common sense.=================
Most user disks today are of size 500gigs or more.I do not concern myself with 1024 megs
for /boot.  I also reserve/use 500megs for /boot/efi 

Better to spend energy on other things.




    On Wednesday, May 10, 2023 at 11:12:20 a.m. EDT, Simo Sorce <simo(a)redhat.com&gt;
wrote:  
 
 On Tue, 2023-05-09 at 12:37 -0400, Neal Gompa wrote:
>  On Tue, May 9, 2023 at 12:31 PM Lennart Poettering
> <mzerqung(a)0pointer.de&gt; wrote:
>  > 
>  > On Di, 09.05.23 08:22, Neal Gompa (ngompa13(a)gmail.com) wrote:
>  > 
>  > > I've been asked to consider converting /boot to a Btrfs subvolume so
>  > > that it no longer has a fixed space allocation to deal with the ever
>  > > increasing amount of firmware required for NVIDIA GPUs[1]. This is
>  > > currently incompatible with how systemd views the world, because the
>  > > "discoverable partition spec" is wired to partitions, and there is
> no
>  > > equivalent spec for subvolumes of a volume. And I imagine that
>  > > XBOOTLDR (whatever that is) also would have a problem with this.
>  > 
>  > This makese no sense. If you want /boot to just be a subvolume of the
>  > rootfs btrfs, then this would imply it's also covered by the same
>  > security choices, i.e. encryption. We want to bind that sooner or
>  > later to things like TPM2, FIDO2, PKCS11. And that's simply not
>  > feasible from a boot loader environment.
>  > 
>  > Hence, the place the kernel is loaded from (regardless if you call it
>  > /efi or /boot or /boot/efi, and regardless what fs it is) must be
>  > accessible from the boot loader easily, without requiring
>  > implementation of TPM2/FIDO2/PKCS11 hookup in the boot loader.
>  > 
>  > Hence: btrfs subvols won't work for this
>  
>  If we're not using LUKS for encryption, then this is not a problem.
>  We're generally looking toward encrypting subvolumes individually
>  using the upcoming Btrfs native encryption capability rather than
>  using LUKS. That allows us to
>  
>  1. Pick which subvolumes are encrypted
>  2. Pick the security binding method per subvolume
>  
>  For example, the root and home subvolumes would use TPM or some other
>  non-interactive binding. The user subvolume in home would decrypt with
>  user login. 
Neal,
I think you are barking up the wrong tree here.

The design of the boot loader *nedds* to be simple.

Simple means, VFAT and *no trust* in the filesystem, individual files
are signed and verified.
Only the bare minimum *necessary* is in the boot partition, which means
kernel images and the bare minimum init image needed to unlock and
mount the root partition.

There is no point in building a more complex system than that and load
tons of garbage drivers in the EFI.

Booting is a staged system, and should be kept as simple as possible to
avoid duplication (which means subtle bugs and a ton of maintenance
overhead).

Simo.

-- 
Simo Sorce
RHEL Crypto Team
Red Hat, Inc


_______________________________________________
devel mailing list -- devel(a)lists.fedoraproject.org
To unsubscribe send an email to devel-leave(a)lists.fedoraproject.org
Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives:
https://lists.fedoraproject.org/archives/list/[e-mail]
Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue

[Neal Gompa]

On Wed, May 10, 2023 at 11:12 AM Simo Sorce <simo(a)redhat.com&gt; wrote:
>  On Tue, 2023-05-09 at 12:37 -0400, Neal Gompa wrote:
>  > On Tue, May 9, 2023 at 12:31 PM Lennart Poettering <mzerqung(a)0pointer.de&gt;
> wrote:
>  > >
>  > > On Di, 09.05.23 08:22, Neal Gompa (ngompa13(a)gmail.com) wrote:
>  > >
>  > > > I've been asked to consider converting /boot to a Btrfs subvolume so
>  > > > that it no longer has a fixed space allocation to deal with the ever
>  > > > increasing amount of firmware required for NVIDIA GPUs[1]. This is
>  > > > currently incompatible with how systemd views the world, because the
>  > > > "discoverable partition spec" is wired to partitions, and there
> is no
>  > > > equivalent spec for subvolumes of a volume. And I imagine that
>  > > > XBOOTLDR (whatever that is) also would have a problem with this.
>  > >
>  > > This makese no sense. If you want /boot to just be a subvolume of the
>  > > rootfs btrfs, then this would imply it's also covered by the same
>  > > security choices, i.e. encryption. We want to bind that sooner or
>  > > later to things like TPM2, FIDO2, PKCS11. And that's simply not
>  > > feasible from a boot loader environment.
>  > >
>  > > Hence, the place the kernel is loaded from (regardless if you call it
>  > > /efi or /boot or /boot/efi, and regardless what fs it is) must be
>  > > accessible from the boot loader easily, without requiring
>  > > implementation of TPM2/FIDO2/PKCS11 hookup in the boot loader.
>  > >
>  > > Hence: btrfs subvols won't work for this
>  >
>  > If we're not using LUKS for encryption, then this is not a problem.
>  > We're generally looking toward encrypting subvolumes individually
>  > using the upcoming Btrfs native encryption capability rather than
>  > using LUKS. That allows us to
>  >
>  > 1. Pick which subvolumes are encrypted
>  > 2. Pick the security binding method per subvolume
>  >
>  > For example, the root and home subvolumes would use TPM or some other
>  > non-interactive binding. The user subvolume in home would decrypt with
>  > user login.
>
>  Neal,
>  I think you are barking up the wrong tree here.
>
>  The design of the boot loader *nedds* to be simple.
>
>  Simple means, VFAT and *no trust* in the filesystem, individual files
>  are signed and verified.
>  Only the bare minimum *necessary* is in the boot partition, which means
>  kernel images and the bare minimum init image needed to unlock and
>  mount the root partition.
>
>  There is no point in building a more complex system than that and load
>  tons of garbage drivers in the EFI.
>
>  Booting is a staged system, and should be kept as simple as possible to
>  avoid duplication (which means subtle bugs and a ton of maintenance
>  overhead).
>  
This proposal isn't better. Neither Windows nor macOS put critical
operating system code in the ESP, and we shouldn't either. But you
want to put kernels in the ESP? That's the wrong approach too.

As soon as you throw UKIs in the mix, you've completely broken that
because now the absolutely most valuable code for your system is in a
"hostile" environment. At least we can make /boot authenticated and
tamper resistant as a Btrfs subvolume.

I would rather have a multi-stage boot process, where the first stage
does just enough to unlock and load the OS volume to load the real
boot environment.





--
真実はいつも一つ！/ Always, there's only one truth!

[Daniel P. Berrangé]

On Wed, May 10, 2023 at 12:00:36PM -0400, Neal Gompa wrote:
>  On Wed, May 10, 2023 at 11:12 AM Simo Sorce <simo(a)redhat.com&gt;
> wrote:
>  >
>  > On Tue, 2023-05-09 at 12:37 -0400, Neal Gompa wrote:
>  > > On Tue, May 9, 2023 at 12:31 PM Lennart Poettering <mzerqung(a)0pointer.de&gt;
> wrote:
>  > > >
>  > > > On Di, 09.05.23 08:22, Neal Gompa (ngompa13(a)gmail.com) wrote:
>  > > >
>  > > > > I've been asked to consider converting /boot to a Btrfs subvolume
> so
>  > > > > that it no longer has a fixed space allocation to deal with the ever
>  > > > > increasing amount of firmware required for NVIDIA GPUs[1]. This is
>  > > > > currently incompatible with how systemd views the world, because the
>  > > > > "discoverable partition spec" is wired to partitions, and
> there is no
>  > > > > equivalent spec for subvolumes of a volume. And I imagine that
>  > > > > XBOOTLDR (whatever that is) also would have a problem with this.
>  > > >
>  > > > This makese no sense. If you want /boot to just be a subvolume of the
>  > > > rootfs btrfs, then this would imply it's also covered by the same
>  > > > security choices, i.e. encryption. We want to bind that sooner or
>  > > > later to things like TPM2, FIDO2, PKCS11. And that's simply not
>  > > > feasible from a boot loader environment.
>  > > >
>  > > > Hence, the place the kernel is loaded from (regardless if you call it
>  > > > /efi or /boot or /boot/efi, and regardless what fs it is) must be
>  > > > accessible from the boot loader easily, without requiring
>  > > > implementation of TPM2/FIDO2/PKCS11 hookup in the boot loader.
>  > > >
>  > > > Hence: btrfs subvols won't work for this
>  > >
>  > > If we're not using LUKS for encryption, then this is not a problem.
>  > > We're generally looking toward encrypting subvolumes individually
>  > > using the upcoming Btrfs native encryption capability rather than
>  > > using LUKS. That allows us to
>  > >
>  > > 1. Pick which subvolumes are encrypted
>  > > 2. Pick the security binding method per subvolume
>  > >
>  > > For example, the root and home subvolumes would use TPM or some other
>  > > non-interactive binding. The user subvolume in home would decrypt with
>  > > user login.
>  >
>  > Neal,
>  > I think you are barking up the wrong tree here.
>  >
>  > The design of the boot loader *nedds* to be simple.
>  >
>  > Simple means, VFAT and *no trust* in the filesystem, individual files
>  > are signed and verified.
>  > Only the bare minimum *necessary* is in the boot partition, which means
>  > kernel images and the bare minimum init image needed to unlock and
>  > mount the root partition.
>  >
>  > There is no point in building a more complex system than that and load
>  > tons of garbage drivers in the EFI.
>  >
>  > Booting is a staged system, and should be kept as simple as possible to
>  > avoid duplication (which means subtle bugs and a ton of maintenance
>  > overhead).
>  >
>  
>  This proposal isn't better. Neither Windows nor macOS put critical
>  operating system code in the ESP, and we shouldn't either. But you
>  want to put kernels in the ESP? That's the wrong approach too.
>  
>  As soon as you throw UKIs in the mix, you've completely broken that
>  because now the absolutely most valuable code for your system is in a
>  "hostile" environment. At least we can make /boot authenticated and
>  tamper resistant as a Btrfs subvolume. 
The firmware will start by booting a binary found in the ESP,
and thus given the ESP is untrusted (hostile) we need a means
to validate that binary's integrity no matter what, most likely
using SecureBoot & TPM PCR measurements. Using this mechanism
for validating the UKI makes sense since it has to exist no
matter what. There's no need for the filesystem to be authenticated
/ tamper resistant if the binaries themselves are authenticated
and tamper resistant.

>  I would rather have a multi-stage boot process, where the first
> stage
>  does just enough to unlock and load the OS volume to load the real
>  boot environment. 
With regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

[Owen Taylor]

On Wed, May 10, 2023 at 12:02 PM Neal Gompa <ngompa13(a)gmail.com&gt; wrote:

>  This proposal isn't better. Neither Windows nor macOS put critical
>  operating system code in the ESP, and we shouldn't either. But you
>  want to put kernels in the ESP? That's the wrong approach too.
>
>  As soon as you throw UKIs in the mix, you've completely broken that
>  because now the absolutely most valuable code for your system is in a
>  "hostile" environment. At least we can make /boot authenticated and
>  tamper resistant as a Btrfs subvolume.
>  
As other people have mentioned, we have a solution for the ESP being
untrusted - secure boot. As far as I understand, there's no tamper
resistance for /boot on btrfs unless it's encrypted, and that would be a
whole other barrel of snakes :-)

Now, there's a second problem with reading everything from the ESP - it's
slow for two reasons. First, because read speeds when going through the
firmware are much slower than the Linux NVMe stack. And second, because we
have to read everything in order to checksum and verify it.

For that reason, Demi's suggestion of moving things onto a dm-verity
protected partition is intriguing. Could that even be a loopback file on
the ESP? It would be like a lazy-read system extension.

The performance geek in me thinks "we could see exciting speedups from
that!" - the practical side of me thinks "it might be a few second faster.
And much more complex! Let's just go with efifb, keep the initramfs small,
and accept any minor regressions".

I would rather have a multi-stage boot process, where the first stage
>  does just enough to unlock and load the OS volume to load the real
>  boot environment.
>  
"unlock and load the OS volume" is pretty much what the initramfs does,
right?

- Owen

[Neal Gompa]

On Wed, May 10, 2023 at 2:24 PM Owen Taylor <otaylor(a)redhat.com&gt; wrote:
>  On Wed, May 10, 2023 at 12:02 PM Neal Gompa <ngompa13(a)gmail.com&gt; wrote:
> >
> >
> > This proposal isn't better. Neither Windows nor macOS put critical
> > operating system code in the ESP, and we shouldn't either. But you
> > want to put kernels in the ESP? That's the wrong approach too.
> >
> > As soon as you throw UKIs in the mix, you've completely broken that
> > because now the absolutely most valuable code for your system is in a
> > "hostile" environment. At least we can make /boot authenticated and
> > tamper resistant as a Btrfs subvolume.
>
>
>  As other people have mentioned, we have a solution for the ESP being untrusted - secure
> boot. As far as I understand, there's no tamper resistance for /boot on btrfs unless
> it's encrypted, and that would be a whole other barrel of snakes :-)
>  
fsverity is separate from fscrypt. We can apply filesystem authentication today.

>  Now, there's a second problem with reading everything from the
> ESP - it's slow for two reasons. First, because read speeds when going through the
> firmware are much slower than the Linux NVMe stack. And second, because we have to read
> everything in order to checksum and verify it.
>
>  For that reason, Demi's suggestion of moving things onto a dm-verity protected
> partition is intriguing. Could that even be a loopback file on the ESP? It would be like a
> lazy-read system extension.
>
>  The performance geek in me thinks "we could see exciting speedups from that!" -
> the practical side of me thinks "it might be a few second faster. And much more
> complex! Let's just go with efifb, keep the initramfs small, and accept any minor
> regressions".
>
> > I would rather have a multi-stage boot process, where the first stage
> > does just enough to unlock and load the OS volume to load the real
> > boot environment.
>
>
>  "unlock and load the OS volume" is pretty much what the initramfs does, right?
>  
No. It initializes the whole operating system, and then pivots the
user-space later. That's why we have to everything in initramfs.
UKIs attempt to standardize the early-stage image without attempting
to solve this problem, because a two-stage boot process requires
changing how we think about operating system initialization.

In Windows, the Windows Boot Manager loads the NT
kernel stub from the NTFS volume, which then loads the minimal
operating system environment, and bootstraps the full Windows
experience. The Windows Boot Manager has just enough to handle
BitLocker and NTFS, and then transfers the rest to Windows itself.

This means that you can do interesting things by simply replacing the
Windows Boot Manager with another boot manager that includes more
features. As an example, Quibble[1] replaces the Windows Boot Manager
to enable booting Windows off Btrfs.

[1]: https://github.com/maharmstone/quibble





--
真実はいつも一つ！/ Always, there's only one truth!

[Luca Boccassi]

>  On Wed, May 10, 2023 at 2:24 PM Owen Taylor
> <otaylor(a)redhat.com&gt; wrote:
>  
>  fsverity is separate from fscrypt. We can apply filesystem authentication today. 
fsverity does not protect metadata, and most importantly it does not protect the
filesystem superblock. It has its uses, but this is not it.

>  No. It initializes the whole operating system, and then pivots the
>  user-space later. That's why we have to everything in initramfs.
>  UKIs attempt to standardize the early-stage image without attempting
>  to solve this problem, because a two-stage boot process requires
>  changing how we think about operating system initialization.
>  
>  In Windows, the Windows Boot Manager loads the NT
>  kernel stub from the NTFS volume, which then loads the minimal
>  operating system environment, and bootstraps the full Windows
>  experience. The Windows Boot Manager has just enough to handle
>  BitLocker and NTFS, and then transfers the rest to Windows itself. 
It is really not that different than the initrd approach. Just the storage is different,
but that's easier when you own both filesystems implementations.

[Lennart Poettering]

On Mi, 10.05.23 14:32, Neal Gompa (ngompa13(a)gmail.com) wrote:

>  On Wed, May 10, 2023 at 2:24 PM Owen Taylor
> <otaylor(a)redhat.com&gt; wrote:
>  >> As soon as you throw UKIs in the mix, you've completely broken that
>  >> because now the absolutely most valuable code for your system is in a
>  >> "hostile" environment. At least we can make /boot authenticated and
>  >> tamper resistant as a Btrfs subvolume.
>  >
>  >
>  > As other people have mentioned, we have a solution for the ESP being untrusted -
> secure boot. As far as I understand, there's no tamper resistance for /boot on btrfs
> unless it's encrypted, and that would be a whole other barrel of snakes :-)
>
>  fsverity is separate from fscrypt. We can apply filesystem
>  authentication today. 
No that's just wrong. fsverity is *not* filesystem
authentication. It's authentication of the content of a single file,
and not more.

And that's just too little, because a complex file system such as
btrfs is simply not considered robust against rogue offline
modification. (Again, I am at LFSMMBPF and if you want I can get you a
quote from the btrfs maintainer about this). Thus you must
authenticate btrfs *before* you mount it, and fsverity is only
available after.

Sorry, fsverity has some usecases, but your usecase here is absolutely
not it.

Seriously, forget about this whole btrfs idea. It's wrong on many many
levels.

>  No. It initializes the whole operating system, and then pivots the
>  user-space later. That's why we have to everything in initramfs.
>  UKIs attempt to standardize the early-stage image without attempting
>  to solve this problem, because a two-stage boot process requires
>  changing how we think about operating system initialization. 
So the initrd is supposed to contain exactly what is necssary to get
access to the root fs, not more. Thing is that Linux is very very
flexible, and people put their root fs on crazy stuff. Now I
personally don't even care much about the crazy storage options people
want to back the rootfs, I only care about the non-crazy part (in my
eyes), i.e. encryption, fido2, tpm stuff, which possibly requires
interactivity so it probably also means a graphical session to some
point. While that generally ends up being a lot, it still is certainly
not *everything*.

You can stick your head in the sand and pretend that nothing of this
mattered, and you don't have to authenticate and things, but then you
simply didn't solve the problem at hand.

Lennart

--
Lennart Poettering, Berlin

[Chris Murphy]

On Wed, May 10, 2023, at 2:24 PM, Owen Taylor wrote:
>  
>  
>  On Wed, May 10, 2023 at 12:02 PM Neal Gompa <ngompa13(a)gmail.com&gt; wrote:
> > 
>  Now, there's a second problem with reading everything from the ESP - it's slow
> for two reasons. First, because read speeds when going through the firmware are much
> slower than the Linux NVMe stack. And second, because we have to read everything in order
> to checksum and verify it.
>  
>  For that reason, Demi's suggestion of moving things onto a dm-verity protected
> partition is intriguing. Could that even be a loopback file on the ESP? It would be like a
> lazy-read system extension.
>  
>  The performance geek in me thinks "we could see exciting speedups from that!" -
> the practical side of me thinks "it might be a few second faster. And much more
> complex! Let's just go with efifb, keep the initramfs small, and accept any minor
> regressions". 
GRUB doesn't support dm-verity, but I don't know if it would make a difference if
it did. Once the kernel has the ability to interact with physical storage, understand
partitions, initialize dm-verity, and read a file systemt... it could just mount `/` . I
think the only way the current initial root file system works with the kernel is for the
bootloader to read an entire initrd file into RAM, and then the kernel can randomly access
things in that RAM disk.

It's early still for UKI details but I assume we will need both a universal UKI (all
use cases), and much smaller use case focused UKIs. I say that because the desktops in the
default installation configuration are the largest single use case. With Btrfs built-into
the kernel, we don't need that much in the initrd to setup block devices, discover
their contents, and perform switch root.

The next most common use case(s), device-mapper and md based, require a pile of user space
programs running to do all the work setting things up. Maybe we can just get away with two
images, a simple fast one and the universal.


--
Chris Murphy

[Owen Taylor]

On Wed, May 10, 2023 at 5:34 PM Chris Murphy <lists(a)colorremedies.com&gt;
wrote:

>  On Wed, May 10, 2023, at 2:24 PM, Owen Taylor wrote:
>
>
>
>  On Wed, May 10, 2023 at 12:02 PM Neal Gompa <ngompa13(a)gmail.com&gt; wrote:
>
>
>  Now, there's a second problem with reading everything from the ESP - it's
>  slow for two reasons. First, because read speeds when going through the
>  firmware are much slower than the Linux NVMe stack. And second, because we
>  have to read everything in order to checksum and verify it.
>
>  For that reason, Demi's suggestion of moving things onto a dm-verity
>  protected partition is intriguing. Could that even be a loopback file on
>  the ESP? It would be like a lazy-read system extension.
>
>  The performance geek in me thinks "we could see exciting speedups from
>  that!" - the practical side of me thinks "it might be a few second faster.
>  And much more complex! Let's just go with efifb, keep the initramfs small,
>  and accept any minor regressions".
>
>
>  GRUB doesn't support dm-verity, but I don't know if it would make a
>  difference if it did. Once the kernel has the ability to interact with
>  physical storage, understand partitions, initialize dm-verity, and read a
>  file systemt... it could just mount `/` . I think the only way the current
>  initial root file system works with the kernel is for the bootloader to
>  read an entire initrd file into RAM, and then the kernel can randomly
>  access things in that RAM disk.
>  
In this idea, the dm-verity parition/file would only be accessed from the
initrd, once we have enough kernel to ability to interact with physical
storage, understand partitions, initialize dm-verity, and read *a*
partition, but potentially not enough to read from a bluetooth keyboard,
show a graphical prompt, mount / from the network, etc.

What dm-verity provides here is a way to trust content without requiring it
to be read ahead of time, checksumed, signature checked and put into ram.

To be clear, I don't think this is necessarily the right way to go - I
think using efifb, not prompting for a passphrase when possible, etc, are
simpler approaches.

>  It's early still for UKI details but I assume we will need both a
>  universal UKI (all use cases), and much smaller use case focused UKIs. I
>  say that because the desktops in the default installation configuration are
>  the largest single use case. With Btrfs built-into the kernel, we don't
>  need that much in the initrd to setup block devices, discover their
>  contents, and perform switch root.
>
>  The next most common use case(s), device-mapper and md based, require a
>  pile of user space programs running to do all the work setting things up.
>  Maybe we can just get away with two images, a simple fast one and the
>  universal.
>  
The elephant in the room is whether the "desktops in the default
installation" UKI requires piles of graphics firmware. It might not be at
all small in that case.

- Owen

[Luca Boccassi]

>  On Wed, May 10, 2023 at 5:34 PM Chris Murphy
> <lists(a)colorremedies.com&gt;
>  wrote:
>  
>  
>  In this idea, the dm-verity parition/file would only be accessed from the
>  initrd, once we have enough kernel to ability to interact with physical
>  storage, understand partitions, initialize dm-verity, and read *a*
>  partition, but potentially not enough to read from a bluetooth keyboard,
>  show a graphical prompt, mount / from the network, etc.
>  
>  What dm-verity provides here is a way to trust content without requiring it
>  to be read ahead of time, checksumed, signature checked and put into ram. 
We already support dm-verity for the rootfs, in both dracut and mkosi-initrd, not sure
what's the issue here?

>  The elephant in the room is whether the "desktops in the
> default
>  installation" UKI requires piles of graphics firmware. It might not be at
>  all small in that case. 
We support optional extensions with UKIs exactly for this reason.

[Chris Murphy]

On Wed, May 10, 2023, at 7:36 PM, Owen Taylor wrote:
>  
>  
>  On Wed, May 10, 2023 at 5:34 PM Chris Murphy <lists(a)colorremedies.com&gt; wrote:
> > __
> > 
> > 
> > On Wed, May 10, 2023, at 2:24 PM, Owen Taylor wrote:
> >> 
> >> 
> >> On Wed, May 10, 2023 at 12:02 PM Neal Gompa <ngompa13(a)gmail.com&gt; wrote:
> >>> 
> >> Now, there's a second problem with reading everything from the ESP - it's
> slow for two reasons. First, because read speeds when going through the firmware are much
> slower than the Linux NVMe stack. And second, because we have to read everything in order
> to checksum and verify it.
> >> 
> >> For that reason, Demi's suggestion of moving things onto a dm-verity
> protected partition is intriguing. Could that even be a loopback file on the ESP? It would
> be like a lazy-read system extension.
> >> 
> >> The performance geek in me thinks "we could see exciting speedups from
> that!" - the practical side of me thinks "it might be a few second faster. And
> much more complex! Let's just go with efifb, keep the initramfs small, and accept any
> minor regressions".
> > 
> > GRUB doesn't support dm-verity, but I don't know if it would make a
> difference if it did. Once the kernel has the ability to interact with physical storage,
> understand partitions, initialize dm-verity, and read a file systemt... it could just
> mount `/` . I think the only way the current initial root file system works with the
> kernel is for the bootloader to read an entire initrd file into RAM, and then the kernel
> can randomly access things in that RAM disk.
>  
>  In this idea, the dm-verity parition/file would only be accessed from the initrd, once we
> have enough kernel to ability to interact with physical storage, understand partitions,
> initialize dm-verity, and read *a* partition, but potentially not enough to read from a
> bluetooth keyboard, show a graphical prompt, mount / from the network, etc.
>  
>  What dm-verity provides here is a way to trust content without requiring it to be read
> ahead of time, checksumed, signature checked and put into ram.
>  
>  To be clear, I don't think this is necessarily the right way to go - I think using
> efifb, not prompting for a passphrase when possible, etc, are simpler approaches. 
OK got it. So it would be some kind of intermediate /usr that bridges between initramfs
and /sysroot mounting? I'm not sure the added complexity helps significantly with
authenticity over either LUKS or fscrypt.


>   
> > 
> > It's early still for UKI details but I assume we will need both a universal UKI
> (all use cases), and much smaller use case focused UKIs. I say that because the desktops
> in the default installation configuration are the largest single use case. With Btrfs
> built-into the kernel, we don't need that much in the initrd to setup block devices,
> discover their contents, and perform switch root.
> > 
> > The next most common use case(s), device-mapper and md based, require a pile of user
> space programs running to do all the work setting things up. Maybe we can just get away
> with two images, a simple fast one and the universal.
>  
>  The elephant in the room is whether the "desktops in the default installation"
> UKI requires piles of graphics firmware. It might not be at all small in that case.
True. I don't know what the limitations/reasons are for firmware blobs needing to be
available so early during boot.

But (a) we have chosen a file system by default that we can shrink online (b) have at
least 128 partitions possible (c) we can extent Boot Loader Spec to permit multiple
XBOOTLDR. So we can make room for these firmware blobs. It's just fricking ugly. And
I'm not sure it'll be RPM's domain if these things really need to exist on
/boot. Something will have to put them there, as needed.



--
Chris Murphy

[Lennart Poettering]

On Mi, 10.05.23 12:00, Neal Gompa (ngompa13(a)gmail.com) wrote:

>  This proposal isn't better. Neither Windows nor macOS put
> critical
>  operating system code in the ESP, and we shouldn't either. 
This is nonsense. I am not sure what definition of "critical" you
have, but the ESP is the entrypoint to the OS, of course it contains
"critical" stuff in there, if you delete what windows places there you
cannot boot anymore.

>  As soon as you throw UKIs in the mix, you've completely broken
> that
>  because now the absolutely most valuable code for your system is in a
>  "hostile" environment. At least we can make /boot authenticated and 
"most valueable code"? what is that even supposed to mean?

>  tamper resistant as a Btrfs subvolume. 
What is a tamper resistent btrfs subvolume? I have not heard of such a
thing. And don't say "fsverity" now. Because that's really not
it. "fsverity" is not some secret sauce that you attach to an fs and
then it cannot be tempered with. It has a much more limited usecase
and requires userspace to first validate the file's metadata and it's
hash manually before using it.

Seriously, fsverity is not an answer for this. And you know what, the
person behind fsverity will tell you the same. (also here at lfsmmbpf,
and I can connect you if you want)

Moreover, you need to validate a complex file system before you access
it, it's too late if you do it the other way round. Hence, unless you
established trust in your btrfs fs *before* you go into and and look
for kernel you are not doing security, you are just doing nonsense.

>  I would rather have a multi-stage boot process, where the first
> stage
>  does just enough to unlock and load the OS volume to load the real
>  boot environment. 
And I'd like a pony!

Lennart

--
Lennart Poettering, Berlin

[Simo Sorce]

On Wed, 2023-05-10 at 12:00 -0400, Neal Gompa wrote:
>  On Wed, May 10, 2023 at 11:12 AM Simo Sorce <simo(a)redhat.com&gt;
> wrote:
>  > 
>  > On Tue, 2023-05-09 at 12:37 -0400, Neal Gompa wrote:
>  > > On Tue, May 9, 2023 at 12:31 PM Lennart Poettering <mzerqung(a)0pointer.de&gt;
> wrote:
>  > > > 
>  > > > On Di, 09.05.23 08:22, Neal Gompa (ngompa13(a)gmail.com) wrote:
>  > > > 
>  > > > > I've been asked to consider converting /boot to a Btrfs subvolume
> so
>  > > > > that it no longer has a fixed space allocation to deal with the ever
>  > > > > increasing amount of firmware required for NVIDIA GPUs[1]. This is
>  > > > > currently incompatible with how systemd views the world, because the
>  > > > > "discoverable partition spec" is wired to partitions, and
> there is no
>  > > > > equivalent spec for subvolumes of a volume. And I imagine that
>  > > > > XBOOTLDR (whatever that is) also would have a problem with this.
>  > > > 
>  > > > This makese no sense. If you want /boot to just be a subvolume of the
>  > > > rootfs btrfs, then this would imply it's also covered by the same
>  > > > security choices, i.e. encryption. We want to bind that sooner or
>  > > > later to things like TPM2, FIDO2, PKCS11. And that's simply not
>  > > > feasible from a boot loader environment.
>  > > > 
>  > > > Hence, the place the kernel is loaded from (regardless if you call it
>  > > > /efi or /boot or /boot/efi, and regardless what fs it is) must be
>  > > > accessible from the boot loader easily, without requiring
>  > > > implementation of TPM2/FIDO2/PKCS11 hookup in the boot loader.
>  > > > 
>  > > > Hence: btrfs subvols won't work for this
>  > > 
>  > > If we're not using LUKS for encryption, then this is not a problem.
>  > > We're generally looking toward encrypting subvolumes individually
>  > > using the upcoming Btrfs native encryption capability rather than
>  > > using LUKS. That allows us to
>  > > 
>  > > 1. Pick which subvolumes are encrypted
>  > > 2. Pick the security binding method per subvolume
>  > > 
>  > > For example, the root and home subvolumes would use TPM or some other
>  > > non-interactive binding. The user subvolume in home would decrypt with
>  > > user login.
>  > 
>  > Neal,
>  > I think you are barking up the wrong tree here.
>  > 
>  > The design of the boot loader *nedds* to be simple.
>  > 
>  > Simple means, VFAT and *no trust* in the filesystem, individual files
>  > are signed and verified.
>  > Only the bare minimum *necessary* is in the boot partition, which means
>  > kernel images and the bare minimum init image needed to unlock and
>  > mount the root partition.
>  > 
>  > There is no point in building a more complex system than that and load
>  > tons of garbage drivers in the EFI.
>  > 
>  > Booting is a staged system, and should be kept as simple as possible to
>  > avoid duplication (which means subtle bugs and a ton of maintenance
>  > overhead).
>  > 
>  
>  This proposal isn't better. Neither Windows nor macOS put critical
>  operating system code in the ESP, and we shouldn't either. But you
>  want to put kernels in the ESP? That's the wrong approach too.
>  
>  As soon as you throw UKIs in the mix, you've completely broken that
>  because now the absolutely most valuable code for your system is in a
>  "hostile" environment. At least we can make /boot authenticated and
>  tamper resistant as a Btrfs subvolume. 
Sorry but this really doesn't compute.
Either you can verify signatures and then it doesn't matter that
someone can tamper with the file system, or you can't and then you
always have evil maid approaches to replace grub with malicious code
anyway. There is no added safety or unsafety about any of these
schemes.

>  I would rather have a multi-stage boot process, where the first
> stage
>  does just enough to unlock and load the OS volume to load the real
>  boot environment. 
I do not see any advantage, loading a kernel is what the first stage
should do, then the kernel+initimage can do everything you need.

Simo.

-- 
Simo Sorce
RHEL Crypto Team
Red Hat, Inc

[Chris Murphy]

On Tue, May 9, 2023, at 12:31 PM, Lennart Poettering wrote:
>  On Di, 09.05.23 08:22, Neal Gompa (ngompa13(a)gmail.com) wrote:
>
> > I've been asked to consider converting /boot to a Btrfs subvolume so
> > that it no longer has a fixed space allocation to deal with the ever
> > increasing amount of firmware required for NVIDIA GPUs[1]. This is
> > currently incompatible with how systemd views the world, because the
> > "discoverable partition spec" is wired to partitions, and there is no
> > equivalent spec for subvolumes of a volume. And I imagine that
> > XBOOTLDR (whatever that is) also would have a problem with this.
>
>  This makese no sense. If you want /boot to just be a subvolume of the
>  rootfs btrfs, then this would imply it's also covered by the same
>  security choices, i.e. encryption. We want to bind that sooner or
>  later to things like TPM2, FIDO2, PKCS11. And that's simply not
>  feasible from a boot loader environment. 
Windows and macOS do this, so it is feasible, the question is what are the consequences of
this and are we willing to live with them? One obvious consequence, it further creates
dependency on a single bootloader, GRUB. Or we need an independent project that provides
an EFI program for unlocking LUKS volumes within the pre-boot environment, thus making
plaintext view available to any bootloader.


>  Hence, the place the kernel is loaded from (regardless if you call
> it
>  /efi or /boot or /boot/efi, and regardless what fs it is) must be
>  accessible from the boot loader easily, without requiring
>  implementation of TPM2/FIDO2/PKCS11 hookup in the boot loader. 
I understand that might be difficult and something we don't want to do for resource
reasons, but there isn't a technical impediment for it.



>  Hence: btrfs subvols won't work for this
>
>  A simple vfat partition however will. 
FAT isn't resizable. The growth requirement for /boot is greater for some use cases
more than others, so there is pressure building because it will create waste for some use
cases and underprovisioning in other use cases. Those unverprovisioned being told they
can't upgrade but need to reprovision to solve this is kindof annoying.


> > Also, as an aside, there is now a "from-scratch" Btrfs EFI driver in
> > development[2] (and for your personal horror, an NTFS one too[3]).
>
>  Not sufficient. You'd also have to implement a LUKS EFI driver, and a
>  TPM2 EFI driver, and a FIDO2 EFI driver, and so on and so
>  on. Basically, you have to reimplement a good chunk of the Linux
>  kernel, of Linux userspace, systemd and so on in EFI mode.
>
>  Good luck with that. 
Right. Hence Linux Boot. Dump all the toy drivers in favor of real ones. And a real user
space.



-- 
Chris Murphy

[Lennart Poettering]

On Di, 09.05.23 15:04, Chris Murphy (lists(a)colorremedies.com) wrote:

>  > This makese no sense. If you want /boot to just be a subvolume
> of the
>  > rootfs btrfs, then this would imply it's also covered by the same
>  > security choices, i.e. encryption. We want to bind that sooner or
>  > later to things like TPM2, FIDO2, PKCS11. And that's simply not
>  > feasible from a boot loader environment.
>
>  Windows and macOS do this, so it is feasible, the question is what
>  are the consequences of this and are we willing to live with them? 
They focus on a much more limited set of functionality.

Also, are you volunteering to implement and maintain a full LUKS2, TPM2,
FIDO2 and PKCS11 stack in UEFI mode? Good luck, man! And it's not
getting any simpler. Next thing coming up is probably PassKey support,
which means networking support. That's going to be fun in UEFI to support!

I mean, these things tend to grow and become more complicated over
time, and if you avoid Linux userspace then you make your life
impossibly hard. And I really don't see Chris Murphy stepping up and
maintaining that mess. Or are you?

As someone who actually occasionally writes UEFI code: ffs, give up on
the idea to reimplement complex auth protocols in UEFI mode! You want
to do *less* stuff in UEFI, not pile on and pile on. Grub's reputation
suffered because they are basically reimplementing a crappy version of
Linux in their boot loader. Get yourself out of that Grub mindset, man!

This idea appears so "out there" to me, I am sorry, but I somehow
can't believe you seriously are proposing this.

>  One obvious consequence, it further creates dependency on a single
>  bootloader, GRUB. Or we need an independent project that provides an
>  EFI program for unlocking LUKS volumes within the pre-boot
>  environment, thus making plaintext view available to any bootloader. 
Sorry, this is *such* *a* *bad* *idea*.

>  > Hence, the place the kernel is loaded from (regardless if you
> call
>  > it /efi or /boot or /boot/efi, and regardless what fs it is) must
>  > be accessible from the boot loader easily, without requiring
>  > implementation of TPM2/FIDO2/PKCS11 hookup in the boot loader.
>
>  I understand that might be difficult and something we don't want to
>  do for resource reasons, but there isn't a technical impediment for
>  it. 
Well, that's true, you can hack anything up that a Turing machine can
execute and also run it in UEFI mode, but I seriously hope that you
are not actually suggesting this.

>  FAT isn't resizable. The growth requirement for /boot is greater
> for
>  some use cases more than others, so there is pressure building
>  because it will create waste for some use cases and
>  underprovisioning in other use cases. Those unverprovisioned being
>  told they can't upgrade but need to reprovision to solve this is
>  kindof annoying. 
If you don't want to resize VFAT and XBOOTLDR is not enough to address
this problem we can relatively easily extend XBOOTLDR to allow more than one
additional such partition we can search through. The code in the boot
loader is relatively straightforward. The limiting factor is more
figuring out where to mount those.

But seriously, you are making up synthetic probems. If ESP is too
small add a large XBOOTLDR and I am pretty sure we'll be fine for a
long long time before this comes an issue again. So long in fact it
might never become.

>  Right. Hence Linux Boot. Dump all the toy drivers in favor of real
>  ones. And a real user space. 
I mean you understand that this just adds another chain to the boot
process? if you do what you are proposing then you need to build a
kernel that supports all that, i.e. all the complex storage, graphics
and so on that you want to boot from, and thus you'll also have alrge
images, and then what did you gain? You ave to put that in the ESP
too, and the size limits still apply. It's illusionary to believe that
the size constraints for a kernel + drivers + complex storage stack +
crypto stack + auth stack would not apply to kernel that just runs in
uefi mode instead of native mode...

Lennart

--
Lennart Poettering, Berlin

[Gerd Hoffmann]

  Hi,

>  > And install kernels to /boot/efi in case /boot is not a
> XBOOTLDR
>  > filesystem?
>  
>  If /boot is not a XBOOTLDR, then we only have one file system which is
>  the ESP. It could be mounted on /boot or on /efi or maybe even /boot/efi (*).
>  The kernels would then go to /boot/EFI/Linux, /efi/EFI/Linux, or /boot/efi/EFI/Linux,
>  respectively. (When you write /boot/efi, it's not clear what exactly you
>  mean. The duplication of "efi" and "EFI" on on case-insensitive
> system
>  is confusing.) 
I meant ESP mounted at /boot/efi (and therefore UKIs in /boot/efi/EFI/Linux).

>  (*) This is actually something that'd need to be figure out.
>  /boot/efi is the worst choice; either /boot or /efi would be OK,
>  but something needs to be chosen. 
/boot/efi is clearly not ideal for a number of reasons, but this is what
we have today and changing this opens up another can of worms.  For
starters this will stop working:

    # rpm -ql shim-x64
    /boot/efi/EFI/BOOT/BOOTX64.EFI
    /boot/efi/EFI/BOOT/fbx64.efi
    /boot/efi/EFI/fedora/BOOTX64.CSV
    [ ... ]

If we are going to change this the only option which makes sense to me
is to go for the systemd default: ESP (if present) mounted at /efi,
XBOOTLDR (if present) mounted at /boot.

take care,
  Gerd

[Zbigniew Jędrzejewski-Szmek]

On Tue, May 09, 2023 at 04:04:33PM +0200, Gerd Hoffmann wrote:
>    Hi,
>  
>  > > And install kernels to /boot/efi in case /boot is not a XBOOTLDR
>  > > filesystem?
>  > 
>  > If /boot is not a XBOOTLDR, then we only have one file system which is
>  > the ESP. It could be mounted on /boot or on /efi or maybe even /boot/efi (*).
>  > The kernels would then go to /boot/EFI/Linux, /efi/EFI/Linux, or
> /boot/efi/EFI/Linux,
>  > respectively. (When you write /boot/efi, it's not clear what exactly you
>  > mean. The duplication of "efi" and "EFI" on on case-insensitive
> system
>  > is confusing.)
>  
>  I meant ESP mounted at /boot/efi (and therefore UKIs in /boot/efi/EFI/Linux). 
OK. I think we want to change this:
- as Lennart mentioned in the other reply, we want automount with
  expiration after a fairly short timeout, and that effectively means
  no nesting. 
- nesting also means that (potentially) we have to modify the upper volume
  to create a directory, at things are ugly if the directory exists but
  is not empty.

Keeping both mount points independent is just nicer.

>  > (*) This is actually something that'd need to be figure
> out.
>  > /boot/efi is the worst choice; either /boot or /efi would be OK,
>  > but something needs to be chosen.
>  
>  /boot/efi is clearly not ideal for a number of reasons, but this is what
>  we have today and changing this opens up another can of worms.  For
>  starters this will stop working:
>  
>      # rpm -ql shim-x64
>      /boot/efi/EFI/BOOT/BOOTX64.EFI
>      /boot/efi/EFI/BOOT/fbx64.efi
>      /boot/efi/EFI/fedora/BOOTX64.CSV 
That's actually an anti-feature that needs to go. Packages should not
use rpm to put files directly in /boot. Systemd doesn't do this, the
kernel doesn't do this (except for %ghost files). grub2 does some
directories, but thankfully no files, and shim must stop too.

Stuff that rpm manages should be under /usr, and scriptlets should
do installation if the config is appropriate. /boot is also subject
to external modifications and management, and rpm just isn't the
right tool for this. Installing files via rpms also makes it hard
to install but not use the package (based on local configuration),
or to consume the package for other things (e.g. just booting vms
or building images, etc.).

>  If we are going to change this the only option which makes sense to
> me
>  is to go for the systemd default: ESP (if present) mounted at /efi,
>  XBOOTLDR (if present) mounted at /boot. 
Yeah. But either choice means that the path harcoding is broken.

Zbyszek

[Gerd Hoffmann]

>  > /boot/efi is clearly not ideal for a number of reasons, but this
> is what
>  > we have today and changing this opens up another can of worms.  For
>  > starters this will stop working:
>  > 
>  >     # rpm -ql shim-x64
>  >     /boot/efi/EFI/BOOT/BOOTX64.EFI
>  >     /boot/efi/EFI/BOOT/fbx64.efi
>  >     /boot/efi/EFI/fedora/BOOTX64.CSV
>  
>  That's actually an anti-feature that needs to go. Packages should not
>  use rpm to put files directly in /boot. Systemd doesn't do this, the
>  kernel doesn't do this (except for %ghost files). grub2 does some
>  directories, but thankfully no files, 
Nope:

    # rpm -ql grub2-efi-x64 | grep EFI
    /boot/efi/EFI/fedora/grub.cfg
    /boot/efi/EFI/fedora/grubx64.efi

Only systemd-boot gets this right today, with files packaged in /usr and
bootctl updating the ESP.  Oh, and fwupd handles it correctly too.

take care,
  Gerd

[Lennart Poettering]

On Di, 09.05.23 12:08, Zbigniew Jędrzejewski-Szmek (zbyszek(a)in.waw.pl) wrote:

>  the ESP. It could be mounted on /boot or on /efi or maybe even
> /boot/efi (*).
>  The kernels would then go to /boot/EFI/Linux, /efi/EFI/Linux, or /boot/efi/EFI/Linux,
>  respectively. (When you write /boot/efi, it's not clear what exactly you
>  mean. The duplication of "efi" and "EFI" on on case-insensitive
> system
>  is confusing.)
>
>  (*) This is actually something that'd need to be figure out.
>  /boot/efi is the worst choice; either /boot or /efi would be OK,
>  but something needs to be chosen. 
I'd strongly advise not to nest them, because that makes mounting them
via autofs (i.e. systemd .automount units) nasty. i.e. /boot/ and
/efi/ are the way to go in my humble opinion.

Given that ESP/XBOOTLDR are likely vfat it's kind crucial to reduce
the time where the partitions are mounted to a minimum, and autofs
makes that very simple and natural, as it means the file systems are
only mounted for a very short period of time when actually accessed,
and unmounted a seconds later.

Lennart

--
Lennart Poettering, Berlin

[Chris Murphy]

On Tue, May 9, 2023, at 8:08 AM, Zbigniew Jędrzejewski-Szmek wrote:
>  On Tue, May 09, 2023 at 12:02:26PM +0200, Gerd Hoffmann wrote:
> >   Hi,
> > 
> > > If we want to change the default here, let's do some proper cleanup:
> > > 1. the split between ESP and XBOOTLDR is only useful in the case where
> > >    ESP already existed and was small. If the installer is *creating*
> > >    an ESP, it should just make it large enough.
> > 
> > And install kernels to /boot/efi in case /boot is not a XBOOTLDR
> > filesystem?
>
>  If /boot is not a XBOOTLDR, then we only have one file system which is
>  the ESP. It could be mounted on /boot or on /efi or maybe even 
>  /boot/efi (*).
>  The kernels would then go to /boot/EFI/Linux, /efi/EFI/Linux, or 
>  /boot/efi/EFI/Linux,
>  respectively. (When you write /boot/efi, it's not clear what exactly you
>  mean. The duplication of "efi" and "EFI" on on case-insensitive
> system
>  is confusing.)
>
>  (*) This is actually something that'd need to be figure out.
>  /boot/efi is the worst choice; either /boot or /efi would be OK,
>  but something needs to be chosen. 
Related but slightly off topic...

What about the increasing growth in linux-firmware and in particular the NVIDIA firmware
requirements? My reading suggests it's significant and the future growth also
significant. There's some preference to put /boot on Btrfs to take advantage of
storage pooling so that we're neither over nor under provisioning the size of /boot.
The problem I have with this is two fold: what about our non-btrfs use cases? Surely those
use cases are equally concerned about this problem? And then what are the impediments to
booting the kernel faster and more quickly getting `/` mounted? Why is there so much
pressure to stuff the firmware blobs into the initramfs or onto /boot in general? Why does
firmware availability need to happen so early during boot, and is it really not possible
to somehow make mounting `/` a higher priority than it has been up until now?

Huge universal initramfs will slow down boot. And it delays mounting `/`. So if there is
some good reason why firmware blobs need to be available soon, it's in direct
opposition to booting fast and that strikes me as a design flaw or need for a feature
request. I just don't know what that could be. But to just keep stuffing more things
into the initramfs doesn't scale either.


Back to the original topic:

ESP and XBOOTLDR are not user domain, should have no user facing configuration in a GUI at
all. It's irrelevant esoteric knowledge that 99% of our users do not need to worry
about. By extension, they don't belong in /etc/fstab nor should they be persistently
mounted. Whether one or the other are temporarily mounted to /boot, /efi, /boot/efi, is up
to the developers creating tools that modify these volumes. I prefer that they are mounted
to a pseudo-random location in /run but I don't really care.

NOTE: We can apply SELinux labels to FAT file systems by using 'mount -o context='
mount option. The limitation is the label applies to that entire mount point, you
don't get per directory labels, it's a one size fits all.

Recent Windows 10/11 images on microsoft.com within the last year produce a 100M EFI
System partition per:
https://learn.microsoft.com/en-us/windows-hardware/manufacture/desktop/co...

I have reinstalled Windows 10 and 11 using the microsoft.com procured installer as of a
year ago and it likewise creates a 100M ESP. Maybe Microsoft didn't get the memo and
the space requirement is really something the OEM's are concerned about? So I expect
this problem is not only our problem to solve.


-- 
Chris Murphy

[Chris Adams]

Once upon a time, Chris Murphy <lists(a)colorremedies.com&gt; said:
>  What about the increasing growth in linux-firmware and in particular
> the NVIDIA firmware requirements? My reading suggests it's significant and the future
> growth also significant. 
Could we use a dumb framebuffer in initrd and get rid of all the GPU
firmware from the image?

-- 
Chris Adams <linux(a)cmadams.net&gt;

[Zbigniew Jędrzejewski-Szmek]

On Tue, May 09, 2023 at 01:31:01PM -0500, Chris Adams wrote:
>  Once upon a time, Chris Murphy <lists(a)colorremedies.com&gt; said:
>  > What about the increasing growth in linux-firmware and in particular the NVIDIA
> firmware requirements? My reading suggests it's significant and the future growth also
> significant.
>  
>  Could we use a dumb framebuffer in initrd and get rid of all the GPU
>  firmware from the image? 
Maybe, probably, who knows… But it's not just the video. The pressure
to add more stuff and more drivers will only grow: bluetooth for keyboards
and FIDO2, sound support for voice assistance, network for remote attestation
or clevis, etc. We can push this can down the road, but it seems we need
to be ready to add move stuff before root is available anyway.

Zbyszek

[Demi Marie Obenour]

On 5/9/23 14:47, Zbigniew Jędrzejewski-Szmek wrote:
>  On Tue, May 09, 2023 at 01:31:01PM -0500, Chris Adams wrote:
> > Once upon a time, Chris Murphy <lists(a)colorremedies.com&gt; said:
> >> What about the increasing growth in linux-firmware and in particular the NVIDIA
> firmware requirements? My reading suggests it's significant and the future growth also
> significant.
> >
> > Could we use a dumb framebuffer in initrd and get rid of all the GPU
> > firmware from the image?
>  
>  Maybe, probably, who knows… But it's not just the video. The pressure
>  to add more stuff and more drivers will only grow: bluetooth for keyboards
>  and FIDO2, sound support for voice assistance, network for remote attestation
>  or clevis, etc. We can push this can down the road, but it seems we need
>  to be ready to add move stuff before root is available anyway.
>  
>  Zbyszek 
I don’t think putting more and more in the initramfs is a good
idea.  I would much rather have a dm-verity protected partition
for early boot stuff, which then uses pivot_root() to switch to
the main system.
-- 
Sincerely,
Demi Marie Obenour (she/her/hers)

[Chris Murphy]

On Tue, May 9, 2023, at 2:47 PM, Zbigniew Jędrzejewski-Szmek wrote:
>  On Tue, May 09, 2023 at 01:31:01PM -0500, Chris Adams wrote:
> > Once upon a time, Chris Murphy <lists(a)colorremedies.com&gt; said:
> > > What about the increasing growth in linux-firmware and in particular the NVIDIA
> firmware requirements? My reading suggests it's significant and the future growth also
> significant.
> > 
> > Could we use a dumb framebuffer in initrd and get rid of all the GPU
> > firmware from the image?
>
>  Maybe, probably, who knows… But it's not just the video. The pressure
>  to add more stuff and more drivers will only grow: bluetooth for keyboards
>  and FIDO2, sound support for voice assistance, network for remote attestation
>  or clevis, etc. We can push this can down the road, but it seems we need
>  to be ready to add move stuff before root is available anyway. 
I still think we need less kernel and initramfs in order to get more by having `/`
available faster. Fast enough that the user isn't looking for or expecting
interactivity in the few seconds it should take to get to `/` being mounted.

Otherwise, next up will be embeding GNOME Shell into the initramfs because we're tired
of waiting for a real a11y and i18n environment to be available so we can prompt the user
for an encryption passphrase with appropriate rules, fonts, locale support, etc.

I can conjure up some use cases for stuffing Firefox in the initramfs. Maybe we should
just put /usr into the initramfs.

We'll just all accept 20+ second linux+initrd times to accommodate everyone getting
through the elevator door simultaneously. No problem.


-- 
Chris Murphy

[Lennart Poettering]

On Di, 09.05.23 15:22, Chris Murphy (lists(a)colorremedies.com) wrote:

>  On Tue, May 9, 2023, at 2:47 PM, Zbigniew Jędrzejewski-Szmek wrote:
>  > On Tue, May 09, 2023 at 01:31:01PM -0500, Chris Adams wrote:
>  >> Once upon a time, Chris Murphy <lists(a)colorremedies.com&gt; said:
>  >> > What about the increasing growth in linux-firmware and in particular the
> NVIDIA firmware requirements? My reading suggests it's significant and the future
> growth also significant.
>  >>
>  >> Could we use a dumb framebuffer in initrd and get rid of all the GPU
>  >> firmware from the image?
>  >
>  > Maybe, probably, who knows… But it's not just the video. The pressure
>  > to add more stuff and more drivers will only grow: bluetooth for keyboards
>  > and FIDO2, sound support for voice assistance, network for remote attestation
>  > or clevis, etc. We can push this can down the road, but it seems we need
>  > to be ready to add move stuff before root is available anyway.
>
>  I still think we need less kernel and initramfs in order to get more
>  by having `/` available faster. Fast enough that the user isn't
>  looking for or expecting interactivity in the few seconds it should
>  take to get to `/` being mounted. 
Aren't you the one who just proposed LinuxBoot, i.e. having one kernel
with initrd that includes complex storage, crypto, drivers, auth and
things chainload a second kernel with initrd that includes all that?

If you want fast boots and minimal disk space usage, then maybe have
tiny boot loader and a single UKI element after that and not play
games of setting up complex storage to load a secondary kernel that
then has to set up complex storage again.

You are arguing here into two opposing directions. One one hand you
want to chainload multiple kernels+initrd (claiming this was a good idea out
of the problem of sizing ESP/XBOOTLDR sufficientlly), and then on the
other hand you claim there's too much kernel+initrd in the boot
process?

What is it now? Pick one: more initrd or less initrd?

Lennart

--
Lennart Poettering, Berlin

[Chris Murphy]

On Wed, May 10, 2023, at 9:42 AM, Lennart Poettering wrote:

>  You are arguing here into two opposing directions. One one hand you
>  want to chainload multiple kernels+initrd (claiming this was a good idea out
>  of the problem of sizing ESP/XBOOTLDR sufficientlly), and then on the
>  other hand you claim there's too much kernel+initrd in the boot
>  process?
>
>  What is it now? Pick one: more initrd or less initrd? 
I've only ever said I want faster boot and smaller initrd. Everything else is pointing
out the consequences of alternatives, not advocacy of them.


-- 
Chris Murphy

[Owen Taylor]

On Tue, May 9, 2023 at 2:31 PM Chris Adams <linux(a)cmadams.net&gt; wrote:

>  Once upon a time, Chris Murphy <lists(a)colorremedies.com&gt; said:
>  > What about the increasing growth in linux-firmware and in particular the
>  NVIDIA firmware requirements? My reading suggests it's significant and the
>  future growth also significant.
>
>  Could we use a dumb framebuffer in initrd and get rid of all the GPU
>  firmware from the image?
>  
The main concern about using efifb during the initrd is handling situations
like:

 - Laptop has the lid closed
 - External monitor is connected to a HDMI port that is only connected to
the discrete GPU or otherwise not lit up by EFI
 - We need to prompt for the passphrase to unlock the root partition

It *mostly* would work fine - and maybe we're OK with that (especially if
we can reduce interaction with the initrd).

- Owen

[Richard Hughes]

On Tue, 9 May 2023 at 10:22, Zbigniew Jędrzejewski-Szmek
<zbyszek(a)in.waw.pl&gt; wrote:
>  This is both too much and not enough. 
Right; and I think a different Fedora feature proposal would be a good
idea for the version of Fedora when we switch to UKIs. Where we can
just have /boot/efi and not /boot -- but that's not what this proposal
is about. This is primarily to keep firmware updates working, and as a
secondary measure maybe more people can test UKIs. I don't want this
simple feature of "unbreak future firmware updates" to become
"completely rework how we boot the system".

>  (Point 2. is not really *necessary* for the size changes, but
> it'd be
>  nice to get rid of this anachronism if this area is being touched.) 
I think that's fine -- but I don't think the onus should be on me to
push it through. Like I said to Lennart, if you want to do a Fedora
proposal to rework all this stuff to remove /boot that's fine, but
please don't hijack this one and make me responsible for doing the
work.

Richard

[Zbigniew Jędrzejewski-Szmek]

On Tue, May 09, 2023 at 01:20:54PM +0100, Richard Hughes wrote:
>  On Tue, 9 May 2023 at 10:22, Zbigniew Jędrzejewski-Szmek
>  <zbyszek(a)in.waw.pl&gt; wrote:
>  > This is both too much and not enough.
>  
>  Right; and I think a different Fedora feature proposal would be a good
>  idea for the version of Fedora when we switch to UKIs. Where we can
>  just have /boot/efi and not /boot -- but that's not what this proposal
>  is about. This is primarily to keep firmware updates working, and as a
>  secondary measure maybe more people can test UKIs. I don't want this
>  simple feature of "unbreak future firmware updates" to become
>  "completely rework how we boot the system".
>  
>  > (Point 2. is not really *necessary* for the size changes, but it'd be
>  > nice to get rid of this anachronism if this area is being touched.)
>  
>  I think that's fine -- but I don't think the onus should be on me to
>  push it through. Like I said to Lennart, if you want to do a Fedora
>  proposal to rework all this stuff to remove /boot that's fine, but
>  please don't hijack this one and make me responsible for doing the
>  work. 
I don't see reply to Lennart anywhere; did you maybe not send to the list?

I wouldn't want to force you (or anyone else) to take on huge tasks.
Especially not by making some small change contingent on moonshot proposals.
But I think that a) the current proposal is just a band-aid, and
b) to make things better we don't need to make huge changes.
And c), there is a real cost to doing a band-aid solution now and
starting another solution in a slightly different direction immediately
after. The layout of partitions generally remains unchanged over the
lifetime of installations, so if we introduce some new layout, we'll have
to make it work over the next 10 years. Every time we introduce a
new scheme, we introduce one more combination that'll need to be
supperted.

To expand on b): we don't really to do anything *new*. It's mostly
about stopping to do things or changing some setting from one value
to another. In particular, Anaconda needs to *lose* a feature where it
refuses VFAT for /boot [1], the various places which create partitions
need to be modified to inject the right partition-type UUID instead of
made-up one, and instead of creating two partitions with different fs
types, create just one. None of this is rocket science. I don't
understand why this needs to be dragged out over multiple years. If
we're already touching this code, it would be really great to make
some real progress, instead of doing the minimal thing that delivers
minimal progress.

[1] https://bugzilla.redhat.com/show_bug.cgi?id=2106706

Zbyszek

[Richard Hughes]

On Wed, 10 May 2023 at 11:55, Zbigniew Jędrzejewski-Szmek
<zbyszek(a)in.waw.pl&gt; wrote:
>  Especially not by making some small change contingent on moonshot
> proposals.
>  But I think that a) the current proposal is just a band-aid, and
>  b) to make things better we don't need to make huge changes. 
Okay, please open a _new_ change proposal with what you want to
happen: expanding the scope like you suggest feels like me getting
tricked to work on your much bigger change that's not terribly
well-defined or tested.

>  ...Anaconda needs to *lose* a feature where it
>  refuses VFAT for /boot [1], the various places which create partitions
>  need to be modified to inject the right partition-type UUID instead of
>  made-up one, and instead of creating two partitions with different fs
>  types, create just one. None of this is rocket science. 
Perhaps you'd like to push this feature. I'd happily retract my
smaller proposal if you've got patches ready for anaconda, have tested
this on all platforms, and have all the votes.

Richard.

[Zbigniew Jędrzejewski-Szmek]

On Wed, May 10, 2023 at 02:25:07PM +0100, Richard Hughes wrote:
>  On Wed, 10 May 2023 at 11:55, Zbigniew Jędrzejewski-Szmek
>  <zbyszek(a)in.waw.pl&gt; wrote:
>  > Especially not by making some small change contingent on moonshot proposals.
>  > But I think that a) the current proposal is just a band-aid, and
>  > b) to make things better we don't need to make huge changes.
>  
>  Okay, please open a _new_ change proposal with what you want to
>  happen: expanding the scope like you suggest feels like me getting
>  tricked to work on your much bigger change that's not terribly
>  well-defined or tested.
>  
>  > ...Anaconda needs to *lose* a feature where it
>  > refuses VFAT for /boot [1], the various places which create partitions
>  > need to be modified to inject the right partition-type UUID instead of
>  > made-up one, and instead of creating two partitions with different fs
>  > types, create just one. None of this is rocket science.
>  
>  Perhaps you'd like to push this feature. I'd happily retract my
>  smaller proposal if you've got patches ready for anaconda, have tested
>  this on all platforms, and have all the votes. 
I'm adding some folks who made some recent Anaconda commits in CC.
For reference, the mail I'm replying to is at [1], and my earlier
proposal is at [2].

Would you be open to modifying Anaconda like this?
I think the code changes would be fairly easy, and I would be happy
to prepare a patch, but I don't want to do this if there's no chance
of it being accepted.

[1]
https://lists.fedoraproject.org/archives/list/[e-mail].o...
[2]
https://lists.fedoraproject.org/archives/list/[e-mail].o...

Zbyszek

[Vladimir Slavik]

Hello!
There are two things to consider, both mean that I can't really ever say
"yes we will do it" for a request like that.

- None of the anaconda devs are qualified enough to decide if we could
accept such a change. It's a bootloader thing. What we would do is delegate
to somebody working with bootloaders to see if it's technically viable.
Likely that would be Peter Jones (CCed).

- Anaconda is intertwined with Fedora, so there would have to be a change
process and all that. The current audience in this thread is likely the
same as you'd get for the hypothetical change, anyway...

Hope that helps?

Best,
Vladimir

On Thu, May 11, 2023 at 12:34 PM Zbigniew Jędrzejewski-Szmek <
zbyszek(a)in.waw.pl&gt; wrote:

>  On Wed, May 10, 2023 at 02:25:07PM +0100, Richard Hughes wrote:
>  > On Wed, 10 May 2023 at 11:55, Zbigniew Jędrzejewski-Szmek
>  > <zbyszek(a)in.waw.pl&gt; wrote:
>  > > Especially not by making some small change contingent on moonshot
>  proposals.
>  > > But I think that a) the current proposal is just a band-aid, and
>  > > b) to make things better we don't need to make huge changes.
>  >
>  > Okay, please open a _new_ change proposal with what you want to
>  > happen: expanding the scope like you suggest feels like me getting
>  > tricked to work on your much bigger change that's not terribly
>  > well-defined or tested.
>  >
>  > > ...Anaconda needs to *lose* a feature where it
>  > > refuses VFAT for /boot [1], the various places which create partitions
>  > > need to be modified to inject the right partition-type UUID instead of
>  > > made-up one, and instead of creating two partitions with different fs
>  > > types, create just one. None of this is rocket science.
>  >
>  > Perhaps you'd like to push this feature. I'd happily retract my
>  > smaller proposal if you've got patches ready for anaconda, have tested
>  > this on all platforms, and have all the votes.
>
>  I'm adding some folks who made some recent Anaconda commits in CC.
>  For reference, the mail I'm replying to is at [1], and my earlier
>  proposal is at [2].
>
>  Would you be open to modifying Anaconda like this?
>  I think the code changes would be fairly easy, and I would be happy
>  to prepare a patch, but I don't want to do this if there's no chance
>  of it being accepted.
>
>  [1]
>
> https://lists.fedoraproject.org/archives/list/[e-mail].o...
>  [2]
>
> https://lists.fedoraproject.org/archives/list/[e-mail].o...
>
>  Zbyszek
>  _______________________________________________
>  devel mailing list -- devel(a)lists.fedoraproject.org
>  To unsubscribe send an email to devel-leave(a)lists.fedoraproject.org
>  Fedora Code of Conduct:
>  https://docs.fedoraproject.org/en-US/project/code-of-conduct/
>  List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
>  List Archives:
>  https://lists.fedoraproject.org/archives/list/[e-mail]
>  Do not reply to spam, report it:
>  https://pagure.io/fedora-infrastructure/new_issue
>  

-- 
Vladimír Slávik <vslavik(a)redhat.com&gt;
Software Engineer, Platform Engineering
Red Hat Czech, s.r.o.

[Lennart Poettering]

On Di, 09.05.23 09:20, Zbigniew Jędrzejewski-Szmek (zbyszek(a)in.waw.pl) wrote:

>  > == Summary ==
>  >
>  > This change will increase the minimum size of the ESP to be 500MB,
>  > which is also the same value used by Microsoft for Windows 10 and
>  > newer.
>
>  This is both too much and not enough. Essentially, this grows the ESP,
>  but also leaves the XBOOTLDR partition large. Overall, the users pays
>  twice, and on some systems 1.5GB is not insignificant. OTOH, 500 or
>  512 MB seems not enough: three big UKIs and a rescue kernel and and
>  some Windows blobs and a firmware update would likely overflow.
>
>  If we want to change the default here, let's do some proper cleanup:
>  1. the split between ESP and XBOOTLDR is only useful in the case where
>     ESP already existed and was small. If the installer is *creating*
>     an ESP, it should just make it large enough.
>  2. having a second partition with a second (different) file system
>     implementation just increases the footprint and attack surface for
>     no gain. If we create XBOOTLDR, make it like the ESP (i.e. VFAT
>     in almost all realistic scenarios).
>  3. if there are bootloaders that don't read one or the other partition
>     as they should, fix them.
>
>  Then we can make the ESP 1 GB *and* save space compared to current
>  defaults.
>
>  (Point 2. is not really *necessary* for the size changes, but it'd be
>  nice to get rid of this anachronism if this area is being touched.) 
I guess this might not be surprising, but this proposal makes a ton of
sense to me and has my full support, FWTW

Lennart

--
Lennart Poettering, Berlin

[Simo Sorce]

On Tue, 2023-05-09 at 18:32 +0200, Lennart Poettering wrote:
>  On Di, 09.05.23 09:20, Zbigniew Jędrzejewski-Szmek
> (zbyszek(a)in.waw.pl) wrote:
>  
>  > > == Summary ==
>  > > 
>  > > This change will increase the minimum size of the ESP to be 500MB,
>  > > which is also the same value used by Microsoft for Windows 10 and
>  > > newer.
>  > 
>  > This is both too much and not enough. Essentially, this grows the ESP,
>  > but also leaves the XBOOTLDR partition large. Overall, the users pays
>  > twice, and on some systems 1.5GB is not insignificant. OTOH, 500 or
>  > 512 MB seems not enough: three big UKIs and a rescue kernel and and
>  > some Windows blobs and a firmware update would likely overflow.
>  > 
>  > If we want to change the default here, let's do some proper cleanup:
>  > 1. the split between ESP and XBOOTLDR is only useful in the case where
>  >    ESP already existed and was small. If the installer is *creating*
>  >    an ESP, it should just make it large enough.
>  > 2. having a second partition with a second (different) file system
>  >    implementation just increases the footprint and attack surface for
>  >    no gain. If we create XBOOTLDR, make it like the ESP (i.e. VFAT
>  >    in almost all realistic scenarios).
>  > 3. if there are bootloaders that don't read one or the other partition
>  >    as they should, fix them.
>  > 
>  > Then we can make the ESP 1 GB *and* save space compared to current
>  > defaults.
>  > 
>  > (Point 2. is not really *necessary* for the size changes, but it'd be
>  > nice to get rid of this anachronism if this area is being touched.)
>  
>  I guess this might not be surprising, but this proposal makes a ton of
>  sense to me and has my full support, FWTW 
It sounds reasonable for sure.
The only concern is, given Microsoft creates at most 500MB ESP
partitions, are we sure all UEFI systems out there will not choke on a
1GB one?

Can't we reduce the number of kernels by having *only* one UKI and a
rescue one that can be used to restore the previous working UKI from
/root if the active one fails?

Or perhaps just have always 2 UKI (current, and former working).
Do we actually need a separate dedicated rescue UKI? Can't rescue be
implemented by booting the previously working UKI with a "rescue"
command line option ?

Simo.

-- 
Simo Sorce
RHEL Crypto Team
Red Hat, Inc

[Luca Boccassi]

>  On Tue, 2023-05-09 at 18:32 +0200, Lennart Poettering wrote:
>  
>  It sounds reasonable for sure.
>  The only concern is, given Microsoft creates at most 500MB ESP
>  partitions, are we sure all UEFI systems out there will not choke on a
>  1GB one?
>  
>  Can't we reduce the number of kernels by having *only* one UKI and a
>  rescue one that can be used to restore the previous working UKI from
>  /root if the active one fails?
>  
>  Or perhaps just have always 2 UKI (current, and former working).
>  Do we actually need a separate dedicated rescue UKI? Can't rescue be
>  implemented by booting the previously working UKI with a "rescue"
>  command line option ? 
Word of caution on 'rescue' images: MSFT just had to essentially render 10 years
of recovery/install media unbootable due to the black lotus vulnerability. It was not (and
still is not) pretty.

When there's signatures and verifications involved, you really want an upgradable
system. But if you set that whole infrastructure up, there's really not that much
difference left with an A/B scheme.

[Daniel P. Berrangé]

On Wed, May 10, 2023 at 03:52:51PM -0000, Luca Boccassi wrote:
>  > On Tue, 2023-05-09 at 18:32 +0200, Lennart Poettering wrote:
>  > 
>  > It sounds reasonable for sure.
>  > The only concern is, given Microsoft creates at most 500MB ESP
>  > partitions, are we sure all UEFI systems out there will not choke on a
>  > 1GB one?
>  > 
>  > Can't we reduce the number of kernels by having *only* one UKI and a
>  > rescue one that can be used to restore the previous working UKI from
>  > /root if the active one fails?
>  > 
>  > Or perhaps just have always 2 UKI (current, and former working).
>  > Do we actually need a separate dedicated rescue UKI? Can't rescue be
>  > implemented by booting the previously working UKI with a "rescue"
>  > command line option ?
>  
>  Word of caution on 'rescue' images: MSFT just had to essentially
>  render 10 years of recovery/install media unbootable due to the
>  black lotus vulnerability. It was not (and still is not) pretty.
>  
>  When there's signatures and verifications involved, you really
>  want an upgradable system. But if you set that whole infrastructure
>  up, there's really not that much difference left with an A/B scheme. 
If the idea to allow a UKI to contain multiple alternate, signed,
cmdline line profiles gets accepted [1], then a "rescue" image
won't neccessarily need to be a separate image anymore. There could
just be an alternative cmdline that caused the initrd to launch in
a "rescue" / "safe" mode, and that would be nicely complemented by
an A/B scheme to cope with bad kernel upgrades.

With regards,
Daniel

[1] https://github.com/systemd/systemd/issues/24539
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

[Chris Adams]

Once upon a time, Daniel P. Berrangé <berrange(a)redhat.com&gt; said:
>  If the idea to allow a UKI to contain multiple alternate, signed,
>  cmdline line profiles gets accepted 
Are those of us who need boot-time kernel options (e.g. for hardware
workarounds or such) just screwed in the signed command-line cases?

-- 
Chris Adams <linux(a)cmadams.net&gt;

[Daniel P. Berrangé]

On Wed, May 10, 2023 at 11:37:34AM -0500, Chris Adams wrote:
>  Once upon a time, Daniel P. Berrangé <berrange(a)redhat.com&gt;
> said:
>  > If the idea to allow a UKI to contain multiple alternate, signed,
>  > cmdline line profiles gets accepted
>  
>  Are those of us who need boot-time kernel options (e.g. for hardware
>  workarounds or such) just screwed in the signed command-line cases? 
Today yes, but in future no. There's very active ongoing discussion
and coding to sort out how to securely allow local customization
of kernel command lines. Note, I said "local customization", I didn't
say "arbitrary interactive override at boot time". IOW, the way it is
achieved will probably look different to what we're used to historically.

Some form of local deployment level customization is clearly critical if
UKIs are to be viable beyond tightly constrained deployment scenarios.
Hardware specific workarounds is one use case that needs to be supported.

If you want to learn more, the systemd ticket I linked to in my mail
has all the discussion (sorry, a very long read, I know) and links to
pull requests where relevant.

With regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

[Luca Boccassi]

>  On Wed, May 10, 2023 at 03:52:51PM -0000, Luca Boccassi wrote:
>  
>  If the idea to allow a UKI to contain multiple alternate, signed,
>  cmdline line profiles gets accepted [1], then a "rescue" image
>  won't neccessarily need to be a separate image anymore. There could
>  just be an alternative cmdline that caused the initrd to launch in
>  a "rescue" / "safe" mode, and that would be nicely complemented by
>  an A/B scheme to cope with bad kernel upgrades.
>  
>  With regards,
>  Daniel
>  
>  [1] https://github.com/systemd/systemd/issues/24539 
Yes, we should definitely have that. Had started scoping how to build it and encode it,
but got distracted by other squirrels.

[Neal Gompa]

On Wed, May 10, 2023 at 11:21 AM Simo Sorce <simo(a)redhat.com&gt; wrote:
>  On Tue, 2023-05-09 at 18:32 +0200, Lennart Poettering wrote:
>  > On Di, 09.05.23 09:20, Zbigniew Jędrzejewski-Szmek (zbyszek(a)in.waw.pl) wrote:
>  >
>  > > > == Summary ==
>  > > >
>  > > > This change will increase the minimum size of the ESP to be 500MB,
>  > > > which is also the same value used by Microsoft for Windows 10 and
>  > > > newer.
>  > >
>  > > This is both too much and not enough. Essentially, this grows the ESP,
>  > > but also leaves the XBOOTLDR partition large. Overall, the users pays
>  > > twice, and on some systems 1.5GB is not insignificant. OTOH, 500 or
>  > > 512 MB seems not enough: three big UKIs and a rescue kernel and and
>  > > some Windows blobs and a firmware update would likely overflow.
>  > >
>  > > If we want to change the default here, let's do some proper cleanup:
>  > > 1. the split between ESP and XBOOTLDR is only useful in the case where
>  > >    ESP already existed and was small. If the installer is *creating*
>  > >    an ESP, it should just make it large enough.
>  > > 2. having a second partition with a second (different) file system
>  > >    implementation just increases the footprint and attack surface for
>  > >    no gain. If we create XBOOTLDR, make it like the ESP (i.e. VFAT
>  > >    in almost all realistic scenarios).
>  > > 3. if there are bootloaders that don't read one or the other partition
>  > >    as they should, fix them.
>  > >
>  > > Then we can make the ESP 1 GB *and* save space compared to current
>  > > defaults.
>  > >
>  > > (Point 2. is not really *necessary* for the size changes, but it'd be
>  > > nice to get rid of this anachronism if this area is being touched.)
>  >
>  > I guess this might not be surprising, but this proposal makes a ton of
>  > sense to me and has my full support, FWTW
>
>  It sounds reasonable for sure.
>  The only concern is, given Microsoft creates at most 500MB ESP
>  partitions, are we sure all UEFI systems out there will not choke on a
>  1GB one?
>
>  Can't we reduce the number of kernels by having *only* one UKI and a
>  rescue one that can be used to restore the previous working UKI from
>  /root if the active one fails? 
At least in the upstream kiwi project, we encountered problems making
bigger ESPs because not all UEFI implementations handle FAT32 (despite
it being part of the spec). In particular, there were a few server
boards and especially AWS EC2 that couldn't handle it.

Reference:
https://github.com/OSInside/kiwi/commit/4b17aaa8b545260bf26ab081d10dfb6a6...

I would be wary of issues like this.




--
真実はいつも一つ！/ Always, there's only one truth!

[Gary Buhrmaster]

On Wed, May 10, 2023 at 3:56 PM Neal Gompa <ngompa13(a)gmail.com&gt; wrote:

>  At least in the upstream kiwi project, we encountered problems
> making
>  bigger ESPs because not all UEFI implementations handle FAT32 (despite
>  it being part of the spec). In particular, there were a few server
>  boards and especially AWS EC2 that couldn't handle it. 
As I recall, modern FAT16 implementations (and yes, not
all UEFI implementations might support modern FAT16)
support a 4GB disk size, which is more than the proposed
1GB size.

[Zbigniew Jędrzejewski-Szmek]

On Wed, May 10, 2023 at 11:54:50AM -0400, Neal Gompa wrote:
>  On Wed, May 10, 2023 at 11:21 AM Simo Sorce <simo(a)redhat.com&gt;
> wrote:
>  >
>  > On Tue, 2023-05-09 at 18:32 +0200, Lennart Poettering wrote:
>  > > On Di, 09.05.23 09:20, Zbigniew Jędrzejewski-Szmek (zbyszek(a)in.waw.pl) wrote:
>  > >
>  > > > > == Summary ==
>  > > > >
>  > > > > This change will increase the minimum size of the ESP to be 500MB,
>  > > > > which is also the same value used by Microsoft for Windows 10 and
>  > > > > newer.
>  > > >
>  > > > This is both too much and not enough. Essentially, this grows the ESP,
>  > > > but also leaves the XBOOTLDR partition large. Overall, the users pays
>  > > > twice, and on some systems 1.5GB is not insignificant. OTOH, 500 or
>  > > > 512 MB seems not enough: three big UKIs and a rescue kernel and and
>  > > > some Windows blobs and a firmware update would likely overflow.
>  > > >
>  > > > If we want to change the default here, let's do some proper cleanup:
>  > > > 1. the split between ESP and XBOOTLDR is only useful in the case where
>  > > >    ESP already existed and was small. If the installer is *creating*
>  > > >    an ESP, it should just make it large enough.
>  > > > 2. having a second partition with a second (different) file system
>  > > >    implementation just increases the footprint and attack surface for
>  > > >    no gain. If we create XBOOTLDR, make it like the ESP (i.e. VFAT
>  > > >    in almost all realistic scenarios).
>  > > > 3. if there are bootloaders that don't read one or the other
> partition
>  > > >    as they should, fix them.
>  > > >
>  > > > Then we can make the ESP 1 GB *and* save space compared to current
>  > > > defaults.
>  > > >
>  > > > (Point 2. is not really *necessary* for the size changes, but it'd be
>  > > > nice to get rid of this anachronism if this area is being touched.)
>  > >
>  > > I guess this might not be surprising, but this proposal makes a ton of
>  > > sense to me and has my full support, FWTW
>  >
>  > It sounds reasonable for sure.
>  > The only concern is, given Microsoft creates at most 500MB ESP
>  > partitions, are we sure all UEFI systems out there will not choke on a
>  > 1GB one?
>  >
>  > Can't we reduce the number of kernels by having *only* one UKI and a
>  > rescue one that can be used to restore the previous working UKI from
>  > /root if the active one fails?
>  
>  At least in the upstream kiwi project, we encountered problems making
>  bigger ESPs because not all UEFI implementations handle FAT32 (despite
>  it being part of the spec). In particular, there were a few server
>  boards and especially AWS EC2 that couldn't handle it.
>  
>  Reference:
> https://github.com/OSInside/kiwi/commit/4b17aaa8b545260bf26ab081d10dfb6a6... 
Are there any more details about this? This commit just does a revert
and doesn't reference any discussion.

Zbyszek

[Neal Gompa]

On Thu, May 11, 2023 at 3:26 AM Zbigniew Jędrzejewski-Szmek
<zbyszek(a)in.waw.pl&gt; wrote:
>  On Wed, May 10, 2023 at 11:54:50AM -0400, Neal Gompa wrote:
>  > On Wed, May 10, 2023 at 11:21 AM Simo Sorce <simo(a)redhat.com&gt; wrote:
>  > >
>  > > On Tue, 2023-05-09 at 18:32 +0200, Lennart Poettering wrote:
>  > > > On Di, 09.05.23 09:20, Zbigniew Jędrzejewski-Szmek (zbyszek(a)in.waw.pl)
> wrote:
>  > > >
>  > > > > > == Summary ==
>  > > > > >
>  > > > > > This change will increase the minimum size of the ESP to be
> 500MB,
>  > > > > > which is also the same value used by Microsoft for Windows 10
> and
>  > > > > > newer.
>  > > > >
>  > > > > This is both too much and not enough. Essentially, this grows the
> ESP,
>  > > > > but also leaves the XBOOTLDR partition large. Overall, the users
> pays
>  > > > > twice, and on some systems 1.5GB is not insignificant. OTOH, 500 or
>  > > > > 512 MB seems not enough: three big UKIs and a rescue kernel and and
>  > > > > some Windows blobs and a firmware update would likely overflow.
>  > > > >
>  > > > > If we want to change the default here, let's do some proper
> cleanup:
>  > > > > 1. the split between ESP and XBOOTLDR is only useful in the case
> where
>  > > > >    ESP already existed and was small. If the installer is *creating*
>  > > > >    an ESP, it should just make it large enough.
>  > > > > 2. having a second partition with a second (different) file system
>  > > > >    implementation just increases the footprint and attack surface
> for
>  > > > >    no gain. If we create XBOOTLDR, make it like the ESP (i.e. VFAT
>  > > > >    in almost all realistic scenarios).
>  > > > > 3. if there are bootloaders that don't read one or the other
> partition
>  > > > >    as they should, fix them.
>  > > > >
>  > > > > Then we can make the ESP 1 GB *and* save space compared to current
>  > > > > defaults.
>  > > > >
>  > > > > (Point 2. is not really *necessary* for the size changes, but
> it'd be
>  > > > > nice to get rid of this anachronism if this area is being touched.)
>  > > >
>  > > > I guess this might not be surprising, but this proposal makes a ton of
>  > > > sense to me and has my full support, FWTW
>  > >
>  > > It sounds reasonable for sure.
>  > > The only concern is, given Microsoft creates at most 500MB ESP
>  > > partitions, are we sure all UEFI systems out there will not choke on a
>  > > 1GB one?
>  > >
>  > > Can't we reduce the number of kernels by having *only* one UKI and a
>  > > rescue one that can be used to restore the previous working UKI from
>  > > /root if the active one fails?
>  >
>  > At least in the upstream kiwi project, we encountered problems making
>  > bigger ESPs because not all UEFI implementations handle FAT32 (despite
>  > it being part of the spec). In particular, there were a few server
>  > boards and especially AWS EC2 that couldn't handle it.
>  >
>  > Reference:
> https://github.com/OSInside/kiwi/commit/4b17aaa8b545260bf26ab081d10dfb6a6...
>
>  Are there any more details about this? This commit just does a revert
>  and doesn't reference any discussion.
>  
It was discussed in the project's Matrix room, if you want more
details, you probably will need to ask him there.



-- 
真実はいつも一つ！/ Always, there's only one truth!

[Lennart Poettering]

On Mi, 10.05.23 11:20, Simo Sorce (simo(a)redhat.com) wrote:

>  It sounds reasonable for sure.
>  The only concern is, given Microsoft creates at most 500MB ESP
>  partitions, are we sure all UEFI systems out there will not choke on a
>  1GB one? 
Well, I don't really think we have a reason to believe that a 1G ESP
was any more problematic than a 0.1G ESP. I am not aware of any
reports, and given that FAT32 is mandated by UEFI since basically
always, I think there's no immediate reason to believe we are in
trouble.

I think the only reasonable approach here is to pick a larger default
in a development distro, and collect feedback.

>  Can't we reduce the number of kernels by having *only* one UKI
> and a
>  rescue one that can be used to restore the previous working UKI from
>  /root if the active one fails? 
I'd kill the rescue concept as a separate kernel. Pre-compiled UKIs
provided by Fedora should be comprehensive and suitable enough to be
rescue images, I don't see why we need a second version of that. The
only difference between a rescue boot and a regular boot should be one
of parameterization, not of picking different kernel.

Lennart

--
Lennart Poettering, Berlin

[Simo Sorce]

On Wed, 2023-05-10 at 18:46 +0200, Lennart Poettering wrote:
>  On Mi, 10.05.23 11:20, Simo Sorce (simo(a)redhat.com) wrote:
>  
>  > It sounds reasonable for sure.
>  > The only concern is, given Microsoft creates at most 500MB ESP
>  > partitions, are we sure all UEFI systems out there will not choke on a
>  > 1GB one?
>  
>  Well, I don't really think we have a reason to believe that a 1G ESP
>  was any more problematic than a 0.1G ESP. I am not aware of any
>  reports, and given that FAT32 is mandated by UEFI since basically
>  always, I think there's no immediate reason to believe we are in
>  trouble.
>  
>  I think the only reasonable approach here is to pick a larger default
>  in a development distro, and collect feedback.
>  
>  > Can't we reduce the number of kernels by having *only* one UKI and a
>  > rescue one that can be used to restore the previous working UKI from
>  > /root if the active one fails?
>  
>  I'd kill the rescue concept as a separate kernel. Pre-compiled UKIs
>  provided by Fedora should be comprehensive and suitable enough to be
>  rescue images, I don't see why we need a second version of that. The
>  only difference between a rescue boot and a regular boot should be one
>  of parameterization, not of picking different kernel. 
The next paragraph you cut off was proposing just that :-)

The reason why I still mentioned the rescue kernel is, as Dan mentioned
that in order to use a single UKI for both regular and rescue function
we'd need to be able to select between multiple signed command lines.
Once that is possible, I definitely would go with A/B images and stop
relying on years old rescue kernels as fallback.

Simo.

>  
>  Lennart
>  
>  --
>  Lennart Poettering, Berlin
>  _______________________________________________
>  devel mailing list -- devel(a)lists.fedoraproject.org
>  To unsubscribe send an email to devel-leave(a)lists.fedoraproject.org
>  Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
>  List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
>  List Archives:
> https://lists.fedoraproject.org/archives/list/[e-mail]
>  Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
-- 
Simo Sorce
RHEL Crypto Team
Red Hat, Inc

[Chris Murphy]

On Mon, Apr 24, 2023, at 12:15 PM, Ben Cotton wrote:
>  https://fedoraproject.org/wiki/Changes/BiggerESP 
>  This change will increase the minimum size of the ESP to be 500MB,
>  which is also the same value used by Microsoft for Windows 10 and
>  newer. 
Issue 1:  Currently anaconda calls mkdosfs on the ESP without any options and historically
are reluctant to add non-default mkfs options. By default, mkdosfs will create a FAT 16
file system on a 500M (either SI or IEC units). The UEFI spec clearly prefers FAT 32 for
the ESP on built-in drives. To my knowledge there haven't been any FAT 16 related bugs
reported during the many years Fedora created FAT 16 ESPs. But it's probably better to
create it as FAT 32.

I don't know where the cutoff is in the mkdosfs code, but a 512MiB (IEC units) does
result in a FAT 32 file system. So you might make it 512MiB.

Issue 2: Last I checked (about 12 months ago), Windows 10 and 11 images from microsoft.com
were still creating ~99M (I forget which units, and it may have been 100). That's
consistent with:

https://learn.microsoft.com/en-us/windows-hardware/manufacture/desktop/co...

"The minimum size of this partition is 100 MB, and must be formatted using the FAT32
file format."

So I'm not sure if Microsoft got the memo, and it's actually vendors' OEM
images that are using large ESP size?



-- 
Chris Murphy

[Neal Gompa]

On Wed, May 10, 2023 at 2:00 PM Chris Murphy <lists(a)colorremedies.com&gt; wrote:
>  On Mon, Apr 24, 2023, at 12:15 PM, Ben Cotton wrote:
>  > https://fedoraproject.org/wiki/Changes/BiggerESP
>
>  > This change will increase the minimum size of the ESP to be 500MB,
>  > which is also the same value used by Microsoft for Windows 10 and
>  > newer.
>
>  Issue 1:  Currently anaconda calls mkdosfs on the ESP without any options and
> historically are reluctant to add non-default mkfs options. By default, mkdosfs will
> create a FAT 16 file system on a 500M (either SI or IEC units). The UEFI spec clearly
> prefers FAT 32 for the ESP on built-in drives. To my knowledge there haven't been any
> FAT 16 related bugs reported during the many years Fedora created FAT 16 ESPs. But
> it's probably better to create it as FAT 32.
>
>  I don't know where the cutoff is in the mkdosfs code, but a 512MiB (IEC units) does
> result in a FAT 32 file system. So you might make it 512MiB.
>
>  Issue 2: Last I checked (about 12 months ago), Windows 10 and 11 images from
> microsoft.com were still creating ~99M (I forget which units, and it may have been 100).
> That's consistent with:
>
>
> https://learn.microsoft.com/en-us/windows-hardware/manufacture/desktop/co...
>
>  "The minimum size of this partition is 100 MB, and must be formatted using the FAT32
> file format."
>
>  So I'm not sure if Microsoft got the memo, and it's actually vendors' OEM
> images that are using large ESP size?
>  
Microsoft has no reason to make it bigger. They have a system boot
architecture that avoids having OS code on the ESP. Some OEMs preload
extra recovery stuff and or extra environments, which might be where
the bigger sizes come from.




--
真実はいつも一つ！/ Always, there's only one truth!