F38 proposal: Unified Kernel Support Phase 1 (System-Wide Change proposal)

[Ben Cotton]

https://fedoraproject.org/wiki/Changes/Unified_Kernel_Support_Phase_1

This document represents a proposed Change. As part of the Changes
process, proposals are publicly announced in order to receive
community feedback. This proposal will only be implemented if approved
by the Fedora Engineering Steering Committee.


== Summary ==
Add support for unified kernels images to Fedora.

== Owner ==
* Name: [[User:kraxel| Gerd Hoffmann]]
* Email: kraxel(a)redhat.com


== Detailed Description ==
The goal is to move away from initrd images being generated on the
installed machine.  They are generated while building the kernel
package instead, then shipped as part of a unified kernel image.

A unified kernel image is an all-in-one efi binary containing kernel,
initrd, cmdline and signature.  The secure boot signature covers
everything, specifically the initrd is included which is not the case
when the initrd gets loaded as separate file from /boot.

Main motivation for this move is to make the distro more robust and more secure.

Switching the whole distro over to unified kernels quickly is not
realistic though.  Too many features are depending on the current
workflow with a host-specific initrd (and host-specific kernel command
line), which is fundamentally incompatible with unified kernels where
everybody will have the same initrd and command line. Thats why there
is 'Phase 1' in title, so we can have more Phases in future releases
😃

A host-specific initrd / command line is needed today for:

* features needing optional dracut modules (initrd rebuild needed to
enable them).
* configuration / secrets baked into the initrd (booting from iscsi
for example).
* configuration being specified on the kernel command line.
** root filesystem being the most important one.
[https://systemd.io/DISCOVERABLE_PARTITIONS/ Discoverable partitions]
allow to remove this.

Phase 1 goals (high priority):

* Ship a unified kernel image as (optional) kernel sub-rpm.  Users can
opt-in to use that kernel by installing the sub-rpm.  Initial focus is
on booting virtual machines where we have a relatively small and well
defined set of drivers / features needed.  Supporting modern physical
machines with standard setup (i.e. boot from local sata/nvme storage)
too should be easy.
* Update kernel install scripts so unified kernels are installed and
updated properly.
* Add bootloader support for unified kernel images.  Add
[https://systemd.io/BOOT_LOADER_SPECIFICATION/#type-2-efi-unified-kernel-i...
unified kernel bls support] to grub2, or support using systemd-boot,
or both.

Phase 1 goals (lower priority, might move to Phase 2):

* Add proper discoverable partitions support to installers (anaconda,
image builder, ...).
** Temporary workaround possible: set types using sfdisk in %post script.
** When using btrfs: configure 'root' subvolume as default volume.
* Add proper systemd-boot support to installers.
** Temporary workaround possible: run 'bootctl install' in %post script.
* Better measurement and remote attestation support.
** store kernel + initrd hashes somewhere (kernel-hashes.rpm ?) to
allow pre-calculate TPM PCR values.
** avoid using grub2 (measures every config file line executed which
is next to impossible to pre-calculate).
* Switch cloud images to use unified kernels.

Phase 2/3 goals (longer-term stuff which is not realistic to complete for F38).

* Move away from using the kernel command line for configuration.
* Move away from storing secrets in the initrd.
* Handle dracut optional modules in a different way.

systemd has some building blocks which can be used, although none of
them are used by fedora today.
[https://www.freedesktop.org/software/systemd/man/systemd-creds.html
systemd credentials] can be used for secrets (also for configuration).
The [https://www.freedesktop.org/software/systemd/man/systemd-stub.html
unified kernel stub] can load credentials from the ESP.

The unified kernel stub can also load
[https://www.freedesktop.org/software/systemd/man/systemd-sysext.html
extensions] from the ESP, which can possibly be used to replace
optional dracut modules.

== Feedback ==


== Benefit to Fedora ==
* Better secure boot support (specifically the initrd is covered by
the signature).
* Better confidential computing support (measurements are much more
useful if we know what hashes to expect for the initrd).
* More robust boot process (generating the initrd on the installed
system is fragile, root cause for kernel bugs reported is simply a
broken initrd sometimes).

== Scope ==
* Proposal owners:
** Update kernel build to create unified kernel sub-package.
*** part one: [https://gitlab.com/cki-project/kernel-ark/-/merge_requests/2179
PR#2179]
*** part two: (wip)
[https://gitlab.com/kraxel/kernel-ark/-/commits/unified/
https://gitlab.com/kraxel/kernel-ark/-/commits/unified/]

* Other developers:
** grub2: add unified kernel support
*** wip code at [https://github.com/osteffenrh/grub2/
https://github.com/osteffenrh/grub2/]
** installer(s): add support for discoverable partitions.
*** [https://bugzilla.redhat.com/show_bug.cgi?id=1075288 Bug#1075288]

* Release engineering: [https://pagure.io/releng/issues #Releng issue number]
* Policies and guidelines: N/A (not needed for this Change)
* Trademark approval: N/A (not needed for this Change)
* Alignment with Objectives:


== Upgrade/compatibility impact ==

None (using unified kernel is opt-in for Phase 1).

== How To Test ==
Try on a existing (uefi) system:
* make sure you are running fedora 37 or rawhide.
* make sure your root filesystem has type "Linux root (x86-64)" (use
`fdisk -l` to check).
** should that not be the case use the fdisk tag command ('t') to
change the partition type.
* when using btrfs: make sure the 'root' subvolume is set as default volume.
* `dnf copr enable kraxel/unified.kernel`
* `dnf update "grub2*"`
* `dnf install kernel-unified-virt`
* `reboot`
You should find two entries in the grub2 boot menu, one for classing
kernel with separate initrd and one for the unified kernel image.
Both should boot fine.

The https://gitlab.com/kraxel/fedora-uki project has kickstart files
and helper scripts for generating virtual machine images.
* image using grub2-efi:
https://gitlab.com/kraxel/fedora-uki/-/raw/master/kickstart/fedora-uki-gr...
* image using systemd-boot:
https://gitlab.com/kraxel/fedora-uki/-/raw/master/kickstart/fedora-uki-sd...

Prebuilt virtual machine images are available from
https://www.kraxel.org/fedora-uki/.

== User Experience ==


== Dependencies ==


== Contingency Plan ==
* Contingency mechanism:
** Probably none (unified kernel images are opt-in for Phase 1).
** In case we tried switching the cloud images to unified kernels:
revert the kickstart config changes.
* Contingency deadline:
* Blocks release? No

== Documentation ==


== Release Notes ==



-- 
Ben Cotton
He / Him / His
Fedora Program Manager
Red Hat
TZ=America/Indiana/Indianapolis

[Dan Horák]

On Tue, 20 Dec 2022 10:22:03 -0500
Ben Cotton <bcotton(a)redhat.com&gt; wrote:

> https://fedoraproject.org/wiki/Changes/Unified_Kernel_Support_Phase_1
>  
>  This document represents a proposed Change. As part of the Changes
>  process, proposals are publicly announced in order to receive
>  community feedback. This proposal will only be implemented if approved
>  by the Fedora Engineering Steering Committee.
>  
>  
>  == Summary ==
>  Add support for unified kernels images to Fedora.
>  
>  == Owner ==
>  * Name: [[User:kraxel| Gerd Hoffmann]]
>  * Email: kraxel(a)redhat.com
>  
>  
>  == Detailed Description ==
>  The goal is to move away from initrd images being generated on the
>  installed machine.  They are generated while building the kernel
>  package instead, then shipped as part of a unified kernel image.
>  
>  A unified kernel image is an all-in-one efi binary containing kernel,
>  initrd, cmdline and signature.  The secure boot signature covers
>  everything, specifically the initrd is included which is not the case
>  when the initrd gets loaded as separate file from /boot. 
What platforms are going to be affected? Is it x86-64 only or does it
include aarch64 (as it's also using UEFI)? Are ppc64le and s390x out of
the current scope?


		Dan

[Daniel P. Berrangé]

On Tue, Dec 20, 2022 at 04:38:37PM +0100, Dan Horák wrote:
>  On Tue, 20 Dec 2022 10:22:03 -0500
>  Ben Cotton <bcotton(a)redhat.com&gt; wrote:
>  
>  > https://fedoraproject.org/wiki/Changes/Unified_Kernel_Support_Phase_1
>  > 
>  > This document represents a proposed Change. As part of the Changes
>  > process, proposals are publicly announced in order to receive
>  > community feedback. This proposal will only be implemented if approved
>  > by the Fedora Engineering Steering Committee.
>  > 
>  > 
>  > == Summary ==
>  > Add support for unified kernels images to Fedora.
>  > 
>  > == Owner ==
>  > * Name: [[User:kraxel| Gerd Hoffmann]]
>  > * Email: kraxel(a)redhat.com
>  > 
>  > 
>  > == Detailed Description ==
>  > The goal is to move away from initrd images being generated on the
>  > installed machine.  They are generated while building the kernel
>  > package instead, then shipped as part of a unified kernel image.
>  > 
>  > A unified kernel image is an all-in-one efi binary containing kernel,
>  > initrd, cmdline and signature.  The secure boot signature covers
>  > everything, specifically the initrd is included which is not the case
>  > when the initrd gets loaded as separate file from /boot.
>  
>  What platforms are going to be affected? Is it x86-64 only or does it
>  include aarch64 (as it's also using UEFI)? Are ppc64le and s390x out of
>  the current scope? 
This change proposal would only be x86_64, since that's the platform
where we need to solve the unsigned initrd problem with SecureBoot
with EDK2/QEMU (and more generally with bare metal, but this change
is limited scope to VMs to make it tractable to pre-built initrds
with a fixed set of kmods).

While aarch64 uses UEFI, there's no viable SecureBoot support for
it in EDK2/QEMU yet. If and when that appears, then it could be
extended to aarch64 at that time.

There's no plans for ppc64le / s390x.

NB, I'm talking from the POV of what we require to address as
virtualization platform maintainers. There might be other use cases
for UKIs from other teams TBD.


With regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

[Gerd Hoffmann]

>  > == Detailed Description ==
>  > The goal is to move away from initrd images being generated on the
>  > installed machine.  They are generated while building the kernel
>  > package instead, then shipped as part of a unified kernel image.
>  > 
>  > A unified kernel image is an all-in-one efi binary containing kernel,
>  > initrd, cmdline and signature.  The secure boot signature covers
>  > everything, specifically the initrd is included which is not the case
>  > when the initrd gets loaded as separate file from /boot.
>  
>  What platforms are going to be affected? Is it x86-64 only or does it
>  include aarch64 (as it's also using UEFI)? 
x86_64 is the primary target.

aarch64 possibly too.  Secure boot isn't a factor there so it's less
important, but having a more robust initrd build and kernel update
process is nice to have ...

>  Are ppc64le and s390x out of the current scope? 
Yes.

take care,
  Gerd

[Zbigniew Jędrzejewski-Szmek]

On Tue, Dec 20, 2022 at 10:22:03AM -0500, Ben Cotton wrote:
>  https://fedoraproject.org/wiki/Changes/Unified_Kernel_Support_Phase_1
It's great to see this happening!

>  Phase 1 goals (high priority):
>  
>  * Ship a unified kernel image as (optional) kernel sub-rpm.  Users can
>  opt-in to use that kernel by installing the sub-rpm.  Initial focus is
>  on booting virtual machines where we have a relatively small and well
>  defined set of drivers / features needed.  Supporting modern physical
>  machines with standard setup (i.e. boot from local sata/nvme storage)
>  too should be easy.
>  * Update kernel install scripts so unified kernels are installed and
>  updated properly.
>  * Add bootloader support for unified kernel images.  Add
>  [https://systemd.io/BOOT_LOADER_SPECIFICATION/#type-2-efi-unified-kernel-i...
>  unified kernel bls support] to grub2, or support using systemd-boot,
>  or both.
>  
>  Phase 1 goals (lower priority, might move to Phase 2):
>  
>  * Add proper discoverable partitions support to installers (anaconda,
>  image builder, ...).
>  ** Temporary workaround possible: set types using sfdisk in %post script.
>  ** When using btrfs: configure 'root' subvolume as default volume.
>  * Add proper systemd-boot support to installers.
>  ** Temporary workaround possible: run 'bootctl install' in %post script.
>  * Better measurement and remote attestation support.
>  ** store kernel + initrd hashes somewhere (kernel-hashes.rpm ?) to
>  allow pre-calculate TPM PCR values.
>  ** avoid using grub2 (measures every config file line executed which
>  is next to impossible to pre-calculate).
>  * Switch cloud images to use unified kernels. 
With my FESCo hat on, I immediately have the following comment:
please narrow down the scope to things that we can actually approve
for F38. E.g. the parts related to replacing grub2 by sd-boot
are IMHO not realistic for F38 (*). And if we use grub2, then also the
pre-calculation of TPM PCR values is not realistic, since they are
too volatile with grub2... I think that those are all very interesting
research tangents, but the stuff that gets a stamp of approval as a
Fedora Change needs to be down-to-earth and users-know-what-to-expect
and you-can-pretty-much-figure-out-how-things-will-look-from-the-change-description.

(*) Or if that is actually the plan, please specify *where* sd-boot
would be supported.

Zbyszek

[Daniel P. Berrangé]

On Tue, Dec 20, 2022 at 04:22:39PM +0000, Zbigniew Jędrzejewski-Szmek wrote:
>  On Tue, Dec 20, 2022 at 10:22:03AM -0500, Ben Cotton wrote:
>  > https://fedoraproject.org/wiki/Changes/Unified_Kernel_Support_Phase_1
>  
>  It's great to see this happening!
>  
>  > Phase 1 goals (high priority):
>  > 
>  > * Ship a unified kernel image as (optional) kernel sub-rpm.  Users can
>  > opt-in to use that kernel by installing the sub-rpm.  Initial focus is
>  > on booting virtual machines where we have a relatively small and well
>  > defined set of drivers / features needed.  Supporting modern physical
>  > machines with standard setup (i.e. boot from local sata/nvme storage)
>  > too should be easy.
>  > * Update kernel install scripts so unified kernels are installed and
>  > updated properly.
>  > * Add bootloader support for unified kernel images.  Add
>  > [https://systemd.io/BOOT_LOADER_SPECIFICATION/#type-2-efi-unified-kernel-i...
>  > unified kernel bls support] to grub2, or support using systemd-boot,
>  > or both.
>  > 
>  > Phase 1 goals (lower priority, might move to Phase 2):
>  > 
>  > * Add proper discoverable partitions support to installers (anaconda,
>  > image builder, ...).
>  > ** Temporary workaround possible: set types using sfdisk in %post script.
>  > ** When using btrfs: configure 'root' subvolume as default volume.
>  > * Add proper systemd-boot support to installers.
>  > ** Temporary workaround possible: run 'bootctl install' in %post script.
>  > * Better measurement and remote attestation support.
>  > ** store kernel + initrd hashes somewhere (kernel-hashes.rpm ?) to
>  > allow pre-calculate TPM PCR values.
>  > ** avoid using grub2 (measures every config file line executed which
>  > is next to impossible to pre-calculate).
>  > * Switch cloud images to use unified kernels.
>  
>  With my FESCo hat on, I immediately have the following comment:
>  please narrow down the scope to things that we can actually approve
>  for F38. E.g. the parts related to replacing grub2 by sd-boot
>  are IMHO not realistic for F38 (*). And if we use grub2, then also the
>  pre-calculation of TPM PCR values is not realistic, since they are
>  too volatile with grub2... I think that those are all very interesting
>  research tangents, but the stuff that gets a stamp of approval as a
>  Fedora Change needs to be down-to-earth and users-know-what-to-expect
>  and you-can-pretty-much-figure-out-how-things-will-look-from-the-change-description.
>  
>  (*) Or if that is actually the plan, please specify *where* sd-boot
>  would be supported. 
The 'description' text there somewhat contradicts the 'Scope' section,
which only mentions grub2, not sd-boot:

    Proposal owners:
        Update kernel build to create unified kernel sub-package.
            part one: PR#2179
            part two: (wip) https://gitlab.com/kraxel/kernel-ark/-/commits/unified/

    Other developers:
        grub2: add unified kernel support
            wip code at https://github.com/osteffenrh/grub2/
        installer(s): add support for discoverable partitions.
            Bug#1075288

While use of sd-boot is desirable in many ways, it is not a pre-requisite
for the use of UKIs. For that matter neither is the grub2 support.

IMHO keeping sd-boot separate from this change though is better to avoid
us getting diverted into a huge debate about bootloader choices for Fedora,
as that's a huge topic in its own right.

As noted there are some WIP patches for grub to allow it to load UKIs.
This would let us address the major hole in grub where initrds are
not signed for SecureBoot. As yuo mention, grub PCRs calculations are
still complex / fragile.

Without grub2 support for UKIs, and without sd-boot signed by Fedora,
UKIs are still useful. shim can be configured to directly boot the
UKI, at which point the PCR calculations are pretty easy [1].

For virtual machines with pre-built disk images, a boot loader can be
considered redundant for common use, as with public cloud users rarely
even see the bootloader phase of boot, let alone bother trying to
interact with it. This is actually desirable for confidential VMs,
as you don't want to be interacting with the guest over a virtual
keyboard/console, as that's not a confidential channel.

IOW, I think scope of this change should be UKIs as the mandatory
deliverable, and grub2 support for UKIs as a nice-to-have, and
anything about sd-boot strictly as a separate Change (whether f38
or F$later TBD)

With regards,
Daniel

[1] I've got an experimental tool for doing PCR pre-calculations

       https://gitlab.com/berrange/tpm-prevision/
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

[Gerd Hoffmann]

  Hi,

>  With my FESCo hat on, I immediately have the following comment:
>  please narrow down the scope to things that we can actually approve
>  for F38. E.g. the parts related to replacing grub2 by sd-boot
>  are IMHO not realistic for F38 (*). 
sd-boot actually replacing grub2 in anaconda installs or cloud images:
Yes, that is unrealistic.

Building images which use sd-boot with a custom kickstart file (doing
'bootctl install' in %post):  That works already.

>  And if we use grub2, then also the
>  pre-calculation of TPM PCR values is not realistic, since they are
>  too volatile with grub2... I think that those are all very interesting
>  research tangents, but the stuff that gets a stamp of approval as a
>  Fedora Change needs to be down-to-earth and users-know-what-to-expect
>  and you-can-pretty-much-figure-out-how-things-will-look-from-the-change-description.
Suggestions how to deal with those best?  The reason this stuff is in
there is that I want also outline the longer-term goal behind this.

I could create a "Phase 2" change page and move over stuff ...

thanks & take care,
  Gerd

[Neal Gompa]

On Tue, Dec 20, 2022 at 10:22 AM Ben Cotton <bcotton(a)redhat.com&gt; wrote:
>  https://fedoraproject.org/wiki/Changes/Unified_Kernel_Support_Phase_1
>
>  This document represents a proposed Change. As part of the Changes
>  process, proposals are publicly announced in order to receive
>  community feedback. This proposal will only be implemented if approved
>  by the Fedora Engineering Steering Committee.
>
>
>  == Summary ==
>  Add support for unified kernels images to Fedora.
>
>  == Owner ==
>  * Name: [[User:kraxel| Gerd Hoffmann]]
>  * Email: kraxel(a)redhat.com
>
>
>  == Detailed Description ==
>  The goal is to move away from initrd images being generated on the
>  installed machine.  They are generated while building the kernel
>  package instead, then shipped as part of a unified kernel image.
>
>  A unified kernel image is an all-in-one efi binary containing kernel,
>  initrd, cmdline and signature.  The secure boot signature covers
>  everything, specifically the initrd is included which is not the case
>  when the initrd gets loaded as separate file from /boot.
>
>  Main motivation for this move is to make the distro more robust and more secure.
>
>  Switching the whole distro over to unified kernels quickly is not
>  realistic though.  Too many features are depending on the current
>  workflow with a host-specific initrd (and host-specific kernel command
>  line), which is fundamentally incompatible with unified kernels where
>  everybody will have the same initrd and command line. Thats why there
>  is 'Phase 1' in title, so we can have more Phases in future releases
>  😃
>
>  A host-specific initrd / command line is needed today for:
>
>  * features needing optional dracut modules (initrd rebuild needed to
>  enable them).
>  * configuration / secrets baked into the initrd (booting from iscsi
>  for example).
>  * configuration being specified on the kernel command line.
>  ** root filesystem being the most important one.
>  [https://systemd.io/DISCOVERABLE_PARTITIONS/ Discoverable partitions]
>  allow to remove this.
>
>  Phase 1 goals (high priority):
>
>  * Ship a unified kernel image as (optional) kernel sub-rpm.  Users can
>  opt-in to use that kernel by installing the sub-rpm.  Initial focus is
>  on booting virtual machines where we have a relatively small and well
>  defined set of drivers / features needed.  Supporting modern physical
>  machines with standard setup (i.e. boot from local sata/nvme storage)
>  too should be easy.
>  * Update kernel install scripts so unified kernels are installed and
>  updated properly.
>  * Add bootloader support for unified kernel images.  Add
>  [https://systemd.io/BOOT_LOADER_SPECIFICATION/#type-2-efi-unified-kernel-i...
>  unified kernel bls support] to grub2, or support using systemd-boot,
>  or both.
>
>  Phase 1 goals (lower priority, might move to Phase 2):
>
>  * Add proper discoverable partitions support to installers (anaconda,
>  image builder, ...).
>  ** Temporary workaround possible: set types using sfdisk in %post script.
>  ** When using btrfs: configure 'root' subvolume as default volume.
>  * Add proper systemd-boot support to installers.
>  ** Temporary workaround possible: run 'bootctl install' in %post script.
>  * Better measurement and remote attestation support.
>  ** store kernel + initrd hashes somewhere (kernel-hashes.rpm ?) to
>  allow pre-calculate TPM PCR values.
>  ** avoid using grub2 (measures every config file line executed which
>  is next to impossible to pre-calculate).
>  * Switch cloud images to use unified kernels.
>
>  Phase 2/3 goals (longer-term stuff which is not realistic to complete for F38).
>
>  * Move away from using the kernel command line for configuration.
>  * Move away from storing secrets in the initrd.
>  * Handle dracut optional modules in a different way.
>
>  systemd has some building blocks which can be used, although none of
>  them are used by fedora today.
>  [https://www.freedesktop.org/software/systemd/man/systemd-creds.html
>  systemd credentials] can be used for secrets (also for configuration).
>  The [https://www.freedesktop.org/software/systemd/man/systemd-stub.html
>  unified kernel stub] can load credentials from the ESP.
>
>  The unified kernel stub can also load
>  [https://www.freedesktop.org/software/systemd/man/systemd-sysext.html
>  extensions] from the ESP, which can possibly be used to replace
>  optional dracut modules.
>
>  == Feedback ==
>
>
>  == Benefit to Fedora ==
>  * Better secure boot support (specifically the initrd is covered by
>  the signature).
>  * Better confidential computing support (measurements are much more
>  useful if we know what hashes to expect for the initrd).
>  * More robust boot process (generating the initrd on the installed
>  system is fragile, root cause for kernel bugs reported is simply a
>  broken initrd sometimes).
>
>  == Scope ==
>  * Proposal owners:
>  ** Update kernel build to create unified kernel sub-package.
>  *** part one: [https://gitlab.com/cki-project/kernel-ark/-/merge_requests/2179
>  PR#2179]
>  *** part two: (wip)
>  [https://gitlab.com/kraxel/kernel-ark/-/commits/unified/
>  https://gitlab.com/kraxel/kernel-ark/-/commits/unified/]
>
>  * Other developers:
>  ** grub2: add unified kernel support
>  *** wip code at [https://github.com/osteffenrh/grub2/
>  https://github.com/osteffenrh/grub2/]
>  ** installer(s): add support for discoverable partitions.
>  *** [https://bugzilla.redhat.com/show_bug.cgi?id=1075288 Bug#1075288]
>
>  * Release engineering: [https://pagure.io/releng/issues #Releng issue number]
>  * Policies and guidelines: N/A (not needed for this Change)
>  * Trademark approval: N/A (not needed for this Change)
>  * Alignment with Objectives:
>
>
>  == Upgrade/compatibility impact ==
>
>  None (using unified kernel is opt-in for Phase 1).
>
>  == How To Test ==
>  Try on a existing (uefi) system:
>  * make sure you are running fedora 37 or rawhide.
>  * make sure your root filesystem has type "Linux root (x86-64)" (use
>  `fdisk -l` to check).
>  ** should that not be the case use the fdisk tag command ('t') to
>  change the partition type.
>  * when using btrfs: make sure the 'root' subvolume is set as default volume.
>  * `dnf copr enable kraxel/unified.kernel`
>  * `dnf update "grub2*"`
>  * `dnf install kernel-unified-virt`
>  * `reboot`
>  You should find two entries in the grub2 boot menu, one for classing
>  kernel with separate initrd and one for the unified kernel image.
>  Both should boot fine.
>
>  The https://gitlab.com/kraxel/fedora-uki project has kickstart files
>  and helper scripts for generating virtual machine images.
>  * image using grub2-efi:
>  https://gitlab.com/kraxel/fedora-uki/-/raw/master/kickstart/fedora-uki-gr...
>  * image using systemd-boot:
>  https://gitlab.com/kraxel/fedora-uki/-/raw/master/kickstart/fedora-uki-sd...
>
>  Prebuilt virtual machine images are available from
>  https://www.kraxel.org/fedora-uki/.
>
>  == User Experience ==
>
>
>  == Dependencies ==
>
>
>  == Contingency Plan ==
>  * Contingency mechanism:
>  ** Probably none (unified kernel images are opt-in for Phase 1).
>  ** In case we tried switching the cloud images to unified kernels:
>  revert the kickstart config changes.
>  * Contingency deadline:
>  * Blocks release? No
>
>  == Documentation ==
>
>
>  == Release Notes ==
>  
I think UKIs are fundamentally flawed and are an idea that came out of
a group that doesn't really interact with real users enough to
understand how much of a problem they actually are. I realize that
this Change is only about VMs, but since it explicitly talks about it
being "phase 1", the implication is that future Changes are coming to
switch fully over. Consequently, I'm going to provide much more
holistic feedback instead of just nitpicking on this case.

In the Fedora case, things are simpler right up until we hit graphics
drivers. This is also a problem for VMs too, because GPU passthrough
is a common case for scientific and gaming workloads. As long as the
NVIDIA driver remains dominant in Linux, UKIs cannot work because by
design you cannot load anything that isn't part of the kernel image.
For bare metal, we *need* these drivers in early boot, though. And
that's another problem: no third-party early boot drivers. Even if you
solve the signing issue, you need to introduce some kind of two-stage
OS boot process so we can bring up the bare minimum to load a second
image containing all the remaining drivers. And at that point, you've
defeated the purpose of UKIs. I've heard from some people that system
extensions (sysexts) would be a way to solve this, and maybe it is.
But again, we've eliminated the value of UKIs by doing so.

Speaking more broadly, there are also problems that will be introduced
as this trickles down from Fedora into prominent downstreams (assuming
this is accepted). In the RHEL case, you've basically broken driver
disks completely. In the UKI model, there's no way to incorporate
early boot storage, network, and graphics drivers. This is
*incredibly* common for RHEL administrators because there's a general
acceptance of proprietary kernel modules from vendors these days. Even
ignoring those, Red Hat's kernel feature support mindset is completely
incompatible with UKIs, because RHEL does default-off rather than
Fedora's default-on model for kernel features. We could debate until
the cows come home on whether it's right or wrong, but their current
mindset essentially means tons of common hardware becomes unsupported
quite regularly. The ELRepo project is popular among RHEL folks
because it restores those drivers and makes it possible to use RHEL on
those machines through a combination of driver disks and kernel module
packages.

The crux of the problem here is *signing*. All of this is tied up in
Secure Boot and TPM, which is the wrong place to actually handle this.
In other operating systems (notably Windows), Secure Boot certificates
are not used for blocking or enabling kernel-level drivers. Instead,
there's a kernel-level certificate store that is used to validate and
enable drivers, allowing everything to be managed in a hardware
platform agnostic way. I've also yet to hear of a decent reason for
TPM measurements too. Block or filesystem verity provides similar
guarantees to provide tamper resistance and are both much easier to
debug than TPM interfaces. I am not convinced that you are providing
security or reliability with this and the trade-offs are *terrible*
for regular users.

With my FESCo hat on, I'm uneasy about this change. With my "Fedora
user and advocate" hat on, I think that the UAPI group has failed to
provide something useful to the Linux world here and I would be
extremely apprehensive about Fedora adopting any portion of this
stuff.



--
真実はいつも一つ！/ Always, there's only one truth!

[Hans de Goede]

Hi,

On 12/20/22 17:28, Neal Gompa wrote:
>  On Tue, Dec 20, 2022 at 10:22 AM Ben Cotton
> <bcotton(a)redhat.com&gt; wrote:
> >
> > https://fedoraproject.org/wiki/Changes/Unified_Kernel_Support_Phase_1
> >
> > This document represents a proposed Change. As part of the Changes
> > process, proposals are publicly announced in order to receive
> > community feedback. This proposal will only be implemented if approved
> > by the Fedora Engineering Steering Committee.
> >
> >
> > == Summary ==
> > Add support for unified kernels images to Fedora.
> >
> > == Owner ==
> > * Name: [[User:kraxel| Gerd Hoffmann]]
> > * Email: kraxel(a)redhat.com
> >
> >
> > == Detailed Description ==
> > The goal is to move away from initrd images being generated on the
> > installed machine.  They are generated while building the kernel
> > package instead, then shipped as part of a unified kernel image.
> >
> > A unified kernel image is an all-in-one efi binary containing kernel,
> > initrd, cmdline and signature.  The secure boot signature covers
> > everything, specifically the initrd is included which is not the case
> > when the initrd gets loaded as separate file from /boot.
> >
> > Main motivation for this move is to make the distro more robust and more secure.
> >
> > Switching the whole distro over to unified kernels quickly is not
> > realistic though.  Too many features are depending on the current
> > workflow with a host-specific initrd (and host-specific kernel command
> > line), which is fundamentally incompatible with unified kernels where
> > everybody will have the same initrd and command line. Thats why there
> > is 'Phase 1' in title, so we can have more Phases in future releases
> > 😃
> >
> > A host-specific initrd / command line is needed today for:
> >
> > * features needing optional dracut modules (initrd rebuild needed to
> > enable them).
> > * configuration / secrets baked into the initrd (booting from iscsi
> > for example).
> > * configuration being specified on the kernel command line.
> > ** root filesystem being the most important one.
> > [https://systemd.io/DISCOVERABLE_PARTITIONS/ Discoverable partitions]
> > allow to remove this.
> >
> > Phase 1 goals (high priority):
> >
> > * Ship a unified kernel image as (optional) kernel sub-rpm.  Users can
> > opt-in to use that kernel by installing the sub-rpm.  Initial focus is
> > on booting virtual machines where we have a relatively small and well
> > defined set of drivers / features needed.  Supporting modern physical
> > machines with standard setup (i.e. boot from local sata/nvme storage)
> > too should be easy.
> > * Update kernel install scripts so unified kernels are installed and
> > updated properly.
> > * Add bootloader support for unified kernel images.  Add
> > [https://systemd.io/BOOT_LOADER_SPECIFICATION/#type-2-efi-unified-kernel-i...
> > unified kernel bls support] to grub2, or support using systemd-boot,
> > or both.
> >
> > Phase 1 goals (lower priority, might move to Phase 2):
> >
> > * Add proper discoverable partitions support to installers (anaconda,
> > image builder, ...).
> > ** Temporary workaround possible: set types using sfdisk in %post script.
> > ** When using btrfs: configure 'root' subvolume as default volume.
> > * Add proper systemd-boot support to installers.
> > ** Temporary workaround possible: run 'bootctl install' in %post script.
> > * Better measurement and remote attestation support.
> > ** store kernel + initrd hashes somewhere (kernel-hashes.rpm ?) to
> > allow pre-calculate TPM PCR values.
> > ** avoid using grub2 (measures every config file line executed which
> > is next to impossible to pre-calculate).
> > * Switch cloud images to use unified kernels.
> >
> > Phase 2/3 goals (longer-term stuff which is not realistic to complete for F38).
> >
> > * Move away from using the kernel command line for configuration.
> > * Move away from storing secrets in the initrd.
> > * Handle dracut optional modules in a different way.
> >
> > systemd has some building blocks which can be used, although none of
> > them are used by fedora today.
> > [https://www.freedesktop.org/software/systemd/man/systemd-creds.html
> > systemd credentials] can be used for secrets (also for configuration).
> > The [https://www.freedesktop.org/software/systemd/man/systemd-stub.html
> > unified kernel stub] can load credentials from the ESP.
> >
> > The unified kernel stub can also load
> > [https://www.freedesktop.org/software/systemd/man/systemd-sysext.html
> > extensions] from the ESP, which can possibly be used to replace
> > optional dracut modules.
> >
> > == Feedback ==
> >
> >
> > == Benefit to Fedora ==
> > * Better secure boot support (specifically the initrd is covered by
> > the signature).
> > * Better confidential computing support (measurements are much more
> > useful if we know what hashes to expect for the initrd).
> > * More robust boot process (generating the initrd on the installed
> > system is fragile, root cause for kernel bugs reported is simply a
> > broken initrd sometimes).
> >
> > == Scope ==
> > * Proposal owners:
> > ** Update kernel build to create unified kernel sub-package.
> > *** part one: [https://gitlab.com/cki-project/kernel-ark/-/merge_requests/2179
> > PR#2179]
> > *** part two: (wip)
> > [https://gitlab.com/kraxel/kernel-ark/-/commits/unified/
> > https://gitlab.com/kraxel/kernel-ark/-/commits/unified/]
> >
> > * Other developers:
> > ** grub2: add unified kernel support
> > *** wip code at [https://github.com/osteffenrh/grub2/
> > https://github.com/osteffenrh/grub2/]
> > ** installer(s): add support for discoverable partitions.
> > *** [https://bugzilla.redhat.com/show_bug.cgi?id=1075288 Bug#1075288]
> >
> > * Release engineering: [https://pagure.io/releng/issues #Releng issue number]
> > * Policies and guidelines: N/A (not needed for this Change)
> > * Trademark approval: N/A (not needed for this Change)
> > * Alignment with Objectives:
> >
> >
> > == Upgrade/compatibility impact ==
> >
> > None (using unified kernel is opt-in for Phase 1).
> >
> > == How To Test ==
> > Try on a existing (uefi) system:
> > * make sure you are running fedora 37 or rawhide.
> > * make sure your root filesystem has type "Linux root (x86-64)" (use
> > `fdisk -l` to check).
> > ** should that not be the case use the fdisk tag command ('t') to
> > change the partition type.
> > * when using btrfs: make sure the 'root' subvolume is set as default volume.
> > * `dnf copr enable kraxel/unified.kernel`
> > * `dnf update "grub2*"`
> > * `dnf install kernel-unified-virt`
> > * `reboot`
> > You should find two entries in the grub2 boot menu, one for classing
> > kernel with separate initrd and one for the unified kernel image.
> > Both should boot fine.
> >
> > The https://gitlab.com/kraxel/fedora-uki project has kickstart files
> > and helper scripts for generating virtual machine images.
> > * image using grub2-efi:
> > https://gitlab.com/kraxel/fedora-uki/-/raw/master/kickstart/fedora-uki-gr...
> > * image using systemd-boot:
> > https://gitlab.com/kraxel/fedora-uki/-/raw/master/kickstart/fedora-uki-sd...
> >
> > Prebuilt virtual machine images are available from
> > https://www.kraxel.org/fedora-uki/.
> >
> > == User Experience ==
> >
> >
> > == Dependencies ==
> >
> >
> > == Contingency Plan ==
> > * Contingency mechanism:
> > ** Probably none (unified kernel images are opt-in for Phase 1).
> > ** In case we tried switching the cloud images to unified kernels:
> > revert the kickstart config changes.
> > * Contingency deadline:
> > * Blocks release? No
> >
> > == Documentation ==
> >
> >
> > == Release Notes ==
> >
>  
>  I think UKIs are fundamentally flawed and are an idea that came out of
>  a group that doesn't really interact with real users enough to
>  understand how much of a problem they actually are. I realize that
>  this Change is only about VMs, but since it explicitly talks about it
>  being "phase 1", the implication is that future Changes are coming to
>  switch fully over. Consequently, I'm going to provide much more
>  holistic feedback instead of just nitpicking on this case.
>  
>  In the Fedora case, things are simpler right up until we hit graphics
>  drivers. This is also a problem for VMs too, because GPU passthrough
>  is a common case for scientific and gaming workloads. 
In case of VMs there certainly is no need to load the GPU driver
from the initrd. 

>  As long as the
>  NVIDIA driver remains dominant in Linux, UKIs cannot work because by
>  design you cannot load anything that isn't part of the kernel image. 
Actually since the NVIDIA binary driver couples the driver and userspace
versions together very tightly using the NVIDIA binary driver inside
the initrd is a bad idea. The rpmfusion pkgs deliberately do not do
this so that the kmods can be rebuild on an nvidia driver update for
all installed kernels without needing to then also regenerate all
initrds (which would be a bad thing to do since we might then cause
all initrds to break at the same time).

>  For bare metal, we *need* these drivers in early boot, though 
Actually I have been toying with the idea to just rely on efifb in
the initrd, at least for the cases where we have an efifb/simpledrm.

When doing a 32 bit Debian install to test some kernel stuff on
quite old hw I noticed that Debian's install images let *grub*
bring up the video-output in vesa mode, with automatic resolution
selection. If we were to do this for legacy BIOS too then the kernel
could inherit that vesa mode with simpledrm and then we would have
legacy BIOS boot also covered with a "firmware framebuffer".

Given the discussion about firmware sizes for GPUs in other threads,
I think this is something to seriously consider. Otherwise the size
of our initrds is going to become absolutely huge.

Using firmware framebuffers inside the initrd could also significantly
speed up booting, since then the graphics card probing, which can be
quite slow when there are multiple external monitors, can be done in
parallel with booting the rest of the system instead of having plymouth
block on this.

Regards,

Hans





>  And that's another problem: no third-party early boot drivers.
> Even if you
>  solve the signing issue, you need to introduce some kind of two-stage
>  OS boot process so we can bring up the bare minimum to load a second
>  image containing all the remaining drivers. And at that point, you've
>  defeated the purpose of UKIs. I've heard from some people that system
>  extensions (sysexts) would be a way to solve this, and maybe it is.
>  But again, we've eliminated the value of UKIs by doing so.
>  
>  Speaking more broadly, there are also problems that will be introduced
>  as this trickles down from Fedora into prominent downstreams (assuming
>  this is accepted). In the RHEL case, you've basically broken driver
>  disks completely. In the UKI model, there's no way to incorporate
>  early boot storage, network, and graphics drivers. This is
>  *incredibly* common for RHEL administrators because there's a general
>  acceptance of proprietary kernel modules from vendors these days. Even
>  ignoring those, Red Hat's kernel feature support mindset is completely
>  incompatible with UKIs, because RHEL does default-off rather than
>  Fedora's default-on model for kernel features. We could debate until
>  the cows come home on whether it's right or wrong, but their current
>  mindset essentially means tons of common hardware becomes unsupported
>  quite regularly. The ELRepo project is popular among RHEL folks
>  because it restores those drivers and makes it possible to use RHEL on
>  those machines through a combination of driver disks and kernel module
>  packages.
>  
>  The crux of the problem here is *signing*. All of this is tied up in
>  Secure Boot and TPM, which is the wrong place to actually handle this.
>  In other operating systems (notably Windows), Secure Boot certificates
>  are not used for blocking or enabling kernel-level drivers. Instead,
>  there's a kernel-level certificate store that is used to validate and
>  enable drivers, allowing everything to be managed in a hardware
>  platform agnostic way. I've also yet to hear of a decent reason for
>  TPM measurements too. Block or filesystem verity provides similar
>  guarantees to provide tamper resistance and are both much easier to
>  debug than TPM interfaces. I am not convinced that you are providing
>  security or reliability with this and the trade-offs are *terrible*
>  for regular users.
>  
>  With my FESCo hat on, I'm uneasy about this change. With my "Fedora
>  user and advocate" hat on, I think that the UAPI group has failed to
>  provide something useful to the Linux world here and I would be
>  extremely apprehensive about Fedora adopting any portion of this
>  stuff.
>  
>  
>  
>  --
>  真実はいつも一つ！/ Always, there's only one truth!
>  _______________________________________________
>  devel mailing list -- devel(a)lists.fedoraproject.org
>  To unsubscribe send an email to devel-leave(a)lists.fedoraproject.org
>  Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
>  List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
>  List Archives:
> https://lists.fedoraproject.org/archives/list/[e-mail]
>  Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue

[Neal Gompa]

On Tue, Dec 20, 2022 at 1:18 PM Hans de Goede <hdegoede(a)redhat.com&gt; wrote:
>  Hi,
>
>  On 12/20/22 17:28, Neal Gompa wrote:
>  > On Tue, Dec 20, 2022 at 10:22 AM Ben Cotton <bcotton(a)redhat.com&gt; wrote:
>  >>
>  >> https://fedoraproject.org/wiki/Changes/Unified_Kernel_Support_Phase_1
>  >>
>  >> This document represents a proposed Change. As part of the Changes
>  >> process, proposals are publicly announced in order to receive
>  >> community feedback. This proposal will only be implemented if approved
>  >> by the Fedora Engineering Steering Committee.
>  >>
>  >>
>  >> == Summary ==
>  >> Add support for unified kernels images to Fedora.
>  >>
>  >> == Owner ==
>  >> * Name: [[User:kraxel| Gerd Hoffmann]]
>  >> * Email: kraxel(a)redhat.com
>  >>
>  >>
>  >> == Detailed Description ==
>  >> The goal is to move away from initrd images being generated on the
>  >> installed machine.  They are generated while building the kernel
>  >> package instead, then shipped as part of a unified kernel image.
>  >>
>  >> A unified kernel image is an all-in-one efi binary containing kernel,
>  >> initrd, cmdline and signature.  The secure boot signature covers
>  >> everything, specifically the initrd is included which is not the case
>  >> when the initrd gets loaded as separate file from /boot.
>  >>
>  >> Main motivation for this move is to make the distro more robust and more
> secure.
>  >>
>  >> Switching the whole distro over to unified kernels quickly is not
>  >> realistic though.  Too many features are depending on the current
>  >> workflow with a host-specific initrd (and host-specific kernel command
>  >> line), which is fundamentally incompatible with unified kernels where
>  >> everybody will have the same initrd and command line. Thats why there
>  >> is 'Phase 1' in title, so we can have more Phases in future releases
>  >> 😃
>  >>
>  >> A host-specific initrd / command line is needed today for:
>  >>
>  >> * features needing optional dracut modules (initrd rebuild needed to
>  >> enable them).
>  >> * configuration / secrets baked into the initrd (booting from iscsi
>  >> for example).
>  >> * configuration being specified on the kernel command line.
>  >> ** root filesystem being the most important one.
>  >> [https://systemd.io/DISCOVERABLE_PARTITIONS/ Discoverable partitions]
>  >> allow to remove this.
>  >>
>  >> Phase 1 goals (high priority):
>  >>
>  >> * Ship a unified kernel image as (optional) kernel sub-rpm.  Users can
>  >> opt-in to use that kernel by installing the sub-rpm.  Initial focus is
>  >> on booting virtual machines where we have a relatively small and well
>  >> defined set of drivers / features needed.  Supporting modern physical
>  >> machines with standard setup (i.e. boot from local sata/nvme storage)
>  >> too should be easy.
>  >> * Update kernel install scripts so unified kernels are installed and
>  >> updated properly.
>  >> * Add bootloader support for unified kernel images.  Add
>  >> [https://systemd.io/BOOT_LOADER_SPECIFICATION/#type-2-efi-unified-kernel-i...
>  >> unified kernel bls support] to grub2, or support using systemd-boot,
>  >> or both.
>  >>
>  >> Phase 1 goals (lower priority, might move to Phase 2):
>  >>
>  >> * Add proper discoverable partitions support to installers (anaconda,
>  >> image builder, ...).
>  >> ** Temporary workaround possible: set types using sfdisk in %post script.
>  >> ** When using btrfs: configure 'root' subvolume as default volume.
>  >> * Add proper systemd-boot support to installers.
>  >> ** Temporary workaround possible: run 'bootctl install' in %post
> script.
>  >> * Better measurement and remote attestation support.
>  >> ** store kernel + initrd hashes somewhere (kernel-hashes.rpm ?) to
>  >> allow pre-calculate TPM PCR values.
>  >> ** avoid using grub2 (measures every config file line executed which
>  >> is next to impossible to pre-calculate).
>  >> * Switch cloud images to use unified kernels.
>  >>
>  >> Phase 2/3 goals (longer-term stuff which is not realistic to complete for F38).
>  >>
>  >> * Move away from using the kernel command line for configuration.
>  >> * Move away from storing secrets in the initrd.
>  >> * Handle dracut optional modules in a different way.
>  >>
>  >> systemd has some building blocks which can be used, although none of
>  >> them are used by fedora today.
>  >> [https://www.freedesktop.org/software/systemd/man/systemd-creds.html
>  >> systemd credentials] can be used for secrets (also for configuration).
>  >> The [https://www.freedesktop.org/software/systemd/man/systemd-stub.html
>  >> unified kernel stub] can load credentials from the ESP.
>  >>
>  >> The unified kernel stub can also load
>  >> [https://www.freedesktop.org/software/systemd/man/systemd-sysext.html
>  >> extensions] from the ESP, which can possibly be used to replace
>  >> optional dracut modules.
>  >>
>  >> == Feedback ==
>  >>
>  >>
>  >> == Benefit to Fedora ==
>  >> * Better secure boot support (specifically the initrd is covered by
>  >> the signature).
>  >> * Better confidential computing support (measurements are much more
>  >> useful if we know what hashes to expect for the initrd).
>  >> * More robust boot process (generating the initrd on the installed
>  >> system is fragile, root cause for kernel bugs reported is simply a
>  >> broken initrd sometimes).
>  >>
>  >> == Scope ==
>  >> * Proposal owners:
>  >> ** Update kernel build to create unified kernel sub-package.
>  >> *** part one: [https://gitlab.com/cki-project/kernel-ark/-/merge_requests/2179
>  >> PR#2179]
>  >> *** part two: (wip)
>  >> [https://gitlab.com/kraxel/kernel-ark/-/commits/unified/
>  >> https://gitlab.com/kraxel/kernel-ark/-/commits/unified/]
>  >>
>  >> * Other developers:
>  >> ** grub2: add unified kernel support
>  >> *** wip code at [https://github.com/osteffenrh/grub2/
>  >> https://github.com/osteffenrh/grub2/]
>  >> ** installer(s): add support for discoverable partitions.
>  >> *** [https://bugzilla.redhat.com/show_bug.cgi?id=1075288 Bug#1075288]
>  >>
>  >> * Release engineering: [https://pagure.io/releng/issues #Releng issue number]
>  >> * Policies and guidelines: N/A (not needed for this Change)
>  >> * Trademark approval: N/A (not needed for this Change)
>  >> * Alignment with Objectives:
>  >>
>  >>
>  >> == Upgrade/compatibility impact ==
>  >>
>  >> None (using unified kernel is opt-in for Phase 1).
>  >>
>  >> == How To Test ==
>  >> Try on a existing (uefi) system:
>  >> * make sure you are running fedora 37 or rawhide.
>  >> * make sure your root filesystem has type "Linux root (x86-64)" (use
>  >> `fdisk -l` to check).
>  >> ** should that not be the case use the fdisk tag command ('t') to
>  >> change the partition type.
>  >> * when using btrfs: make sure the 'root' subvolume is set as default
> volume.
>  >> * `dnf copr enable kraxel/unified.kernel`
>  >> * `dnf update "grub2*"`
>  >> * `dnf install kernel-unified-virt`
>  >> * `reboot`
>  >> You should find two entries in the grub2 boot menu, one for classing
>  >> kernel with separate initrd and one for the unified kernel image.
>  >> Both should boot fine.
>  >>
>  >> The https://gitlab.com/kraxel/fedora-uki project has kickstart files
>  >> and helper scripts for generating virtual machine images.
>  >> * image using grub2-efi:
>  >> https://gitlab.com/kraxel/fedora-uki/-/raw/master/kickstart/fedora-uki-gr...
>  >> * image using systemd-boot:
>  >>
> https://gitlab.com/kraxel/fedora-uki/-/raw/master/kickstart/fedora-uki-sd...
>  >>
>  >> Prebuilt virtual machine images are available from
>  >> https://www.kraxel.org/fedora-uki/.
>  >>
>  >> == User Experience ==
>  >>
>  >>
>  >> == Dependencies ==
>  >>
>  >>
>  >> == Contingency Plan ==
>  >> * Contingency mechanism:
>  >> ** Probably none (unified kernel images are opt-in for Phase 1).
>  >> ** In case we tried switching the cloud images to unified kernels:
>  >> revert the kickstart config changes.
>  >> * Contingency deadline:
>  >> * Blocks release? No
>  >>
>  >> == Documentation ==
>  >>
>  >>
>  >> == Release Notes ==
>  >>
>  >
>  > I think UKIs are fundamentally flawed and are an idea that came out of
>  > a group that doesn't really interact with real users enough to
>  > understand how much of a problem they actually are. I realize that
>  > this Change is only about VMs, but since it explicitly talks about it
>  > being "phase 1", the implication is that future Changes are coming to
>  > switch fully over. Consequently, I'm going to provide much more
>  > holistic feedback instead of just nitpicking on this case.
>  >
>  > In the Fedora case, things are simpler right up until we hit graphics
>  > drivers. This is also a problem for VMs too, because GPU passthrough
>  > is a common case for scientific and gaming workloads.
>
>  In case of VMs there certainly is no need to load the GPU driver
>  from the initrd.
>
>  > As long as the
>  > NVIDIA driver remains dominant in Linux, UKIs cannot work because by
>  > design you cannot load anything that isn't part of the kernel image.
>
>  Actually since the NVIDIA binary driver couples the driver and userspace
>  versions together very tightly using the NVIDIA binary driver inside
>  the initrd is a bad idea. The rpmfusion pkgs deliberately do not do
>  this so that the kmods can be rebuild on an nvidia driver update for
>  all installed kernels without needing to then also regenerate all
>  initrds (which would be a bad thing to do since we might then cause
>  all initrds to break at the same time).
>
>  > For bare metal, we *need* these drivers in early boot, though
>
>  Actually I have been toying with the idea to just rely on efifb in
>  the initrd, at least for the cases where we have an efifb/simpledrm.
>
>  When doing a 32 bit Debian install to test some kernel stuff on
>  quite old hw I noticed that Debian's install images let *grub*
>  bring up the video-output in vesa mode, with automatic resolution
>  selection. If we were to do this for legacy BIOS too then the kernel
>  could inherit that vesa mode with simpledrm and then we would have
>  legacy BIOS boot also covered with a "firmware framebuffer".
>  
I think Fedora is the only major distro that doesn't do this,
actually. Mageia and openSUSE do this too. They also use graphical
GRUB by default instead of plain text GRUB.

>  Given the discussion about firmware sizes for GPUs in other threads,
>  I think this is something to seriously consider. Otherwise the size
>  of our initrds is going to become absolutely huge.
>
>  Using firmware framebuffers inside the initrd could also significantly
>  speed up booting, since then the graphics card probing, which can be
>  quite slow when there are multiple external monitors, can be done in
>  parallel with booting the rest of the system instead of having plymouth
>  block on this.
>  
I agree, though the driver handover process needs to work first. It
doesn't solve the problem for storage and network drivers though.



--
真実はいつも一つ！/ Always, there's only one truth!

[Kevin Kofler]

Neal Gompa wrote:
>  I think Fedora is the only major distro that doesn't do this,
>  actually. Mageia and openSUSE do this too. They also use graphical
>  GRUB by default instead of plain text GRUB. 
IIRC, the reason that Fedora does not use graphical GRUB by default is that 
it at least used to break the NVidia proprietary driver.

        Kevin Kofler

[Neal Gompa]

On Tue, Dec 20, 2022 at 9:04 PM Kevin Kofler via devel
<devel(a)lists.fedoraproject.org&gt; wrote:
>  Neal Gompa wrote:
>  > I think Fedora is the only major distro that doesn't do this,
>  > actually. Mageia and openSUSE do this too. They also use graphical
>  > GRUB by default instead of plain text GRUB.
>
>  IIRC, the reason that Fedora does not use graphical GRUB by default is that
>  it at least used to break the NVidia proprietary driver.
>  
That must have not been true for a very long time, because I've never
seen this issue in the past 7 years of working on both distros with
users with NVIDIA GPUs with the proprietary driver.



--
真実はいつも一つ！/ Always, there's only one truth!

[Vitaly Zaitsev]

On 21/12/2022 03:03, Kevin Kofler via devel wrote:
>  IIRC, the reason that Fedora does not use graphical GRUB by default
> is that
>  it at least used to break the NVidia proprietary driver. 
It works fine on modern NVIDIA driver releases.

-- 
Sincerely,
   Vitaly Zaitsev (vitaly(a)easycoding.org)

[Daniel P. Berrangé]

On Tue, Dec 20, 2022 at 11:28:48AM -0500, Neal Gompa wrote:
>  On Tue, Dec 20, 2022 at 10:22 AM Ben Cotton
> <bcotton(a)redhat.com&gt; wrote:
>  >
>  > https://fedoraproject.org/wiki/Changes/Unified_Kernel_Support_Phase_1
>  >
>  > This document represents a proposed Change. As part of the Changes
>  > process, proposals are publicly announced in order to receive
>  > community feedback. This proposal will only be implemented if approved
>  > by the Fedora Engineering Steering Committee.
>  >
>  >
>  > == Summary ==
>  > Add support for unified kernels images to Fedora.
>  >
>  > == Owner ==
>  > * Name: [[User:kraxel| Gerd Hoffmann]]
>  > * Email: kraxel(a)redhat.com
>  >
>  >
>  > == Detailed Description ==
>  > The goal is to move away from initrd images being generated on the
>  > installed machine.  They are generated while building the kernel
>  > package instead, then shipped as part of a unified kernel image.
>  >
>  > A unified kernel image is an all-in-one efi binary containing kernel,
>  > initrd, cmdline and signature.  The secure boot signature covers
>  > everything, specifically the initrd is included which is not the case
>  > when the initrd gets loaded as separate file from /boot.
>  >
>  > Main motivation for this move is to make the distro more robust and more secure.
>  >
>  > Switching the whole distro over to unified kernels quickly is not
>  > realistic though.  Too many features are depending on the current
>  > workflow with a host-specific initrd (and host-specific kernel command
>  > line), which is fundamentally incompatible with unified kernels where
>  > everybody will have the same initrd and command line. Thats why there
>  > is 'Phase 1' in title, so we can have more Phases in future releases
>  > 😃
>  >
>  > A host-specific initrd / command line is needed today for:
>  >
>  > * features needing optional dracut modules (initrd rebuild needed to
>  > enable them).
>  > * configuration / secrets baked into the initrd (booting from iscsi
>  > for example).
>  > * configuration being specified on the kernel command line.
>  > ** root filesystem being the most important one.
>  > [https://systemd.io/DISCOVERABLE_PARTITIONS/ Discoverable partitions]
>  > allow to remove this.
>  >
>  > Phase 1 goals (high priority):
>  >
>  > * Ship a unified kernel image as (optional) kernel sub-rpm.  Users can
>  > opt-in to use that kernel by installing the sub-rpm.  Initial focus is
>  > on booting virtual machines where we have a relatively small and well
>  > defined set of drivers / features needed.  Supporting modern physical
>  > machines with standard setup (i.e. boot from local sata/nvme storage)
>  > too should be easy.
>  > * Update kernel install scripts so unified kernels are installed and
>  > updated properly.
>  > * Add bootloader support for unified kernel images.  Add
>  > [https://systemd.io/BOOT_LOADER_SPECIFICATION/#type-2-efi-unified-kernel-i...
>  > unified kernel bls support] to grub2, or support using systemd-boot,
>  > or both.
>  >
>  > Phase 1 goals (lower priority, might move to Phase 2):
>  >
>  > * Add proper discoverable partitions support to installers (anaconda,
>  > image builder, ...).
>  > ** Temporary workaround possible: set types using sfdisk in %post script.
>  > ** When using btrfs: configure 'root' subvolume as default volume.
>  > * Add proper systemd-boot support to installers.
>  > ** Temporary workaround possible: run 'bootctl install' in %post script.
>  > * Better measurement and remote attestation support.
>  > ** store kernel + initrd hashes somewhere (kernel-hashes.rpm ?) to
>  > allow pre-calculate TPM PCR values.
>  > ** avoid using grub2 (measures every config file line executed which
>  > is next to impossible to pre-calculate).
>  > * Switch cloud images to use unified kernels.
>  >
>  > Phase 2/3 goals (longer-term stuff which is not realistic to complete for F38).
>  >
>  > * Move away from using the kernel command line for configuration.
>  > * Move away from storing secrets in the initrd.
>  > * Handle dracut optional modules in a different way. 
snip

>  
>  I think UKIs are fundamentally flawed and are an idea that came out of
>  a group that doesn't really interact with real users enough to
>  understand how much of a problem they actually are. I realize that
>  this Change is only about VMs, but since it explicitly talks about it
>  being "phase 1", the implication is that future Changes are coming to
>  switch fully over. Consequently, I'm going to provide much more
>  holistic feedback instead of just nitpicking on this case. 
The future "phase 2/3" goal are mentioned above, and don't
mention fully switching to UKIs.

I don't anticipate such a switch being realistic any time
in the forseeable future. There are too many unknowns, in
addition to some points you mentioned, to even consider
that right now. The in place upgrade story gogin from
locally generated initrd, with arbitrary user chosen
content to a UKI doesn't even bare thinking about. I
don't think that's even possible to achieve in a seemless
way given the infinite number of variations. IOW, I would
anticipate the current kernel+initrd scheme being around
indefinitely, in parallel with any UKI support.

>  In the Fedora case, things are simpler right up until we hit
> graphics
>  drivers. This is also a problem for VMs too, because GPU passthrough
>  is a common case for scientific and gaming workloads. As long as the
>  NVIDIA driver remains dominant in Linux, UKIs cannot work because by
>  design you cannot load anything that isn't part of the kernel image.
>  For bare metal, we *need* these drivers in early boot, though. And
>  that's another problem: no third-party early boot drivers. Even if you
>  solve the signing issue, you need to introduce some kind of two-stage
>  OS boot process so we can bring up the bare minimum to load a second
>  image containing all the remaining drivers. And at that point, you've
>  defeated the purpose of UKIs. I've heard from some people that system
>  extensions (sysexts) would be a way to solve this, and maybe it is. 
Yes, system extensions are one mechanism for making the initrd
content more flexible when using UKIs. There would be one base
layer definiing the 90% common case, and a number of add-ons
that can cope with niche use cases. This avoids the core UKI
having to be huge and ship with every possible feature present.

This is something that might be considered at a later phase,
but is not a priority. What's proposed in phase 1 is sufficient
to cope with the cloud VM common case. While device passthrough
may be common in some industries/domains usage, it is not the
common case for cloud computing in general.

>  But again, we've eliminated the value of UKIs by doing so.
That is not correct. There are a number of benefits of UKIs.

The most critical is that the initrd content and cmdline is
covered by the SecureBoot signature. This remains true even
with system extensions, as such extensions would be signed
too. They do not neccessarily need to be signed by the OS
vendor, they could use a 3rd party SecureBoot signing key,
or the users' own key. That is TBD and not something we're
actively considering - its not even mentioned in phase 2/3
ideas in the change proposal. 

A further benefit of UKIs is supportability, since we know
exactly what content is in the initrd that user is booting.
With locally generated initrds there are an arbitrary
number of combinations and many ways for the initrd creation
process to go wrong, which are hard for Fedora maintainers
to understand and debug.

>  Speaking more broadly, there are also problems that will be
> introduced
>  as this trickles down from Fedora into prominent downstreams (assuming
>  this is accepted). In the RHEL case, you've basically broken driver
>  disks completely. In the UKI model, there's no way to incorporate
>  early boot storage, network, and graphics drivers. This is
>  *incredibly* common for RHEL administrators because there's a general
>  acceptance of proprietary kernel modules from vendors these days. Even
>  ignoring those, Red Hat's kernel feature support mindset is completely
>  incompatible with UKIs, because RHEL does default-off rather than
>  Fedora's default-on model for kernel features. We could debate until
>  the cows come home on whether it's right or wrong, but their current
>  mindset essentially means tons of common hardware becomes unsupported
>  quite regularly. The ELRepo project is popular among RHEL folks
>  because it restores those drivers and makes it possible to use RHEL on
>  those machines through a combination of driver disks and kernel module
>  packages. 
Yes, this desire for UKIs in Fedora is matched by a desire to
ship the same UKIs in RHEL too.

No one is proposing to take away the existing ability to use
locally generated initrds, either in Fedora or RHEL. Everything
that is possible todo today will remain possible indefinitely.
UKIs are adding a new mechanism in parallel, in order to solve
a number of use cases that are not possible today, due to
unsigned content being part of the boot chain when SecureBoot
is active.

>  The crux of the problem here is *signing*. All of this is tied up in
>  Secure Boot and TPM, which is the wrong place to actually handle this.
>  In other operating systems (notably Windows), Secure Boot certificates
>  are not used for blocking or enabling kernel-level drivers. Instead,
>  there's a kernel-level certificate store that is used to validate and
>  enable drivers, allowing everything to be managed in a hardware
>  platform agnostic way. I've also yet to hear of a decent reason for
>  TPM measurements too. Block or filesystem verity provides similar
>  guarantees to provide tamper resistance and are both much easier to
>  debug than TPM interfaces. I am not convinced that you are providing
>  security or reliability with this and the trade-offs are *terrible*
>  for regular users. 
The immediate need for UKIs is indeed related to SecureBoot and
TPMs. These are a core technology foundation of the confidential
virtual machine stack. On Azure today, if you request an Ubuntu
confidential VM, Azure will pre-encrypt the root filesystem and
seal the LUKS key against predicted TPM PCR values. It guarantees
that the root disk can only be decrypted by the specific VM
instance that is requested, when it is running in SecureBoot
mode with the expected measurments on AMD SEV-SNP confidential
hardware. The Ubuntu image in Azure already uses UKIs, and boots
them directly from shim, with no bootloader involved. The usage
of SecureBoot and TPMs is all transparent to the user, since all
the integration is handled by the OS vendor on their behalf.

>  With my FESCo hat on, I'm uneasy about this change. With my
> "Fedora
>  user and advocate" hat on, I think that the UAPI group has failed to
>  provide something useful to the Linux world here and I would be
>  extremely apprehensive about Fedora adopting any portion of this
>  stuff. 
As mentioned above, UKIs have provided a means to close the
SecureBoot hole with unsigned initrd content that has existed
in more or less every mainstream Linux distro. That is a
clearly useful outcome, regardless of whether Fedora is interested
in any other aspect of what that UAPI groups is proposing.

In proposing the UKI support for Fedora, we're not coming at this
as representatives of the UAPI group or its vision. We're trying
to solve the problem of having a fully verified secureboot chain
for VMs with no unsigned content, and to be able to use this to
support confidential VMs with disk encryption sealed to TPMs, as
required by Azure today, and likely KVM in future.

With regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

[Neal Gompa]

On Tue, Dec 20, 2022 at 2:02 PM Daniel P. Berrangé <berrange(a)redhat.com&gt; wrote:
>  On Tue, Dec 20, 2022 at 11:28:48AM -0500, Neal Gompa wrote:
>  > On Tue, Dec 20, 2022 at 10:22 AM Ben Cotton <bcotton(a)redhat.com&gt; wrote:
>  > >
>  > > https://fedoraproject.org/wiki/Changes/Unified_Kernel_Support_Phase_1
>  > >
>  > > This document represents a proposed Change. As part of the Changes
>  > > process, proposals are publicly announced in order to receive
>  > > community feedback. This proposal will only be implemented if approved
>  > > by the Fedora Engineering Steering Committee.
>  > >
>  > >
>  > > == Summary ==
>  > > Add support for unified kernels images to Fedora.
>  > >
>  > > == Owner ==
>  > > * Name: [[User:kraxel| Gerd Hoffmann]]
>  > > * Email: kraxel(a)redhat.com
>  > >
>  > >
>  > > == Detailed Description ==
>  > > The goal is to move away from initrd images being generated on the
>  > > installed machine.  They are generated while building the kernel
>  > > package instead, then shipped as part of a unified kernel image.
>  > >
>  > > A unified kernel image is an all-in-one efi binary containing kernel,
>  > > initrd, cmdline and signature.  The secure boot signature covers
>  > > everything, specifically the initrd is included which is not the case
>  > > when the initrd gets loaded as separate file from /boot.
>  > >
>  > > Main motivation for this move is to make the distro more robust and more
> secure.
>  > >
>  > > Switching the whole distro over to unified kernels quickly is not
>  > > realistic though.  Too many features are depending on the current
>  > > workflow with a host-specific initrd (and host-specific kernel command
>  > > line), which is fundamentally incompatible with unified kernels where
>  > > everybody will have the same initrd and command line. Thats why there
>  > > is 'Phase 1' in title, so we can have more Phases in future releases
>  > > 😃
>  > >
>  > > A host-specific initrd / command line is needed today for:
>  > >
>  > > * features needing optional dracut modules (initrd rebuild needed to
>  > > enable them).
>  > > * configuration / secrets baked into the initrd (booting from iscsi
>  > > for example).
>  > > * configuration being specified on the kernel command line.
>  > > ** root filesystem being the most important one.
>  > > [https://systemd.io/DISCOVERABLE_PARTITIONS/ Discoverable partitions]
>  > > allow to remove this.
>  > >
>  > > Phase 1 goals (high priority):
>  > >
>  > > * Ship a unified kernel image as (optional) kernel sub-rpm.  Users can
>  > > opt-in to use that kernel by installing the sub-rpm.  Initial focus is
>  > > on booting virtual machines where we have a relatively small and well
>  > > defined set of drivers / features needed.  Supporting modern physical
>  > > machines with standard setup (i.e. boot from local sata/nvme storage)
>  > > too should be easy.
>  > > * Update kernel install scripts so unified kernels are installed and
>  > > updated properly.
>  > > * Add bootloader support for unified kernel images.  Add
>  > >
> [https://systemd.io/BOOT_LOADER_SPECIFICATION/#type-2-efi-unified-kernel-i...
>  > > unified kernel bls support] to grub2, or support using systemd-boot,
>  > > or both.
>  > >
>  > > Phase 1 goals (lower priority, might move to Phase 2):
>  > >
>  > > * Add proper discoverable partitions support to installers (anaconda,
>  > > image builder, ...).
>  > > ** Temporary workaround possible: set types using sfdisk in %post script.
>  > > ** When using btrfs: configure 'root' subvolume as default volume.
>  > > * Add proper systemd-boot support to installers.
>  > > ** Temporary workaround possible: run 'bootctl install' in %post
> script.
>  > > * Better measurement and remote attestation support.
>  > > ** store kernel + initrd hashes somewhere (kernel-hashes.rpm ?) to
>  > > allow pre-calculate TPM PCR values.
>  > > ** avoid using grub2 (measures every config file line executed which
>  > > is next to impossible to pre-calculate).
>  > > * Switch cloud images to use unified kernels.
>  > >
>  > > Phase 2/3 goals (longer-term stuff which is not realistic to complete for
> F38).
>  > >
>  > > * Move away from using the kernel command line for configuration.
>  > > * Move away from storing secrets in the initrd.
>  > > * Handle dracut optional modules in a different way.
>
>  snip
>
>  >
>  > I think UKIs are fundamentally flawed and are an idea that came out of
>  > a group that doesn't really interact with real users enough to
>  > understand how much of a problem they actually are. I realize that
>  > this Change is only about VMs, but since it explicitly talks about it
>  > being "phase 1", the implication is that future Changes are coming to
>  > switch fully over. Consequently, I'm going to provide much more
>  > holistic feedback instead of just nitpicking on this case.
>
>  The future "phase 2/3" goal are mentioned above, and don't
>  mention fully switching to UKIs.
>
>  I don't anticipate such a switch being realistic any time
>  in the forseeable future. There are too many unknowns, in
>  addition to some points you mentioned, to even consider
>  that right now. The in place upgrade story gogin from
>  locally generated initrd, with arbitrary user chosen
>  content to a UKI doesn't even bare thinking about. I
>  don't think that's even possible to achieve in a seemless
>  way given the infinite number of variations. IOW, I would
>  anticipate the current kernel+initrd scheme being around
>  indefinitely, in parallel with any UKI support.
>  
Forgive me for being pessimistic here, because Change proposals like
this are not written in such a way. You also bundled in bootloader
changes and PCR/TPM stuff too. All of which are irrelevant, not
useful, and don't actually work for Linux users.

>  > In the Fedora case, things are simpler right up until we hit
> graphics
>  > drivers. This is also a problem for VMs too, because GPU passthrough
>  > is a common case for scientific and gaming workloads. As long as the
>  > NVIDIA driver remains dominant in Linux, UKIs cannot work because by
>  > design you cannot load anything that isn't part of the kernel image.
>  > For bare metal, we *need* these drivers in early boot, though. And
>  > that's another problem: no third-party early boot drivers. Even if you
>  > solve the signing issue, you need to introduce some kind of two-stage
>  > OS boot process so we can bring up the bare minimum to load a second
>  > image containing all the remaining drivers. And at that point, you've
>  > defeated the purpose of UKIs. I've heard from some people that system
>  > extensions (sysexts) would be a way to solve this, and maybe it is.
>
>  Yes, system extensions are one mechanism for making the initrd
>  content more flexible when using UKIs. There would be one base
>  layer definiing the 90% common case, and a number of add-ons
>  that can cope with niche use cases. This avoids the core UKI
>  having to be huge and ship with every possible feature present.
>
>  This is something that might be considered at a later phase,
>  but is not a priority. What's proposed in phase 1 is sufficient
>  to cope with the cloud VM common case. While device passthrough
>  may be common in some industries/domains usage, it is not the
>  common case for cloud computing in general.
>
>  > But again, we've eliminated the value of UKIs by doing so.
>
>  That is not correct. There are a number of benefits of UKIs.
>
>  The most critical is that the initrd content and cmdline is
>  covered by the SecureBoot signature. This remains true even
>  with system extensions, as such extensions would be signed
>  too. They do not neccessarily need to be signed by the OS
>  vendor, they could use a 3rd party SecureBoot signing key,
>  or the users' own key. That is TBD and not something we're
>  actively considering - its not even mentioned in phase 2/3
>  ideas in the change proposal.
>  
Secure Boot keys are flawed. There are many documented problems with
relying on Secure Boot keys in the real world. Redesign to not use
them, please.

>  A further benefit of UKIs is supportability, since we know
>  exactly what content is in the initrd that user is booting.
>  With locally generated initrds there are an arbitrary
>  number of combinations and many ways for the initrd creation
>  process to go wrong, which are hard for Fedora maintainers
>  to understand and debug.
>
>  > Speaking more broadly, there are also problems that will be introduced
>  > as this trickles down from Fedora into prominent downstreams (assuming
>  > this is accepted). In the RHEL case, you've basically broken driver
>  > disks completely. In the UKI model, there's no way to incorporate
>  > early boot storage, network, and graphics drivers. This is
>  > *incredibly* common for RHEL administrators because there's a general
>  > acceptance of proprietary kernel modules from vendors these days. Even
>  > ignoring those, Red Hat's kernel feature support mindset is completely
>  > incompatible with UKIs, because RHEL does default-off rather than
>  > Fedora's default-on model for kernel features. We could debate until
>  > the cows come home on whether it's right or wrong, but their current
>  > mindset essentially means tons of common hardware becomes unsupported
>  > quite regularly. The ELRepo project is popular among RHEL folks
>  > because it restores those drivers and makes it possible to use RHEL on
>  > those machines through a combination of driver disks and kernel module
>  > packages.
>
>  Yes, this desire for UKIs in Fedora is matched by a desire to
>  ship the same UKIs in RHEL too.
>
>  No one is proposing to take away the existing ability to use
>  locally generated initrds, either in Fedora or RHEL. Everything
>  that is possible todo today will remain possible indefinitely.
>  UKIs are adding a new mechanism in parallel, in order to solve
>  a number of use cases that are not possible today, due to
>  unsigned content being part of the boot chain when SecureBoot
>  is active.
>
>  > The crux of the problem here is *signing*. All of this is tied up in
>  > Secure Boot and TPM, which is the wrong place to actually handle this.
>  > In other operating systems (notably Windows), Secure Boot certificates
>  > are not used for blocking or enabling kernel-level drivers. Instead,
>  > there's a kernel-level certificate store that is used to validate and
>  > enable drivers, allowing everything to be managed in a hardware
>  > platform agnostic way. I've also yet to hear of a decent reason for
>  > TPM measurements too. Block or filesystem verity provides similar
>  > guarantees to provide tamper resistance and are both much easier to
>  > debug than TPM interfaces. I am not convinced that you are providing
>  > security or reliability with this and the trade-offs are *terrible*
>  > for regular users.
>
>  The immediate need for UKIs is indeed related to SecureBoot and
>  TPMs. These are a core technology foundation of the confidential
>  virtual machine stack. On Azure today, if you request an Ubuntu
>  confidential VM, Azure will pre-encrypt the root filesystem and
>  seal the LUKS key against predicted TPM PCR values. It guarantees
>  that the root disk can only be decrypted by the specific VM
>  instance that is requested, when it is running in SecureBoot
>  mode with the expected measurments on AMD SEV-SNP confidential
>  hardware. The Ubuntu image in Azure already uses UKIs, and boots
>  them directly from shim, with no bootloader involved. The usage
>  of SecureBoot and TPMs is all transparent to the user, since all
>  the integration is handled by the OS vendor on their behalf.
>  
Well, how is that relevant to Fedora? We don't have Azure images, and
Red Hat still blocks us from meaningfully existing in that ecosystem,
which is why we don't have official WSL images or Azure images.

I'll reiterate that we should not rely on SB signing for this.

>  > With my FESCo hat on, I'm uneasy about this change. With my
> "Fedora
>  > user and advocate" hat on, I think that the UAPI group has failed to
>  > provide something useful to the Linux world here and I would be
>  > extremely apprehensive about Fedora adopting any portion of this
>  > stuff.
>
>  As mentioned above, UKIs have provided a means to close the
>  SecureBoot hole with unsigned initrd content that has existed
>  in more or less every mainstream Linux distro. That is a
>  clearly useful outcome, regardless of whether Fedora is interested
>  in any other aspect of what that UAPI groups is proposing.
>
>  In proposing the UKI support for Fedora, we're not coming at this
>  as representatives of the UAPI group or its vision. We're trying
>  to solve the problem of having a fully verified secureboot chain
>  for VMs with no unsigned content, and to be able to use this to
>  support confidential VMs with disk encryption sealed to TPMs, as
>  required by Azure today, and likely KVM in future. 
Yeah, I seriously doubt this. Linux's model for supporting
confidential computing is not user-friendly, so I expect low adoption
and resistance once the flaws become apparent to would-be users.



--
真実はいつも一つ！/ Always, there's only one truth!

[Neal Gompa]

On Tue, Dec 20, 2022, 2:29 PM Neal Gompa <ngompa13(a)gmail.com&gt; wrote:

>  On Tue, Dec 20, 2022 at 2:02 PM Daniel P. Berrangé
> <berrange(a)redhat.com
> >  
>  wrote:
> >  
>  > On Tue, Dec 20, 2022 at 11:28:48AM -0500, Neal Gompa
> wrote:
>  > > On Tue, Dec 20, 2022 at 10:22 AM Ben Cotton <bcotton(a)redhat.com
> >  
>  wrote:
>  > >
> >  
>  > >
> >  
> https://fedoraproject.org/wiki/Changes/Unified_Kernel_Support_Phase_1
>  > >
> >  
>  > > > This document
> represents a proposed Change. As part of the Changes
>  > > > process, proposals are publicly announced in order to receive
>  > > > community feedback. This proposal will only be implemented if
>  approved
>  > > > by the Fedora Engineering Steering Committee.
>  > >
> >  
>  > >
> >  
>  > > > == Summary ==
>  > > > Add support for unified kernels images to Fedora.
>  > >
> >  
>  > > > == Owner ==
>  > > > * Name: [[User:kraxel| Gerd Hoffmann]]
>  > > > * Email: kraxel(a)redhat.com
>  > >
> >  
>  > >
> >  
>  > > > == Detailed Description ==
>  > > > The goal is to move away from initrd images being generated on the
>  > > > installed machine.  They are generated while building the kernel
>  > > > package instead, then shipped as part of a unified kernel image.
>  > >
> >  
>  > > > A unified kernel
> image is an all-in-one efi binary containing kernel,
>  > > > initrd, cmdline and signature.  The secure boot signature covers
>  > > > everything, specifically the initrd is included which is not the case
>  > > > when the initrd gets loaded as separate file from /boot.
>  > >
> >  
>  > > > Main motivation for
> this move is to make the distro more robust and
>  more secure.
>  > >
> >  
>  > > > Switching the whole
> distro over to unified kernels quickly is not
>  > > > realistic though.  Too many features are depending on the current
>  > > > workflow with a host-specific initrd (and host-specific kernel
>  command
>  > > > line), which is fundamentally incompatible with unified kernels where
>  > > > everybody will have the same initrd and command line. Thats why there
>  > > > is 'Phase 1' in title, so we can have more Phases in future
> releases
>  > > > 😃
>  > >
> >  
>  > > > A host-specific
> initrd / command line is needed today for:
>  > >
> >  
>  > > > * features needing
> optional dracut modules (initrd rebuild needed to
>  > > > enable them).
>  > > > * configuration / secrets baked into the initrd (booting from iscsi
>  > > > for example).
>  > > > * configuration being specified on the kernel command line.
>  > > > ** root filesystem being the most important one.
>  > > > [https://systemd.io/DISCOVERABLE_PARTITIONS/ Discoverable
>  partitions]
>  > > > allow to remove this.
>  > >
> >  
>  > > > Phase 1 goals (high
> priority):
>  > >
> >  
>  > > > * Ship a unified
> kernel image as (optional) kernel sub-rpm.  Users
>  can
>  > > > opt-in to use that kernel by installing the sub-rpm.  Initial focus
>  is
>  > > > on booting virtual machines where we have a relatively small and well
>  > > > defined set of drivers / features needed.  Supporting modern physical
>  > > > machines with standard setup (i.e. boot from local sata/nvme storage)
>  > > > too should be easy.
>  > > > * Update kernel install scripts so unified kernels are installed and
>  > > > updated properly.
>  > > > * Add bootloader support for unified kernel images.  Add
>  > > > [
>  https://systemd.io/BOOT_LOADER_SPECIFICATION/#type-2-efi-unified-kernel-i...
>  > > > unified kernel bls support] to grub2, or support using systemd-boot,
>  > > > or both.
>  > >
> >  
>  > > > Phase 1 goals (lower
> priority, might move to Phase 2):
>  > >
> >  
>  > > > * Add proper
> discoverable partitions support to installers (anaconda,
>  > > > image builder, ...).
>  > > > ** Temporary workaround possible: set types using sfdisk in %post
>  script.
>  > > > ** When using btrfs: configure 'root' subvolume as default
> volume.
>  > > > * Add proper systemd-boot support to installers.
>  > > > ** Temporary workaround possible: run 'bootctl install' in %post
>  script.
>  > > > * Better measurement and remote attestation support.
>  > > > ** store kernel + initrd hashes somewhere (kernel-hashes.rpm ?) to
>  > > > allow pre-calculate TPM PCR values.
>  > > > ** avoid using grub2 (measures every config file line executed which
>  > > > is next to impossible to pre-calculate).
>  > > > * Switch cloud images to use unified kernels.
>  > >
> >  
>  > > > Phase 2/3 goals
> (longer-term stuff which is not realistic to
>  complete for F38).
>  > >
> >  
>  > > > * Move away from
> using the kernel command line for configuration.
>  > > > * Move away from storing secrets in the initrd.
>  > > > * Handle dracut optional modules in a different way.
> >  
>  > snip
> >  
>  >
> >  
>  > > I think UKIs are
> fundamentally flawed and are an idea that came out of
>  > > a group that doesn't really interact with real users enough to
>  > > understand how much of a problem they actually are. I realize that
>  > > this Change is only about VMs, but since it explicitly talks about it
>  > > being "phase 1", the implication is that future Changes are coming
> to
>  > > switch fully over. Consequently, I'm going to provide much more
>  > > holistic feedback instead of just nitpicking on this case.
> >  
>  > The future "phase 2/3" goal are mentioned
> above, and don't
>  > mention fully switching to UKIs.
> >  
>  > I don't anticipate such a switch being realistic any
> time
>  > in the forseeable future. There are too many unknowns, in
>  > addition to some points you mentioned, to even consider
>  > that right now. The in place upgrade story gogin from
>  > locally generated initrd, with arbitrary user chosen
>  > content to a UKI doesn't even bare thinking about. I
>  > don't think that's even possible to achieve in a seemless
>  > way given the infinite number of variations. IOW, I would
>  > anticipate the current kernel+initrd scheme being around
>  > indefinitely, in parallel with any UKI support.
> >  
>  Forgive me for being pessimistic here, because Change proposals like
>  this are not written in such a way. You also bundled in bootloader
>  changes and PCR/TPM stuff too. All of which are irrelevant, not
>  useful, and don't actually work for Linux users.
>
>  > > In the Fedora case, things are simpler right up until we hit graphics
>  > > drivers. This is also a problem for VMs too, because GPU passthrough
>  > > is a common case for scientific and gaming workloads. As long as the
>  > > NVIDIA driver remains dominant in Linux, UKIs cannot work because by
>  > > design you cannot load anything that isn't part of the kernel image.
>  > > For bare metal, we *need* these drivers in early boot, though. And
>  > > that's another problem: no third-party early boot drivers. Even if you
>  > > solve the signing issue, you need to introduce some kind of two-stage
>  > > OS boot process so we can bring up the bare minimum to load a second
>  > > image containing all the remaining drivers. And at that point, you've
>  > > defeated the purpose of UKIs. I've heard from some people that system
>  > > extensions (sysexts) would be a way to solve this, and maybe it is.
> >  
>  > Yes, system extensions are one mechanism for making the
> initrd
>  > content more flexible when using UKIs. There would be one base
>  > layer definiing the 90% common case, and a number of add-ons
>  > that can cope with niche use cases. This avoids the core UKI
>  > having to be huge and ship with every possible feature present.
> >  
>  > This is something that might be considered at a later
> phase,
>  > but is not a priority. What's proposed in phase 1 is sufficient
>  > to cope with the cloud VM common case. While device passthrough
>  > may be common in some industries/domains usage, it is not the
>  > common case for cloud computing in general.
> >  
>  > > But again, we've eliminated the value of UKIs by
> doing so.
> >  
>  > That is not correct. There are a number of benefits of
> UKIs.
> >  
>  > The most critical is that the initrd content and cmdline
> is
>  > covered by the SecureBoot signature. This remains true even
>  > with system extensions, as such extensions would be signed
>  > too. They do not neccessarily need to be signed by the OS
>  > vendor, they could use a 3rd party SecureBoot signing key,
>  > or the users' own key. That is TBD and not something we're
>  > actively considering - its not even mentioned in phase 2/3
>  > ideas in the change proposal.
> >  
>  Secure Boot keys are flawed. There are many documented problems with
>  relying on Secure Boot keys in the real world. Redesign to not use
>  them, please.
>
>  > A further benefit of UKIs is supportability, since we know
>  > exactly what content is in the initrd that user is booting.
>  > With locally generated initrds there are an arbitrary
>  > number of combinations and many ways for the initrd creation
>  > process to go wrong, which are hard for Fedora maintainers
>  > to understand and debug.
> >  
>  > > Speaking more broadly, there are also problems that
> will be introduced
>  > > as this trickles down from Fedora into prominent downstreams (assuming
>  > > this is accepted). In the RHEL case, you've basically broken driver
>  > > disks completely. In the UKI model, there's no way to incorporate
>  > > early boot storage, network, and graphics drivers. This is
>  > > *incredibly* common for RHEL administrators because there's a general
>  > > acceptance of proprietary kernel modules from vendors these days. Even
>  > > ignoring those, Red Hat's kernel feature support mindset is completely
>  > > incompatible with UKIs, because RHEL does default-off rather than
>  > > Fedora's default-on model for kernel features. We could debate until
>  > > the cows come home on whether it's right or wrong, but their current
>  > > mindset essentially means tons of common hardware becomes unsupported
>  > > quite regularly. The ELRepo project is popular among RHEL folks
>  > > because it restores those drivers and makes it possible to use RHEL on
>  > > those machines through a combination of driver disks and kernel module
>  > > packages.
> >  
>  > Yes, this desire for UKIs in Fedora is matched by a
> desire to
>  > ship the same UKIs in RHEL too.
> >  
>  > No one is proposing to take away the existing ability to
> use
>  > locally generated initrds, either in Fedora or RHEL. Everything
>  > that is possible todo today will remain possible indefinitely.
>  > UKIs are adding a new mechanism in parallel, in order to solve
>  > a number of use cases that are not possible today, due to
>  > unsigned content being part of the boot chain when SecureBoot
>  > is active.
> >  
>  > > The crux of the problem here is *signing*. All of
> this is tied up in
>  > > Secure Boot and TPM, which is the wrong place to actually handle this.
>  > > In other operating systems (notably Windows), Secure Boot certificates
>  > > are not used for blocking or enabling kernel-level drivers. Instead,
>  > > there's a kernel-level certificate store that is used to validate and
>  > > enable drivers, allowing everything to be managed in a hardware
>  > > platform agnostic way. I've also yet to hear of a decent reason for
>  > > TPM measurements too. Block or filesystem verity provides similar
>  > > guarantees to provide tamper resistance and are both much easier to
>  > > debug than TPM interfaces. I am not convinced that you are providing
>  > > security or reliability with this and the trade-offs are *terrible*
>  > > for regular users.
> >  
>  > The immediate need for UKIs is indeed related to
> SecureBoot and
>  > TPMs. These are a core technology foundation of the confidential
>  > virtual machine stack. On Azure today, if you request an Ubuntu
>  > confidential VM, Azure will pre-encrypt the root filesystem and
>  > seal the LUKS key against predicted TPM PCR values. It guarantees
>  > that the root disk can only be decrypted by the specific VM
>  > instance that is requested, when it is running in SecureBoot
>  > mode with the expected measurments on AMD SEV-SNP confidential
>  > hardware. The Ubuntu image in Azure already uses UKIs, and boots
>  > them directly from shim, with no bootloader involved. The usage
>  > of SecureBoot and TPMs is all transparent to the user, since all
>  > the integration is handled by the OS vendor on their behalf.
> >  
>  Well, how is that relevant to Fedora? We don't have Azure images, and
>  Red Hat still blocks us from meaningfully existing in that ecosystem,
>  which is why we don't have official WSL images or Azure images.
>
>  I'll reiterate that we should not rely on SB signing for this.
>
>  > > With my FESCo hat on, I'm uneasy about this change. With my "Fedora
>  > > user and advocate" hat on, I think that the UAPI group has failed to
>  > > provide something useful to the Linux world here and I would be
>  > > extremely apprehensive about Fedora adopting any portion of this
>  > > stuff.
> >  
>  > As mentioned above, UKIs have provided a means to close
> the
>  > SecureBoot hole with unsigned initrd content that has existed
>  > in more or less every mainstream Linux distro. That is a
>  > clearly useful outcome, regardless of whether Fedora is interested
>  > in any other aspect of what that UAPI groups is proposing.
> >  
>  > In proposing the UKI support for Fedora, we're not
> coming at this
>  > as representatives of the UAPI group or its vision. We're trying
>  > to solve the problem of having a fully verified secureboot chain
>  > for VMs with no unsigned content, and to be able to use this to
>  > support confidential VMs with disk encryption sealed to TPMs, as
>  > required by Azure today, and likely KVM in future.
>
>  Yeah, I seriously doubt this. Linux's model for supporting
>  confidential computing is not user-friendly, so I expect low adoption
>  and resistance once the flaws become apparent to would-be users.
>  

Relevant blog article about in-VM disk encryption in the cloud being
potentially harmful:
https://tmp.bearblog.dev/disk-encryption-aws-is-close-to-useless-and-pote...

>

[Simo Sorce]

On Tue, 2022-12-20 at 14:29 -0500, Neal Gompa wrote:
>  Yeah, I seriously doubt this. Linux's model for supporting
>  confidential computing is not user-friendly, so I expect low adoption
>  and resistance once the flaws become apparent to would-be users.
>   
Neal, you are being unnecessarily negative. And user-friendliness is
directly related to the fact we do not have good support for it. This
proposal would make SecureBoot fundamentally transparent, and if you
don't see it and it works, I see no resistance happening.

SecureBoot may not be to your liking but is what is installed on 99% of
modern hardware sold, so it is a good idea to first show you can
support it. Then if you have interested you can propose "something
better".

Finally, unless this proposal harms Fedora I do not see why oppose it.
If, as you fear, it won't work ... then it won't and we'll try
something else. However, having some knowledge of the (security side of
the) matter I do not see why it wouldn't work, once all the pieces fall
in place.

In fact I would love to be able to test this, every time I run updates
I dread the many minutes wasted regenerating initrd when I have a
pretty standard configuration that requires really no special
drivers... the only issue probably being the use of LVM for the root
filesystem, which I hope we'll have a way to deal with (but I can do
without on the laptop).

Simo.

-- 
Simo Sorce
RHEL Crypto Team
Red Hat, Inc

[Neal Gompa]

On Tue, Dec 20, 2022, 4:27 PM Simo Sorce <simo(a)redhat.com&gt; wrote:

>  On Tue, 2022-12-20 at 14:29 -0500, Neal Gompa wrote:
>  > Yeah, I seriously doubt this. Linux's model for supporting
>  > confidential computing is not user-friendly, so I expect low adoption
>  > and resistance once the flaws become apparent to would-be users.
> >  
>  Neal, you are being unnecessarily negative. And user-friendliness is
>  directly related to the fact we do not have good support for it. This
>  proposal would make SecureBoot fundamentally transparent, and if you
>  don't see it and it works, I see no resistance happening.
>
>  SecureBoot may not be to your liking but is what is installed on 99% of
>  modern hardware sold, so it is a good idea to first show you can
>  support it. Then if you have interested you can propose "something
>  better".
>  
We have supported Secure Boot for over a decade now. In that timeframe,
literally nobody did anything to make all the workflows you talk about
easier and friendlier.


In fact, everyone I talk to seems to think it's basically impossible
because of how it works at the firmware level.


It's telling that neither Windows nor macOS use Secure Boot like Linux
does. And they don't precisely for the reasons I outlined.

Finally, unless this proposal harms Fedora I do not see why oppose it.
>  If, as you fear, it won't work ... then it won't and
> we'll try
>  something else. However, having some knowledge of the (security side of
>  the) matter I do not see why it wouldn't work, once all the pieces fall
>  in place.
>  
This adds significant complexity to the Fedora kernel package and it
effectively increases what we need to test for virtualized Fedora Linux
environments.

I assert that the proposal has not yet met the bar to overcome those issues.


In fact I would love to be able to test this, every time I run updates
>  I dread the many minutes wasted regenerating initrd when I have a
>  pretty standard configuration that requires really no special
>  drivers... the only issue probably being the use of LVM for the root
>  filesystem, which I hope we'll have a way to deal with (but I can do
>  without on the laptop).
>  
UKIs need more work to be generally useful, but that would be nice to
eliminate if the issues could be fixed.

>

[Vitaly Zaitsev]

On 20/12/2022 23:11, Neal Gompa wrote:
>  We have supported Secure Boot for over a decade now. In that
> timeframe, 
>  literally nobody did anything to make all the workflows you talk about 
>  easier and friendlier. 
Microsoft wants to limit[1] the use of non-Windows operating systems on 
new hardware:

>  To trust and boot operating systems, like Linux, and components
> signed by the UEFI signature, Secured-core PCs can be configured in the BIOS menu to add
> the signature in the UEFI database by following these step 
Users will need to go to
the UEFI BIOS settings menu, find the Security 
tab there, and import the Microsoft 3rd Party UEFI CA.

Lenovo already started doing that[2].

[1]: 
https://learn.microsoft.com/en-us/windows/security/information-protection...
[2]: https://www.phoronix.com/news/Lenovo-Pluton-Windows-Default

-- 
Sincerely,
   Vitaly Zaitsev (vitaly(a)easycoding.org)

[Daniel P. Berrangé]

On Tue, Dec 20, 2022 at 05:11:57PM -0500, Neal Gompa wrote:
>  On Tue, Dec 20, 2022, 4:27 PM Simo Sorce <simo(a)redhat.com&gt;
> wrote:
>  
>  > On Tue, 2022-12-20 at 14:29 -0500, Neal Gompa wrote:
>  > > Yeah, I seriously doubt this. Linux's model for supporting
>  > > confidential computing is not user-friendly, so I expect low adoption
>  > > and resistance once the flaws become apparent to would-be users.
>  > >
>  >
>  > Neal, you are being unnecessarily negative. And user-friendliness is
>  > directly related to the fact we do not have good support for it. This
>  > proposal would make SecureBoot fundamentally transparent, and if you
>  > don't see it and it works, I see no resistance happening.
>  >
>  > SecureBoot may not be to your liking but is what is installed on 99% of
>  > modern hardware sold, so it is a good idea to first show you can
>  > support it. Then if you have interested you can propose "something
>  > better".
>  >
>  
>  We have supported Secure Boot for over a decade now. In that timeframe,
>  literally nobody did anything to make all the workflows you talk about
>  easier and friendlier. 
What's done/not done in the past should not constrain us from
trying to improve the future.

>  In fact, everyone I talk to seems to think it's basically
> impossible
>  because of how it works at the firmware level. 
That is a challenge at bare metal, but with virtualization many
of the problems can be addressed. We should not in fact even need
to rely on Microsoft signing certs for virtualization. It is possible
to pre-enroll the certs associated with the guest OS the user intends
to boot. In fact it is desirable todo this, since it gives the VM
much greater protection if it only trusts the guest OS vendor certs.

>  Finally, unless this proposal harms Fedora I do not see why oppose
> it.
>  > If, as you fear, it won't work ... then it won't and we'll try
>  > something else. However, having some knowledge of the (security side of
>  > the) matter I do not see why it wouldn't work, once all the pieces fall
>  > in place.
>  >
>  
>  This adds significant complexity to the Fedora kernel package and it
>  effectively increases what we need to test for virtualized Fedora Linux
>  environments. 
The complexity to the kernel pacakge is negligible and has been
presented to the kernel maintainers already and not had any
unsolvable concerns raised in response.

There is an increase in testing matrix, but this path is chosen
primarily to ensure we can limit the burden long term. What we
don't want is to end up in a situation where confidential VMs
use a completely different OS setup / integration to normal VMs,
which would entail shipping many extra cloud disk images. The
approach we're taking aims to make it possible to have a single
cloud disk image cover all virtualization deployment scenarios,
legacy BIOS, UEFI (with SB+TPM), and UEFI with confidential VMs
to minimize the burden of what we deliver for cloud.

With regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

[Lennart Poettering]

On Di, 20.12.22 17:11, Neal Gompa (ngompa13(a)gmail.com) wrote:

>  > SecureBoot may not be to your liking but is what is installed on
> 99% of
>  > modern hardware sold, so it is a good idea to first show you can
>  > support it. Then if you have interested you can propose "something
>  > better".
>
>  We have supported Secure Boot for over a decade now. In that timeframe,
>  literally nobody did anything to make all the workflows you talk about
>  easier and friendlier.
>
>  In fact, everyone I talk to seems to think it's basically impossible
>  because of how it works at the firmware level.
>
>  It's telling that neither Windows nor macOS use Secure Boot like Linux
>  does. And they don't precisely for the reasons I outlined. 
Yes, Linux never solved the initrd hole so far, but that's not really
the fault of SecureBoot, it's the fault of Linux if you so
will. And this proposal is an attempt to finally do something about
this, and get things in order to actually deliver what the other OSes
all are delivering.

I understand the big issue you have is that the certificate store for
Linux (i.e. the mokutil db) is limited in size because EFI variable
NVRAM is limited in size? If that's the big issue you are having then
that's absolutely something the Linux community can fix, and can
address. Specifically, shim could allow storing the cert store on
disk, and authenticate it at boot via the TPM. Not trivial, but
doable. And certainly not the firmware's fault...

I mean, UEFI has its warts, but I don't see that it's UEFI's fault if
the way Linux/shim/mokutil implement the cert db is done the way it is
currently done.

>  > Finally, unless this proposal harms Fedora I do not see why
> oppose it.
>  > If, as you fear, it won't work ... then it won't and we'll try
>  > something else. However, having some knowledge of the (security side of
>  > the) matter I do not see why it wouldn't work, once all the pieces fall
>  > in place.
>
>  This adds significant complexity to the Fedora kernel package and it
>  effectively increases what we need to test for virtualized Fedora Linux
>  environments. 
Nah. UKIs + sysext are ultimately something that is simpler and much
better to test than the current mess. Yeah, for a while we'll have to
add complexity because to mechanisms will have to be supported, but
eventually things are going to be simple and easier to test.

>  I assert that the proposal has not yet met the bar to overcome those
>  issues. 
If the "those issues" is supposed to be that you hit the size of the
mokutil cert store once, then I fail to see the relationship to
UKI/sysext. After all, the fedora signing keys for sysexts would be
built into the kernel and hence not be charged against NVRAM. And the
fedora UKI is signed with the same key as our current kernels, which
are also not charged against NVRAM but are built into fedora shim. And
the shim signature key is the msft key.

Yeah, if you want to build your own UKIs + sysext, then you have to
use mokutil to enroll a cert, but it would be entirely sufficient to
enroll one for that, and sign UKIS, kmods, sysext with them.

(or, maybe you actually hit the NVRAM size limits because you enrolled
hashes, not certs? if so, then maybe address that?)

Lennart

--
Lennart Poettering, Berlin

[Neal Gompa]

On Wed, Dec 21, 2022 at 1:56 PM Lennart Poettering <mzerqung(a)0pointer.de&gt; wrote:
>  On Di, 20.12.22 17:11, Neal Gompa (ngompa13(a)gmail.com) wrote:
>
>  > > SecureBoot may not be to your liking but is what is installed on 99% of
>  > > modern hardware sold, so it is a good idea to first show you can
>  > > support it. Then if you have interested you can propose "something
>  > > better".
>  >
>  > We have supported Secure Boot for over a decade now. In that timeframe,
>  > literally nobody did anything to make all the workflows you talk about
>  > easier and friendlier.
>  >
>  > In fact, everyone I talk to seems to think it's basically impossible
>  > because of how it works at the firmware level.
>  >
>  > It's telling that neither Windows nor macOS use Secure Boot like Linux
>  > does. And they don't precisely for the reasons I outlined.
>
>  Yes, Linux never solved the initrd hole so far, but that's not really
>  the fault of SecureBoot, it's the fault of Linux if you so
>  will. And this proposal is an attempt to finally do something about
>  this, and get things in order to actually deliver what the other OSes
>  all are delivering.
>
>  I understand the big issue you have is that the certificate store for
>  Linux (i.e. the mokutil db) is limited in size because EFI variable
>  NVRAM is limited in size? If that's the big issue you are having then
>  that's absolutely something the Linux community can fix, and can
>  address. Specifically, shim could allow storing the cert store on
>  disk, and authenticate it at boot via the TPM. Not trivial, but
>  doable. And certainly not the firmware's fault...
>
>  I mean, UEFI has its warts, but I don't see that it's UEFI's fault if
>  the way Linux/shim/mokutil implement the cert db is done the way it is
>  currently done.
>  
Frankly, I'm aggravated by the increased reliance on UEFI features
without fixing Linux's problems with UEFI features. If we stopped
relying on UEFI for the cert store, a lot of my issues would go away,
because then we can design better workflows for managing certificates.
It makes automation possible, it makes management possible, and it
opens the doors to experiment with how to do layered images for boot
(e.g. UKIs + system extension images) without bricking people's
computers.

That also has the side effect of us having half a chance of being able
to port this security capability to non-UEFI platforms where we use
fake UEFI implementations to bootstrap our boot process, because the
amount of coverage required for fake UEFI implementations is
considerably lower now.

More generally, relying on UEFI-specific security features when most
platforms are not being brought up with UEFI is foolish. ARM is almost
entirely non-UEFI (except the tiny proportion of servers that almost
no one has) and RISC-V is not a UEFI platform either. POWER uses
OpenFirmware instead of UEFI, and IBM Z does something else entirely
different.

You *need* to think about these problems when designing this stuff.
You can't take a myopic x86-only view to this. I have not heard of
anything about UKI security for non-x86 because it seems to all be
tied up in UEFI stuff.

Coming back to "my issue", I've had this problem for six years, and
for most of it "the Linux community" (specifically the Linux boot
community) has brushed me off saying my issue doesn't matter. Even if
it leads to bricked systems. Even if it makes it hard for people to
use their computers. Even if it leads to people turning off Secure
Boot. And finally, even if it means people don't use Linux at all
because they can't make it work.

>  > > Finally, unless this proposal harms Fedora I do not see why
> oppose it.
>  > > If, as you fear, it won't work ... then it won't and we'll try
>  > > something else. However, having some knowledge of the (security side of
>  > > the) matter I do not see why it wouldn't work, once all the pieces fall
>  > > in place.
>  >
>  > This adds significant complexity to the Fedora kernel package and it
>  > effectively increases what we need to test for virtualized Fedora Linux
>  > environments.
>
>  Nah. UKIs + sysext are ultimately something that is simpler and much
>  better to test than the current mess. Yeah, for a while we'll have to
>  add complexity because to mechanisms will have to be supported, but
>  eventually things are going to be simple and easier to test.
>  
Maybe. It's not super intuitive from where I'm standing, and I know
how all this stuff works.

>  > I assert that the proposal has not yet met the bar to overcome
> those
>  > issues.
>
>  If the "those issues" is supposed to be that you hit the size of the
>  mokutil cert store once, then I fail to see the relationship to
>  UKI/sysext. After all, the fedora signing keys for sysexts would be
>  built into the kernel and hence not be charged against NVRAM. And the
>  fedora UKI is signed with the same key as our current kernels, which
>  are also not charged against NVRAM but are built into fedora shim. And
>  the shim signature key is the msft key.
>
>  Yeah, if you want to build your own UKIs + sysext, then you have to
>  use mokutil to enroll a cert, but it would be entirely sufficient to
>  enroll one for that, and sign UKIS, kmods, sysext with them.
>
>  (or, maybe you actually hit the NVRAM size limits because you enrolled
>  hashes, not certs? if so, then maybe address that?)
>  
I do not want to add more things to Fedora that *will* cause people to
potentially brick their systems because they have to screw around with
UEFI to be able to extend what they can use to boot their systems.

Just because today it's only about VMs doesn't mean I can't figure out
you want to do more with it down the road. I want to see some
demonstration of someone actually caring about the user experience for
once with the boot stuff.




--
真実はいつも一つ！/ Always, there's only one truth!

[Daniel P. Berrangé]

On Thu, Dec 22, 2022 at 05:38:01AM -0500, Neal Gompa wrote:
>  On Wed, Dec 21, 2022 at 1:56 PM Lennart Poettering
> <mzerqung(a)0pointer.de&gt; wrote:
>  >
>  > On Di, 20.12.22 17:11, Neal Gompa (ngompa13(a)gmail.com) wrote:
>  >
>  > > > SecureBoot may not be to your liking but is what is installed on 99% of
>  > > > modern hardware sold, so it is a good idea to first show you can
>  > > > support it. Then if you have interested you can propose "something
>  > > > better".
>  > >
>  > > We have supported Secure Boot for over a decade now. In that timeframe,
>  > > literally nobody did anything to make all the workflows you talk about
>  > > easier and friendlier.
>  > >
>  > > In fact, everyone I talk to seems to think it's basically impossible
>  > > because of how it works at the firmware level.
>  > >
>  > > It's telling that neither Windows nor macOS use Secure Boot like Linux
>  > > does. And they don't precisely for the reasons I outlined.
>  >
>  > Yes, Linux never solved the initrd hole so far, but that's not really
>  > the fault of SecureBoot, it's the fault of Linux if you so
>  > will. And this proposal is an attempt to finally do something about
>  > this, and get things in order to actually deliver what the other OSes
>  > all are delivering.
>  >
>  > I understand the big issue you have is that the certificate store for
>  > Linux (i.e. the mokutil db) is limited in size because EFI variable
>  > NVRAM is limited in size? If that's the big issue you are having then
>  > that's absolutely something the Linux community can fix, and can
>  > address. Specifically, shim could allow storing the cert store on
>  > disk, and authenticate it at boot via the TPM. Not trivial, but
>  > doable. And certainly not the firmware's fault...
>  >
>  > I mean, UEFI has its warts, but I don't see that it's UEFI's fault if
>  > the way Linux/shim/mokutil implement the cert db is done the way it is
>  > currently done.
>  >
>  
>  Frankly, I'm aggravated by the increased reliance on UEFI features
>  without fixing Linux's problems with UEFI features. If we stopped
>  relying on UEFI for the cert store, a lot of my issues would go away,
>  because then we can design better workflows for managing certificates.
>  It makes automation possible, it makes management possible, and it
>  opens the doors to experiment with how to do layered images for boot
>  (e.g. UKIs + system extension images) without bricking people's
>  computers. 
So your objections are completely unrelated to the use of UKIs,
and should not be a blocker for this feature proposal. You're
just asking people to work on other UEFI related problems.

>  > > I assert that the proposal has not yet met the bar to
> overcome those
>  > > issues.
>  >
>  > If the "those issues" is supposed to be that you hit the size of the
>  > mokutil cert store once, then I fail to see the relationship to
>  > UKI/sysext. After all, the fedora signing keys for sysexts would be
>  > built into the kernel and hence not be charged against NVRAM. And the
>  > fedora UKI is signed with the same key as our current kernels, which
>  > are also not charged against NVRAM but are built into fedora shim. And
>  > the shim signature key is the msft key.
>  >
>  > Yeah, if you want to build your own UKIs + sysext, then you have to
>  > use mokutil to enroll a cert, but it would be entirely sufficient to
>  > enroll one for that, and sign UKIS, kmods, sysext with them.
>  >
>  > (or, maybe you actually hit the NVRAM size limits because you enrolled
>  > hashes, not certs? if so, then maybe address that?)
>  >
>  
>  I do not want to add more things to Fedora that *will* cause people to
>  potentially brick their systems because they have to screw around with
>  UEFI to be able to extend what they can use to boot their systems. 
Bricking systems is not an issue for VMs, where this proposal
is targetted.

>  Just because today it's only about VMs doesn't mean I
> can't figure out
>  you want to do more with it down the road. I want to see some
>  demonstration of someone actually caring about the user experience for
>  once with the boot stuff. 
If something is proposed for bare metal in the future, then raise
the problems at that point. It is unreasonable to demand that we
fix problems for a use case that is not in scope for what is being
proposed.  Anything related to bare metal was explicitly out of
scope, precisely because it will massively increase the complexity
of what needs to be addressed, making the task infeasible. We need
an incremental path where we can tackle individual practical tasks
rather than trying to solve everything in one go.

With regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

[Neal Gompa]

On Thu, Dec 22, 2022 at 6:29 AM Daniel P. Berrangé <berrange(a)redhat.com&gt; wrote:
>  On Thu, Dec 22, 2022 at 05:38:01AM -0500, Neal Gompa wrote:
>  > On Wed, Dec 21, 2022 at 1:56 PM Lennart Poettering <mzerqung(a)0pointer.de&gt;
> wrote:
>  > >
>  > > On Di, 20.12.22 17:11, Neal Gompa (ngompa13(a)gmail.com) wrote:
>  > >
>  > > > > SecureBoot may not be to your liking but is what is installed on 99%
> of
>  > > > > modern hardware sold, so it is a good idea to first show you can
>  > > > > support it. Then if you have interested you can propose
> "something
>  > > > > better".
>  > > >
>  > > > We have supported Secure Boot for over a decade now. In that timeframe,
>  > > > literally nobody did anything to make all the workflows you talk about
>  > > > easier and friendlier.
>  > > >
>  > > > In fact, everyone I talk to seems to think it's basically impossible
>  > > > because of how it works at the firmware level.
>  > > >
>  > > > It's telling that neither Windows nor macOS use Secure Boot like
> Linux
>  > > > does. And they don't precisely for the reasons I outlined.
>  > >
>  > > Yes, Linux never solved the initrd hole so far, but that's not really
>  > > the fault of SecureBoot, it's the fault of Linux if you so
>  > > will. And this proposal is an attempt to finally do something about
>  > > this, and get things in order to actually deliver what the other OSes
>  > > all are delivering.
>  > >
>  > > I understand the big issue you have is that the certificate store for
>  > > Linux (i.e. the mokutil db) is limited in size because EFI variable
>  > > NVRAM is limited in size? If that's the big issue you are having then
>  > > that's absolutely something the Linux community can fix, and can
>  > > address. Specifically, shim could allow storing the cert store on
>  > > disk, and authenticate it at boot via the TPM. Not trivial, but
>  > > doable. And certainly not the firmware's fault...
>  > >
>  > > I mean, UEFI has its warts, but I don't see that it's UEFI's fault
> if
>  > > the way Linux/shim/mokutil implement the cert db is done the way it is
>  > > currently done.
>  > >
>  >
>  > Frankly, I'm aggravated by the increased reliance on UEFI features
>  > without fixing Linux's problems with UEFI features. If we stopped
>  > relying on UEFI for the cert store, a lot of my issues would go away,
>  > because then we can design better workflows for managing certificates.
>  > It makes automation possible, it makes management possible, and it
>  > opens the doors to experiment with how to do layered images for boot
>  > (e.g. UKIs + system extension images) without bricking people's
>  > computers.
>
>  So your objections are completely unrelated to the use of UKIs,
>  and should not be a blocker for this feature proposal. You're
>  just asking people to work on other UEFI related problems.
>  
It is not unreasonable to have to deal with it for VMs either. Having
custom drivers needed for workloads where hardware passthrough occurs
is not unusual. While you can kind of handwave graphics, I've seen
storage and network devices passed in too.

>  > > > I assert that the proposal has not yet met the bar to
> overcome those
>  > > > issues.
>  > >
>  > > If the "those issues" is supposed to be that you hit the size of the
>  > > mokutil cert store once, then I fail to see the relationship to
>  > > UKI/sysext. After all, the fedora signing keys for sysexts would be
>  > > built into the kernel and hence not be charged against NVRAM. And the
>  > > fedora UKI is signed with the same key as our current kernels, which
>  > > are also not charged against NVRAM but are built into fedora shim. And
>  > > the shim signature key is the msft key.
>  > >
>  > > Yeah, if you want to build your own UKIs + sysext, then you have to
>  > > use mokutil to enroll a cert, but it would be entirely sufficient to
>  > > enroll one for that, and sign UKIS, kmods, sysext with them.
>  > >
>  > > (or, maybe you actually hit the NVRAM size limits because you enrolled
>  > > hashes, not certs? if so, then maybe address that?)
>  > >
>  >
>  > I do not want to add more things to Fedora that *will* cause people to
>  > potentially brick their systems because they have to screw around with
>  > UEFI to be able to extend what they can use to boot their systems.
>
>  Bricking systems is not an issue for VMs, where this proposal
>  is targetted.
>
>  > Just because today it's only about VMs doesn't mean I can't figure out
>  > you want to do more with it down the road. I want to see some
>  > demonstration of someone actually caring about the user experience for
>  > once with the boot stuff.
>
>  If something is proposed for bare metal in the future, then raise
>  the problems at that point. It is unreasonable to demand that we
>  fix problems for a use case that is not in scope for what is being
>  proposed.  Anything related to bare metal was explicitly out of
>  scope, precisely because it will massively increase the complexity
>  of what needs to be addressed, making the task infeasible. We need
>  an incremental path where we can tackle individual practical tasks
>  rather than trying to solve everything in one go.
>  
Yeah, and what happens when it gets punted again when that happens? I
do not think it's unreasonable to bring these objections up front when
this is clearly marked as a "phase 1" Change that implies UKIs will be
used in more and more scenarios over time.

Also, just because *you* intend it for VMs doesn't mean that's what is
going to happen. Someone is going to do something to try to use that
UKI in a bare metal deployment, possibly by using sysexts or squashfs
or god forbid OCI images.

I have to think about what happens when the cat is out of the bag.
What I want is not necessarily a solution to this now, but a
commitment that someone will actively work on fixing these problems
*before* proposing the next phase and have it ready *before* making
any future proposals on UKIs. If you can't do that, I can't in good
faith consider incrementally supporting UKIs, because there's
effectively no plan to make them work as a future default way of
shipping the Linux kernel.




--
真実はいつも一つ！/ Always, there's only one truth!

[Daniel P. Berrangé]

On Thu, Dec 22, 2022 at 06:38:56AM -0500, Neal Gompa wrote:
>  On Thu, Dec 22, 2022 at 6:29 AM Daniel P. Berrangé
> <berrange(a)redhat.com&gt; wrote:
>  >
>  > On Thu, Dec 22, 2022 at 05:38:01AM -0500, Neal Gompa wrote:
>  > > On Wed, Dec 21, 2022 at 1:56 PM Lennart Poettering <mzerqung(a)0pointer.de&gt;
> wrote:
>  > > >
>  > > > On Di, 20.12.22 17:11, Neal Gompa (ngompa13(a)gmail.com) wrote:
>  > > >
>  > > > > > SecureBoot may not be to your liking but is what is installed on
> 99% of
>  > > > > > modern hardware sold, so it is a good idea to first show you
> can
>  > > > > > support it. Then if you have interested you can propose
> "something
>  > > > > > better".
>  > > > >
>  > > > > We have supported Secure Boot for over a decade now. In that
> timeframe,
>  > > > > literally nobody did anything to make all the workflows you talk
> about
>  > > > > easier and friendlier.
>  > > > >
>  > > > > In fact, everyone I talk to seems to think it's basically
> impossible
>  > > > > because of how it works at the firmware level.
>  > > > >
>  > > > > It's telling that neither Windows nor macOS use Secure Boot like
> Linux
>  > > > > does. And they don't precisely for the reasons I outlined.
>  > > >
>  > > > Yes, Linux never solved the initrd hole so far, but that's not really
>  > > > the fault of SecureBoot, it's the fault of Linux if you so
>  > > > will. And this proposal is an attempt to finally do something about
>  > > > this, and get things in order to actually deliver what the other OSes
>  > > > all are delivering.
>  > > >
>  > > > I understand the big issue you have is that the certificate store for
>  > > > Linux (i.e. the mokutil db) is limited in size because EFI variable
>  > > > NVRAM is limited in size? If that's the big issue you are having then
>  > > > that's absolutely something the Linux community can fix, and can
>  > > > address. Specifically, shim could allow storing the cert store on
>  > > > disk, and authenticate it at boot via the TPM. Not trivial, but
>  > > > doable. And certainly not the firmware's fault...
>  > > >
>  > > > I mean, UEFI has its warts, but I don't see that it's UEFI's
> fault if
>  > > > the way Linux/shim/mokutil implement the cert db is done the way it is
>  > > > currently done.
>  > > >
>  > >
>  > > Frankly, I'm aggravated by the increased reliance on UEFI features
>  > > without fixing Linux's problems with UEFI features. If we stopped
>  > > relying on UEFI for the cert store, a lot of my issues would go away,
>  > > because then we can design better workflows for managing certificates.
>  > > It makes automation possible, it makes management possible, and it
>  > > opens the doors to experiment with how to do layered images for boot
>  > > (e.g. UKIs + system extension images) without bricking people's
>  > > computers.
>  >
>  > So your objections are completely unrelated to the use of UKIs,
>  > and should not be a blocker for this feature proposal. You're
>  > just asking people to work on other UEFI related problems.
>  >
>  
>  It is not unreasonable to have to deal with it for VMs either. Having
>  custom drivers needed for workloads where hardware passthrough occurs
>  is not unusual. While you can kind of handwave graphics, I've seen
>  storage and network devices passed in too. 
The initrd  in the UKI only needs to provide kmods needed for getting
the rootfs up, once that's available all other kmods are available.
It is reasonable to assume that any passthrough storage is not likely
to be used for the VM rootfs, instead for data volumes.


>  > > Just because today it's only about VMs doesn't mean
> I can't figure out
>  > > you want to do more with it down the road. I want to see some
>  > > demonstration of someone actually caring about the user experience for
>  > > once with the boot stuff.
>  >
>  > If something is proposed for bare metal in the future, then raise
>  > the problems at that point. It is unreasonable to demand that we
>  > fix problems for a use case that is not in scope for what is being
>  > proposed.  Anything related to bare metal was explicitly out of
>  > scope, precisely because it will massively increase the complexity
>  > of what needs to be addressed, making the task infeasible. We need
>  > an incremental path where we can tackle individual practical tasks
>  > rather than trying to solve everything in one go.
>  >
>  
>  Yeah, and what happens when it gets punted again when that happens? I
>  do not think it's unreasonable to bring these objections up front when
>  this is clearly marked as a "phase 1" Change that implies UKIs will be
>  used in more and more scenarios over time. 
The proposal gives an indication of what's expected

>  Also, just because *you* intend it for VMs doesn't mean
> that's what is
>  going to happen. Someone is going to do something to try to use that
>  UKI in a bare metal deployment, possibly by using sysexts or squashfs
>  or god forbid OCI images. 
Well people can do all this today, because they can use dracut
to build a UKI and sign it with certs they're enrolled with
shim. That doesn't imply that Fedora is on the hook to give them
solutions to problems they may hit when doing this.

>  I have to think about what happens when the cat is out of the bag.
>  What I want is not necessarily a solution to this now, but a
>  commitment that someone will actively work on fixing these problems
>  *before* proposing the next phase and have it ready *before* making
>  any future proposals on UKIs. If you can't do that, I can't in good
>  faith consider incrementally supporting UKIs, because there's
>  effectively no plan to make them work as a future default way of
>  shipping the Linux kernel. 
No one can accurately predict what they'll have the time to work
on in the future, especially if it is for a problem that is not
impacting a supported usage scenario they're addressing. So while
someone could make a committment, I wouldn't put any weight on it
being followed up in any particular timeframe.

If there is a future proposal for Fedora that actually targets the
scenarios with problems identified, then you can reasonably expect
someone to commit to addressing them & have confidence in them
following through on the commitment, because it would be a core
part of the Change proposed.

With regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

[Gerd Hoffmann]

  Hi,

>  > If something is proposed for bare metal in the future, then
> raise
>  > the problems at that point. It is unreasonable to demand that we
>  > fix problems for a use case that is not in scope for what is being
>  > proposed.  Anything related to bare metal was explicitly out of
>  > scope, precisely because it will massively increase the complexity
>  > of what needs to be addressed, making the task infeasible. We need
>  > an incremental path where we can tackle individual practical tasks
>  > rather than trying to solve everything in one go.
>  
>  Yeah, and what happens when it gets punted again when that happens? I
>  do not think it's unreasonable to bring these objections up front when
>  this is clearly marked as a "phase 1" Change that implies UKIs will be
>  used in more and more scenarios over time. 
I want UKIs becoming an option in more and more use cases.  I don't
expect non-UKIs disappearing anytime soon though.  From the updated
Change page:

<quote>
long term plan:
    Phase 1: Get the basic building blocks into place, so it is possible
             to work with and develop for UKIs in virtual machines.
    Phase 2+: Expand UKI support, tackling the use cases which depend on
              a host-specific initrd or command line (see below) one by one.
    Phase X: Once UKIs have feature parity with non-UKI kernels discuss
             whenever they should be used by default everywhere (specific
             use cases like cloud images might switch earlier).
    NOT planed: remove support for non-UKI kernels.
</quote>

I think at the end of the day this will be somewhat simliar to the Xorg
-> Wayland transition.  First get basic support there, so it is possible
to try out stuff, figure what works, figure what needs changes etc.
Then a (probably long) phase adapting software, fixing bugs found etc,
making more more use cases being able to work with the new stuff.
Then at some point eventually flip the default.

One notable difference is that with UKIs there isn't something simliar
to Xwayland, so flipping the default requires really everything being
able to work with UKIs.  And flipping the default can only happen for
new installs, I don't think trying to migrate existing installs to UKIs
automatically is a sane idea.

take care,
  Gerd

[Neal Gompa]

On Thu, Dec 22, 2022 at 7:32 AM Gerd Hoffmann <kraxel(a)redhat.com&gt; wrote:
>    Hi,
>
>  > > If something is proposed for bare metal in the future, then raise
>  > > the problems at that point. It is unreasonable to demand that we
>  > > fix problems for a use case that is not in scope for what is being
>  > > proposed.  Anything related to bare metal was explicitly out of
>  > > scope, precisely because it will massively increase the complexity
>  > > of what needs to be addressed, making the task infeasible. We need
>  > > an incremental path where we can tackle individual practical tasks
>  > > rather than trying to solve everything in one go.
>  >
>  > Yeah, and what happens when it gets punted again when that happens? I
>  > do not think it's unreasonable to bring these objections up front when
>  > this is clearly marked as a "phase 1" Change that implies UKIs will be
>  > used in more and more scenarios over time.
>
>  I want UKIs becoming an option in more and more use cases.  I don't
>  expect non-UKIs disappearing anytime soon though.  From the updated
>  Change page:
>
>  <quote>
>  long term plan:
>      Phase 1: Get the basic building blocks into place, so it is possible
>               to work with and develop for UKIs in virtual machines.
>      Phase 2+: Expand UKI support, tackling the use cases which depend on
>                a host-specific initrd or command line (see below) one by one.
>      Phase X: Once UKIs have feature parity with non-UKI kernels discuss
>               whenever they should be used by default everywhere (specific
>               use cases like cloud images might switch earlier).
>      NOT planed: remove support for non-UKI kernels.
>  </quote>
>
>  I think at the end of the day this will be somewhat simliar to the Xorg
>  -> Wayland transition.  First get basic support there, so it is possible
>  to try out stuff, figure what works, figure what needs changes etc.
>  Then a (probably long) phase adapting software, fixing bugs found etc,
>  making more more use cases being able to work with the new stuff.
>  Then at some point eventually flip the default.
>
>  One notable difference is that with UKIs there isn't something simliar
>  to Xwayland, so flipping the default requires really everything being
>  able to work with UKIs.  And flipping the default can only happen for
>  new installs, I don't think trying to migrate existing installs to UKIs
>  automatically is a sane idea.
>  
Hmm, the updated Change is mostly okay. I disagree that you have any
real security benefits since all the Secure Boot stuff in Linux is
still in a bad place. I would like the Change document to be updated
to include feedback about Secure Boot in there to further justify how
restricted the scope will remain for Phase 1.

Additionally, "discoverable partitions" fixes nothing for Fedora right
now. There are two problems here:
* We don't have a discoverable subvolume specification for Btrfs
* Discoverable partition specification falls over dead with snapshots.

Don't plan on using systemd-boot. I've said why before in other
threads, so I'm not repeating it again.

Drop locking out modified kernel command lines. That's pretty much a
non-starter.

If you want to pursue a Fedora Cloud image with UKIs, please bring it
up with the Cloud SIG, keeping in mind the feedback I listed.



-- 
真実はいつも一つ！/ Always, there's only one truth!

[Gerd Hoffmann]

  Hi,

>  Hmm, the updated Change is mostly okay. I disagree that you have any
>  real security benefits since all the Secure Boot stuff in Linux is
>  still in a bad place. 
It's a step into the right direction, but I agree, it alone doesn't
improve the situation much.  I think we need to overthink the whole
secure boot signing process in Fedora.  For starters we need to use
different signing keys for UKI and non-UKI kernels, so it is possible
(for the user if he chooses so) to disable booting non-UKI kernels to
really plug the unsigned initrd hole.

shim project planning to move from compiled-in certificates to signed
certificates in separate files should make it easier to manage this.

But all that is probably worth a separate change proposal and a separate
discussion.

>  I would like the Change document to be updated
>  to include feedback about Secure Boot in there to further justify how
>  restricted the scope will remain for Phase 1.
>  
>  Additionally, "discoverable partitions" fixes nothing for Fedora right
>  now. There are two problems here:
>  * We don't have a discoverable subvolume specification for Btrfs
>  * Discoverable partition specification falls over dead with snapshots.
>  
>  Don't plan on using systemd-boot. I've said why before in other
>  threads, so I'm not repeating it again.
>  
>  Drop locking out modified kernel command lines. That's pretty much a
>  non-starter.
>  
>  If you want to pursue a Fedora Cloud image with UKIs, please bring it
>  up with the Cloud SIG, keeping in mind the feedback I listed. 
Noted.  I'll rework the Change proposal early next year, I'm almost off
into my xmas holidays.

take care,
  Gerd

[Neal Gompa]

On Thu, Dec 22, 2022 at 10:35 AM Gerd Hoffmann <kraxel(a)redhat.com&gt; wrote:
>    Hi,
>
>  > Hmm, the updated Change is mostly okay. I disagree that you have any
>  > real security benefits since all the Secure Boot stuff in Linux is
>  > still in a bad place.
>
>  It's a step into the right direction, but I agree, it alone doesn't
>  improve the situation much.  I think we need to overthink the whole
>  secure boot signing process in Fedora.  For starters we need to use
>  different signing keys for UKI and non-UKI kernels, so it is possible
>  (for the user if he chooses so) to disable booting non-UKI kernels to
>  really plug the unsigned initrd hole.
>
>  shim project planning to move from compiled-in certificates to signed
>  certificates in separate files should make it easier to manage this.
>  
As part of this, we *really* need to make the process for managing
supported/enabled certificates reasonable and independent of the
firmware restrictions.

>  But all that is probably worth a separate change proposal and a
> separate
>  discussion.
>
>  > I would like the Change document to be updated
>  > to include feedback about Secure Boot in there to further justify how
>  > restricted the scope will remain for Phase 1.
>  >
>  > Additionally, "discoverable partitions" fixes nothing for Fedora right
>  > now. There are two problems here:
>  > * We don't have a discoverable subvolume specification for Btrfs
>  > * Discoverable partition specification falls over dead with snapshots.
>  >
>  > Don't plan on using systemd-boot. I've said why before in other
>  > threads, so I'm not repeating it again.
>  >
>  > Drop locking out modified kernel command lines. That's pretty much a
>  > non-starter.
>  >
>  > If you want to pursue a Fedora Cloud image with UKIs, please bring it
>  > up with the Cloud SIG, keeping in mind the feedback I listed.
>
>  Noted.  I'll rework the Change proposal early next year, I'm almost off
>  into my xmas holidays.
>  
Makes sense. Have a merry Christmas and a happy new year! :)



-- 
真実はいつも一つ！/ Always, there's only one truth!

[Lennart Poettering]

On Do, 22.12.22 06:38, Neal Gompa (ngompa13(a)gmail.com) wrote:

>  I have to think about what happens when the cat is out of the bag.
>  What I want is not necessarily a solution to this now, but a
>  commitment that someone will actively work on fixing these problems
>  *before* proposing the next phase and have it ready *before* making
>  any future proposals on UKIs. If you can't do that, I can't in good
>  faith consider incrementally supporting UKIs, because there's
>  effectively no plan to make them work as a future default way of
>  shipping the Linux kernel. 
BTW, you keep talking of "these" problems, and are extremely vague
about those. I think I understood that you hit the NVRAM size limits
before, by enrolling private certs?

What are those issues precisely? Did you file bugs? How many certs did
you try to enroll? Or did you try to enroll hashes? Were was this
dicussed?

Without any of that the vagueness just constitutes FUD... Hence,
please be more specific!

Lennart

--
Lennart Poettering, Berlin

[Neal Gompa]

On Thu, Dec 22, 2022 at 10:46 AM Lennart Poettering
<mzerqung(a)0pointer.de&gt; wrote:
>  On Do, 22.12.22 06:38, Neal Gompa (ngompa13(a)gmail.com) wrote:
>
>  > I have to think about what happens when the cat is out of the bag.
>  > What I want is not necessarily a solution to this now, but a
>  > commitment that someone will actively work on fixing these problems
>  > *before* proposing the next phase and have it ready *before* making
>  > any future proposals on UKIs. If you can't do that, I can't in good
>  > faith consider incrementally supporting UKIs, because there's
>  > effectively no plan to make them work as a future default way of
>  > shipping the Linux kernel.
>
>  BTW, you keep talking of "these" problems, and are extremely vague
>  about those. I think I understood that you hit the NVRAM size limits
>  before, by enrolling private certs?
>  
Yes. Specifically for dealing with the NVIDIA driver, backports of
Intel network card drivers, OpenZFS, ELRepo modules, etc.

>  What are those issues precisely? Did you file bugs? How many certs
> did
>  you try to enroll? Or did you try to enroll hashes? Were was this
>  dicussed?
>  
Anywhere between 1 to 3 UEFI certs, representing different vendors
(machine-local, ELRepo, and OpenZFS).

>  Without any of that the vagueness just constitutes FUD... Hence,
>  please be more specific!
>  
You know better than that. Stop it.


-- 
真実はいつも一つ！/ Always, there's only one truth!

[Lennart Poettering]

On Do, 22.12.22 10:52, Neal Gompa (ngompa13(a)gmail.com) wrote:
65;6800;1c
>  > BTW, you keep talking of "these" problems, and are
> extremely vague
>  > about those. I think I understood that you hit the NVRAM size limits
>  > before, by enrolling private certs?
>
>  Yes. Specifically for dealing with the NVIDIA driver, backports of
>  Intel network card drivers, OpenZFS, ELRepo modules, etc.
>
>  > What are those issues precisely? Did you file bugs? How many certs did
>  > you try to enroll? Or did you try to enroll hashes? Were was this
>  > dicussed?
>
>  Anywhere between 1 to 3 UEFI certs, representing different vendors
>  (machine-local, ELRepo, and OpenZFS). 
OK, so what's left is exactly one issue you had: you tried to enroll 3
UEFI certs via mokutil and what exactly happened then? machine
bricked how precisely?

link for a bug report?

Please be specific, otherwise noone will be able to help you!

Lennart

--
Lennart Poettering, Berlin

[Neal Gompa]

On Thu, Dec 22, 2022 at 10:56 AM Lennart Poettering
<mzerqung(a)0pointer.de&gt; wrote:
>  On Do, 22.12.22 10:52, Neal Gompa (ngompa13(a)gmail.com) wrote:
>  65;6800;1c
>  > > BTW, you keep talking of "these" problems, and are extremely vague
>  > > about those. I think I understood that you hit the NVRAM size limits
>  > > before, by enrolling private certs?
>  >
>  > Yes. Specifically for dealing with the NVIDIA driver, backports of
>  > Intel network card drivers, OpenZFS, ELRepo modules, etc.
>  >
>  > > What are those issues precisely? Did you file bugs? How many certs did
>  > > you try to enroll? Or did you try to enroll hashes? Were was this
>  > > dicussed?
>  >
>  > Anywhere between 1 to 3 UEFI certs, representing different vendors
>  > (machine-local, ELRepo, and OpenZFS).
>
>  OK, so what's left is exactly one issue you had: you tried to enroll 3
>  UEFI certs via mokutil and what exactly happened then? machine
>  bricked how precisely?
>  
The UEFI environment failed to import without reporting failure and
the system failed to boot, showing garbage instead.

>  link for a bug report?
>  
Sorry, I can't.

>  Please be specific, otherwise noone will be able to help you!
>  
No one helped anyway, even back when I could provide more information.
They're even less likely now that I can't provide that information.


-- 
真実はいつも一つ！/ Always, there's only one truth!

[Lennart Poettering]

On Do, 22.12.22 11:00, Neal Gompa (ngompa13(a)gmail.com) wrote:

>  > OK, so what's left is exactly one issue you had: you tried
> to enroll 3
>  > UEFI certs via mokutil and what exactly happened then? machine
>  > bricked how precisely?
>
>  The UEFI environment failed to import without reporting failure and
>  the system failed to boot, showing garbage instead. 
By "uefi environment" you mean shim? Sounds like something to report
to the shim project then.

>  > link for a bug report?
>
>  Sorry, I can't. 
That makes it very hard to do anything about this. To fix a situation
one needs actionable reports.

>  No one helped anyway, even back when I could provide more
> information.
>  They're even less likely now that I can't provide that information. 
You do realise that you are are complaining but not giving anyone the
chance to do anything about your complaints.

Lennart

--
Lennart Poettering, Berlin

[Gerd Hoffmann]

On Thu, Dec 22, 2022 at 10:52:05AM -0500, Neal Gompa wrote:
>  On Thu, Dec 22, 2022 at 10:46 AM Lennart Poettering
>  <mzerqung(a)0pointer.de&gt; wrote:
>  >
>  > BTW, you keep talking of "these" problems, and are extremely vague
>  > about those. I think I understood that you hit the NVRAM size limits
>  > before, by enrolling private certs?
>  
>  Yes. 
Physical hardware or virtual machines?

For virtual machines:  Fedora uses for historical reasons 2M builds of
OVMF, which have not much NVRAM space.  Alternative 4M builds which have
much more space are available in /usr/share/edk2/ovmf-4m.  Using these
needs manual work right now, hashing out a plan to use these by default
for new VMs without changing/breaking existing VMs is not (yet) done.

take care,
  Gerd

[Gerd Hoffmann]

  Hi,

>  > I mean, UEFI has its warts, but I don't see that it's
> UEFI's fault if
>  > the way Linux/shim/mokutil implement the cert db is done the way it is
>  > currently done. 
Well, UEFI *not* defining some standard way to enroll user certificates
actually is part of the problem.  It is one of reasons why shim+mokutil
exist in the first place.

>  Frankly, I'm aggravated by the increased reliance on UEFI
> features
>  without fixing Linux's problems with UEFI features. If we stopped
>  relying on UEFI for the cert store, a lot of my issues would go away,
>  because then we can design better workflows for managing certificates. 
Ignoring the UEFI cert store is just not possible, that is the only one
available and used initially.  For booting the bootloader to have to
work with the capabilities provided by the firmware.  Doing something
simliar for ppc would likewise depend on ppc-specific firmware features.

Once shim is loaded both uefi and shim cert stores are available.
Once the kernel is loaded that expands to uefi + shim + kernel.

>  It makes automation possible, it makes management possible, and it
>  opens the doors to experiment with how to do layered images for boot
>  (e.g. UKIs + system extension images) without bricking people's
>  computers. 
system extensions are verified after the kernel is loaded, so you are
not limited to the uefi/shim cert stores for them.

>  That also has the side effect of us having half a chance of being
> able
>  to port this security capability to non-UEFI platforms where we use
>  fake UEFI implementations to bootstrap our boot process, because the
>  amount of coverage required for fake UEFI implementations is
>  considerably lower now. 
Yep.  Because it's a standard.  Having u-boot (assuming this is what you
are referring to) provide standard UEFI protocols, then use standard efi
boot process is much less work because there is only one piece in the
boot process which needs to know the hardware specifics.  Which is
u-boot (playing the firmware role on many SBCs).

>  More generally, relying on UEFI-specific security features when most
>  platforms are not being brought up with UEFI is foolish. ARM is almost
>  entirely non-UEFI (except the tiny proportion of servers that almost
>  no one has) 
Any aarch64 server I boot in some cloud is UEFI.
aarch64 virtual machines use UEFI.
UEFI implementations exist for RPI 3+4.

>  and RISC-V is not a UEFI platform either. 
https://github.com/tianocore/edk2-platforms/tree/master/Platform/RISC-V/P...

>  You *need* to think about these problems when designing this stuff.
>  You can't take a myopic x86-only view to this. I have not heard of
>  anything about UKI security for non-x86 because it seems to all be
>  tied up in UEFI stuff. 
Booting UKIs in BIOS mode works with a patched grub (needed for hybrid
bios/uefi cloud images).  Of course that doesn't improve security
because the bios lacks the features needed for that.  You can still get
the other advantages UKIs have like a more robust kernel update process.

ppc uses grub too, so being able to boot UKIs on ppc too should be
doable without too much effort when the platform maintainers think
it is useful.

>  > Nah. UKIs + sysext are ultimately something that is simpler and
> much
>  > better to test than the current mess. Yeah, for a while we'll have to
>  > add complexity because to mechanisms will have to be supported, but
>  > eventually things are going to be simple and easier to test.
>  
>  Maybe. It's not super intuitive from where I'm standing, and I know
>  how all this stuff works. 
Because today fedora doesn't use sysexts and doesn't provide much
tooling.

Ideally the packages which drop some dracut module to
/usr/lib/dracut/modules.d/ today will also drop a sysext to the correct
place in the future.  No manual steps needed.  Users don't even need to
know how this works under the hood.

take care,
  Gerd

[Lennart Poettering]

On Do, 22.12.22 05:38, Neal Gompa (ngompa13(a)gmail.com) wrote:

>  > I understand the big issue you have is that the certificate
> store for
>  > Linux (i.e. the mokutil db) is limited in size because EFI variable
>  > NVRAM is limited in size? If that's the big issue you are having then
>  > that's absolutely something the Linux community can fix, and can
>  > address. Specifically, shim could allow storing the cert store on
>  > disk, and authenticate it at boot via the TPM. Not trivial, but
>  > doable. And certainly not the firmware's fault...
>  >
>  > I mean, UEFI has its warts, but I don't see that it's UEFI's fault if
>  > the way Linux/shim/mokutil implement the cert db is done the way it is
>  > currently done.
>
>  Frankly, I'm aggravated by the increased reliance on UEFI features
>  without fixing Linux's problems with UEFI features. If we stopped
>  relying on UEFI for the cert store, a lot of my issues would go away,
>  because then we can design better workflows for managing
>  certificates. 
Well, the thing is: a chain of trust is a *chain*, hence you must
ultimately hook validation to what the firmware provides you with as
root. And that ultimately is the SecureBoot db on commodity hardware.

If I buy a boring laptop at a store it will come with a UEFI cert
store, and nothing else. Linux cannot really ignore that. If it did,
then you'd not have a trusted boot chain, hence the whole excercsie
would be pointless.

>  It makes automation possible, it makes management possible, and it
>  opens the doors to experiment with how to do layered images for boot
>  (e.g. UKIs + system extension images) without bricking people's
>  computers. 
As mentioned before: note that the Fedora signing keys are only built
into shim and the kernel, not stored in NVRAM.

But anyway, I think it would be great if shim could manage a cert
store on disk, I already said that. But that's truly orthogonal to
UKIs and sysext really. You keep trying to change topics away from
UKI/sysext towards your general discontent with UEFI.

>  That also has the side effect of us having half a chance of being
> able
>  to port this security capability to non-UEFI platforms where we use
>  fake UEFI implementations to bootstrap our boot process, because the
>  amount of coverage required for fake UEFI implementations is
>  considerably lower now.
>
>  More generally, relying on UEFI-specific security features when most
>  platforms are not being brought up with UEFI is foolish. ARM is almost
>  entirely non-UEFI (except the tiny proportion of servers that almost
>  no one has) and RISC-V is not a UEFI platform either. POWER uses
>  OpenFirmware instead of UEFI, and IBM Z does something else entirely
>  different. 
Well, "foolish", eyh?

The thing is that chain of trust works quite differently on each such
system (if it exists at all). UEFI is a standardizing and unifying
force that simplifies things greatly, since while not universal it's
still the most widely adopted mechanism (and one of the more
advanced).

Generally, I am very sure we should focus on making the more limited
stuff work like the modern stuff, and not the other way round. For
example, there has already been work on making UKIs boot on grub/MBR,
as mentioned, which is exactly how I think it should be done: move the
old/legacy/simple stuff work more like the
new/modern/featureful/standardized stuff, rather than the other way
round.

>  You *need* to think about these problems when designing this stuff.
>  You can't take a myopic x86-only view to this. I have not heard of
>  anything about UKI security for non-x86 because it seems to all be
>  tied up in UEFI stuff. 
I work for a company that is working on ARM UEFI systems booting UKIs
in massive scale.

Lennart

--
Lennart Poettering, Berlin

[Neal Gompa]

On Thu, Dec 22, 2022 at 10:39 AM Lennart Poettering
<mzerqung(a)0pointer.de&gt; wrote:
>  On Do, 22.12.22 05:38, Neal Gompa (ngompa13(a)gmail.com) wrote:
>
>  > > I understand the big issue you have is that the certificate store for
>  > > Linux (i.e. the mokutil db) is limited in size because EFI variable
>  > > NVRAM is limited in size? If that's the big issue you are having then
>  > > that's absolutely something the Linux community can fix, and can
>  > > address. Specifically, shim could allow storing the cert store on
>  > > disk, and authenticate it at boot via the TPM. Not trivial, but
>  > > doable. And certainly not the firmware's fault...
>  > >
>  > > I mean, UEFI has its warts, but I don't see that it's UEFI's fault
> if
>  > > the way Linux/shim/mokutil implement the cert db is done the way it is
>  > > currently done.
>  >
>  > Frankly, I'm aggravated by the increased reliance on UEFI features
>  > without fixing Linux's problems with UEFI features. If we stopped
>  > relying on UEFI for the cert store, a lot of my issues would go away,
>  > because then we can design better workflows for managing
>  > certificates.
>
>  Well, the thing is: a chain of trust is a *chain*, hence you must
>  ultimately hook validation to what the firmware provides you with as
>  root. And that ultimately is the SecureBoot db on commodity hardware.
>
>  If I buy a boring laptop at a store it will come with a UEFI cert
>  store, and nothing else. Linux cannot really ignore that. If it did,
>  then you'd not have a trusted boot chain, hence the whole excercsie
>  would be pointless.
>  
Shim trusts MS and the main distro cert, grub is trusted from there,
then the OS trusts those and anything else the admin adds. That's how
It should work. Trust chains are sensible as long as they're extensible.

In a perfect world, they would work properly at the lowest levels, but
between commercial and community malpractice, they don't.

>  > It makes automation possible, it makes management possible, and
> it
>  > opens the doors to experiment with how to do layered images for boot
>  > (e.g. UKIs + system extension images) without bricking people's
>  > computers.
>
>  As mentioned before: note that the Fedora signing keys are only built
>  into shim and the kernel, not stored in NVRAM.
>
>  But anyway, I think it would be great if shim could manage a cert
>  store on disk, I already said that. But that's truly orthogonal to
>  UKIs and sysext really. You keep trying to change topics away from
>  UKI/sysext towards your general discontent with UEFI.
>
>  > That also has the side effect of us having half a chance of being able
>  > to port this security capability to non-UEFI platforms where we use
>  > fake UEFI implementations to bootstrap our boot process, because the
>  > amount of coverage required for fake UEFI implementations is
>  > considerably lower now.
>  >
>  > More generally, relying on UEFI-specific security features when most
>  > platforms are not being brought up with UEFI is foolish. ARM is almost
>  > entirely non-UEFI (except the tiny proportion of servers that almost
>  > no one has) and RISC-V is not a UEFI platform either. POWER uses
>  > OpenFirmware instead of UEFI, and IBM Z does something else entirely
>  > different.
>
>  Well, "foolish", eyh?
>
>  The thing is that chain of trust works quite differently on each such
>  system (if it exists at all). UEFI is a standardizing and unifying
>  force that simplifies things greatly, since while not universal it's
>  still the most widely adopted mechanism (and one of the more
>  advanced).
>
>  Generally, I am very sure we should focus on making the more limited
>  stuff work like the modern stuff, and not the other way round. For
>  example, there has already been work on making UKIs boot on grub/MBR,
>  as mentioned, which is exactly how I think it should be done: move the
>  old/legacy/simple stuff work more like the
>  new/modern/featureful/standardized stuff, rather than the other way
>  round.
>
>  > You *need* to think about these problems when designing this stuff.
>  > You can't take a myopic x86-only view to this. I have not heard of
>  > anything about UKI security for non-x86 because it seems to all be
>  > tied up in UEFI stuff.
>
>  I work for a company that is working on ARM UEFI systems booting UKIs
>  in massive scale.
>  
One company is one company. It's not a variety of people and users.
Scale means nothing if it's not distributed.




--
真実はいつも一つ！/ Always, there's only one truth!

[Lennart Poettering]

On Do, 22.12.22 10:43, Neal Gompa (ngompa13(a)gmail.com) wrote:

>  On Thu, Dec 22, 2022 at 10:39 AM Lennart Poettering
>  <mzerqung(a)0pointer.de&gt; wrote:
>  >
>  > On Do, 22.12.22 05:38, Neal Gompa (ngompa13(a)gmail.com) wrote:
>  >
>  > > > I understand the big issue you have is that the certificate store for
>  > > > Linux (i.e. the mokutil db) is limited in size because EFI variable
>  > > > NVRAM is limited in size? If that's the big issue you are having then
>  > > > that's absolutely something the Linux community can fix, and can
>  > > > address. Specifically, shim could allow storing the cert store on
>  > > > disk, and authenticate it at boot via the TPM. Not trivial, but
>  > > > doable. And certainly not the firmware's fault...
>  > > >
>  > > > I mean, UEFI has its warts, but I don't see that it's UEFI's
> fault if
>  > > > the way Linux/shim/mokutil implement the cert db is done the way it is
>  > > > currently done.
>  > >
>  > > Frankly, I'm aggravated by the increased reliance on UEFI features
>  > > without fixing Linux's problems with UEFI features. If we stopped
>  > > relying on UEFI for the cert store, a lot of my issues would go away,
>  > > because then we can design better workflows for managing
>  > > certificates.
>  >
>  > Well, the thing is: a chain of trust is a *chain*, hence you must
>  > ultimately hook validation to what the firmware provides you with as
>  > root. And that ultimately is the SecureBoot db on commodity hardware.
>  >
>  > If I buy a boring laptop at a store it will come with a UEFI cert
>  > store, and nothing else. Linux cannot really ignore that. If it did,
>  > then you'd not have a trusted boot chain, hence the whole excercsie
>  > would be pointless.
>  >
>
>  Shim trusts MS and the main distro cert, grub is trusted from there,
>  then the OS trusts those and anything else the admin adds. That's how
>  It should work. Trust chains are sensible as long as they're
>  extensible. 
Hmm, that chain is partly backwards? it's the firmware that has to
trust the msft cert which trusts shim, which trusts the distro cert,
which trusts grub and the kernel. The thing that comes first
at boot must trust the next, otherwise we'd have to boot into
untrusted stuff first, which really misses the point? not grokking
what you are trying tosay?

>  > > You *need* to think about these problems when designing
> this stuff.
>  > > You can't take a myopic x86-only view to this. I have not heard of
>  > > anything about UKI security for non-x86 because it seems to all be
>  > > tied up in UEFI stuff.
>  >
>  > I work for a company that is working on ARM UEFI systems booting UKIs
>  > in massive scale.
>
>  One company is one company. It's not a variety of people and users.
>  Scale means nothing if it's not distributed. 
Well, you said you have not "heard" of anyone doing UKI security on
non-x86. Now you have.

Lennart

--
Lennart Poettering, Berlin

[Neal Gompa]

On Thu, Dec 22, 2022 at 10:51 AM Lennart Poettering
<mzerqung(a)0pointer.de&gt; wrote:
>  On Do, 22.12.22 10:43, Neal Gompa (ngompa13(a)gmail.com) wrote:
>
>  > On Thu, Dec 22, 2022 at 10:39 AM Lennart Poettering
>  > <mzerqung(a)0pointer.de&gt; wrote:
>  > >
>  > > On Do, 22.12.22 05:38, Neal Gompa (ngompa13(a)gmail.com) wrote:
>  > >
>  > > > > I understand the big issue you have is that the certificate store
> for
>  > > > > Linux (i.e. the mokutil db) is limited in size because EFI variable
>  > > > > NVRAM is limited in size? If that's the big issue you are having
> then
>  > > > > that's absolutely something the Linux community can fix, and can
>  > > > > address. Specifically, shim could allow storing the cert store on
>  > > > > disk, and authenticate it at boot via the TPM. Not trivial, but
>  > > > > doable. And certainly not the firmware's fault...
>  > > > >
>  > > > > I mean, UEFI has its warts, but I don't see that it's
> UEFI's fault if
>  > > > > the way Linux/shim/mokutil implement the cert db is done the way it
> is
>  > > > > currently done.
>  > > >
>  > > > Frankly, I'm aggravated by the increased reliance on UEFI features
>  > > > without fixing Linux's problems with UEFI features. If we stopped
>  > > > relying on UEFI for the cert store, a lot of my issues would go away,
>  > > > because then we can design better workflows for managing
>  > > > certificates.
>  > >
>  > > Well, the thing is: a chain of trust is a *chain*, hence you must
>  > > ultimately hook validation to what the firmware provides you with as
>  > > root. And that ultimately is the SecureBoot db on commodity hardware.
>  > >
>  > > If I buy a boring laptop at a store it will come with a UEFI cert
>  > > store, and nothing else. Linux cannot really ignore that. If it did,
>  > > then you'd not have a trusted boot chain, hence the whole excercsie
>  > > would be pointless.
>  > >
>  >
>  > Shim trusts MS and the main distro cert, grub is trusted from there,
>  > then the OS trusts those and anything else the admin adds. That's how
>  > It should work. Trust chains are sensible as long as they're
>  > extensible.
>
>  Hmm, that chain is partly backwards? it's the firmware that has to
>  trust the msft cert which trusts shim, which trusts the distro cert,
>  which trusts grub and the kernel. The thing that comes first
>  at boot must trust the next, otherwise we'd have to boot into
>  untrusted stuff first, which really misses the point? not grokking
>  what you are trying tosay?
>  
Basically, I'm saying that the model of trust is flawed because users
are unable to work with it.

And besides, each level up is a smaller scope from the previous. A
cert trusted by shim can execute anything the firmware trusts, a cert
trusted by grub can only execute things it trusts, and finally a cert
trusted by the OS can only execute things in its context. Once we
reach the OS-level, we don't need pre-boot trust anymore. So enrolling
certificates to trust kernel modules/sysexts/etc. should not require
going down the trust levels. The OS should be able to establish its
own trust to those pieces or reject them independently. It should
certainly trust everything the lower levels trust, but there's no
reason to not allow the higher levels to establish their own scoped
trust.

This is the flaw we have right now: we can't do that.



-- 
真実はいつも一つ！/ Always, there's only one truth!

[Luca Boccassi]

>  On Thu, Dec 22, 2022 at 10:51 AM Lennart Poettering
>  <mzerqung(a)0pointer.de&gt; wrote:
>  
>  Basically, I'm saying that the model of trust is flawed because users
>  are unable to work with it.
>  
>  And besides, each level up is a smaller scope from the previous. A
>  cert trusted by shim can execute anything the firmware trusts, a cert
>  trusted by grub can only execute things it trusts, and finally a cert
>  trusted by the OS can only execute things in its context. Once we
>  reach the OS-level, we don't need pre-boot trust anymore. So enrolling
>  certificates to trust kernel modules/sysexts/etc. should not require
>  going down the trust levels. The OS should be able to establish its
>  own trust to those pieces or reject them independently. It should
>  certainly trust everything the lower levels trust, but there's no
>  reason to not allow the higher levels to establish their own scoped
>  trust.
>  
>  This is the flaw we have right now: we can't do that. 
Of course there's a reason to only allow a fully validated trust chain - not only
that, but it's the very basic of the entire model, and without it the entire premise
falls flat on its face.
The way to enroll your own certs is via firmware-mediated mechanisms such as shim+mok, and
via built-in trusted keyring. Adding arbitrary trust anchors at the OS level completely
ignores the foundational principle of the whole thing.

[Neal Gompa]

On Thu, Dec 22, 2022 at 12:00 PM Luca Boccassi <bluca(a)debian.org&gt; wrote:
>  > On Thu, Dec 22, 2022 at 10:51 AM Lennart Poettering
>  > <mzerqung(a)0pointer.de&gt; wrote:
>  >
>  > Basically, I'm saying that the model of trust is flawed because users
>  > are unable to work with it.
>  >
>  > And besides, each level up is a smaller scope from the previous. A
>  > cert trusted by shim can execute anything the firmware trusts, a cert
>  > trusted by grub can only execute things it trusts, and finally a cert
>  > trusted by the OS can only execute things in its context. Once we
>  > reach the OS-level, we don't need pre-boot trust anymore. So enrolling
>  > certificates to trust kernel modules/sysexts/etc. should not require
>  > going down the trust levels. The OS should be able to establish its
>  > own trust to those pieces or reject them independently. It should
>  > certainly trust everything the lower levels trust, but there's no
>  > reason to not allow the higher levels to establish their own scoped
>  > trust.
>  >
>  > This is the flaw we have right now: we can't do that.
>
>  Of course there's a reason to only allow a fully validated trust chain - not only
> that, but it's the very basic of the entire model, and without it the entire premise
> falls flat on its face.
>  The way to enroll your own certs is via firmware-mediated mechanisms such as shim+mok,
> and via built-in trusted keyring. Adding arbitrary trust anchors at the OS level
> completely ignores the foundational principle of the whole thing. 
Your concept only works in *some* enterprise hardware where it's even
possible to have full control without breaking something. Even in my
server gear, I can't do that without breaking the network firmware. If
I can't safely distrust Microsoft reliably, then it's already broken.

Firmware trust has been broken since the very beginning, and it was
broken by design in the PC world.

Consumer PC equipment is even worse off because of how Microsoft's
Windows requirements influence how UEFI implementations work.



--
真実はいつも一つ！/ Always, there's only one truth!

[Luca Boccassi]

>  On Thu, Dec 22, 2022 at 12:00 PM Luca Boccassi
> <bluca(a)debian.org&gt; wrote:
>  
>  Your concept only works in *some* enterprise hardware where it's even
>  possible to have full control without breaking something. Even in my
>  server gear, I can't do that without breaking the network firmware. If
>  I can't safely distrust Microsoft reliably, then it's already broken.
>  
>  Firmware trust has been broken since the very beginning, and it was
>  broken by design in the PC world. 
"My" (not really mine, but whatever) concept works on pretty much every single
x86 consumer device sold in the past ~12 years, in every public cloud provider, a
gazillion arm64 use cases, and more. It's only "broken" if by that you mean
that an entire class of very real and very much alive in the wild malware infections are
no longer possible.
Of course if you do things like deleting the 3rd party UEFI CA on machines that require
option roms to work without allow-listing or re-signing, then things break, but that has
absolutely nothing to do with the "concept" - if you delete a trust anchor,
objects trusted by it are no longer trusted in the absence of an alternative chain,
that's pretty much expected behaviour.
Obviously things can always be improved, and we are working on that, SBAT is the first
step in that direction. But saying that the trust chain model is broken is simply untrue.

>  Consumer PC equipment is even worse off because of how
> Microsoft's
>  Windows requirements influence how UEFI implementations work. 
You meant to say "light years ahead" here - it is not even funny anymore how far
behind Linux is to Windows w.r.t. security, especially in the boot process. We have been
playing catch-up for 10 years and are nowhere near done. It is very fortunate for the
ecosystem at large that there are people who actually understand the threat models pushing
the industry forward, because if it was up to the hardware vendors computer security would
still be where it was in the mid 90s.

But we are working on it, and making progress - in some technical concepts we are even
ahead of them right now (eg: signed TPM policies, FIDO2 for luks, Windows doesn't have
anything like that), but only in PoCs. Now we need the wide adoption, and this proposal is
one timid step forward in that direction. The fact that in 2022 there is still no
mainstream distro that has closed the glaring security gaping hole of writable, untrusted
initrd (yes some distros have non-default specialized flavours for this, but it's
niche) should be a source of embarrassment for anybody who works on OS development. It is
a difficult problem, but by no means impossible, and it really ought to have been fixed at
least for the generic use case by now. We need to get this sorted at long last.

[Demi Marie Obenour]

On 12/22/22 12:33, Neal Gompa wrote:
>  On Thu, Dec 22, 2022 at 12:00 PM Luca Boccassi
> <bluca(a)debian.org&gt; wrote:
> >
> >> On Thu, Dec 22, 2022 at 10:51 AM Lennart Poettering
> >> <mzerqung(a)0pointer.de&gt; wrote:
> >>
> >> Basically, I'm saying that the model of trust is flawed because users
> >> are unable to work with it.
> >>
> >> And besides, each level up is a smaller scope from the previous. A
> >> cert trusted by shim can execute anything the firmware trusts, a cert
> >> trusted by grub can only execute things it trusts, and finally a cert
> >> trusted by the OS can only execute things in its context. Once we
> >> reach the OS-level, we don't need pre-boot trust anymore. So enrolling
> >> certificates to trust kernel modules/sysexts/etc. should not require
> >> going down the trust levels. The OS should be able to establish its
> >> own trust to those pieces or reject them independently. It should
> >> certainly trust everything the lower levels trust, but there's no
> >> reason to not allow the higher levels to establish their own scoped
> >> trust.
> >>
> >> This is the flaw we have right now: we can't do that.
> >
> > Of course there's a reason to only allow a fully validated trust chain -
> > not only that, but it's the very basic of the entire model, and without
> > it the entire premise falls flat on its face.  The way to enroll your own
> > certs is via firmware-mediated mechanisms such as shim+mok, and via
> > built-in trusted keyring. Adding arbitrary trust anchors at the OS level
> > completely ignores the foundational principle of the whole thing.
>  
>  Your concept only works in *some* enterprise hardware where it's even
>  possible to have full control without breaking something. Even in my
>  server gear, I can't do that without breaking the network firmware. 
What???  That is strange.

>  If I can't safely distrust Microsoft reliably, then it's
> already broken. 
Absolutely.  Otherwise, an attacker can just boot into Windows and
use any of the numerous bring-your-own-vulnerable-driver exploits to
get code execution in the kernel.  To stop this, one needs to remove
the Microsoft signing key from the root store.

>  Firmware trust has been broken since the very beginning, and it was
>  broken by design in the PC world.
>  
>  Consumer PC equipment is even worse off because of how Microsoft's
>  Windows requirements influence how UEFI implementations work. 
IMO a much more realistic approach for bare hardware is measured boot,
using Dynamic Root of Trust for Measurement (DRTM) where available.
The latter is the basis of Qubes OS’s Anti Evil Maid.
-- 
Sincerely,
Demi Marie Obenour (she/her/hers)

[Demi Marie Obenour]

On 12/22/22 12:00, Luca Boccassi wrote:
> > On Thu, Dec 22, 2022 at 10:51 AM Lennart Poettering
> > <mzerqung(a)0pointer.de&gt; wrote:
> >
> > Basically, I'm saying that the model of trust is flawed because users
> > are unable to work with it.
> >
> > And besides, each level up is a smaller scope from the previous. A
> > cert trusted by shim can execute anything the firmware trusts, a cert
> > trusted by grub can only execute things it trusts, and finally a cert
> > trusted by the OS can only execute things in its context. Once we
> > reach the OS-level, we don't need pre-boot trust anymore. So enrolling
> > certificates to trust kernel modules/sysexts/etc. should not require
> > going down the trust levels. The OS should be able to establish its
> > own trust to those pieces or reject them independently. It should
> > certainly trust everything the lower levels trust, but there's no
> > reason to not allow the higher levels to establish their own scoped
> > trust.
> >
> > This is the flaw we have right now: we can't do that.
>  
>  Of course there's a reason to only allow a fully validated trust chain - not only
> that, but it's the very basic of the entire model, and without it the entire premise
> falls flat on its face.
>  The way to enroll your own certs is via firmware-mediated mechanisms such as shim+mok,
> and via built-in trusted keyring. Adding arbitrary trust anchors at the OS level
> completely ignores the foundational principle of the whole thing. 
As Neal mentioned earlier, these mechanisms are often broken.  Not only
does some firmware wind up bricking the machine when they are used, they
also may not support removing the key used to sign Windows.  If the
attacker can boot Windows and measured boot is not in use, they have won.

A much better approach is to install a TPM-generated key in the TPM’s
NVRAM, with a policy that only allows the key to be used once a trusted
operating system has booted.  That can be used as a trust anchor even
without support from buggy UEFI firmware.  Furthermore, measured boot
allows tying e.g. LUKS keys to a combination of the actual OS booted and
a passphrase needed to unlock the TPM.  This allows the TPM’s protection
against brute-force attacks to be used.
-- 
Sincerely,
Demi Marie Obenour (she/her/hers)

[Luca Boccassi]

>  On 12/22/22 12:00, Luca Boccassi wrote:
>  
>  As Neal mentioned earlier, these mechanisms are often broken.  Not only
>  does some firmware wind up bricking the machine when they are used, they
>  also may not support removing the key used to sign Windows.  If the
>  attacker can boot Windows and measured boot is not in use, they have won. 
Nah, it's the other way around. Linux is severely behind Windows, and I mean really
severely.

[Gerd Hoffmann]

  Hi,

>  A much better approach is to install a TPM-generated key in the
> TPM’s
>  NVRAM, with a policy that only allows the key to be used once a trusted
>  operating system has booted.  That can be used as a trust anchor even
>  without support from buggy UEFI firmware. 
Side note: measuring kernel + initrd happens using UEFI firmware services.

(once the kernel is up'n'running it will use its own tpm drivers instead
of depending on the firmware services).

>  Furthermore, measured boot allows tying e.g. LUKS keys to a
>  combination of the actual OS booted and a passphrase needed to unlock
>  the TPM.  This allows the TPM’s protection against brute-force attacks
>  to be used. 
You also want protect the initrd against modifications to make sure an
attacker can't sniff your passphrase.  Unified kernels help here too
because the initrd for a given kernel has a fixed and known hash.

take care,
  Gerd

[Jóhann B. Guðmundsson]

On 12/22/22 15:39, Lennart Poettering wrote:

>  Well, the thing is: a chain of trust is a*chain*, hence you must
>  ultimately hook validation to what the firmware provides you with as
>  root. And that ultimately is the SecureBoot db on commodity hardware. 

Well, the thing with a chain of trust is the fact that the only chain 
the user can trust is the one that he himself or the host device he owns 
and operates generated that trust of chain, from link 0 in that chain. ( 
And we all know how browsers handle self signed certificates who are no 
less secure than those issued )

If the user does not generate or otherwise have control over *all* the 
links in the trust chain, that chain cant be considered trusted now can 
it, which in turn begs the question why partake in this industry 
security theater which may brick or otherwise make the end users life 
more miserable or even exclude certain types of devices, if in the end 
of the day, the host or the end user is not  "secure" for it.

Are those efforts truly for the end user or just to meet some 
industry/government requirements ( some governments require backdoor 
entrance(s) from vendors for "lawful inspection", backdoor(s) that might 
be implement or otherwise supported in the trust chain itself if the 
host or user has not full control over that chain ).


JBG

[Luca Boccassi]

>  On 12/22/22 15:39, Lennart Poettering wrote:
>  
>  
>  
>  Well, the thing with a chain of trust is the fact that the only chain 
>  the user can trust is the one that he himself or the host device he owns 
>  and operates generated that trust of chain, from link 0 in that chain. ( 
>  And we all know how browsers handle self signed certificates who are no 
>  less secure than those issued )
>  
>  If the user does not generate or otherwise have control over *all* the 
>  links in the trust chain, that chain cant be considered trusted now can 
>  it, which in turn begs the question why partake in this industry 
>  security theater which may brick or otherwise make the end users life 
>  more miserable or even exclude certain types of devices, if in the end 
>  of the day, the host or the end user is not  "secure" for it.
>  
>  Are those efforts truly for the end user or just to meet some 
>  industry/government requirements ( some governments require backdoor 
>  entrance(s) from vendors for "lawful inspection", backdoor(s) that might 
>  be implement or otherwise supported in the trust chain itself if the 
>  host or user has not full control over that chain ). 
For the vast, vast, vast majority of users a chain of trust that anchors to the device
manufacturer is more than enough, and solves real-world problems with rootkits and such
things. End users have to trust the hardware vendors anyway.

For the tiny minority of use cases where that is not enough (and I'm pretty sure
nobody here is actually interesting enough to fall into that group) rolling your own PK is
always possible.

Seriously, this "debate" is no longer a debate, it's been done and dusted in
the 15 years or so secure boot has been a thing, it's been over for a long while and
it's really time to stop rehashing tired and debunked nonsense. We have more important
things to do.

[Vitaly Zaitsev]

On 20/12/2022 22:27, Simo Sorce wrote:
>  SecureBoot may not be to your liking but is what is installed on 99%
> of
>  modern hardware sold, so it is a good idea to first show you can
>  support it. Then if you have interested you can propose "something
>  better". 
Secure Boot with shim and the use of a Microsoft key defeats all purpose 
of the __Secure__ Boot.

Also new computers will be able to boot only Microsoft Windows as 
described here[1]:

>  Secured-core PCs require Secure Boot to be enabled and configured to
> distrust the Microsoft 3rd Party UEFI CA signature, by default, to provide customers with
> the most secure configuration of their PCs possible. 
>  To trust and boot operating systems, like Linux, and components
> signed by the UEFI signature, Secured-core PCs can be configured in the BIOS menu to add
> the signature in the UEFI database by following these steps 
[1]: 
https://learn.microsoft.com/en-us/windows/security/information-protection...

-- 
Sincerely,
   Vitaly Zaitsev (vitaly(a)easycoding.org)

[Jóhann B. Guðmundsson]

On 12/20/22 21:27, Simo Sorce wrote:

>  Finally, unless this proposal harms Fedora I do not see why oppose
> it.
>  If, as you fear, it won't work ... then it won't and we'll try
>  something else. 

You do realize that the day that Lenovo started to sell it's hardware 
with Fedora pre-installed ( as it was convinced to do) , the days of 
Fedora doing somekind of experiments on it's users base to see what 
sticks or making decisions like hey people if this does not work let's 
try something else, were over right? ( atleast for the workstation 
working group ).


JBG

[Daniel P. Berrangé]

On Tue, Dec 20, 2022 at 02:29:22PM -0500, Neal Gompa wrote:
>  On Tue, Dec 20, 2022 at 2:02 PM Daniel P. Berrangé
> <berrange(a)redhat.com&gt; wrote:
>  > > In the Fedora case, things are simpler right up until we hit graphics
>  > > drivers. This is also a problem for VMs too, because GPU passthrough
>  > > is a common case for scientific and gaming workloads. As long as the
>  > > NVIDIA driver remains dominant in Linux, UKIs cannot work because by
>  > > design you cannot load anything that isn't part of the kernel image.
>  > > For bare metal, we *need* these drivers in early boot, though. And
>  > > that's another problem: no third-party early boot drivers. Even if you
>  > > solve the signing issue, you need to introduce some kind of two-stage
>  > > OS boot process so we can bring up the bare minimum to load a second
>  > > image containing all the remaining drivers. And at that point, you've
>  > > defeated the purpose of UKIs. I've heard from some people that system
>  > > extensions (sysexts) would be a way to solve this, and maybe it is.
>  >
>  > Yes, system extensions are one mechanism for making the initrd
>  > content more flexible when using UKIs. There would be one base
>  > layer definiing the 90% common case, and a number of add-ons
>  > that can cope with niche use cases. This avoids the core UKI
>  > having to be huge and ship with every possible feature present.
>  >
>  > This is something that might be considered at a later phase,
>  > but is not a priority. What's proposed in phase 1 is sufficient
>  > to cope with the cloud VM common case. While device passthrough
>  > may be common in some industries/domains usage, it is not the
>  > common case for cloud computing in general.
>  >
>  > > But again, we've eliminated the value of UKIs by doing so.
>  >
>  > That is not correct. There are a number of benefits of UKIs.
>  >
>  > The most critical is that the initrd content and cmdline is
>  > covered by the SecureBoot signature. This remains true even
>  > with system extensions, as such extensions would be signed
>  > too. They do not neccessarily need to be signed by the OS
>  > vendor, they could use a 3rd party SecureBoot signing key,
>  > or the users' own key. That is TBD and not something we're
>  > actively considering - its not even mentioned in phase 2/3
>  > ideas in the change proposal.
>  >
>  
>  Secure Boot keys are flawed. There are many documented problems with
>  relying on Secure Boot keys in the real world. Redesign to not use
>  them, please. 
We're aiming to fix the flawed usage of Secure Boot in the
Fedora / RHEL distros, so it becomes useful. Simply ignoring
the problems and carrying on with our current approach is
not sustainable.


>  > TPMs. These are a core technology foundation of the
> confidential
>  > virtual machine stack. On Azure today, if you request an Ubuntu
>  > confidential VM, Azure will pre-encrypt the root filesystem and
>  > seal the LUKS key against predicted TPM PCR values. It guarantees
>  > that the root disk can only be decrypted by the specific VM
>  > instance that is requested, when it is running in SecureBoot
>  > mode with the expected measurments on AMD SEV-SNP confidential
>  > hardware. The Ubuntu image in Azure already uses UKIs, and boots
>  > them directly from shim, with no bootloader involved. The usage
>  > of SecureBoot and TPMs is all transparent to the user, since all
>  > the integration is handled by the OS vendor on their behalf.
>  >
>  
>  Well, how is that relevant to Fedora? We don't have Azure images, and
>  Red Hat still blocks us from meaningfully existing in that ecosystem,
>  which is why we don't have official WSL images or Azure images. 
Users can upload their own images, and 3rd parties have uploaded
public Fedora images too. It is still relevant, even if Fedora
itself doesn't officially publish images.

As mentioned in another part of this thread, we're also expecting
to support the same technology with KVM, so this use of UKIS will
benefit Fedora on a fully open source hypervisor platform.

>  > > With my FESCo hat on, I'm uneasy about this change.
> With my "Fedora
>  > > user and advocate" hat on, I think that the UAPI group has failed to
>  > > provide something useful to the Linux world here and I would be
>  > > extremely apprehensive about Fedora adopting any portion of this
>  > > stuff.
>  >
>  > As mentioned above, UKIs have provided a means to close the
>  > SecureBoot hole with unsigned initrd content that has existed
>  > in more or less every mainstream Linux distro. That is a
>  > clearly useful outcome, regardless of whether Fedora is interested
>  > in any other aspect of what that UAPI groups is proposing.
>  >
>  > In proposing the UKI support for Fedora, we're not coming at this
>  > as representatives of the UAPI group or its vision. We're trying
>  > to solve the problem of having a fully verified secureboot chain
>  > for VMs with no unsigned content, and to be able to use this to
>  > support confidential VMs with disk encryption sealed to TPMs, as
>  > required by Azure today, and likely KVM in future.
>  
>  Yeah, I seriously doubt this. Linux's model for supporting
>  confidential computing is not user-friendly, so I expect low adoption
>  and resistance once the flaws become apparent to would-be users. 
What exists in Linux today is not be user friendly. We don't have
to resign ourselves to stick with the status quo. It is entirely
possible to build a solution that works, without the user needing
to get involved in understanding confidential computing, secure
boot or TPMs. The hypervisor vendor and the guest OS vendor take
responsibility for understanding the hard bits. The user ends up
merely ticking a box to say they want an encrypted confidential
VM. If it ends up being not user-friendly then we've failed in
our goals, and of course it is not our expectation to fail.

With regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

[Kevin Kofler]

Daniel P. Berrangé wrote:
>  That is not correct. There are a number of benefits of UKIs.
>   
>  The most critical is that the initrd content and cmdline is
>  covered by the SecureBoot signature. 
From an end user point of view, having more stuff covered by Restricted Boot 
is not a benefit.

        Kevin Kofler

[Neal Gompa]

On Tue, Dec 20, 2022 at 9:10 PM Kevin Kofler via devel
<devel(a)lists.fedoraproject.org&gt; wrote:
>  Daniel P. Berrangé wrote:
>  > That is not correct. There are a number of benefits of UKIs.
>  >
>  > The most critical is that the initrd content and cmdline is
>  > covered by the SecureBoot signature.
>
>  From an end user point of view, having more stuff covered by Restricted Boot
>  is not a benefit.
>  
I do believe Secure Boot can provide value, but we need to consider it
only as "bootstrap security" rather than "boot security". On its
face,
it's pretty absurd to restrict operating system content to pre-boot
signatures. The original goal of secure boot support in Linux was to
support the de minimis effort to enable Free platforms to work on PCs
where it may not be easy or possible to turn off Secure Boot. There
was a very real fear that it would be a reality, which is why we sign
our boot code with the Microsoft certificate. But our design
deliberately uses the Microsoft certificate to chain into our own.

Where we fell apart compared to Windows and macOS is that we decided
that we wanted to have all kernel code locked down exclusively by
either firmware certificates or hardcoded certificates in shim. In
retrospect, this was a bad move because it crippled our ability to
enable driver integrity protection for the operating system using
similar techniques used by other operating systems. On the flip side,
because our own ability to enable and support operating system trust
was hamstrung by firmware certificates, we did finally get NVIDIA to
open up[1]. Was it the right decision anyway? I don't think so. The
way the UAPI group envisions system security clearly shows where this
falls apart: we increasingly rely on parts of the system that we have
no influence on to guide our system integrity protection.

Secure Boot would have been great if it was designed as a
user-friendly way to establish and control system content trust. But
that's not how it works in practice at all. And the sheer amount of
resistance around making it work that way demonstrates how much of a
failure it is for that purpose. And it bears repeating: Windows
doesn't use Secure Boot for system integrity protection. It has its
own in-OS mechanisms instead.

As someone whose day job involves working with teams who develop both
Windows and Linux drivers (and in the past, even macOS drivers!), I
can categorically say that Windows driver engineering processes are
way friendlier than Linux ones, precisely because Windows drivers are
deliberately not exposed to Secure Boot *at all*. Running development
versions of drivers just requires installing a development certificate
into the NT kernel certificate store. The equivalent process for Linux
is to load a custom secure boot certificate into firmware, which is
fraught with peril and potentially not even possible. I cannot tell
you how many times I've bricked a system by attempting to load another
cert only to find out there's not enough space and the firmware didn't
handle that well.

We've failed here, badly. And nobody cares.


[1]: https://developer.nvidia.com/blog/nvidia-releases-open-source-gpu-kernel-...


-- 
真実はいつも一つ！/ Always, there's only one truth!

[Vitaly Zaitsev]

On 21/12/2022 03:32, Neal Gompa wrote:
>  As someone whose day job involves working with teams who develop
> both
>  Windows and Linux drivers (and in the past, even macOS drivers!), I
>  can categorically say that Windows driver engineering processes are
>  way friendlier than Linux ones, precisely because Windows drivers are
>  deliberately not exposed to Secure Boot*at all*. 
All Windows drivers must be signed by WHQL[1], i.e. by Microsoft 
Corporation.

Unsigned Windows drivers can be loaded only in a special "Driver 
testing" mode.

[1]: 
https://learn.microsoft.com/en-us/windows-hardware/drivers/install/whql-r...

-- 
Sincerely,
   Vitaly Zaitsev (vitaly(a)easycoding.org)

[Lennart Poettering]

On Di, 20.12.22 21:32, Neal Gompa (ngompa13(a)gmail.com) wrote:

>  As someone whose day job involves working with teams who develop
> both
>  Windows and Linux drivers (and in the past, even macOS drivers!), I
>  can categorically say that Windows driver engineering processes are
>  way friendlier than Linux ones, precisely because Windows drivers are
>  deliberately not exposed to Secure Boot *at all*. Running development
>  versions of drivers just requires installing a development certificate
>  into the NT kernel certificate store. The equivalent process for Linux
>  is to load a custom secure boot certificate into firmware, which is
>  fraught with peril and potentially not even possible. 
This is a bit of a misrepresentation how this currently works.

The Linux kernel populates its keyrings these days from both built-in
keys and from those imported from uefi vars (both uefi db and mokutil
stuff).

So, yes, SecureBoot keys are imported into the kernel keyring, but
they aren't the only thing. You can enroll your own via mokutil and
the OS vendor can enroll by building them into the kernel. Neither of
those two has much to do with firmware, i.e. there's no need to update
the UEFI db, updating the mokutil efi vars is enough.

>  I cannot tell you how many times I've bricked a system by
> attempting
>  to load another cert only to find out there's not enough space and
>  the firmware didn't handle that well. 
Well, sure, space in the efi vars is limited. But christ, how many
certs did you actually enroll via mokutil? (if you are working with
local throw-away certs, then I guess it's just not build for that)

I mean, sure, it would make sense of shim would be able to manage a
keyring on a file in the ESP instead of purely via an efi var, but
this massively raises the question of authenticating that. i.e. would
require some TPM protection usually, but TPM from UEFI mode is not
fun, since (except for PCR measurements) you have to roll the TPM
communication yourself (or embedd a fill TPM stack, but thta's going
to make shim even larger than it already is).

I mean, file an RFE against shim.

But this really has nothing to do with UKIs/sysext, we just use the
same authentication mechanisms as before pretty much: UKIs are
authenticated like any kernels, just as before, and sysext are
authenticated via the same keyring as kmods basically. Hence there's
nothing really new here when it comes to key management.

If you have beef with how Linux manages the keys for authenticating
kmods then bring that up with the kernel community, it has nothing to
do with the way UKIs/sysext works.

Lennart

--
Lennart Poettering, Berlin

[Gerd Hoffmann]

On Wed, Dec 21, 2022 at 03:06:26AM +0100, Kevin Kofler via devel wrote:
>  Daniel P. Berrangé wrote:
>  > That is not correct. There are a number of benefits of UKIs.
>  >  
>  > The most critical is that the initrd content and cmdline is
>  > covered by the SecureBoot signature.
>  
>  From an end user point of view, having more stuff covered by Restricted Boot 
>  is not a benefit. 
It's an option for those who care.  If you don't just turn off secure
boot. Nobody forces anything on users.

take care,
  Gerd

[Kevin Kofler]

PS (adding to my previous reply):

Daniel P. Berrangé wrote:
>  The immediate need for UKIs is indeed related to SecureBoot and
>  TPMs. These are a core technology foundation of the confidential
>  virtual machine stack. On Azure today, if you request an Ubuntu
>  confidential VM, Azure will pre-encrypt the root filesystem and 
So basically this change proposal is about supporting a feature of the 
Microsoft cloud platform (Azure) in Fedora and will be pretty useless to any 
user not using Microsoft's platform.

>  seal the LUKS key against predicted TPM PCR values. It guarantees
>  that the root disk can only be decrypted by the specific VM
>  instance that is requested, when it is running in SecureBoot
>  mode with the expected measurments on AMD SEV-SNP confidential
>  hardware. 
Does it really guarantee that, and not just that it can only be decrypted by 
any VM using the same UKI?

How reliably does it ensure that the user can only get root in the decrypted 
image with the root password (or SSH key or similar) stored inside the image 
and not through some other means?

In the end, if you store data on a "cloud", you are storing it on other 
people's computers. You are also relying on their confidentiality 
guarantees. How can you trust the "cloud" provider to actually perform the 
encryption steps they claim to perform when you check that checkbox, and 
also to not have a backdoor (such as a fixed master key in an extra LUKS key 
slot, or a custom, possibly software-emulated, TPM that does not actually 
keep the key sealed) that allows them to decrypt anything anyway?

You are handing off your data to a third party and then trying to rely on 
Treacherous Computing technologies preventing that third party from doing 
some things (such as copying the encryption key) on their own computers. I 
do not think that this is in either party's interest.

        Kevin Kofler

[Daniel P. Berrangé]

On Wed, Dec 21, 2022 at 03:31:10AM +0100, Kevin Kofler via devel wrote:
>  PS (adding to my previous reply):
>  
>  Daniel P. Berrangé wrote:
>  > The immediate need for UKIs is indeed related to SecureBoot and
>  > TPMs. These are a core technology foundation of the confidential
>  > virtual machine stack. On Azure today, if you request an Ubuntu
>  > confidential VM, Azure will pre-encrypt the root filesystem and
>  
>  So basically this change proposal is about supporting a feature of the 
>  Microsoft cloud platform (Azure) in Fedora and will be pretty useless to any 
>  user not using Microsoft's platform. 
No it is not useless. Most major public public clouds have support
for running VMs with SecureBoot and TPMs, which they typically
refer to as "Trusted Boot". Fedora (and Linux in general) support
for this is a pretty poor from a security POV since the initrd is
not covered by the signature. Making use of UKIs allows building
VMs that take proper advantage of Trusted Boot to allow data to
be secrely sealed to the TPM.

The Azure confidential VMs feature is something that then builds
on top of Trusted Boot, to provide confidentiality against a
malicious hypervisor OS/admin.

I mention Azure because it is first to ship something in this
space. We are also working on confidential VMs for KVM, which
will likewise use Trusted Boot as a foundation for the boot
process. So we will have a fully open source hypervisor stack
that can take advantage of UKIs in Fedora. I can't promise any
timeframe to deliver it in Fedora yet, though probably not F38.
Maybe some bits in F39 if everything aligns.

Overall our goal is to make "confidential VMs" into a boring
feature that "just works" without users needing to understand
the technology details (unless they really want to, or have
specific extra requirements for verification). We want to be
able to build a VM disk image that is capable of booting in
a guest using legacy BIOS, or UEFI, or UEFI with SecureBoot
+ TPM, or UEFI with SecureBoot + TPM + Confidential VM. This
is one of the reasons for adding UKI support to grub. The
complex bits are all to be handled by the OS / hypervisor
vendors, not made into a burden no the end users.

>  > seal the LUKS key against predicted TPM PCR values. It
> guarantees
>  > that the root disk can only be decrypted by the specific VM
>  > instance that is requested, when it is running in SecureBoot
>  > mode with the expected measurments on AMD SEV-SNP confidential
>  > hardware.
>  
>  Does it really guarantee that, and not just that it can only be decrypted by 
>  any VM using the same UKI? 
The root disk has to be encrypted for *each* instance of the VM,
because you can't have the LUKS primary key shared beween instances.
Similarly each VM has to have a distinct TPM with its own keys.
When you seal the LUKS passphrase using keys associated with one
TPM, its impossible to unseal using the TPM from any other VM.

The signed UKI is needed so that the initrd is running trustworthy
code, to ensure it is not stashing a copy of the LUKS passpharse
some an attacker can then access.

>  How reliably does it ensure that the user can only get root in the
> decrypted 
>  image with the root password (or SSH key or similar) stored inside the image 
>  and not through some other means? 
There is a challenge with the way tools like cloud-init traditionally
work, because the data they inject into the guest, is marshalled by
the hypervisor. This hypervisor is considered untrustworthy in a
confidential compute environment.

Thus the runtime injection of the SSH key to the VM is dubious.
So I can't claim every attack vector is solved yet with confidential
VMs. People are working to incrementally tackle different pieces.
Most likely it will either involve a different channel that cloud-init,
or will involve providing signatures over cloud-init data that can be
securely verified, to prove the right SSH key is injected.

>  In the end, if you store data on a "cloud", you are storing
> it on other 
>  people's computers. You are also relying on their confidentiality 
>  guarantees. How can you trust the "cloud" provider to actually perform the 
>  encryption steps they claim to perform when you check that checkbox, and 
>  also to not have a backdoor (such as a fixed master key in an extra LUKS key 
>  slot, or a custom, possibly software-emulated, TPM that does not actually 
>  keep the key sealed) that allows them to decrypt anything anyway? 
There are theoretically ways to prove this, but it is getting
quite complicated to go into much detail, so I'll make two
general points.

First there is a difference between trusting the hypervisor and
trusting the cloud vendor. Even if you still need to trust the
cloud vendor, it is beneficial to be able to distrust their
individual compute hosts. If a VM manages to break out into the
host, this gives protection to your VM that doesn't exist in
traditional virtualization environments. Being able to distrust
the cloud vendor entirely is a subsequent logical step.

I think that open source should be a better position here, since
being able to verify the source running on the cloud, and have
reprodicable builds are a key part to establishing confidence.
The disk image encryption stuff would have to be verifiable &
reproducible, and itself be running inside a confidential VM
that the user can verify, to avoid the possibility of the cloud
owner backdooring it. Hard but by no means impossible, and what
we want to achieve with KVM eventually.

>  You are handing off your data to a third party and then trying to
> rely on 
>  Treacherous Computing technologies preventing that third party from doing 
>  some things (such as copying the encryption key) on their own computers. I 
>  do not think that this is in either party's interest. 
Compare with the current state of public cloud, where you hand off
your data to a third party and have no guarantees about anything,
just have to blindly trust all of the hardware and software and
vendor. With confidential VMs we're incrementally reducing the
amount of trust needed on the vendor/software, by taking advantage
of the hardware. Hardware (and its associated firmware) of course
is not foolproof, but users already trust the hardware today.

Ultimately some people will be paranoid enough they'll never use
public cloud for certain workloads. Confidential VMs may move the
needle such that some more workloads become appropriate for public
cloud, while also giving greater protection to workloads that are
already using public cloud. 

With regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

[Gerd Hoffmann]

  Hi,

>  I think UKIs are fundamentally flawed and are an idea that came out
> of
>  a group that doesn't really interact with real users enough to
>  understand how much of a problem they actually are. I realize that
>  this Change is only about VMs, but since it explicitly talks about it
>  being "phase 1", the implication is that future Changes are coming to
>  switch fully over. Consequently, I'm going to provide much more
>  holistic feedback instead of just nitpicking on this case. 
That is correct, although I expect we'll support both ukis and
traditional initrds in parallel for quite a while and the day where all
use cases can be handled with ukis is far away.  Too many use cases
depend on todays host-generated initrd workflow.

>  In the Fedora case, things are simpler right up until we hit
> graphics
>  drivers. This is also a problem for VMs too, because GPU passthrough
>  is a common case for scientific and gaming workloads. As long as the
>  NVIDIA driver remains dominant in Linux, UKIs cannot work because by
>  design you cannot load anything that isn't part of the kernel image.
>  For bare metal, we *need* these drivers in early boot, though. 
No.  The graphics drivers don't need be part of the initrd.  Booting
on vgacon or efifb until the root filesystem is mounted and drivers can
be loaded from there works just fine.

>  And that's another problem: no third-party early boot drivers.
> Even if
>  you solve the signing issue, you need to introduce some kind of
>  two-stage OS boot process so we can bring up the bare minimum to load
>  a second image containing all the remaining drivers. And at that
>  point, you've defeated the purpose of UKIs. I've heard from some
>  people that system extensions (sysexts) would be a way to solve this,
>  and maybe it is. 
Yes, that problem is not solved yet.  And, yes, sysexts are the IMHO
most promising option to handle that.  I suspect using them requires
putting dracut upside down or replace dracut with something else, so
I expect that will take a while to sort out (disclaimer: I don't have
much insight into the inner workings of dracut).

>  But again, we've eliminated the value of UKIs by doing so.
No.  sysexts can be signed.

>  Speaking more broadly, there are also problems that will be
> introduced
>  as this trickles down from Fedora into prominent downstreams (assuming
>  this is accepted). In the RHEL case, you've basically broken driver
>  disks completely. 
Hmm?  A driver disk could provide a sysext ...

>  The crux of the problem here is *signing*. All of this is tied up in
>  Secure Boot and TPM, which is the wrong place to actually handle this. 
sysext signatures are independent from secure boot.

take care,
  Gerd

[Zbigniew Jędrzejewski-Szmek]

On Tue, Dec 20, 2022 at 11:28:48AM -0500, Neal Gompa wrote:
>  On Tue, Dec 20, 2022 at 10:22 AM Ben Cotton
> <bcotton(a)redhat.com&gt; wrote:
>  > https://fedoraproject.org/wiki/Changes/Unified_Kernel_Support_Phase_1
>  I think UKIs are fundamentally flawed and are an idea that came out of
>  a group that doesn't really interact with real users enough to
>  understand how much of a problem they actually are. 
You're painting with a very wide brush here. The concept of UKIs has been in
development for a long time. [1] adds the initial implementation of sd-stub in
systemd, and that commit is from 2015!  Many, many people have been working on
the details of the concept and vying to make the implementation better. That
group includes most maintainers of systemd, but also external contributors.
Accusing us of not interacting with "real users" is strange.

[1] https://github.com/systemd/systemd/commit/0fa2cac4f0cdefaf1

>  I realize that
>  this Change is only about VMs, but since it explicitly talks about it
>  being "phase 1", the implication is that future Changes are coming to
>  switch fully over. Consequently, I'm going to provide much more
>  holistic feedback instead of just nitpicking on this case. 
I think that limiting this Change to VMs is a very smart move. It is a very
clear case that we can actually tackle right now, and we generally know what
concepts we need to cover it, and the code is either there or at least under
significant development. Let's do this one thing right and learn from the
experience.

>  In the Fedora case, things are simpler right up until we hit
> graphics
>  drivers. This is also a problem for VMs too, because GPU passthrough
>  is a common case for scientific and gaming workloads. As long as the
>  NVIDIA driver remains dominant in Linux, UKIs cannot work because by
>  design you cannot load anything that isn't part of the kernel image.
>  For bare metal, we *need* these drivers in early boot, though. And
>  that's another problem: no third-party early boot drivers. Even if you
>  solve the signing issue, you need to introduce some kind of two-stage
>  OS boot process so we can bring up the bare minimum to load a second
>  image containing all the remaining drivers. And at that point, you've
>  defeated the purpose of UKIs. I've heard from some people that system
>  extensions (sysexts) would be a way to solve this, and maybe it is.
>  But again, we've eliminated the value of UKIs by doing so.
>
>  Speaking more broadly, there are also problems that will be introduced
>  as this trickles down from Fedora into prominent downstreams (assuming
>  this is accepted). In the RHEL case, you've basically broken driver
>  disks completely. In the UKI model, there's no way to incorporate
>  early boot storage, network, and graphics drivers. This is
>  *incredibly* common for RHEL administrators because there's a general
>  acceptance of proprietary kernel modules from vendors these days. Even
>  ignoring those, Red Hat's kernel feature support mindset is completely
>  incompatible with UKIs, because RHEL does default-off rather than
>  Fedora's default-on model for kernel features. We could debate until
>  the cows come home on whether it's right or wrong, but their current
>  mindset essentially means tons of common hardware becomes unsupported
>  quite regularly. The ELRepo project is popular among RHEL folks
>  because it restores those drivers and makes it possible to use RHEL on
>  those machines through a combination of driver disks and kernel module
>  packages.
>  
>  The crux of the problem here is *signing*. All of this is tied up in
>  Secure Boot and TPM, which is the wrong place to actually handle this.
>  In other operating systems (notably Windows), Secure Boot certificates
>  are not used for blocking or enabling kernel-level drivers. Instead,
>  there's a kernel-level certificate store that is used to validate and
>  enable drivers, allowing everything to be managed in a hardware
>  platform agnostic way. 
Whether the kernel is installed as an UKI or as a separate kernel+initrd, doesn't
change much for driver loading. In SecureBoot mode, the kernel will refuse to
load any module that is not signed. But there is a big difference for userspace
code: with a UKI, the early boot userspace code is also signed and the signature
can be verified like with the kernel. (And once this code has been loaded, and
we have normal userspace, we can do futher verifications on other code that is
loaded later.)

If additional modules need to be loaded from outside of the UKI, this is
entirely doable. Systexts are one solution that is proposed, but it may not be
even be the best fit in this case, because sysexts are designed to extend the
initrd file system with additional code. They are a good solution when for
example you want to add a bluetooth stack to the initrd, but overkill for the
case of a module file. For modules, what we need is a way for the initrd code to
access some additional storage and e.g. try to load all files matching a certain
name pattern as modules. This isn't rocket science, and once we have that, we're
back in similar situation as with a separate initrd (apart from the part where
we check the signature of initrd code before starting it).

Whether we should change the verification and requirements on (external) modules
is an interesting question, but mostly orthogonal to the topic at hand.

>  I've also yet to hear of a decent reason for TPM measurements
> too. Block or
>  filesystem verity provides similar guarantees to provide tamper resistance and
>  are both much easier to debug than TPM interfaces. 
TPM measurements haven't been used much in the Linux world yet, primarily
because doing anything with them is very hard. UKIs (when combined with a boot loader
that supports this) actually make things significantly easier:
- PCR values after boot become predictable and can be pre-calculated
  (see www.freedesktop.org/software/systemd/man/systemd-measure.html).
- since PCR values are pre-calculated, PCR statements (policies) on them can be
  signed by the distro and attached to the UKI (see
  https://github.com/systemd/systemd/blob/main/src/ukify/ukify.py).
- This means that you can bind TPM-protected secrets to the certificate used by
  the distro (instead of specific PCR values, which necessarilly change with each
  update).
- This means that you can build policies for TPM-protected secrets bound to various
  aspects of the machine state. A typical example would be allowing
  password-less decryption of the root LUKS volume, iff you're runnning Fedora with
  an official kernel and initrd and unchanged kernel command line, and the
  machine hasn't yet transitioned from the initrd to the host system.

Another use of TPM measurements is obviously remote attestation. I'm personally
less interested in this, but for cloud computing it's apparently pretty
important.

This is all still a bit vague and underdeveloped. We are finally starting to
have technological pieces in hand to play with this at the distro level.

>  I am not convinced that you are providing security or reliability
> with this
>  and the trade-offs are *terrible* for regular users. 
Dunno, you make it sound like this would have some big impact on users. But
after all, this is essentially a proposal to have an additional initrd packaging
method in one corner of the distro. The old packaging method is not going
away. The way that the initrd is built is not changed. If it works for those
users who try it — great. If not, they can immediately switch back to the
existing approach.

Zbyszek

[Lennart Poettering]

On Di, 20.12.22 11:28, Neal Gompa (ngompa13(a)gmail.com) wrote:

>  I think UKIs are fundamentally flawed and are an idea that came out
> of
>  a group that doesn't really interact with real users enough to
>  understand how much of a problem they actually are. 
I presume this is supposed to be a comment on systemd upstream, which
includes me?

Dunno, I think I have quite a good idea how people use systemd, and
how general purpose distros like Fedora use it. Because I get the
firehose of feedback on it. From *all* distros, and from all ways how
peole use it.

>  In the Fedora case, things are simpler right up until we hit
> graphics
>  drivers. This is also a problem for VMs too, because GPU passthrough
>  is a common case for scientific and gaming workloads. As long as the
>  NVIDIA driver remains dominant in Linux, UKIs cannot work because by
>  design you cannot load anything that isn't part of the kernel image. 
That's simply not true. Along with UKIs we developed the "sysext"
concept, to make them extensible. Primary usecase: extend the basic
initrd with drivers/subsystems that you want on some systems but not all.

>  For bare metal, we *need* these drivers in early boot, though. And
>  that's another problem: no third-party early boot drivers. Even if you
>  solve the signing issue, you need to introduce some kind of two-stage
>  OS boot process so we can bring up the bare minimum to load a second
>  image containing all the remaining drivers. 
In the UKI model the UEFI stub (i.e. systemd-stub) actually loads the
sysext images from UEFI mode still, via UEFI fs APIs. i.e. they area
already in memory when the initrd initializes, and can be used
instantly. They are validated via verity/kernel certification
keyring.

>  And at that point, you've
>  defeated the purpose of UKIs. I've heard from some people that system
>  extensions (sysexts) would be a way to solve this, and maybe it is.
>  But again, we've eliminated the value of UKIs by doing so. 
I don't see any value being "eliminated".

>  Speaking more broadly, there are also problems that will be
> introduced
>  as this trickles down from Fedora into prominent downstreams (assuming
>  this is accepted). In the RHEL case, you've basically broken driver
>  disks completely. In the UKI model, there's no way to incorporate
>  early boot storage, network, and graphics drivers. 
There is, sysext.

>  The crux of the problem here is *signing*. All of this is tied up in
>  Secure Boot and TPM, which is the wrong place to actually handle
>  this. 
sysext authentication is tied to kernel keyring authentication
actually, exactly like kernel modules.

The kernel keyring can be populated via mokutil.

>  In other operating systems (notably Windows), Secure Boot
> certificates
>  are not used for blocking or enabling kernel-level drivers. Instead,
>  there's a kernel-level certificate store that is used to validate and
>  enable drivers, allowing everything to be managed in a hardware
>  platform agnostic way. 
I think people currently accept that mokutil is the certificate store
for Linux. If you don't like that, I might even agree to some point
with that, but this isn't new in the context of UKIs or sysext, we
just use the very same infrastructure kmods use.

>  I've also yet to hear of a decent reason for
>  TPM measurements too. Block or filesystem verity provides similar
>  guarantees to provide tamper resistance and are both much easier to
>  debug than TPM interfaces. 
Unlocking secrets via TPM is usually what you want for write
acccess. Verity is read-only only.

>  With my FESCo hat on, I'm uneasy about this change. With my
> "Fedora
>  user and advocate" hat on, I think that the UAPI group has failed to
>  provide something useful to the Linux world here and I would be
>  extremely apprehensive about Fedora adopting any portion of this
>  stuff. 
Umpfh. This is FUD. Please read up before writing scathing comments
like that.

Lennart

--
Lennart Poettering, Berlin

[Chris Murphy]

On Tue, Dec 20, 2022, at 10:22 AM, Ben Cotton wrote:

>  == Detailed Description ==
>  The goal is to move away from initrd images being generated on the
>  installed machine.  They are generated while building the kernel
>  package instead, then shipped as part of a unified kernel image.
>
>  A unified kernel image is an all-in-one efi binary containing kernel,
>  initrd, cmdline and signature.  The secure boot signature covers
>  everything, specifically the initrd is included which is not the case
>  when the initrd gets loaded as separate file from /boot.
>
>  Main motivation for this move is to make the distro more robust and more secure.
>
>  Switching the whole distro over to unified kernels quickly is not
>  realistic though.  Too many features are depending on the current
>  workflow with a host-specific initrd (and host-specific kernel command
>  line), which is fundamentally incompatible with unified kernels where
>  everybody will have the same initrd and command line. Thats why there
>  is 'Phase 1' in title, so we can have more Phases in future releases
>  😃
>
>  A host-specific initrd / command line is needed today for:
>
>  * features needing optional dracut modules (initrd rebuild needed to
>  enable them).
>  * configuration / secrets baked into the initrd (booting from iscsi
>  for example).
>  * configuration being specified on the kernel command line.
>  ** root filesystem being the most important one.
>  [https://systemd.io/DISCOVERABLE_PARTITIONS/ Discoverable partitions]
>  allow to remove this. 
Discoverable partitions is the first barrier in the proposal. I agree with everything up
to this point. I like discoverable partitions specification, the gotcha is there is the
significant amount of inertia still against trusting GPT partition type GUIDs. They tend
to be entirely ignored in Linux. Yes systemd honors them, but the installer is pretty weak
in its understanding of them, and by extention parted/libparted lacks support for (a) the
entire discoverable partitions spec's listing of type GUIDs (b) arbitrary GUIDs. 

So I think the first big barrier to entry is answering the questions:

* Enhance parted/libparted to support arbitrary GUIDs and enhance blivet to understand the
full listing of GUIDs? Or

* Enhance parted/libparted to support full listing of GUIDs? Or

* switch anaconda/blivet to using fdisk/libfdisk? It already supports pretty much all GPT
partition type GUIDs and receives regular updates with additions; I'm not sure if it
supports arbitrary GUIDs, the CLI tool seems not to support it but maybe fdisk does.


>  Phase 1 goals (high priority):
>
>  * Ship a unified kernel image as (optional) kernel sub-rpm.  Users can
>  opt-in to use that kernel by installing the sub-rpm.  Initial focus is
>  on booting virtual machines where we have a relatively small and well
>  defined set of drivers / features needed.  Supporting modern physical
>  machines with standard setup (i.e. boot from local sata/nvme storage)
>  too should be easy. 
I like the focus on VM's so long as it drives us forward incrementally in a way we can
then converge baremetal back toward rather than fragmenting. I'm skeptical this is
going to be easy to do on baremetal... more in a bit.


>  * Add bootloader support for unified kernel images.  Add
>  [https://systemd.io/BOOT_LOADER_SPECIFICATION/#type-2-efi-unified-kernel-i...
>  unified kernel bls support] to grub2, or support using systemd-boot,
>  or both. 
Great. Just be aware the bootloader team is somewhere between disinterested in Boot Loader
Specification, and hostile to it. They support the BLS snippet file format. Beyond that,
there is defiance of the specification as an actual bonefide spec that Fedora does or even
should support. And because of that you can expect breakage.

For example:
https://bugzilla.redhat.com/show_bug.cgi?id=2120845

For that matter, grubby likewise steps on *all* BLS snippets found in /boot/loader/entries
when using the --update=ALL flag, not just the BLS snippets

>  ** When using btrfs: configure 'root' subvolume as default
> volume. 
The installer understands `btrfs subvolume set-default` so it could set the
"root" subvolume. The difficulty I've always had with this, it's that
it's not discoverable. There's no way someone not already familiar with the
concept of changing the default subvolume would ever figure this out on their own, and
I'm a big fan of the boot process being verbose and explicit. Anytime there's
silent leaps of faith, it's harder to trouble shoot, and harder still for mortal users
to understand how their system functions.

But maybe this can be mitigated by e.g. the kernel's btrfs driver announcing the
subvolume/subvolume ID it's mounting when it's not explicit (i.e. when it's
honoring a subvolume other than ID 5, set as the default subvolume)

Previously I've mentioned expanding the discoverable partitions conceptually, a sort
of "discoverable subvolumes spec" so that Btrfs and OSTree and LVM could benefit
from discoverability via a naming convention that a new systemd generator could parse and
take action on. LVM and OSTree don't have an approximate equivalent of btrfs default
subvolumes, so they are in need of a solution to find the proper root without it being
explicit. If we're going to accept that specifying root trees must be explicit for LVM
and OSTree, then I think we might as well leave things be for Btrfs and explicitly list
the root tree there too.


>  * Add proper systemd-boot support to installers. 

Great. The gotcha though is this in effect requires a change in the file system currently
mounted at /boot, which is ext4. And ext4 isn't supported by sd-boot or UEFI firmware.
So if you're going to support sd-boot, the installer needs to be aware that either the
ESP is big enough to be used as /boot, or if it's not big enough then it will be
mounted on /efi *and* a new partition XBOOTLDR formatted as FAT will be used as /boot.

This is not a major consideration for this feature proposal, but the sooner some of these
decisions are made the easier it's going to be by not having to unwind mistakes down
the road:

For the purposes of a VM, it's a single OS setup so it makes sense to go with the
simplest option which is an ESP on /boot that serves the purposes of both ESP and
"boot" (bootloader, bootloader configuration, and kernels all on one volume).
However...

There is the still common and supported baremetal dual-boot scenario were the ESP will be
too small, so we'll need ESP on /efi, and XBOOTLDR on /boot. While we could support
both EFI only and EFI+XTBOOTLDR layouts for the different use cases, I'd encourage
just supporting one size fits all: EFI+XTBOOTLDR. It's more compatible (dual boot),
and more like what we have today. 



>  * Better secure boot support (specifically the initrd is covered by
>  the signature). 
We need to solve the glaring hole that is the initrd. There's no question about it. I
can't really assess if this feature is the best way to do that. Or if it would be
adequate for dracut to self-sign every locally generated initrd with a unique key pair and
throw away the private key after each initrd is generated.  Or if we could do enough
strict standardization in the boot chain with a possibly larger kernel to avoid needing an
initrd, i.e. get to sysroot mount faster thereby obviating the need for a large initrd.

The problem with a large initrd anyway is that we have to load the whole thing in memory
and unpack it. And this will increase boot times. For whatever reason, the bootloaders
don't support particularly fast reading off modern storage. Maybe it's a pre-boot
environment limitation, not a bootloader limitation, I'm not sure.


-- 
Chris Murphy

[Chris Murphy]

On Tue, Dec 20, 2022, at 1:56 PM, Chris Murphy wrote:

>  So I think the first big barrier to entry is answering the
> questions:
>
>  * Enhance parted/libparted to support arbitrary GUIDs and enhance 
>  blivet to understand the full listing of GUIDs? Or
>
>  * Enhance parted/libparted to support full listing of GUIDs? Or
>
>  * switch anaconda/blivet to using fdisk/libfdisk? It already supports 
>  pretty much all GPT partition type GUIDs and receives regular updates 
>  with additions; I'm not sure if it supports arbitrary GUIDs, the CLI 
>  tool seems not to support it but maybe fdisk does. 
Typo: maybe libfdisk does (support arbitrary partition type GUIDs)


-- 
Chris Murphy

[Chris Murphy]

On Tue, Dec 20, 2022, at 1:56 PM, Chris Murphy wrote:

>  For example:
>  https://bugzilla.redhat.com/show_bug.cgi?id=2120845
>
>  For that matter, grubby likewise steps on *all* BLS snippets found in 
>  /boot/loader/entries when using the --update=ALL flag, not just the BLS 
>  snippets 
Incomplete thought:

--update=ALL modifies all snippets in /boot/loader/entries, not just the BLS snippets the
currently running root owns. This brings up the concept of each installed distro, whether
literally a different distro or merely an additional instance of the same distro, having
ownership of its BLS snippets and not owning other BLS snippets.

Fedora A should not be modifying Fedora B's BLS snipets, but it does. And now breaks
them since the change earlier this year mentioned in the bug.



-- 
Chris Murphy

[Daniel P. Berrangé]

On Tue, Dec 20, 2022 at 01:56:57PM -0500, Chris Murphy wrote:
>  
>  
>  On Tue, Dec 20, 2022, at 10:22 AM, Ben Cotton wrote:
>  
>  > == Detailed Description ==
>  > The goal is to move away from initrd images being generated on the
>  > installed machine.  They are generated while building the kernel
>  > package instead, then shipped as part of a unified kernel image.
>  >
>  > A unified kernel image is an all-in-one efi binary containing kernel,
>  > initrd, cmdline and signature.  The secure boot signature covers
>  > everything, specifically the initrd is included which is not the case
>  > when the initrd gets loaded as separate file from /boot.
>  >
>  > Main motivation for this move is to make the distro more robust and more secure.
>  >
>  > Switching the whole distro over to unified kernels quickly is not
>  > realistic though.  Too many features are depending on the current
>  > workflow with a host-specific initrd (and host-specific kernel command
>  > line), which is fundamentally incompatible with unified kernels where
>  > everybody will have the same initrd and command line. Thats why there
>  > is 'Phase 1' in title, so we can have more Phases in future releases
>  > 😃
>  >
>  > A host-specific initrd / command line is needed today for:
>  >
>  > * features needing optional dracut modules (initrd rebuild needed to
>  > enable them).
>  > * configuration / secrets baked into the initrd (booting from iscsi
>  > for example).
>  > * configuration being specified on the kernel command line.
>  > ** root filesystem being the most important one.
>  > [https://systemd.io/DISCOVERABLE_PARTITIONS/ Discoverable partitions]
>  > allow to remove this.
>  
>  Discoverable partitions is the first barrier in the proposal. I agree with everything up
> to this point. I like discoverable partitions specification, the gotcha is there is the
> significant amount of inertia still against trusting GPT partition type GUIDs. They tend
> to be entirely ignored in Linux. Yes systemd honors them, but the installer is pretty weak
> in its understanding of them, and by extention parted/libparted lacks support for (a) the
> entire discoverable partitions spec's listing of type GUIDs (b) arbitrary GUIDs. 
>  
>  So I think the first big barrier to entry is answering the questions:
>  
>  * Enhance parted/libparted to support arbitrary GUIDs and
>    enhance blivet to understand the full listing of GUIDs? Or 
parted/libparted already have support for handling GUIDs since
their 3.5 release.

I added pyparted support in

  https://github.com/dcantrell/pyparted/pull/95

and I've got work in progress for blivet support

  https://github.com/storaged-project/blivet/pull/1080

in theory that should merely then require anaconda to set a flag
in blivet to automagicaly enable setting well known GUIDs.

I've also got some changes to pykickstart to let the GUID be
set in kickstart file part commands.


>  > Phase 1 goals (high priority):
>  >
>  > * Ship a unified kernel image as (optional) kernel sub-rpm.  Users can
>  > opt-in to use that kernel by installing the sub-rpm.  Initial focus is
>  > on booting virtual machines where we have a relatively small and well
>  > defined set of drivers / features needed.  Supporting modern physical
>  > machines with standard setup (i.e. boot from local sata/nvme storage)
>  > too should be easy.
>  
>  I like the focus on VM's so long as it drives us forward incrementally
>  in a way we can then converge baremetal back toward rather than
>  fragmenting. I'm skeptical this is going to be easy to do on baremetal...
>  more in a bit. 
Yes, bare metal is certainly harder due to the much wider variance
in boot hardware and also users tend todo much more complex thing.

The focus on VMs was intentional to make the problem more tractable
to solve. Even for VMs this proposal doesn't solve every use case.
It is anticipated things will evolve incrementally over time to
cover more use cases for  both VMs & bare metal.

The existing way of booting kernels+locally created initrds isn't
going anywhere, so we're not loosing any features.


>  > * Add bootloader support for unified kernel images.  Add
>  > [https://systemd.io/BOOT_LOADER_SPECIFICATION/#type-2-efi-unified-kernel-i...
>  > unified kernel bls support] to grub2, or support using systemd-boot,
>  > or both.
>  
>  Great. Just be aware the bootloader team is somewhere between
>  disinterested in Boot Loader Specification, and hostile to it.
>  They support the BLS snippet file format. Beyond that, there
>  is defiance of the specification as an actual bonefide spec
>  that Fedora does or even should support. And because of that
>  you can expect breakage.
>  
>  For example:
>  https://bugzilla.redhat.com/show_bug.cgi?id=2120845
>  
>  For that matter, grubby likewise steps on *all* BLS snippets
>  found in /boot/loader/entries when using the --update=ALL
>  flag, not just the BLS snippets 
As part of virtualization work with confidential VMs we are
in active discussions/collaboration with people invovled in
bootloaders from the Red Hat side. I'm not going to comment
on likes/dislikes anyone might have historically. We're just
looking to work positively find acceptable solutions to the
goals for confidential VMs, ideally with little disruption
for existing practices.

>  > * Add proper systemd-boot support to installers.
>  
>  
>  Great. The gotcha though is this in effect requires a
>  change in the file system currently mounted at /boot,
>  which is ext4. And ext4 isn't supported by sd-boot or
>  UEFI firmware. So if you're going to support sd-boot,
>  the installer needs to be aware that either the ESP
>  is big enough to be used as /boot, or if it's not big
>  enough then it will be mounted on /efi *and* a new
>  partition XBOOTLDR formatted as FAT will be used as
>  /boot. 
Note, for cloud VM disk images, there's little reason
to even have a /boot partition as a separate volume.
We can size the ESP such that it is large enough for
our needs in cloud kickstart files.

For bare metal this would need to be considered though,
since vendors can ship insufficiently large ESPs. THis
is out of scope for anything in the near future, sine
our focus is on VMs, especially cloud VMs.

>  > * Better secure boot support (specifically the initrd is covered
> by
>  > the signature).
>  
>  We need to solve the glaring hole that is the initrd.
>  There's no question about it. I can't really assess if
>  this feature is the best way to do that. Or if it would
>  be adequate for dracut to self-sign every locally
>  generated initrd with a unique key pair and throw
>  away the private key after each initrd is generated. 
Pre-generated initrds solves the SecureBoot hole, but it
does other things too. Most specifically it gives us
predictable measurements of the boot components. We don't
really want to seal secrets (LUKS key) to measurements
that are variable across upgrades, and that means you
have to reseal periodically. We do, however, want to be
able to do attestation of a running system, and that is
more scalable if we can have a more predictable set of
measurements across every VM instance in the fleet.
Locally generated initrds will always be impractical
in this respect, unless you can guaranee the locally
generated content will always be identical on every
install. At that point though, there's little point in
building it locally at all.

>  Or if we could do enough strict standardization in
>  the boot chain with a possibly larger kernel to avoid
>  needing an initrd, i.e. get to sysroot mount faster
>  thereby obviating the need for a large initrd. 
Completely eliminating the need for an initrd is an
interesting option. I'm not sure that would be a thing
that can be delivered in a practical timeframe though,
given its potential impact on the distro as a whole.
UKIs have the benefit that they have near zero impact
on anything that's done today. They're just enabling
new use cases that aren't possible today. The main
burden of course is the need to support them as a
concept in parallel with existing local initrds. I
think that burden is quite modest though, and offset
by the use cases they unlock.

With regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

[Chris Murphy]

On Tue, Dec 20, 2022, at 2:22 PM, Daniel P. Berrangé wrote:

>  parted/libparted already have support for handling GUIDs since
>  their 3.5 release.
>
>  I added pyparted support in
>
>    https://github.com/dcantrell/pyparted/pull/95
>
>  and I've got work in progress for blivet support
>
>    https://github.com/storaged-project/blivet/pull/1080
>
>  in theory that should merely then require anaconda to set a flag
>  in blivet to automagicaly enable setting well known GUIDs.
>
>  I've also got some changes to pykickstart to let the GUID be
>  set in kickstart file part commands. 
Wow, great news!



-- 
Chris Murphy

[Zbigniew Jędrzejewski-Szmek]

On Tue, Dec 20, 2022 at 07:22:34PM +0000, Daniel P. Berrangé wrote:
>  On Tue, Dec 20, 2022 at 01:56:57PM -0500, Chris Murphy wrote:
>  > Or if we could do enough strict standardization in
>  > the boot chain with a possibly larger kernel to avoid
>  > needing an initrd, i.e. get to sysroot mount faster
>  > thereby obviating the need for a large initrd.
>  
>  Completely eliminating the need for an initrd is an
>  interesting option. I'm not sure that would be a thing
>  that can be delivered in a practical timeframe though,
>  given its potential impact on the distro as a whole.
>  UKIs have the benefit that they have near zero impact
>  on anything that's done today. They're just enabling
>  new use cases that aren't possible today. The main
>  burden of course is the need to support them as a
>  concept in parallel with existing local initrds. I
>  think that burden is quite modest though, and offset
>  by the use cases they unlock. 
There might be some specialized cases where initrd-less boot can be done,
but in general I think those will be even more of a minority in the future
than now.

Without an initrd we immediately have the following limitations:
- all kernel modules needed to mount root must be compiled in
- all that code is always loaded and remains in unswappable memory
- root= syntax is limited to what the kernel understands, i.e.
  no root=UUID=… o root=/dev/disk/by-path/… or other udev links,
  no encryption or dm-verity.
- no bluetooth keyboards or other fancy peripherals
- recovery is pretty hard

Initrd are now more complicated and costly (e.g. in the sense of time
required for installation) than they need to be. We should make them
simpler and cheaper, but trying to eliminate them, and thus also
forbid any userspace code from running before root is mounted, is a
red herring.

Zbyszek

[Chris Adams]

Once upon a time, Zbigniew Jędrzejewski-Szmek <zbyszek(a)in.waw.pl&gt; said:
>  Without an initrd we immediately have the following limitations:
>  - all kernel modules needed to mount root must be compiled in
>  - all that code is always loaded and remains in unswappable memory
>  - root= syntax is limited to what the kernel understands, i.e.
>    no root=UUID=… o root=/dev/disk/by-path/… or other udev links,
>    no encryption or dm-verity.
>  - no bluetooth keyboards or other fancy peripherals
>  - recovery is pretty hard 
Also, the security lock-down for the kernel command line means:
- no network root filesystem
- no boot-time-only kernel/module configuration

The idea of switching from grub2 to sd-boot would also drop network boot
and BIOS support.  Supporting boot loaders seems to be a bit of a issue
sometimes, so trying to support multiple boot loaders seems like a bad
idea to me.

-- 
Chris Adams <linux(a)cmadams.net&gt;

[Tomasz Torcz]

On Wed, Dec 21, 2022 at 10:16:58AM -0600, Chris Adams wrote:
>  Once upon a time, Zbigniew Jędrzejewski-Szmek
> <zbyszek(a)in.waw.pl&gt; said:
>  > Without an initrd we immediately have the following limitations:
>  > - all kernel modules needed to mount root must be compiled in
>  > - all that code is always loaded and remains in unswappable memory
>  > - root= syntax is limited to what the kernel understands, i.e.
>  >   no root=UUID=… o root=/dev/disk/by-path/… or other udev links,
>  >   no encryption or dm-verity.
>  > - no bluetooth keyboards or other fancy peripherals
>  > - recovery is pretty hard
>  
>  Also, the security lock-down for the kernel command line means:
>  - no network root filesystem
>  - no boot-time-only kernel/module configuration
>  
>  The idea of switching from grub2 to sd-boot would also drop network boot
>  and BIOS support.  Supporting boot loaders seems to be a bit of a issue
>  sometimes, so trying to support multiple boot loaders seems like a bad
>  idea to me. 
  I'm using network boot without grub2, just iPXE and kernel+initrd.
It worth noting that NFS root doesn't work with dracut from rawhide,
for months now.

-- 
Tomasz Torcz                 “God, root, what's the difference?”
tomek(a)pipebreaker.pl               “God is more forgiving.”

[Daniel P. Berrangé]

On Wed, Dec 21, 2022 at 10:16:58AM -0600, Chris Adams wrote:
>  Once upon a time, Zbigniew Jędrzejewski-Szmek
> <zbyszek(a)in.waw.pl&gt; said:
>  > Without an initrd we immediately have the following limitations:
>  > - all kernel modules needed to mount root must be compiled in
>  > - all that code is always loaded and remains in unswappable memory
>  > - root= syntax is limited to what the kernel understands, i.e.
>  >   no root=UUID=… o root=/dev/disk/by-path/… or other udev links,
>  >   no encryption or dm-verity.
>  > - no bluetooth keyboards or other fancy peripherals
>  > - recovery is pretty hard
>  
>  Also, the security lock-down for the kernel command line means:
>  - no network root filesystem
>  - no boot-time-only kernel/module configuration
>  
>  The idea of switching from grub2 to sd-boot would also drop network boot
>  and BIOS support.  Supporting boot loaders seems to be a bit of a issue
>  sometimes, so trying to support multiple boot loaders seems like a bad
>  idea to me. 
I certainly wouldn't suggest supporting multiple boot loaders with the
complexity level of grub.  sd-boot is quite a different beast though.
It is drastically limited in scope/features, with the actual loading
process offloaded to UEFI services, so probably better thought of as
merely a boot menu. It can't replace grub given its limited scope,
but it is not unreasonable to consider supporting it in parallel for
the cases where the kitchen sink is not required, since it makes it
massively easier to predict TPM measurements than grub.

With regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

[Lennart Poettering]

On Mi, 21.12.22 10:16, Chris Adams (linux(a)cmadams.net) wrote:

>  Once upon a time, Zbigniew Jędrzejewski-Szmek
> <zbyszek(a)in.waw.pl&gt; said:
>  > Without an initrd we immediately have the following limitations:
>  > - all kernel modules needed to mount root must be compiled in
>  > - all that code is always loaded and remains in unswappable memory
>  > - root= syntax is limited to what the kernel understands, i.e.
>  >   no root=UUID=… o root=/dev/disk/by-path/… or other udev links,
>  >   no encryption or dm-verity.
>  > - no bluetooth keyboards or other fancy peripherals
>  > - recovery is pretty hard
>
>  Also, the security lock-down for the kernel command line means:
>  - no network root filesystem
>  - no boot-time-only kernel/module configuration
>
>  The idea of switching from grub2 to sd-boot would also drop network boot
>  and BIOS support.  Supporting boot loaders seems to be a bit of a issue
>  sometimes, so trying to support multiple boot loaders seems like a bad
>  idea to me. 
I am not sure why you think that kernel command line lockdown would mean
no network root fs anymore?

I mean, sure, if you use the Fedora supplied vanilla signed UKI
without anything else then it won't boot from a network, because that
would be a security hole. But I see no reason why a network boot UKI
couldn't be build that maybe be booted via PXE and derives root fs
info from DHCP alone.

(I mean, this is going to be as secure or insecure as your DHCP leases
are protected, but the security is definitely not going to be worse
than in the status quo ante that way)

(you could also build/sign your own UKIs with the network boot info
baked in. Or you could use TPM-bound systemd "credentials" to provide
the network server info to the booted kernel securely)

Lennart

--
Lennart Poettering, Berlin

[Gerd Hoffmann]

On Wed, Dec 21, 2022 at 05:44:36PM +0100, Lennart Poettering wrote:
>  
>  I mean, sure, if you use the Fedora supplied vanilla signed UKI
>  without anything else then it won't boot from a network, because that
>  would be a security hole. But I see no reason why a network boot UKI
>  couldn't be build that maybe be booted via PXE and derives root fs
>  info from DHCP alone. 
I expect that is an area where we have to research how we want build
stuff in the future.  The UKI for virtual machines / cloud images is
simple and static enough that we can just build that as part of the
kernel build process.  For other use cases that might not be the best
approach though.

Also note that UKIs can be booted without an bootloader.  It's an EFI
binary the firmware can handle on its own.  That actually simplifies
network booting, your dhcp server can hand out URLs to the UKI.

take care,
  Gerd

[Gerd Hoffmann]

On Wed, Dec 21, 2022 at 10:16:58AM -0600, Chris Adams wrote:
>  Once upon a time, Zbigniew Jędrzejewski-Szmek
> <zbyszek(a)in.waw.pl&gt; said:
>  > Without an initrd we immediately have the following limitations:
>  > - all kernel modules needed to mount root must be compiled in
>  > - all that code is always loaded and remains in unswappable memory
>  > - root= syntax is limited to what the kernel understands, i.e.
>  >   no root=UUID=… o root=/dev/disk/by-path/… or other udev links,
>  >   no encryption or dm-verity.
>  > - no bluetooth keyboards or other fancy peripherals
>  > - recovery is pretty hard
>  
>  Also, the security lock-down for the kernel command line means:
>  - no network root filesystem
>  - no boot-time-only kernel/module configuration
>  
>  The idea of switching from grub2 to sd-boot would also drop network boot
>  and BIOS support.  Supporting boot loaders seems to be a bit of a issue
>  sometimes, so trying to support multiple boot loaders seems like a bad
>  idea to me. 
The switch to BootLoaderSpec made switching boot loaders *much* easier.
The actual boot loader config is fixed and never touched, kernel updates
add/remove BLS config snippets, and all boot loaders handle that just
fine.

You can even have *two* bootloaders, sd-boot for efi mode and grub2 for
bios mode.

take care,
  Gerd

[Gerd Hoffmann]

  Hi,

>  * Enhance parted/libparted to support arbitrary GUIDs and enhance
> blivet to understand the full listing of GUIDs? Or 
parted is done, blivet still todo.
https://bugzilla.redhat.com/show_bug.cgi?id=1075288#c7

>  the CLI tool seems not to support it but maybe fdisk does. 
sfdisk supports it.  The workaround mentioned in the change page looks
like this:

# setup discoverable partitions
/usr/sbin/sfdisk --part-type /dev/sda 3 4F68BCE3-E8CD-4DB1-96E7-FBCAF984B709  # Linux root
(x86-64)

>  > * Add proper systemd-boot support to installers.
>  
>  Great. The gotcha though is this in effect requires a change in the
>  file system currently mounted at /boot, which is ext4. And ext4 isn't
>  supported by sd-boot or UEFI firmware. So if you're going to support
>  sd-boot, the installer needs to be aware that either the ESP is big
>  enough to be used as /boot, or if it's not big enough then it will be
>  mounted on /efi *and* a new partition XBOOTLDR formatted as FAT will
>  be used as /boot. 
Well, for the virtual machine use case isn't a problem, you can just
make the ESP big enough and even drop the separate /boot filesystem
then.

For the general case we need some other option.  Could be just stick to
grub2 for those cases (we'll continue to need grub2 anyway for bios boot
and ppc64le).  Or drop an ext4 driver to /boot/efi/EFI/systemd/drivers,
for example this one:
  https://github.com/tianocore/edk2-platforms/tree/master/Features/Ext4Pkg

>  > * Better secure boot support (specifically the initrd is covered
> by
>  > the signature).
>  
>  We need to solve the glaring hole that is the initrd. There's no
>  question about it. I can't really assess if this feature is the best
>  way to do that. Or if it would be adequate for dracut to self-sign
>  every locally generated initrd with a unique key pair and throw away
>  the private key after each initrd is generated. 
The private key management problem unfortunately isn't solved that
easily ... 

When using distro signatures you don't have the key management problem
in the first place.

>  Or if we could do enough strict standardization in the boot chain
> with
>  a possibly larger kernel to avoid needing an initrd, i.e. get to
>  sysroot mount faster thereby obviating the need for a large initrd. 
The initrd does more than carrying drivers.  Finding the root filesystem
for example.  The kernel's builtin feaure set for root= is small
compared to what a dracut initrd is able to do.

take care,
  Gerd

[Lennart Poettering]

On Mi, 21.12.22 10:03, Gerd Hoffmann (kraxel(a)redhat.com) wrote:

>  For the general case we need some other option.  Could be just stick
> to
>  grub2 for those cases (we'll continue to need grub2 anyway for bios boot
>  and ppc64le).  Or drop an ext4 driver to /boot/efi/EFI/systemd/drivers,
>  for example this one:
>    https://github.com/tianocore/edk2-platforms/tree/master/Features/Ext4Pkg 
I am pretty strongly against the idea of ext4 for /boot/.

First of all, the firmware can't read it natively, but what's worse,
uefi cannot even make sense of any of the features it provides above
vfat. i.e. file ownership, access modes, ACLs, selinux labels, xattrs,
symlinks, … are all complete nonsense to the UEFI fs layer (and to
boot loaders that natively implement the fs drivers, the same). The
UEFI fs API has no concepts for *any* of these features. Hence, if
this is supposed to carry data intended for consumption by the boot
loader, why the f**k would you use a file system it cannot even
remotely take benefit of?

Also, I am pretty sure boot loaders should have the ability to make
changes to the file systems they read UKIs from, to implement boot
counting and random seed management. grub fs drivers and the stuff
derived thereof (i.e. efifs stuff) typically is read-only though.

At least for the systemd stuff, we carefully made sure that our access
patterns to the ESP both from sd-boot/sd-stub and from userspace are
by default as minimal and robust as we can make them, to minimize
chance of corruption, given that vfat is not particularly good with
that. (i.e. we sync a lot, and the whole ESP mount is by default an
autofs instance with an extremely short idle timeout so that it
basically remains unmounted — and thus — clean during almost all
times).

Anyway, if you want to know more about choice of the fs for /boot/,
see my ideas here:

https://0pointer.net/blog/linux-boot-partitions.html

Lennart

--
Lennart Poettering, Berlin

[Demi Marie Obenour]

On 12/21/22 12:00, Lennart Poettering wrote:
>  On Mi, 21.12.22 10:03, Gerd Hoffmann (kraxel(a)redhat.com) wrote:
>  
> > For the general case we need some other option.  Could be just stick to
> > grub2 for those cases (we'll continue to need grub2 anyway for bios boot
> > and ppc64le).  Or drop an ext4 driver to /boot/efi/EFI/systemd/drivers,
> > for example this one:
> >   https://github.com/tianocore/edk2-platforms/tree/master/Features/Ext4Pkg
>  
>  I am pretty strongly against the idea of ext4 for /boot/.
>  
>  First of all, the firmware can't read it natively, but what's worse,
>  uefi cannot even make sense of any of the features it provides above
>  vfat. i.e. file ownership, access modes, ACLs, selinux labels, xattrs,
>  symlinks, … are all complete nonsense to the UEFI fs layer (and to
>  boot loaders that natively implement the fs drivers, the same). The
>  UEFI fs API has no concepts for *any* of these features. Hence, if
>  this is supposed to carry data intended for consumption by the boot
>  loader, why the f**k would you use a file system it cannot even
>  remotely take benefit of?
>  
>  Also, I am pretty sure boot loaders should have the ability to make
>  changes to the file systems they read UKIs from, to implement boot
>  counting and random seed management. grub fs drivers and the stuff
>  derived thereof (i.e. efifs stuff) typically is read-only though.
>  
>  At least for the systemd stuff, we carefully made sure that our access
>  patterns to the ESP both from sd-boot/sd-stub and from userspace are
>  by default as minimal and robust as we can make them, to minimize
>  chance of corruption, given that vfat is not particularly good with
>  that. (i.e. we sync a lot, and the whole ESP mount is by default an
>  autofs instance with an extremely short idle timeout so that it
>  basically remains unmounted — and thus — clean during almost all
>  times).
>  
>  Anyway, if you want to know more about choice of the fs for /boot/,
>  see my ideas here:
>  
>  https://0pointer.net/blog/linux-boot-partitions.html
>  
>  Lennart 
Does vfat support atomic rename?  Is it possible to atomically upgrade
a bootloader/UKI/etc?
-- 
Sincerely,
Demi Marie Obenour (she/her/hers)

[Lennart Poettering]

On Mi, 21.12.22 12:12, Demi Marie Obenour (demiobenour(a)gmail.com) wrote:

>  > At least for the systemd stuff, we carefully made sure that our
> access
>  > patterns to the ESP both from sd-boot/sd-stub and from userspace are
>  > by default as minimal and robust as we can make them, to minimize
>  > chance of corruption, given that vfat is not particularly good with
>  > that. (i.e. we sync a lot, and the whole ESP mount is by default an
>  > autofs instance with an extremely short idle timeout so that it
>  > basically remains unmounted — and thus — clean during almost all
>  > times).
>  >
>  > Anyway, if you want to know more about choice of the fs for /boot/,
>  > see my ideas here:
>  >
>  > https://0pointer.net/blog/linux-boot-partitions.html
>  >
>  > Lennart
>
>  Does vfat support atomic rename?  Is it possible to atomically upgrade
>  a bootloader/UKI/etc? 
Depends on the fs driver. But yeah, the filename is stored at exactly
one place, and hence typically a single-sector update is
possible. (well, within bounds: for example if you rename a file from
a short filename to a long one, fs driver might need to allocate a
bunch of separate long file name dir entries, which might then span
multiple sectors)

Lennart

--
Lennart Poettering, Berlin

[Demi Marie Obenour]

On 12/21/22 12:17, Lennart Poettering wrote:
>  On Mi, 21.12.22 12:12, Demi Marie Obenour (demiobenour(a)gmail.com)
> wrote:
>  
> >> At least for the systemd stuff, we carefully made sure that our access
> >> patterns to the ESP both from sd-boot/sd-stub and from userspace are
> >> by default as minimal and robust as we can make them, to minimize
> >> chance of corruption, given that vfat is not particularly good with
> >> that. (i.e. we sync a lot, and the whole ESP mount is by default an
> >> autofs instance with an extremely short idle timeout so that it
> >> basically remains unmounted — and thus — clean during almost all
> >> times).
> >>
> >> Anyway, if you want to know more about choice of the fs for /boot/,
> >> see my ideas here:
> >>
> >> https://0pointer.net/blog/linux-boot-partitions.html
> >>
> >> Lennart
> >
> > Does vfat support atomic rename?  Is it possible to atomically upgrade
> > a bootloader/UKI/etc?
>  
>  Depends on the fs driver. But yeah, the filename is stored at exactly
>  one place, and hence typically a single-sector update is
>  possible. (well, within bounds: for example if you rename a file from
>  a short filename to a long one, fs driver might need to allocate a
>  bunch of separate long file name dir entries, which might then span
>  multiple sectors)
>  
>  Lennart 
Under what circumstances do the Linux kernel driver and the firmware
driver actually make such guarantees?  For instance, is renaming a file
with a longer name to a shorter name (and replacing the old one) atomic?
-- 
Sincerely,
Demi Marie Obenour (she/her/hers)

[Javier Martinez Canillas]

On Wed, Dec 21, 2022 at 6:13 PM Demi Marie Obenour
<demiobenour(a)gmail.com&gt; wrote:
>  
[...]

>  
> Does vfat support atomic rename?  Is it possible to
atomically upgrade
> a bootloader/UKI/etc?
> --

For Linux, renameat2 RENAME_EXCHANGE is supported in vfat since
version v6.0. The relevant commits FYI are:

https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit...
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit...
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit...

Best regards,
Javier

[Chris Murphy]

On Wed, Dec 21, 2022, at 12:00 PM, Lennart Poettering wrote:
>  On Mi, 21.12.22 10:03, Gerd Hoffmann (kraxel(a)redhat.com) wrote:
>
> > For the general case we need some other option.  Could be just stick to
> > grub2 for those cases (we'll continue to need grub2 anyway for bios boot
> > and ppc64le).  Or drop an ext4 driver to /boot/efi/EFI/systemd/drivers,
> > for example this one:
> >   https://github.com/tianocore/edk2-platforms/tree/master/Features/Ext4Pkg
>
>  I am pretty strongly against the idea of ext4 for /boot/.
>
>  First of all, the firmware can't read it natively, but what's worse,
>  uefi cannot even make sense of any of the features it provides above
>  vfat. i.e. file ownership, access modes, ACLs, selinux labels, xattrs,
>  symlinks, … are all complete nonsense to the UEFI fs layer (and to
>  boot loaders that natively implement the fs drivers, the same). The
>  UEFI fs API has no concepts for *any* of these features. Hence, if
>  this is supposed to carry data intended for consumption by the boot
>  loader, why the f**k would you use a file system it cannot even
>  remotely take benefit of? 
And for btrfs, the GRUB btrfs.mod doesn't do any checksum checking at all, so
there's not even an incremental improvement to integrity, let alone any support for
fs-verity. So while I like btrfs for /boot due to the space efficiency advantage (I
don't have to ask how big boot should be, no space is wasted and I don't run out
either), I'm willing to give it up for practical matters like simplicity and reduced
attack and maintenance surface area.



-- 
Chris Murphy

[Vitaly Zaitsev]

On 20/12/2022 19:56, Chris Murphy wrote:
>  Great. The gotcha though is this in effect requires a change in the
> file system currently mounted at /boot, which is ext4. And ext4 isn't supported by
> sd-boot or UEFI firmware. So if you're going to support sd-boot, the installer needs
> to be aware that either the ESP is big enough to be used as /boot, or if it's not big
> enough then it will be mounted on /efi*and*  a new partition XBOOTLDR formatted as FAT
> will be used as /boot. 
Nobody should use FAT for /boot. efifs[1] should be used instead.

systemd-boot can load these drivers from ESP out of the box[2].

[1]: https://github.com/pbatard/efifs
[2]: https://github.com/systemd/systemd/issues/15617

-- 
Sincerely,
   Vitaly Zaitsev (vitaly(a)easycoding.org)

[Neal Gompa]

On Wed, Dec 21, 2022 at 6:23 AM Vitaly Zaitsev via devel
<devel(a)lists.fedoraproject.org&gt; wrote:
>  On 20/12/2022 19:56, Chris Murphy wrote:
>  > Great. The gotcha though is this in effect requires a change in the file system
> currently mounted at /boot, which is ext4. And ext4 isn't supported by sd-boot or UEFI
> firmware. So if you're going to support sd-boot, the installer needs to be aware that
> either the ESP is big enough to be used as /boot, or if it's not big enough then it
> will be mounted on /efi*and*  a new partition XBOOTLDR formatted as FAT will be used as
> /boot.
>
>  Nobody should use FAT for /boot. efifs[1] should be used instead.
>
>  systemd-boot can load these drivers from ESP out of the box[2].
>
>  [1]: https://github.com/pbatard/efifs
>  [2]: https://github.com/systemd/systemd/issues/15617
>  
Indeed. We should not endeavor to create more problems by putting more
stuff in the ESP. By doing so, we lose features and capabilities from
our own native filesystem. No matter what boot manager we use, we
should not be required to give that up. We already have efifs[1]
packaged in Fedora, let's use that.

[1]: https://src.fedoraproject.org/rpms/efifs

-- 
真実はいつも一つ！/ Always, there's only one truth!

[Vitaly Zaitsev]

On 21/12/2022 12:27, Neal Gompa wrote:
>  We should not endeavor to create more problems by putting more
>  stuff in the ESP. 
These drivers must be located in firmware-readable location, i.e. ESP.

>  No matter what boot manager we use, we
>  should not be required to give that up. We already have efifs[1]
>  packaged in Fedora, let's use that. 
Yes, but the contents of this package must be signed with Fedora keys or 
the firmware will reject them if Secure Boot is enabled.

-- 
Sincerely,
   Vitaly Zaitsev (vitaly(a)easycoding.org)

[Lennart Poettering]

On Mi, 21.12.22 06:27, Neal Gompa (ngompa13(a)gmail.com) wrote:

>  On Wed, Dec 21, 2022 at 6:23 AM Vitaly Zaitsev via devel
>  <devel(a)lists.fedoraproject.org&gt; wrote:
>  >
>  > On 20/12/2022 19:56, Chris Murphy wrote:
>  > > Great. The gotcha though is this in effect requires a change in the file system
> currently mounted at /boot, which is ext4. And ext4 isn't supported by sd-boot or UEFI
> firmware. So if you're going to support sd-boot, the installer needs to be aware that
> either the ESP is big enough to be used as /boot, or if it's not big enough then it
> will be mounted on /efi*and*  a new partition XBOOTLDR formatted as FAT will be used as
> /boot.
>  >
>  > Nobody should use FAT for /boot. efifs[1] should be used instead.
>  >
>  > systemd-boot can load these drivers from ESP out of the box[2].
>  >
>  > [1]: https://github.com/pbatard/efifs
>  > [2]: https://github.com/systemd/systemd/issues/15617
>
>  Indeed. We should not endeavor to create more problems by putting more
>  stuff in the ESP. By doing so, we lose features and capabilities from
>  our own native filesystem. No matter what boot manager we use, we
>  should not be required to give that up. We already have efifs[1]
>  packaged in Fedora, let's use that. 
That's such a weird logic. /boot is supposed to contain data consumed
by the boot loader, not so much by the OS.

What do you think a boot laoder is going to do with access modes, file
ownership, selinux labels?

I am pretty sure we should reduce problems and simplify things by just
using VFAT for boot loader stuff, and put everything at the least
number of places possible.

Lennart

--
Lennart Poettering, Berlin

[Daniel P. Berrangé]

On Wed, Dec 21, 2022 at 12:22:25PM +0100, Vitaly Zaitsev via devel wrote:
>  On 20/12/2022 19:56, Chris Murphy wrote:
>  > Great. The gotcha though is this in effect requires a change in the file system
> currently mounted at /boot, which is ext4. And ext4 isn't supported by sd-boot or UEFI
> firmware. So if you're going to support sd-boot, the installer needs to be aware that
> either the ESP is big enough to be used as /boot, or if it's not big enough then it
> will be mounted on /efi*and*  a new partition XBOOTLDR formatted as FAT will be used as
> /boot.
>  
>  Nobody should use FAT for /boot. efifs[1] should be used instead. 
Why shouldn't FAT be used for /boot.  In an EFI world, /boot
is used for the same functional pupose as the ESP, which is
already going to use FAT. 

>  systemd-boot can load these drivers from ESP out of the box[2].
Yes, you can, but there needs to be a compelling benefit to
adding this extra functionality, as opposed to using FAT
for /boot. Such drivers would need to be signed to be used
under SecureBoot, thus expanding the set of components you
need to audit & trust for security purposes.

With regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

[Vitaly Zaitsev]

On 21/12/2022 12:38, Daniel P. Berrangé wrote:
>  Why shouldn't FAT be used for /boot.  In an EFI world, /boot
>  is used for the same functional pupose as the ESP, which is
>  already going to use FAT. 
Doesn't support links, lournaling and ACLs.

Everyone can do whatever they want with the files, and a trivial power 
outage can easily wipe out all of its contents.

>  Such drivers would need to be signed to be used
>  under SecureBoot, thus expanding the set of components you
>  need to audit & trust for security purposes. 
These drivers are backports from the grub2 code. If we trust GRUB, we 
can trust them too.

Fedora Infra can be configured to sign the contents of the efifs package 
with a Fedora SB key.

-- 
Sincerely,
   Vitaly Zaitsev (vitaly(a)easycoding.org)

[Gerd Hoffmann]

On Wed, Dec 21, 2022 at 12:53:05PM +0100, Vitaly Zaitsev via devel wrote:
>  On 21/12/2022 12:38, Daniel P. Berrangé wrote:
>  > Why shouldn't FAT be used for /boot.  In an EFI world, /boot
>  > is used for the same functional pupose as the ESP, which is
>  > already going to use FAT.
>  
>  Doesn't support links, lournaling and ACLs. 
Is that something you need in /boot?

And journaling actually is more a problem than a solution due to
firmware (or grub) filesystem drivers often not having full support for
the journal.  Luckily this is rarely a problem in practice because /boot
is rarely written to.

>  Everyone can do whatever they want with the files, 
Hmm?

From the booted system:  No, depends on the configuration and usually
it's mounted to be only writable by root.

When you plug the disk elsewhere:  Yes, but all other filesystems have
the same problem then.  With physical access to the disk only encryption
can prevent that.

take care,
  Gerd

[Neal Gompa]

On Wed, Dec 21, 2022 at 7:17 AM Gerd Hoffmann <kraxel(a)redhat.com&gt; wrote:
>  On Wed, Dec 21, 2022 at 12:53:05PM +0100, Vitaly Zaitsev via devel wrote:
>  > On 21/12/2022 12:38, Daniel P. Berrangé wrote:
>  > > Why shouldn't FAT be used for /boot.  In an EFI world, /boot
>  > > is used for the same functional pupose as the ESP, which is
>  > > already going to use FAT.
>  >
>  > Doesn't support links, lournaling and ACLs.
>
>  Is that something you need in /boot?
>  
Yes, because content is directly installed there.

>  And journaling actually is more a problem than a solution due to
>  firmware (or grub) filesystem drivers often not having full support for
>  the journal.  Luckily this is rarely a problem in practice because /boot
>  is rarely written to.
>  
We could just make those read-only from the bootloader side then. I
have /boot on btrfs, which grub/efifs doesn't support writing to at
all anyway. Write grubenv settings into the BIOS boot partition or ESP
rather than /boot.




--
真実はいつも一つ！/ Always, there's only one truth!

[Lennart Poettering]

On Mi, 21.12.22 07:40, Neal Gompa (ngompa13(a)gmail.com) wrote:

>  > And journaling actually is more a problem than a solution due
> to
>  > firmware (or grub) filesystem drivers often not having full support for
>  > the journal.  Luckily this is rarely a problem in practice because /boot
>  > is rarely written to.
>
>  We could just make those read-only from the bootloader side then. I
>  have /boot on btrfs, which grub/efifs doesn't support writing to at
>  all anyway. Write grubenv settings into the BIOS boot partition or ESP
>  rather than /boot. 
The ESP is more important for booting than /boot. It's a bit weird you
are fun with writes to the former, but have issues with the latter.

Generally: boot counting/assement/fallback data is best attached to
the resources it's supposed to cover, to make rotating/vacuuming of
that data obvious and robust. Otherwise if you remove a boot entry you
need to find the matching data at a completely different place and
remove that too, which is quite fragile.

sd-boot implements boot counting by stashing the counter inside the
UKI (or bootspec type #1 conf file) file name, so that the counter is
impossible to ever get lost, pile up or so.

Lennart

--
Lennart Poettering, Berlin

[Neal Gompa]

On Wed, Dec 21, 2022 at 12:33 PM Lennart Poettering
<mzerqung(a)0pointer.de&gt; wrote:
>  On Mi, 21.12.22 07:40, Neal Gompa (ngompa13(a)gmail.com) wrote:
>
>  > > And journaling actually is more a problem than a solution due to
>  > > firmware (or grub) filesystem drivers often not having full support for
>  > > the journal.  Luckily this is rarely a problem in practice because /boot
>  > > is rarely written to.
>  >
>  > We could just make those read-only from the bootloader side then. I
>  > have /boot on btrfs, which grub/efifs doesn't support writing to at
>  > all anyway. Write grubenv settings into the BIOS boot partition or ESP
>  > rather than /boot.
>
>  The ESP is more important for booting than /boot. It's a bit weird you
>  are fun with writes to the former, but have issues with the latter.
>
>  Generally: boot counting/assement/fallback data is best attached to
>  the resources it's supposed to cover, to make rotating/vacuuming of
>  that data obvious and robust. Otherwise if you remove a boot entry you
>  need to find the matching data at a completely different place and
>  remove that too, which is quite fragile.
>
>  sd-boot implements boot counting by stashing the counter inside the
>  UKI (or bootspec type #1 conf file) file name, so that the counter is
>  impossible to ever get lost, pile up or so.
>  
Because the ESP is intended to be shared, and these days /boot is not.
Unshared boot content should not be in a shared space.

We can only really have one boot manager, especially with how broken
multiple ESPs on a system are.


-- 
真実はいつも一つ！/ Always, there's only one truth!

[Lennart Poettering]

On Mi, 21.12.22 12:38, Neal Gompa (ngompa13(a)gmail.com) wrote:

>  > sd-boot implements boot counting by stashing the counter inside
> the
>  > UKI (or bootspec type #1 conf file) file name, so that the counter is
>  > impossible to ever get lost, pile up or so.
>
>  Because the ESP is intended to be shared, and these days /boot is not.
>  Unshared boot content should not be in a shared space. 
This is wrong on two levels:

1. The boot loader should process all UKIs from all installed OSes,
   hence the UKIs should be dropped in in a *shared* directory, not a
   private directory. It's the idea of the boot loader spec to make
   the drop-in dir shared between Linux OSes.

2. The UEFI spec defines clearly where OSes should put private stuff
   in the ESP if they have any (see https://uefi.org/registry for
   example, fedora is listed there btw). The ESP is very clearly
   defined both in terms for shared resources and for private
   resources.

I am not sure what you intend to put in /boot? If it's UKIs then
please notice that these are a single file per kernel/initrd combo,
and that is inherently placed in a shared dir, you need nothing else
really.

>  We can only really have one boot manager, especially with how broken
>  multiple ESPs on a system are. 
Why would you have multiple ESPs? I don't follow?

Lennart

--
Lennart Poettering, Berlin

[Vitaly Zaitsev]

On 21/12/2022 13:17, Gerd Hoffmann wrote:
>  Is that something you need in /boot? 
Yes.

>  And journaling actually is more a problem than a solution due to
>  firmware (or grub) filesystem drivers often not having full support for
>  the journal.  Luckily this is rarely a problem in practice because /boot
>  is rarely written to. 
The journal is very useful when writing kernels there.

>  No, depends on the configuration and usually
>  it's mounted to be only writable by root. 
The big hole is covered with a carpet.

-- 
Sincerely,
   Vitaly Zaitsev (vitaly(a)easycoding.org)

[Lennart Poettering]

On Mi, 21.12.22 14:00, Fedora Development ML (devel(a)lists.fedoraproject.org) wrote:

>  > And journaling actually is more a problem than a solution due
> to
>  > firmware (or grub) filesystem drivers often not having full support for
>  > the journal.  Luckily this is rarely a problem in practice because /boot
>  > is rarely written to.
>
>  The journal is very useful when writing kernels there. 
how so? iirc grub's fs drivers can't even replay the journal?

if you drop single files into vfat by creating a file under a random
name, then writing the file linearly in full, then syncing, and then
renaming it to the final name (and syncing again) you should get
equal/better guarantees from the fs, in a way actually compatible with
the pre-boot env.

Lennart

--
Lennart Poettering, Berlin

[Daniel P. Berrangé]

On Wed, Dec 21, 2022 at 12:53:05PM +0100, Vitaly Zaitsev via devel wrote:
>  On 21/12/2022 12:38, Daniel P. Berrangé wrote:
>  > Why shouldn't FAT be used for /boot.  In an EFI world, /boot
>  > is used for the same functional pupose as the ESP, which is
>  > already going to use FAT.
>  
>  Doesn't support links, lournaling and ACLs.
>  
>  Everyone can do whatever they want with the files, and a trivial power
>  outage can easily wipe out all of its contents.
>  
>  > Such drivers would need to be signed to be used
>  > under SecureBoot, thus expanding the set of components you
>  > need to audit & trust for security purposes.
>  
>  These drivers are backports from the grub2 code. If we trust GRUB, we can
>  trust them too. 
Historically we decided to trust Grub because we had little other
practical choice. We shouldn't neccessarily need to continue that,
as it would be beneficial to have less complex code needing to be
trusted in the boot path.

With regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

[Lennart Poettering]

On Mi, 21.12.22 12:53, Fedora Development ML (devel(a)lists.fedoraproject.org) wrote:

>  On 21/12/2022 12:38, Daniel P. Berrangé wrote:
>  > Why shouldn't FAT be used for /boot.  In an EFI world, /boot
>  > is used for the same functional pupose as the ESP, which is
>  > already going to use FAT.
>
>  Doesn't support links, lournaling and ACLs. 
What you want in a boot loader: native read access, write access.

What UEFI doesn't have any understanding of and just ignores: symlinks, ACLs.

What grub's fs drivers doesn't even implement: journalling, write access.

hence, ext4 via efis might be OK as a compat solution, but it delivers
only stuff one doesn't want, and lacks everything one does want.

>  Everyone can do whatever they want with the files, and a trivial
> power
>  outage can easily wipe out all of its contents. 
As mentioned elsewhere, write access to the ESP are generally done at
a very limited set of operations:

1. new kernel installed/old one removed
2. boot loader updates
3. random seed refreshes
4. boot counter updates

All of these are extremely simple operations, that either are single
sector writes, or implemented via whole file replacement, which
typically can be implemented robustly even on vfat. i.e there's no
long-running access, no random access, nothing like that.

systemd's autofs logic for the ESP (and the write patterns bootctl
implements) are designed to give you data safety for the ESP to the
level this could be possible. Sure, it's still no journalling, but
it's as close as it can get.

--
Lennart Poettering, Berlin

[Neal Gompa]

On Wed, Dec 21, 2022 at 12:15 PM Lennart Poettering
<mzerqung(a)0pointer.de&gt; wrote:
>  On Mi, 21.12.22 12:53, Fedora Development ML (devel(a)lists.fedoraproject.org) wrote:
>
>  > On 21/12/2022 12:38, Daniel P. Berrangé wrote:
>  > > Why shouldn't FAT be used for /boot.  In an EFI world, /boot
>  > > is used for the same functional pupose as the ESP, which is
>  > > already going to use FAT.
>  >
>  > Doesn't support links, lournaling and ACLs.
>
>  What you want in a boot loader: native read access, write access.
>  
Actually no, I don't want the boot loader / boot manager writing anything.

>  What UEFI doesn't have any understanding of and just ignores:
> symlinks, ACLs.
>
>  What grub's fs drivers doesn't even implement: journalling, write access.
>
>  hence, ext4 via efis might be OK as a compat solution, but it delivers
>  only stuff one doesn't want, and lacks everything one does want.
>  
Sounds great! It means write operations only happen in the OS and
nowhere else. That's what I want.


-- 
真実はいつも一つ！/ Always, there's only one truth!

[Lennart Poettering]

On Mi, 21.12.22 12:21, Neal Gompa (ngompa13(a)gmail.com) wrote:

>  > > > Why shouldn't FAT be used for /boot.  In an EFI
> world, /boot
>  > > > is used for the same functional pupose as the ESP, which is
>  > > > already going to use FAT.
>  > >
>  > > Doesn't support links, lournaling and ACLs.
>  >
>  > What you want in a boot loader: native read access, write access.
>
>  Actually no, I don't want the boot loader / boot manager writing
>  anything. 
Well, good for you.

But I think it would be wise for Fedora to implement automatic
fallback for hosed kernels, and that requires counting boot attempts,
and that requires storing the counters somewhere. And that means we
need to store something somewhere. Hence write access from pre-boot is
typically desirable. Basically, there are too places a boot loader can
write stuff: file system and NVRAM. The latter is problematic to write to
since cheap hw supposedly doesn't allow too many write cycles before
breaking. Hence file system it must be.

If Fedora every intends to be useful for people who cannot recover a
hosed system on their own because they are Linux guru themselves, I am
pretty sure we want automatic boot assessment/fallback logic in place.

And similar for server/embedded stuff. If fedora wants to be deployed
in such worlds, it's kinda nice if we can automatically recover from
hosed updates.

I am sure Neal Gompa can recover his own machines, but not every
Fedora system comes with a Neal Gompa deployment included.

Lennart

--
Lennart Poettering, Berlin

[Neal Gompa]

On Wed, Dec 21, 2022 at 12:29 PM Lennart Poettering
<mzerqung(a)0pointer.de&gt; wrote:
>  On Mi, 21.12.22 12:21, Neal Gompa (ngompa13(a)gmail.com) wrote:
>
>  > > > > Why shouldn't FAT be used for /boot.  In an EFI world, /boot
>  > > > > is used for the same functional pupose as the ESP, which is
>  > > > > already going to use FAT.
>  > > >
>  > > > Doesn't support links, lournaling and ACLs.
>  > >
>  > > What you want in a boot loader: native read access, write access.
>  >
>  > Actually no, I don't want the boot loader / boot manager writing
>  > anything.
>
>  Well, good for you.
>
>  But I think it would be wise for Fedora to implement automatic
>  fallback for hosed kernels, and that requires counting boot attempts,
>  and that requires storing the counters somewhere. And that means we
>  need to store something somewhere. Hence write access from pre-boot is
>  typically desirable. Basically, there are too places a boot loader can
>  write stuff: file system and NVRAM. The latter is problematic to write to
>  since cheap hw supposedly doesn't allow too many write cycles before
>  breaking. Hence file system it must be.
>
>  If Fedora every intends to be useful for people who cannot recover a
>  hosed system on their own because they are Linux guru themselves, I am
>  pretty sure we want automatic boot assessment/fallback logic in place.
>
>  And similar for server/embedded stuff. If fedora wants to be deployed
>  in such worlds, it's kinda nice if we can automatically recover from
>  hosed updates.
>  
None of those things require us to write data to /boot. Even in your
model, if you *must* write to a filesystem, the counters can live on
the ESP even if all the system-installed content exists in /boot. I'm
sure you could envision a simple file in the ESP for that. None of
that is permanent configuration, just transient stuff.

>  I am sure Neal Gompa can recover his own machines, but not every
>  Fedora system comes with a Neal Gompa deployment included.
>  
Okay, if you're going to be a jerk about it, no Fedora system comes
with a helpful version of Lennart to work out problems people
encounter with this stuff.


-- 
真実はいつも一つ！/ Always, there's only one truth!

[Lennart Poettering]

On Mi, 21.12.22 12:35, Neal Gompa (ngompa13(a)gmail.com) wrote:

>  > And similar for server/embedded stuff. If fedora wants to be
> deployed
>  > in such worlds, it's kinda nice if we can automatically recover from
>  > hosed updates.
>
>  None of those things require us to write data to /boot. Even in your
>  model, if you *must* write to a filesystem, the counters can live on
>  the ESP even if all the system-installed content exists in /boot. I'm
>  sure you could envision a simple file in the ESP for that. None of
>  that is permanent configuration, just transient stuff. 
I don't follow your thinking at all. On one hand you want /boot/ to be
ext4, supposedly for data safety reasons. But you don't want writes
from pre-boot environment to go there. You are fine if pre-boot writes
to ESP (i.e. VFAT) however for boot counting.

So, ESP is more important for booting than /boot/ (simply because a
hosed kernel doesn't matter, if you have another — a hosed boot loader
is much more problematic however since you typically have no other),
hence if anything you should be more concerned about writes there than
on /boot.

If you accept that writes to the ESP/VFAT are actually OK, then I
think it's just a minor step to say that /boot/ as VFAT is also OK
given these writes are more seldom, are done from the safer OS
environment, and can be tightly controlled.

Lennart

--
Lennart Poettering, Berlin

[Chris Murphy]

On Wed, Dec 21, 2022, at 6:53 AM, Vitaly Zaitsev via devel wrote:
>  On 21/12/2022 12:38, Daniel P. Berrangé wrote:
> > Why shouldn't FAT be used for /boot.  In an EFI world, /boot
> > is used for the same functional pupose as the ESP, which is
> > already going to use FAT.
>
>  Doesn't support links, lournaling and ACLs. 
What's the use case? Is it a nice to have or is it a hard requirement and why?

The Linux FAT driver does support SELinux context with a mount option. We aren't using
that right now but we can have a suitable SELinux label enforced file system wide on ESP
and XBOOTLDR.


>  Everyone can do whatever they want with the files, and a trivial
> power 
>  outage can easily wipe out all of its contents. 
This is a rare but real problem. I think we should be looking at ESP and XBOOTLDR updates
doing A/B updates, i.e. write the updates to temporary files or directories and then use
renameat(2) which is atomic at the VFS layer, and should get pretty close to atomic at the
FAT layer to the degree that worse case scenario we have either the old *and* new files
following a crash. Ideally we'd get old *or* new. But that's probably not possible
with FAT. But we can still boot. 


> > Such drivers would need to be signed to be used
> > under SecureBoot, thus expanding the set of components you
> > need to audit & trust for security purposes.
>
>  These drivers are backports from the grub2 code. If we trust GRUB, we 
>  can trust them too.
>
>  Fedora Infra can be configured to sign the contents of the efifs package 
>  with a Fedora SB key. 
Yeah but then try getting all the distros to agree on signing efifs. How many distros are
there? It's not unfair to say  all distros should be able to put their signed efifs
drivers on the ESP because that's the only way their bootloader can read
loader/entries to properly draw a boot menu.


-- 
Chris Murphy

[Vitaly Zaitsev]

On 22/12/2022 21:29, Chris Murphy wrote:
>  This is a rare but real problem. 
I don't think so. Power outage is a very common problem in some countries.

I still remember how unreliable FAT32 was in the Windows 9x era. You 
needed to run a scandisk check after every power failure or pressing the 
reset button. And sometimes your documents or other files disappeared. I 
really don't want a repeat of this.

-- 
Sincerely,
   Vitaly Zaitsev (vitaly(a)easycoding.org)

[Luca Boccassi]

>  On 22/12/2022 21:29, Chris Murphy wrote:
>  
>  I don't think so. Power outage is a very common problem in some countries.
>  
>  I still remember how unreliable FAT32 was in the Windows 9x era. You 
>  needed to run a scandisk check after every power failure or pressing the 
>  reset button. And sometimes your documents or other files disappeared. I 
>  really don't want a repeat of this. 
As mentioned many times already, vfat here is not used to keep files open for continuous
editing or things like that, where that experience might be repeated. It's
single-block or as-atomic-as-it-can-be single-file swap (and with newer kernels it looks
like there's actual atomic rename too). So "files disappearing" due to power
outages are extremely unlikely in this particular use case. The worst case scenario is
that you end up with both old-and-new UKIs, which means you can still boot (and
there's no "valuable" data to be lost here, everything that goes in the ESP
can be regenerated on the next boot). On the other hand reimplementing filesystem drivers
in the bootloader can result in broken filesystems or worse, security vulnerabilities.
This is something that actually happens and keeps happening.

[Lennart Poettering]

On Fr, 23.12.22 08:51, Fedora Development ML (devel(a)lists.fedoraproject.org) wrote:

>  On 22/12/2022 21:29, Chris Murphy wrote:
>  > This is a rare but real problem.
>
>  I don't think so. Power outage is a very common problem in some countries.
>
>  I still remember how unreliable FAT32 was in the Windows 9x era. You needed
>  to run a scandisk check after every power failure or pressing the reset
>  button. And sometimes your documents or other files disappeared. I really
>  don't want a repeat of this. 
Nobody is proposing to run the full Linux OS from FAT. I mean, yeah,
in /var and /home people usually do complex write patterns, and the
files there basically are pinned all the time thus cannot be unmounted
during runtime to ensure the file system stays clean most of the time.

But this is different for XBOOTLDR: the autofs logic in systemd
ensures it remains unmounted almost all of the time, and is only very
briefly mounted whenever something actually accesses it. And the write
patterns on XBOOTLDR are comparatively simple: drop in whole file,
erase whole file, plus single-sector writes for some corner cases.

This is *massively* different from the write patterns Windows 9x did
on its file systems.

Lennart

--
Lennart Poettering, Berlin

[Lennart Poettering]

On Mi, 21.12.22 12:22, Fedora Development ML (devel(a)lists.fedoraproject.org) wrote:

>  On 20/12/2022 19:56, Chris Murphy wrote: > Great. The gotcha
> though
>  is this in effect requires a change in the file system currently
>  mounted at /boot, which is ext4. And ext4 isn't supported by sd-boot
>  or UEFI firmware. So if you're going to support sd-boot, the
>  installer needs to be aware that either the ESP is big enough to be
>  used as /boot, or if it's not big enough then it will be mounted on
>  /efi*and* a new partition XBOOTLDR formatted as FAT will be used as
>  /boot.
>
>  Nobody should use FAT for /boot. efifs[1] should be used instead. 
I vehemently disagree. VFAT is the way to go. It's simply a pointless
excercise to use anything else.

https://0pointer.net/blog/linux-boot-partitions.html

I think it might be OK to stick to ext4 for upgrades, but outside of
that, for new installs it's something that really should be avoided I
am sure.

Lennart

--
Lennart Poettering, Berlin

[Chris Murphy]

On Wed, Dec 21, 2022, at 6:22 AM, Vitaly Zaitsev via devel wrote:
>  On 20/12/2022 19:56, Chris Murphy wrote:
> > Great. The gotcha though is this in effect requires a change in the file system
> currently mounted at /boot, which is ext4. And ext4 isn't supported by sd-boot or UEFI
> firmware. So if you're going to support sd-boot, the installer needs to be aware that
> either the ESP is big enough to be used as /boot, or if it's not big enough then it
> will be mounted on /efi*and*  a new partition XBOOTLDR formatted as FAT will be used as
> /boot.
>
>  Nobody should use FAT for /boot. efifs[1] should be used instead.
>
>  systemd-boot can load these drivers from ESP out of the box[2]. 
The founding principle in Boot Loader Spec is that multiboot between Linux distros sucks.
The cooperation between distros, is shit. And BLS strives to present an opportunity to
compromise and fix that problem.

It's harder to fix this problem if XBOOTLDR is not FAT. efifs drivers need to be
Secure Boot signed just like the bootloader. The firmware already trusts its built-in FAT
driver, for better or worse, so what is the exact problem with just using that so we
don't have to deal with UEFI SB signing efifs drivers, and the much harder job of
expecting every distro to include signed efifs drivers *on the ESP* for multiboot to work?


If /boot is ext4, then every Linux distro must include a signed ext4 efifs driver in order
to properly render the boot menu. But what if (open)SUSE doesn't want to use ext4,
they want Btrfs? Compromise dictates that every distro now needs to provide a signed btrfs
efifs driver too. OK Red Hat uses XFS for boot, so now every distro needs to include ext4,
btrfs, and XFS signed efifs drivers with every installation. It's explosively more
complicated to implement let alone to agree upon than just use the one driver we know
everyone has and can use.

XBOOTLDR in practice needs to be FAT. I don't like it. But I like it better than
choosing batshit as the alternative, and having a bunch of signed efifs drivers on the ESP
per distro sounds like batshit to me. And not in the good way.


-- 
Chris Murphy

[Vitaly Zaitsev]

On 22/12/2022 21:18, Chris Murphy wrote:
>  XBOOTLDR in practice needs to be FAT. I don't like it. But I like
> it better than choosing batshit as the alternative, and having a bunch of signed efifs
> drivers on the ESP per distro sounds like batshit to me. And not in the good way. 
I don't think so. XBOOTLDR on FAT32 should be rejected as a defective by 
design due to a FAT32 unreliability.

>  It's harder to fix this problem if XBOOTLDR is not FAT. efifs
> drivers need to be Secure Boot signed just like the bootloader. The firmware already
> trusts its built-in FAT driver, for better or worse, so what is the exact problem with
> just using that so we don't have to deal with UEFI SB signing efifs drivers, and the
> much harder job of expecting every distro to include signed efifs drivers *on the ESP* for
> multiboot to work?  
Who we are to make decisions for other Linux distributions? Every 
distribution can use whatever they want.

I doubt that Fedora's shim+grub2 can boot Ubuntu kernels in Secure Boot 
mode and vice versa.

-- 
Sincerely,
   Vitaly Zaitsev (vitaly(a)easycoding.org)

[Lennart Poettering]

On Fr, 23.12.22 09:01, Fedora Development ML (devel(a)lists.fedoraproject.org) wrote:

>  On 22/12/2022 21:18, Chris Murphy wrote: 
>  > XBOOTLDR in practice needs to be FAT. I don't like it. But I
> like
>  > it better than choosing batshit as the alternative, and having a
>  > bunch of signed efifs drivers on the ESP per distro sounds like
>  > batshit to me. And not in the good way.
>
>  I don't think so. XBOOTLDR on FAT32 should be rejected as a defective by
>  design due to a FAT32 unreliability. 
It's not the best file system if you intend to do random access writes
all the time. But if you don't do that, restrict your write patterns
to a certain reasonably safe subset, and ensure that you keep the file
system unmounted most of the time then it should be OK. I mean, UEFI
effectively mandates FAT for one partition (i.e. the ESP), you can't
avoid it. And at the bare minimum the boot loader is stored in the
ESP, and you need to update that as regularly as any other software
package, hence it's illusionary that you could avoid regular write
patterns onto FAT if you just make XBOOTLDR something non-FAT.

>  I doubt that Fedora's shim+grub2 can boot Ubuntu kernels in
> Secure Boot mode
>  and vice versa. 
After enrolling the Ubuntu key via mokutil that should be fine. Sure,
if you have the shim belonging to distro X then this means only
kernels of distro X can be just booted, since only X' certificate will
be built-in. But once you enroll other certs things should be fine.

Lennart

--
Lennart Poettering, Berlin

[Gerd Hoffmann]

On Fri, Dec 23, 2022 at 09:01:52AM +0100, Vitaly Zaitsev via devel wrote:

>  I doubt that Fedora's shim+grub2 can boot Ubuntu kernels in
> Secure Boot mode
>  and vice versa. 
Needs installing the signing keys to the shim key database using
mokutil, but should otherwise work just fine.

take care,
  Gerd

[Lennart Poettering]

On Di, 20.12.22 13:56, Chris Murphy (lists(a)colorremedies.com) wrote:

>  > * Better secure boot support (specifically the initrd is covered
> by
>  > the signature).
>
>  We need to solve the glaring hole that is the initrd. There's no
>  question about it. I can't really assess if this feature is the best
>  way to do that. Or if it would be adequate for dracut to self-sign
>  every locally generated initrd with a unique key pair and throw away
>  the private key after each initrd is generated.  Or if we could do
>  enough strict standardization in the boot chain with a possibly
>  larger kernel to avoid needing an initrd, i.e. get to sysroot mount
>  faster thereby obviating the need for a large initrd. 
Systems without initrd are unrealistic outside of corner cases. I am
pretty sure that if you care about SecureBoot then you must care about
protecting the root fs somehow, too. Otherwise fixing the initrd hole
is a pretty pointless excercise. Protecting the root fs means
encryption/LUKS, Verity or dm-integrity in some way. But that implies
an initrd, in particular if you want to hook that up with TPM or FIDO2
or so, which I am pretty sure should be considered a pretty common
case sooner or later.

I think initrd-less systems only make sense as a corner case for
certain low-security systems, but certainly not as a default.

Lennart

--
Lennart Poettering, Berlin

[Björn Persson]

>  Main motivation for this move is to make the distro more robust and
> more secure. 
Improving security would be great, but it must be done in a way that
allows the sysadmin to configure and repair the system and authorize
the new configuration.

>  Switching the whole distro over to unified kernels quickly is not
>  realistic though.  Too many features are depending on the current
>  workflow with a host-specific initrd (and host-specific kernel command
>  line), which is fundamentally incompatible with unified kernels where
>  everybody will have the same initrd and command line. Thats why there
>  is 'Phase 1' in title, so we can have more Phases in future releases
>  😃 
Whew! So usable kernels won't go away in F38. I only have to worry
about being forced to build my own kernels in some unspecified future
phase. Doom is still coming but no one knows when. That's *such* a
relief.

>  A host-specific initrd / command line is needed today for:
[...]
>  * configuration being specified on the kernel command line.
>  ** root filesystem being the most important one.
>  [https://systemd.io/DISCOVERABLE_PARTITIONS/ Discoverable partitions]
>  allow to remove this. 
Why link to a page that only contains a link to another page? Why not
link directly to
https://uapi-group.org/specifications/specs/discoverable_partitions_speci...

That page makes it clear that omitting root= from the kernel command
line is only an *option* which is proposed only "for simpler,
appliance-like installations". My workstation and my laptop aren't
appliance-like in the slightest. And on appliances you want a stable,
reliable operating system, not a fast-moving, unstable one like Fedora.

** Troubleshooting being the second most important one. When the system
won't boot, it's necessary to remove "quiet" and "rhgb" from
the kernel
command line to see how far the boot process gets and what error
messages are printed. It may also be necessary to configure a serial
console for example.

The root filesystem is also relevant for troubleshooting, when I take a
storage device out of a broken computer and connect it to another
computer to inspect it. Suddenly there are two "discoverable" root
partitions, and the kernel parameter is necessary to specify which one
is the root filesystem, as stated under "Doesn’t this break multi-boot
scenarios?":
https://uapi-group.org/specifications/specs/discoverable_partitions_speci...

>  Phase 2/3 goals (longer-term stuff which is not realistic to complete
> for F38).
>  
>  * Move away from using the kernel command line for configuration. 
I note that taking away the kernel command line is indeed a clearly
stated goal, which will limit Fedora to simple, appliance-like uses.

If any of what I wrote above misrepresents the change owner's
intentions, then the change proposal is badly written and needs
reworking to communicate the true intentions.

Björn Persson

[Simo Sorce]

On Tue, 2022-12-20 at 20:42 +0100, Björn Persson wrote:
>  I note that taking away the kernel command line is indeed a clearly
>  stated goal, which will limit Fedora to simple, appliance-like uses. 
I for one, haven't touched once the command line in this laptop that
has 4 years. So I welcome simplification for that *common* case.

Given nobody is taking away the initrd way all you will have to do (at
most, if you use it) is to disable secure boot and regain the ability
to change the kernel command line and build your own initrd or even
your own kernel if you so like.

And if you chose your HW carefully you may even be able to register
your own public keys, generate and sign your own built UKIs and re-
enable SecureBoot after that... your choice!

Simo.

-- 
Simo Sorce
RHEL Crypto Team
Red Hat, Inc

[Neal Gompa]

On Tue, Dec 20, 2022, 4:31 PM Simo Sorce <simo(a)redhat.com&gt; wrote:

>  On Tue, 2022-12-20 at 20:42 +0100, Björn Persson wrote:
>  > I note that taking away the kernel command line is indeed a clearly
>  > stated goal, which will limit Fedora to simple, appliance-like uses.
>
>  I for one, haven't touched once the command line in this laptop that
>  has 4 years. So I welcome simplification for that *common* case.
>  

Every person who chooses basic graphics mode is triggering this case. The
kernel command line is pretty much the first stop for troubleshooting and
bypassing driver problems. It's incredibly common even now, which is why we
keep hitting bugs and fixing them each release. We are finally starting to
automate testing of these modes, so I do not appreciate you saying they are
uncommon, because the reality is they are.



>  Given nobody is taking away the initrd way all you will have to do (at
>  most, if you use it) is to disable secure boot and regain the ability
>  to change the kernel command line and build your own initrd or even
>  your own kernel if you so like.
>  
This is becoming harder and harder to do. Consumer hardware has not been
required to offer that ability in years and I have friends and colleagues
who have such hardware today. This is not a realistic expectation to have.
And while most still have the capacity to disable it, it is difficult and
something we cannot expect users to be able to accomplish.


And if you chose your HW carefully you may even be able to register
>  your own public keys, generate and sign your own built UKIs and re-
>  enable SecureBoot after that... your choice!
>  
That doesn't mean we should offer any in Fedora itself unless we can be
assured someone cares about how regular people are supposed to deal with
problems caused by this.

>

[Gerd Hoffmann]

On Tue, Dec 20, 2022 at 04:31:20PM -0500, Simo Sorce wrote:
>  On Tue, 2022-12-20 at 20:42 +0100, Björn Persson wrote:
>  > I note that taking away the kernel command line is indeed a clearly
>  > stated goal, which will limit Fedora to simple, appliance-like uses.
>  
>  And if you chose your HW carefully you may even be able to register
>  your own public keys, generate and sign your own built UKIs and re-
>  enable SecureBoot after that... your choice! 
And when your hardware doesn't allow that you can still add your own
keys with mokutil so shim.efi will accept your self-signed UKIs.

take care,
  Gerd

[Björn Persson]

Gerd Hoffmann wrote:
>  On Tue, Dec 20, 2022 at 04:31:20PM -0500, Simo Sorce wrote:
>  > And if you chose your HW carefully you may even be able to register
>  > your own public keys, generate and sign your own built UKIs and re-
>  > enable SecureBoot after that... your choice!  
>  
>  And when your hardware doesn't allow that you can still add your own
>  keys with mokutil so shim.efi will accept your self-signed UKIs. 
I'll see when I can take the time for a research project to figure out
how customized UKIs can be produced, and develop a tool to do that.
Probably never, because I have way too much to do already.

Apparently there is no such tool and no plan to provide one, because
surely that would have been mentioned under "User Experience".

Björn Persson

[Zbigniew Jędrzejewski-Szmek]

On Wed, Dec 21, 2022 at 06:56:05PM +0100, Björn Persson wrote:
>  Gerd Hoffmann wrote:
>  > On Tue, Dec 20, 2022 at 04:31:20PM -0500, Simo Sorce wrote:
>  > > And if you chose your HW carefully you may even be able to register
>  > > your own public keys, generate and sign your own built UKIs and re-
>  > > enable SecureBoot after that... your choice!  
>  > 
>  > And when your hardware doesn't allow that you can still add your own
>  > keys with mokutil so shim.efi will accept your self-signed UKIs.
>  
>  I'll see when I can take the time for a research project to figure out
>  how customized UKIs can be produced, and develop a tool to do that.
>  Probably never, because I have way too much to do already.
>  
>  Apparently there is no such tool and no plan to provide one, because
>  surely that would have been mentioned under "User Experience". 
The tools are already there. Essentially, any tool used in koji by the official
builds is also available to you on your machine.

There are at three ways that are used: 'dracut --uefi', systemd's ukify, and
a
manual objcopy invocation. The first two are wrappers around objcopy. I'm biased
because I wrote 'ukify', but I think that's the way to go. What dracut does
is
somewhat primitive and doesn't get the offsets right. Obviously it could be
improved, but putting the code to generate UKIs inside of an initrd generator is
a historical accident, and bash is not the nicest language to do offset
calculations. Thus I hope we can standarize on ukify instead.

In particular, if some functionality is missing from ukify, feel free to submit
a PR or an issue. It's in Python, so fairly nice to modify. So far it's been
written to take a kernel and an initrd and the stub, and produce an efi binary.
It could be extended to start with an existing efi binary and e.g. a new initrd
or a different commandline, but so far nobody has requested that.

Zbyszek

[Neal Gompa]

On Thu, Dec 22, 2022 at 5:58 AM Zbigniew Jędrzejewski-Szmek
<zbyszek(a)in.waw.pl&gt; wrote:
>  On Wed, Dec 21, 2022 at 06:56:05PM +0100, Björn Persson wrote:
>  > Gerd Hoffmann wrote:
>  > > On Tue, Dec 20, 2022 at 04:31:20PM -0500, Simo Sorce wrote:
>  > > > And if you chose your HW carefully you may even be able to register
>  > > > your own public keys, generate and sign your own built UKIs and re-
>  > > > enable SecureBoot after that... your choice!
>  > >
>  > > And when your hardware doesn't allow that you can still add your own
>  > > keys with mokutil so shim.efi will accept your self-signed UKIs.
>  >
>  > I'll see when I can take the time for a research project to figure out
>  > how customized UKIs can be produced, and develop a tool to do that.
>  > Probably never, because I have way too much to do already.
>  >
>  > Apparently there is no such tool and no plan to provide one, because
>  > surely that would have been mentioned under "User Experience".
>
>  The tools are already there. Essentially, any tool used in koji by the official
>  builds is also available to you on your machine.
>
>  There are at three ways that are used: 'dracut --uefi', systemd's ukify, and
> a
>  manual objcopy invocation. The first two are wrappers around objcopy. I'm biased
>  because I wrote 'ukify', but I think that's the way to go. What dracut does
> is
>  somewhat primitive and doesn't get the offsets right. Obviously it could be
>  improved, but putting the code to generate UKIs inside of an initrd generator is
>  a historical accident, and bash is not the nicest language to do offset
>  calculations. Thus I hope we can standarize on ukify instead.
>
>  In particular, if some functionality is missing from ukify, feel free to submit
>  a PR or an issue. It's in Python, so fairly nice to modify. So far it's been
>  written to take a kernel and an initrd and the stub, and produce an efi binary.
>  It could be extended to start with an existing efi binary and e.g. a new initrd
>  or a different commandline, but so far nobody has requested that.
>  
It would definitely make sense to add some documentation about
preferred tools, since other distributions will eventually reference
our Change document to do UKIs on their own too.



-- 
真実はいつも一つ！/ Always, there's only one truth!

[Daniel P. Berrangé]

On Thu, Dec 22, 2022 at 10:58:11AM +0000, Zbigniew Jędrzejewski-Szmek wrote:
>  On Wed, Dec 21, 2022 at 06:56:05PM +0100, Björn Persson wrote:
>  > Gerd Hoffmann wrote:
>  > > On Tue, Dec 20, 2022 at 04:31:20PM -0500, Simo Sorce wrote:
>  > > > And if you chose your HW carefully you may even be able to register
>  > > > your own public keys, generate and sign your own built UKIs and re-
>  > > > enable SecureBoot after that... your choice!  
>  > > 
>  > > And when your hardware doesn't allow that you can still add your own
>  > > keys with mokutil so shim.efi will accept your self-signed UKIs.
>  > 
>  > I'll see when I can take the time for a research project to figure out
>  > how customized UKIs can be produced, and develop a tool to do that.
>  > Probably never, because I have way too much to do already.
>  > 
>  > Apparently there is no such tool and no plan to provide one, because
>  > surely that would have been mentioned under "User Experience".
>  
>  The tools are already there. Essentially, any tool used in koji by the official
>  builds is also available to you on your machine.
>  
>  There are at three ways that are used: 'dracut --uefi', systemd's ukify, and
> a
>  manual objcopy invocation. The first two are wrappers around objcopy. I'm biased
>  because I wrote 'ukify', but I think that's the way to go. What dracut does
> is
>  somewhat primitive and doesn't get the offsets right. Obviously it could be
>  improved, but putting the code to generate UKIs inside of an initrd generator is
>  a historical accident, and bash is not the nicest language to do offset
>  calculations. Thus I hope we can standarize on ukify instead. 
When you say it dooesn't get the offsets right, can you elaborate ?
We've been using dracut --uefi to build the UKIs in koji and they
appear to work as expected. Are the offsets only incorrect in
certain circumstances.

We're pretty ambivalent about choice of tool though. We just picked
dracut as it was the first we came across and appeared to work. Any
other tool is viable as then should all (generally) end up doing the
same thing, whichever is most maintainable sounds good.

I can't  argue with Python being better than shell, so ukify certain
wins on that score :-)


>  In particular, if some functionality is missing from ukify, feel free
> to submit
>  a PR or an issue. It's in Python, so fairly nice to modify. So far it's been
>  written to take a kernel and an initrd and the stub, and produce an efi binary. 
Currently we just ask dracut to create the UKI and it creates
the intird as part of that process. If using ukify, we would
ask dracut to create the initrd, and then ask ukify to build
that into a UKI. Its more work, but if ukify does a better
job at the UKI bits that's fine, especially if we'd be suggesting
use of ukify for users in general.


With regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

[Lennart Poettering]

On Do, 22.12.22 14:49, Daniel P. Berrangé (berrange(a)redhat.com) wrote:

>  When you say it dooesn't get the offsets right, can you elaborate
> ? 
dracut uses fixed offsets for the sections to be placed in memory
in. The values are simply hardcoded, literally specified address
offsets, that worked for the original authors. This typically works –
as long as your sections are not much larger than they were for the
people wo came up with these offsets initially. But as it turns out
this doesn't work for some cases. In such cases the sections will be
loaded into memory overlapped and bad things happen.

ukify hence calculates the offsets manually (by adding up the section
sizes so that this cannot happen.

Lennart

--
Lennart Poettering, Berlin

[Zbigniew Jędrzejewski-Szmek]

On Thu, Dec 22, 2022 at 04:24:11PM +0100, Lennart Poettering wrote:
>  On Do, 22.12.22 14:49, Daniel P. Berrangé (berrange(a)redhat.com)
> wrote:
>
>  > When you say it dooesn't get the offsets right, can you elaborate ?
>
>  dracut uses fixed offsets for the sections to be placed in memory
>  in. The values are simply hardcoded, literally specified address
>  offsets, that worked for the original authors. This typically works –
>  as long as your sections are not much larger than they were for the
>  people wo came up with these offsets initially. But as it turns out
>  this doesn't work for some cases. In such cases the sections will be
>  loaded into memory overlapped and bad things happen.
>
>  ukify hence calculates the offsets manually (by adding up the section
>  sizes so that this cannot happen. 
The issue was detected in CI [1]. Some code changes made the .text
section bigger, causing other sections to overlap, causing an actual
failure during boot. But it seems that the problem is more widespread and
we were just being lucky ;(  We're figuring out the details,

See the attached program:
$ dracut --uefi /tmp/initrd 6.0.13-300.fc37.x86_64
$ python info.py /tmp/initrd
...
#   4 .rela         000010c8  000000000001f000  000000000001f000  00017f40  2**2
  start=126976 end=131272
#   5 .osrel        000002df  0000000000020000  0000000000020000  00019140  2**2
  start=131072 end=131807
  vma overlap with previous section: 200 bytes
...

I plan to return to this after the holidays.

Zbyszek

[1] https://github.com/systemd/systemd/pull/23706#issuecomment-1354729112

[Björn Persson]

Lennart Poettering wrote:
>  dracut uses fixed offsets for the sections to be placed in memory
>  in. The values are simply hardcoded, literally specified address
>  offsets, that worked for the original authors. This typically works –
>  as long as your sections are not much larger than they were for the
>  people wo came up with these offsets initially. But as it turns out
>  this doesn't work for some cases. In such cases the sections will be
>  loaded into memory overlapped and bad things happen. 
Oh yuck! And the manpage is written as if dracut --uefi is expected to
work reliably. I see no big eye-catching warning that such-and-such
must be smaller than x bytes.

Björn Persson

[Gerd Hoffmann]

On Thu, Dec 22, 2022 at 02:49:37PM +0000, Daniel P. Berrangé wrote:
>  > There are at three ways that are used: 'dracut --uefi',
> systemd's ukify, and a
>  > manual objcopy invocation. The first two are wrappers around objcopy. I'm
> biased
>  > because I wrote 'ukify', but I think that's the way to go. What dracut
> does is
>  > somewhat primitive and doesn't get the offsets right. Obviously it could be
>  > improved, but putting the code to generate UKIs inside of an initrd generator is
>  > a historical accident, and bash is not the nicest language to do offset
>  > calculations. Thus I hope we can standarize on ukify instead.
>  
>  When you say it dooesn't get the offsets right, can you elaborate ?
>  We've been using dracut --uefi to build the UKIs in koji and they
>  appear to work as expected. Are the offsets only incorrect in
>  certain circumstances. 
When the kernel binary happens to be too big things break because the
end of the kernel will overlap with the start of the initrd due to the
fixed offsets used and thus the fixes space being available for the
kernel binary (and the other things being added to the uki, except
initrd which comes last).

Which is not the case right now with fedora kernels, so everything works
fine.

take care,
  Gerd

[Gerd Hoffmann]

On Tue, Dec 20, 2022 at 08:42:14PM +0100, Björn Persson wrote:
>  > Main motivation for this move is to make the distro more robust
> and more secure.
>  
>  Improving security would be great, but it must be done in a way that
>  allows the sysadmin to configure and repair the system and authorize
>  the new configuration.
>  
>  > Switching the whole distro over to unified kernels quickly is not
>  > realistic though.  Too many features are depending on the current
>  > workflow with a host-specific initrd (and host-specific kernel command
>  > line), which is fundamentally incompatible with unified kernels where
>  > everybody will have the same initrd and command line. Thats why there
>  > is 'Phase 1' in title, so we can have more Phases in future releases
>  > 😃
>  
>  Whew! So usable kernels won't go away in F38. I only have to worry
>  about being forced to build my own kernels in some unspecified future
>  phase. Doom is still coming but no one knows when. That's *such* a
>  relief. 
Note the proposal talks about adding support for ukis, not about
removing support for current separate kernel+initrd setup.

It surely is the goal to incrementally improve uki support to cover more
use cases.  I expect it will take quite some time to have ukis reach
feature parity, and only then it makes sense to start
thinking/discussing about eventually dropping support for non-uki
kernels.  Which may very well never happen, for starters switching
existing installs automatically is a rather hard problem because the
local generated initrd may have unknown customizations.

>  > A host-specific initrd / command line is needed today for:
>  [...]
>  > * configuration being specified on the kernel command line.
>  > ** root filesystem being the most important one.
>  > [https://systemd.io/DISCOVERABLE_PARTITIONS/ Discoverable partitions]
>  > allow to remove this.
>  
>  Why link to a page that only contains a link to another page? Why not
>  link directly to
>  https://uapi-group.org/specifications/specs/discoverable_partitions_speci...
Oh, it moved.  Didn't notice that yet, will update the link.

>  > Phase 2/3 goals (longer-term stuff which is not realistic to
> complete for F38).
>  > 
>  > * Move away from using the kernel command line for configuration.
>  
>  I note that taking away the kernel command line is indeed a clearly
>  stated goal, which will limit Fedora to simple, appliance-like uses. 
Ok, looks like I should clarify that.  I want fedora not depend on the
kernel command line for configuration, so a unified kernel with the
compiled-in default command line boots the system fine without needing
any customization.

If you want change the command line you surely can do that (assuming
secure boot is disabled, otherwise systemd-stub ignores the command
line).

take care,
  Gerd

[Björn Persson]

Gerd Hoffmann wrote:
>  On Tue, Dec 20, 2022 at 08:42:14PM +0100, Björn Persson wrote:
>  > > Switching the whole distro over to unified kernels quickly is not
>  > > realistic though.  Too many features are depending on the current
>  > > workflow with a host-specific initrd (and host-specific kernel command
>  > > line), which is fundamentally incompatible with unified kernels where
>  > > everybody will have the same initrd and command line. Thats why there
>  > > is 'Phase 1' in title, so we can have more Phases in future releases
>  > > 😃  
>  > 
>  > Whew! So usable kernels won't go away in F38. I only have to worry
>  > about being forced to build my own kernels in some unspecified future
>  > phase. Doom is still coming but no one knows when. That's *such* a
>  > relief.  
>  
>  Note the proposal talks about adding support for ukis, not about
>  removing support for current separate kernel+initrd setup. 
That's not how I read "switching the whole distro over". Switching over
means to remove the old thing and use only the new thing.

I see you have changed that in the wiki. The change proposal looks less
scary now.

Björn Persson

[Daniel P. Berrangé]

On Tue, Dec 20, 2022 at 08:42:14PM +0100, Björn Persson wrote:
>  > Main motivation for this move is to make the distro more robust
> and more secure.
>  
>  Improving security would be great, but it must be done in a way that
>  allows the sysadmin to configure and repair the system and authorize
>  the new configuration.
>  
>  > Switching the whole distro over to unified kernels quickly is not
>  > realistic though.  Too many features are depending on the current
>  > workflow with a host-specific initrd (and host-specific kernel command
>  > line), which is fundamentally incompatible with unified kernels where
>  > everybody will have the same initrd and command line. Thats why there
>  > is 'Phase 1' in title, so we can have more Phases in future releases
>  > 😃
>  
>  Whew! So usable kernels won't go away in F38. I only have to worry
>  about being forced to build my own kernels in some unspecified future
>  phase. Doom is still coming but no one knows when. That's *such* a
>  relief.
>  
>  > A host-specific initrd / command line is needed today for:
>  [...]
>  > * configuration being specified on the kernel command line.
>  > ** root filesystem being the most important one.
>  > [https://systemd.io/DISCOVERABLE_PARTITIONS/ Discoverable partitions]
>  > allow to remove this.
>  
>  Why link to a page that only contains a link to another page? Why not
>  link directly to
>  https://uapi-group.org/specifications/specs/discoverable_partitions_speci...
>  
>  That page makes it clear that omitting root= from the kernel command
>  line is only an *option* which is proposed only "for simpler,
>  appliance-like installations". My workstation and my laptop aren't
>  appliance-like in the slightest. And on appliances you want a stable,
>  reliable operating system, not a fast-moving, unstable one like Fedora.
>  
>  ** Troubleshooting being the second most important one. When the system
>  won't boot, it's necessary to remove "quiet" and "rhgb" from
> the kernel
>  command line to see how far the boot process gets and what error
>  messages are printed. It may also be necessary to configure a serial
>  console for example. 
Current UKIs only support a single embedded cmdline, but I've proposed
supporting multiple. The intent would be to ship with two cmdlines,
one for "normal" production usage which would be 'quiet rhgb' and one
for debug usage, which would omit 'quiet rhgb'. The bootloader would
need to allow the user to pick between the two options, but this
would allow SecureBoot to be sustained.

>  The root filesystem is also relevant for troubleshooting, when I take
> a
>  storage device out of a broken computer and connect it to another
>  computer to inspect it. Suddenly there are two "discoverable" root
>  partitions, and the kernel parameter is necessary to specify which one
>  is the root filesystem, as stated under "Doesn’t this break multi-boot
>  scenarios?":
>
> https://uapi-group.org/specifications/specs/discoverable_partitions_speci...
This proposal for UKIs is targetting VM usage where multi-boot is
not something that is important. You can take the guest disk image
and access it using tools like 'guestfish' to access data, and
discoverable partitions won't make that aspect of it harder.

Note that when systemd discovers partitions, it only discovers
partitions on the same physical disk that the kernel is booting
from. So even in the bare metal case, it wouldn't even consider
the discoverable partitions on your second disk. The problems
only arise if you have multiple OS installs on the same disk.

>  > Phase 2/3 goals (longer-term stuff which is not realistic to
> complete for F38).
>  > 
>  > * Move away from using the kernel command line for configuration.
>  
>  I note that taking away the kernel command line is indeed a clearly
>  stated goal, which will limit Fedora to simple, appliance-like uses. 
That bullet point is a little open to mis-interpretation. If
it is possible to do something on the kernel command line today,
it isn't proposing to remove that ability. This point is rather
about providing an alternative way to achieve the same goal
without relying on the command line. IOW there would be a choice,
with the right answer picked depending on the usage scenario.


With regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

[Lennart Poettering]

On Di, 20.12.22 20:42, Björn Persson (Bjorn(a)xn--rombobjrn-67a.se) wrote:

>  The root filesystem is also relevant for troubleshooting, when I take
> a
>  storage device out of a broken computer and connect it to another
>  computer to inspect it. Suddenly there are two "discoverable" root
>  partitions, and the kernel parameter is necessary to specify which one
>  is the root filesystem, as stated under "Doesn’t this break multi-boot
>  scenarios?":
>
> https://uapi-group.org/specifications/specs/discoverable_partitions_speci...
Nah. When booting without root= so that the root partition is
auto-discovered then this is done on the disk the ESP that was used
for booting is found on and no other disks. Thus: once you pick a disk
for booting in the firmware menu, auto-discovery won't "leave" this
disk anymore.

Lennart

--
Lennart Poettering, Berlin

[Demi Marie Obenour]

On 12/20/22 10:22, Ben Cotton wrote:
> https://fedoraproject.org/wiki/Changes/Unified_Kernel_Support_Phase_1
>  
>  This document represents a proposed Change. As part of the Changes
>  process, proposals are publicly announced in order to receive
>  community feedback. This proposal will only be implemented if approved
>  by the Fedora Engineering Steering Committee.
>  
>  
>  == Summary ==
>  Add support for unified kernels images to Fedora.
>  
>  == Owner ==
>  * Name: [[User:kraxel| Gerd Hoffmann]]
>  * Email: kraxel(a)redhat.com
>  
>  
>  == Detailed Description ==
>  The goal is to move away from initrd images being generated on the
>  installed machine.  They are generated while building the kernel
>  package instead, then shipped as part of a unified kernel image.
>  
>  A unified kernel image is an all-in-one efi binary containing kernel,
>  initrd, cmdline and signature.  The secure boot signature covers
>  everything, specifically the initrd is included which is not the case
>  when the initrd gets loaded as separate file from /boot.
>  
>  Main motivation for this move is to make the distro more robust and more secure.
>  
>  Switching the whole distro over to unified kernels quickly is not
>  realistic though.  Too many features are depending on the current
>  workflow with a host-specific initrd (and host-specific kernel command
>  line), which is fundamentally incompatible with unified kernels where
>  everybody will have the same initrd and command line. Thats why there
>  is 'Phase 1' in title, so we can have more Phases in future releases
>  😃
>  
>  A host-specific initrd / command line is needed today for:
>  
>  * features needing optional dracut modules (initrd rebuild needed to
>  enable them).
>  * configuration / secrets baked into the initrd (booting from iscsi
>  for example).
>  * configuration being specified on the kernel command line.
>  ** root filesystem being the most important one.
>  [https://systemd.io/DISCOVERABLE_PARTITIONS/ Discoverable partitions]
>  allow to remove this.
>  
>  Phase 1 goals (high priority):
>  
>  * Ship a unified kernel image as (optional) kernel sub-rpm.  Users can
>  opt-in to use that kernel by installing the sub-rpm.  Initial focus is
>  on booting virtual machines where we have a relatively small and well
>  defined set of drivers / features needed.  Supporting modern physical
>  machines with standard setup (i.e. boot from local sata/nvme storage)
>  too should be easy.
>  * Update kernel install scripts so unified kernels are installed and
>  updated properly.
>  * Add bootloader support for unified kernel images.  Add
>  [https://systemd.io/BOOT_LOADER_SPECIFICATION/#type-2-efi-unified-kernel-i...
>  unified kernel bls support] to grub2, or support using systemd-boot,
>  or both.
>  
>  Phase 1 goals (lower priority, might move to Phase 2):
>  
>  * Add proper discoverable partitions support to installers (anaconda,
>  image builder, ...).
>  ** Temporary workaround possible: set types using sfdisk in %post script.
>  ** When using btrfs: configure 'root' subvolume as default volume.
>  * Add proper systemd-boot support to installers.
>  ** Temporary workaround possible: run 'bootctl install' in %post script.
>  * Better measurement and remote attestation support.
>  ** store kernel + initrd hashes somewhere (kernel-hashes.rpm ?) to
>  allow pre-calculate TPM PCR values.
>  ** avoid using grub2 (measures every config file line executed which
>  is next to impossible to pre-calculate).
>  * Switch cloud images to use unified kernels.
>  
>  Phase 2/3 goals (longer-term stuff which is not realistic to complete for F38).
>  
>  * Move away from using the kernel command line for configuration.
>  * Move away from storing secrets in the initrd.
>  * Handle dracut optional modules in a different way.
>  
>  systemd has some building blocks which can be used, although none of
>  them are used by fedora today.
>  [https://www.freedesktop.org/software/systemd/man/systemd-creds.html
>  systemd credentials] can be used for secrets (also for configuration).
>  The [https://www.freedesktop.org/software/systemd/man/systemd-stub.html
>  unified kernel stub] can load credentials from the ESP.
>  
>  The unified kernel stub can also load
>  [https://www.freedesktop.org/software/systemd/man/systemd-sysext.html
>  extensions] from the ESP, which can possibly be used to replace
>  optional dracut modules.
>  
>  == Feedback ==
>  
>  
>  == Benefit to Fedora ==
>  * Better secure boot support (specifically the initrd is covered by
>  the signature).
>  * Better confidential computing support (measurements are much more
>  useful if we know what hashes to expect for the initrd).
>  * More robust boot process (generating the initrd on the installed
>  system is fragile, root cause for kernel bugs reported is simply a
>  broken initrd sometimes).
>  
>  == Scope ==
>  * Proposal owners:
>  ** Update kernel build to create unified kernel sub-package.
>  *** part one: [https://gitlab.com/cki-project/kernel-ark/-/merge_requests/2179
>  PR#2179]
>  *** part two: (wip)
>  [https://gitlab.com/kraxel/kernel-ark/-/commits/unified/
>  https://gitlab.com/kraxel/kernel-ark/-/commits/unified/]
>  
>  * Other developers:
>  ** grub2: add unified kernel support
>  *** wip code at [https://github.com/osteffenrh/grub2/
>  https://github.com/osteffenrh/grub2/]
>  ** installer(s): add support for discoverable partitions.
>  *** [https://bugzilla.redhat.com/show_bug.cgi?id=1075288 Bug#1075288]
>  
>  * Release engineering: [https://pagure.io/releng/issues #Releng issue number]
>  * Policies and guidelines: N/A (not needed for this Change)
>  * Trademark approval: N/A (not needed for this Change)
>  * Alignment with Objectives:
>  
>  
>  == Upgrade/compatibility impact ==
>  
>  None (using unified kernel is opt-in for Phase 1).
>  
>  == How To Test ==
>  Try on a existing (uefi) system:
>  * make sure you are running fedora 37 or rawhide.
>  * make sure your root filesystem has type "Linux root (x86-64)" (use
>  `fdisk -l` to check).
>  ** should that not be the case use the fdisk tag command ('t') to
>  change the partition type.
>  * when using btrfs: make sure the 'root' subvolume is set as default volume.
>  * `dnf copr enable kraxel/unified.kernel`
>  * `dnf update "grub2*"`
>  * `dnf install kernel-unified-virt`
>  * `reboot`
>  You should find two entries in the grub2 boot menu, one for classing
>  kernel with separate initrd and one for the unified kernel image.
>  Both should boot fine.
>  
>  The https://gitlab.com/kraxel/fedora-uki project has kickstart files
>  and helper scripts for generating virtual machine images.
>  * image using grub2-efi:
>  https://gitlab.com/kraxel/fedora-uki/-/raw/master/kickstart/fedora-uki-gr...
>  * image using systemd-boot:
>  https://gitlab.com/kraxel/fedora-uki/-/raw/master/kickstart/fedora-uki-sd...
>  
>  Prebuilt virtual machine images are available from
>  https://www.kraxel.org/fedora-uki/.
>  
>  == User Experience ==
>  
>  
>  == Dependencies ==
>  
>  
>  == Contingency Plan ==
>  * Contingency mechanism:
>  ** Probably none (unified kernel images are opt-in for Phase 1).
>  ** In case we tried switching the cloud images to unified kernels:
>  revert the kickstart config changes.
>  * Contingency deadline:
>  * Blocks release? No
>  
>  == Documentation ==
>  
>  
>  == Release Notes == 
How do you plan to handle system recovery?  For VMs this is much
less of a concern, but on bare metal there needs to be a way for
a local, authenticated administrator to obtain a root shell on
the system console even if the root filesystem cannot be mounted.
This has saved my system more than once.

Also, how will Xen be supported in this model?  Will the hypervisor
be part of an alternate UKI?  CCing Marek Marczykowski-Gorécki of
Qubes OS.
-- 
Sincerely,
Demi Marie Obenour (she/her/hers)

[Simo Sorce]

On Tue, 2022-12-20 at 14:56 -0500, Demi Marie Obenour wrote:
>  How do you plan to handle system recovery?  For VMs this is much
>  less of a concern, but on bare metal there needs to be a way for
>  a local, authenticated administrator to obtain a root shell on
>  the system console even if the root filesystem cannot be mounted.
>  This has saved my system more than once.
>  
>  Also, how will Xen be supported in this model?  Will the hypervisor
>  be part of an alternate UKI?  CCing Marek Marczykowski-Gorécki of
>  Qubes OS. 
It is all answered in the large amount of text you quoted, if you read
it carefully.
The old kernel+inird does not go away, so you disable secure boot and
just use the good old methods, or worst case you use a recovery disk
(or USB drive, or whatever you use to install) if you damaged the boot
partition.

Anything that is not explicitly supported likewise will use the old
kernel + custom initrd, you just disable secure boot.

Simo.

-- 
Simo Sorce
RHEL Crypto Team
Red Hat, Inc

[Demi Marie Obenour]

On 12/20/22 16:34, Simo Sorce wrote:
>  On Tue, 2022-12-20 at 14:56 -0500, Demi Marie Obenour wrote:
> > How do you plan to handle system recovery?  For VMs this is much
> > less of a concern, but on bare metal there needs to be a way for
> > a local, authenticated administrator to obtain a root shell on
> > the system console even if the root filesystem cannot be mounted.
> > This has saved my system more than once.
> >
> > Also, how will Xen be supported in this model?  Will the hypervisor
> > be part of an alternate UKI?  CCing Marek Marczykowski-Gorécki of
> > Qubes OS.
>  
>  It is all answered in the large amount of text you quoted, if you read
>  it carefully.
>  The old kernel+inird does not go away, so you disable secure boot and
>  just use the good old methods, or worst case you use a recovery disk
>  (or USB drive, or whatever you use to install) if you damaged the boot
>  partition.
>  
>  Anything that is not explicitly supported likewise will use the old
>  kernel + custom initrd, you just disable secure boot. 
If rescuing a system means disabling secure boot, then there is no point
in having secure boot in the first place, because malware can reliably
cause the user to disable it.  There needs to be a way to rescue the
system *without* disabling secure boot, at least as long as the UKI can
itself be loaded.  Furthermore, this must not require the user to enter
secrets until they have (via TPM PCR-based attestation) verified that
the system is booting trusted code.  This means that the initramfs must
be able to authenticate the user.  And having Xen be incompatible with
secure boot is not something Xen users are going to be happy with,
especially because Xen fully supports UKIs that include the hypervisor.

Having this be out of scope for phase 1 is fine, but it should be
supported eventually.
-- 
Sincerely,
Demi Marie Obenour (she/her/hers)

[Daniel P. Berrangé]

On Tue, Dec 20, 2022 at 02:56:41PM -0500, Demi Marie Obenour wrote:
>  On 12/20/22 10:22, Ben Cotton wrote:
>  > https://fedoraproject.org/wiki/Changes/Unified_Kernel_Support_Phase_1
>  > 
>  > This document represents a proposed Change. As part of the Changes
>  > process, proposals are publicly announced in order to receive
>  > community feedback. This proposal will only be implemented if approved
>  > by the Fedora Engineering Steering Committee.
>  > 
>  > 
>  > == Summary ==
>  > Add support for unified kernels images to Fedora.
>  > 
>  > == Owner ==
>  > * Name: [[User:kraxel| Gerd Hoffmann]]
>  > * Email: kraxel(a)redhat.com
>  > 
>  > 
>  > == Detailed Description ==
>  > The goal is to move away from initrd images being generated on the
>  > installed machine.  They are generated while building the kernel
>  > package instead, then shipped as part of a unified kernel image.
>  > 
>  > A unified kernel image is an all-in-one efi binary containing kernel,
>  > initrd, cmdline and signature.  The secure boot signature covers
>  > everything, specifically the initrd is included which is not the case
>  > when the initrd gets loaded as separate file from /boot.
>  > 
>  > Main motivation for this move is to make the distro more robust and more secure.
>  > 
>  > Switching the whole distro over to unified kernels quickly is not
>  > realistic though.  Too many features are depending on the current
>  > workflow with a host-specific initrd (and host-specific kernel command
>  > line), which is fundamentally incompatible with unified kernels where
>  > everybody will have the same initrd and command line. Thats why there
>  > is 'Phase 1' in title, so we can have more Phases in future releases
>  > 😃
>  > 
>  > A host-specific initrd / command line is needed today for:
>  > 
>  > * features needing optional dracut modules (initrd rebuild needed to
>  > enable them).
>  > * configuration / secrets baked into the initrd (booting from iscsi
>  > for example).
>  > * configuration being specified on the kernel command line.
>  > ** root filesystem being the most important one.
>  > [https://systemd.io/DISCOVERABLE_PARTITIONS/ Discoverable partitions]
>  > allow to remove this.
>  > 
>  > Phase 1 goals (high priority):
>  > 
>  > * Ship a unified kernel image as (optional) kernel sub-rpm.  Users can
>  > opt-in to use that kernel by installing the sub-rpm.  Initial focus is
>  > on booting virtual machines where we have a relatively small and well
>  > defined set of drivers / features needed.  Supporting modern physical
>  > machines with standard setup (i.e. boot from local sata/nvme storage)
>  > too should be easy.
>  > * Update kernel install scripts so unified kernels are installed and
>  > updated properly.
>  > * Add bootloader support for unified kernel images.  Add
>  > [https://systemd.io/BOOT_LOADER_SPECIFICATION/#type-2-efi-unified-kernel-i...
>  > unified kernel bls support] to grub2, or support using systemd-boot,
>  > or both.
>  > 
>  > Phase 1 goals (lower priority, might move to Phase 2):
>  > 
>  > * Add proper discoverable partitions support to installers (anaconda,
>  > image builder, ...).
>  > ** Temporary workaround possible: set types using sfdisk in %post script.
>  > ** When using btrfs: configure 'root' subvolume as default volume.
>  > * Add proper systemd-boot support to installers.
>  > ** Temporary workaround possible: run 'bootctl install' in %post script.
>  > * Better measurement and remote attestation support.
>  > ** store kernel + initrd hashes somewhere (kernel-hashes.rpm ?) to
>  > allow pre-calculate TPM PCR values.
>  > ** avoid using grub2 (measures every config file line executed which
>  > is next to impossible to pre-calculate).
>  > * Switch cloud images to use unified kernels.
>  
>  How do you plan to handle system recovery?  For VMs this is much
>  less of a concern, but on bare metal there needs to be a way for
>  a local, authenticated administrator to obtain a root shell on
>  the system console even if the root filesystem cannot be mounted.
>  This has saved my system more than once. 
Note this proposal is only targetting making UKIs available
as an optional feature for use with VMs. As such needs for
a recovery root shell are minimal, and bare metal is not
impacted wrt its featureset today.

>  Also, how will Xen be supported in this model?  Will the hypervisor
>  be part of an alternate UKI?  CCing Marek Marczykowski-Gorécki of
>  Qubes OS. 
Xen has only been considered from the POV of the guest, since
this proposal is only targetting guest usage. The UKIs are
proposed to include the Xen block driver in the initrd. Nothing
is changing that impacts the Hypervisor side.

With regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

[Marc Pervaz Boocha]

This entire proposal although has write ideas (I also like to see UKI
in Fedora as I have use UKI on Arch and undestand its advantages) is in
the wrong.

But why start doing UKI without first fixing the need of host specific
initrd and commandline. I am sure even non-UEFI users will be better of
of not having a host specific initrd. We should first start working on
shipping initrd on dedora as an rpm with atleast one use which can be
extended by sysext or someother mechanism without the need of UKI
first. Once That is possible that UKI can be easily be intgrated and
tested. We will be also provide a subset of benefits to the non-UEFI
users(for example on other arches). Let first get any gpt or lvm rootfs
work with this model then look.

Thanks & Regards
Marc Pervaz Boocha

[Gerd Hoffmann]

  Hi,

>  But why start doing UKI without first fixing the need of host
> specific
>  initrd and commandline. 
We are trying to tackle that in parallel.  At least when generating
virtual machine images we can temporarily workaround that with short
%post scripts in kickstart files.

>  I am sure even non-UEFI users will be better of of not having a host
>  specific initrd. We should first start working on shipping initrd on
>  dedora as an rpm with atleast one use which can be extended by sysext
>  or someother mechanism without the need of UKI first. 
We'll have to continue support kernel + host-generated initrd for the
foreseeable future.  UKIs are a second option.  Adding kernel + shipped
initrd as third option IMHO just adds complexity for no real benefit.

>  Once That is possible that UKI can be easily be intgrated and
>  tested. We will be also provide a subset of benefits to the non-UEFI
>  users(for example on other arches). 
We have experimental grub patches which can boot unified kernel on bios
systems.  Intended use case: hybrid bios/uefi cloud images with
UKIs.  Extending that to ppc is probably possible too should the ppc
arch maintainers be interested.

take care,
  Gerd

[Vitaly Zaitsev]

On 20/12/2022 16:22, Ben Cotton wrote:
>  The goal is to move away from initrd images being generated on the
>  installed machine.  They are generated while building the kernel
>  package instead, then shipped as part of a unified kernel image. 
+1.

>  * Add proper systemd-boot support to installers. 
+100. Finally we will be able to get rid of GRUB2 aka a collection of 
patches and hacks.

I use systemd-boot + SB enabled with my own CA on my laptop for two 
years and everything works fine there.

-- 
Sincerely,
   Vitaly Zaitsev (vitaly(a)easycoding.org)

[Dennis Gilmore]

In my case, I have Network Manager config files included in my initrd
and bootargs to bring up the network so that I get automatic disk
decryption while on my home network, and prompted for a password when
I am not at home. I think this a reasonable enough use case it should
be considered in the long term plan. There was an effort many years
ago that built the initramfs with the kernel, it was abandoned due to
not being able to guarantee sources for the binaries in the initramfs,
trying to dig up the details I am having trouble finding it, but legal
blocked it there is a reference to it in an old FESCo meeting
https://lists.fedoraproject.org/pipermail/devel/2013-February/178220.html.
Additionally, we should also consider
https://fedoraproject.org/wiki/Features/DracutHostOnly and the size
implications and why we moved to have kernel-core, kernel-modules, and
kernel-modules-extra for cloud and different use cases.

Dennis

On Tue, Dec 20, 2022 at 9:22 AM Ben Cotton <bcotton(a)redhat.com&gt; wrote:
>
> https://fedoraproject.org/wiki/Changes/Unified_Kernel_Support_Phase_1
>
> This document represents a proposed Change. As part of the Changes
> process, proposals are publicly announced in order to receive
> community feedback. This proposal will only be implemented if approved
> by the Fedora Engineering Steering Committee.
>
>
> == Summary ==
> Add support for unified kernels images to Fedora.
>
> == Owner ==
> * Name: [[User:kraxel| Gerd Hoffmann]]
> * Email: kraxel(a)redhat.com
>
>
> == Detailed Description ==
> The goal is to move away from initrd images being generated on the
> installed machine.  They are generated while building the kernel
> package instead, then shipped as part of a unified kernel image.
>
> A unified kernel image is an all-in-one efi binary containing kernel,
> initrd, cmdline and signature.  The secure boot signature covers
> everything, specifically the initrd is included which is not the case
> when the initrd gets loaded as separate file from /boot.
>
> Main motivation for this move is to make the distro more robust and more secure.
>
> Switching the whole distro over to unified kernels quickly is not
> realistic though.  Too many features are depending on the current
> workflow with a host-specific initrd (and host-specific kernel command
> line), which is fundamentally incompatible with unified kernels where
> everybody will have the same initrd and command line. Thats why there
> is 'Phase 1' in title, so we can have more Phases in future releases
> 😃
>
> A host-specific initrd / command line is needed today for:
>
> * features needing optional dracut modules (initrd rebuild needed to
> enable them).
> * configuration / secrets baked into the initrd (booting from iscsi
> for example).
> * configuration being specified on the kernel command line.
> ** root filesystem being the most important one.
> [https://systemd.io/DISCOVERABLE_PARTITIONS/ Discoverable partitions]
> allow to remove this.
>
> Phase 1 goals (high priority):
>
> * Ship a unified kernel image as (optional) kernel sub-rpm.  Users can
> opt-in to use that kernel by installing the sub-rpm.  Initial focus is
> on booting virtual machines where we have a relatively small and well
> defined set of drivers / features needed.  Supporting modern physical
> machines with standard setup (i.e. boot from local sata/nvme storage)
> too should be easy.
> * Update kernel install scripts so unified kernels are installed and
> updated properly.
> * Add bootloader support for unified kernel images.  Add
> [https://systemd.io/BOOT_LOADER_SPECIFICATION/#type-2-efi-unified-kernel-i...
> unified kernel bls support] to grub2, or support using systemd-boot,
> or both.
>
> Phase 1 goals (lower priority, might move to Phase 2):
>
> * Add proper discoverable partitions support to installers (anaconda,
> image builder, ...).
> ** Temporary workaround possible: set types using sfdisk in %post script.
> ** When using btrfs: configure 'root' subvolume as default volume.
> * Add proper systemd-boot support to installers.
> ** Temporary workaround possible: run 'bootctl install' in %post script.
> * Better measurement and remote attestation support.
> ** store kernel + initrd hashes somewhere (kernel-hashes.rpm ?) to
> allow pre-calculate TPM PCR values.
> ** avoid using grub2 (measures every config file line executed which
> is next to impossible to pre-calculate).
> * Switch cloud images to use unified kernels.
>
> Phase 2/3 goals (longer-term stuff which is not realistic to complete for F38).
>
> * Move away from using the kernel command line for configuration.
> * Move away from storing secrets in the initrd.
> * Handle dracut optional modules in a different way.
>
> systemd has some building blocks which can be used, although none of
> them are used by fedora today.
> [https://www.freedesktop.org/software/systemd/man/systemd-creds.html
> systemd credentials] can be used for secrets (also for configuration).
> The [https://www.freedesktop.org/software/systemd/man/systemd-stub.html
> unified kernel stub] can load credentials from the ESP.
>
> The unified kernel stub can also load
> [https://www.freedesktop.org/software/systemd/man/systemd-sysext.html
> extensions] from the ESP, which can possibly be used to replace
> optional dracut modules.
>
> == Feedback ==
>
>
> == Benefit to Fedora ==
> * Better secure boot support (specifically the initrd is covered by
> the signature).
> * Better confidential computing support (measurements are much more
> useful if we know what hashes to expect for the initrd).
> * More robust boot process (generating the initrd on the installed
> system is fragile, root cause for kernel bugs reported is simply a
> broken initrd sometimes).
>
> == Scope ==
> * Proposal owners:
> ** Update kernel build to create unified kernel sub-package.
> *** part one: [https://gitlab.com/cki-project/kernel-ark/-/merge_requests/2179
> PR#2179]
> *** part two: (wip)
> [https://gitlab.com/kraxel/kernel-ark/-/commits/unified/
> https://gitlab.com/kraxel/kernel-ark/-/commits/unified/]
>
> * Other developers:
> ** grub2: add unified kernel support
> *** wip code at [https://github.com/osteffenrh/grub2/
> https://github.com/osteffenrh/grub2/]
> ** installer(s): add support for discoverable partitions.
> *** [https://bugzilla.redhat.com/show_bug.cgi?id=1075288 Bug#1075288]
>
> * Release engineering: [https://pagure.io/releng/issues #Releng issue number]
> * Policies and guidelines: N/A (not needed for this Change)
> * Trademark approval: N/A (not needed for this Change)
> * Alignment with Objectives:
>
>
> == Upgrade/compatibility impact ==
>
> None (using unified kernel is opt-in for Phase 1).
>
> == How To Test ==
> Try on a existing (uefi) system:
> * make sure you are running fedora 37 or rawhide.
> * make sure your root filesystem has type "Linux root (x86-64)" (use
> `fdisk -l` to check).
> ** should that not be the case use the fdisk tag command ('t') to
> change the partition type.
> * when using btrfs: make sure the 'root' subvolume is set as default volume.
> * `dnf copr enable kraxel/unified.kernel`
> * `dnf update "grub2*"`
> * `dnf install kernel-unified-virt`
> * `reboot`
> You should find two entries in the grub2 boot menu, one for classing
> kernel with separate initrd and one for the unified kernel image.
> Both should boot fine.
>
> The https://gitlab.com/kraxel/fedora-uki project has kickstart files
> and helper scripts for generating virtual machine images.
> * image using grub2-efi:
> https://gitlab.com/kraxel/fedora-uki/-/raw/master/kickstart/fedora-uki-gr...
> * image using systemd-boot:
> https://gitlab.com/kraxel/fedora-uki/-/raw/master/kickstart/fedora-uki-sd...
>
> Prebuilt virtual machine images are available from
> https://www.kraxel.org/fedora-uki/.
>
> == User Experience ==
>
>
> == Dependencies ==
>
>
> == Contingency Plan ==
> * Contingency mechanism:
> ** Probably none (unified kernel images are opt-in for Phase 1).
> ** In case we tried switching the cloud images to unified kernels:
> revert the kickstart config changes.
> * Contingency deadline:
> * Blocks release? No
>
> == Documentation ==
>
>
> == Release Notes ==
>
>
>
> --
> Ben Cotton
> He / Him / His
> Fedora Program Manager
> Red Hat
> TZ=America/Indiana/Indianapolis
> _______________________________________________
> devel mailing list -- devel(a)lists.fedoraproject.org
> To unsubscribe send an email to devel-leave(a)lists.fedoraproject.org
> Fedora Code of Conduct:
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives:
https://lists.fedoraproject.org/archives/list/[e-mail]
> Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue

[Zbigniew Jędrzejewski-Szmek]

On Wed, Dec 21, 2022 at 11:56:32AM -0600, Dennis Gilmore via devel wrote:
>  In my case, I have Network Manager config files included in my
> initrd
>  and bootargs to bring up the network so that I get automatic disk
>  decryption while on my home network, and prompted for a password when
>  I am not at home. I think this a reasonable enough use case it should
>  be considered in the long term plan. There was an effort many years
>  ago that built the initramfs with the kernel, it was abandoned due to
>  not being able to guarantee sources for the binaries in the initramfs, 
If a UKI is built in koji, the initrd it embeds must also be built in koji. And
when the initrd is built in koji, it is just taking some binaries from rpms and
repacking them. We ensure that we have an srpm for any official srpm. Thus, going
back from the UKI, you look up the initrd, and the logs for the initrd build,
and get a list of rpms, and then look the appropriate srpms from that, and from
the srpms you get the sources. There's a few more steps, but the availability of
sources is guaranteed using the mechanism existing for normal rpms.

>  trying to dig up the details I am having trouble finding it, but
> legal
>  blocked it there is a reference to it in an old FESCo meeting
>  https://lists.fedoraproject.org/pipermail/devel/2013-February/178220.html.
>  Additionally, we should also consider
>  https://fedoraproject.org/wiki/Features/DracutHostOnly and the size
>  implications and why we moved to have kernel-core, kernel-modules, and
>  kernel-modules-extra for cloud and different use cases. 
Yes. That's why this proposal explicitly targets a narrow use case which doesn't
need many drivers and will support many different machines with the same
(relatively small) initrd.

Zbyszek

[Dennis Gilmore]

On Thu, Dec 22, 2022 at 5:25 AM Zbigniew Jędrzejewski-Szmek
<zbyszek(a)in.waw.pl&gt; wrote:
>  On Wed, Dec 21, 2022 at 11:56:32AM -0600, Dennis Gilmore via devel wrote:
>  > In my case, I have Network Manager config files included in my initrd
>  > and bootargs to bring up the network so that I get automatic disk
>  > decryption while on my home network, and prompted for a password when
>  > I am not at home. I think this a reasonable enough use case it should
>  > be considered in the long term plan. There was an effort many years
>  > ago that built the initramfs with the kernel, it was abandoned due to
>  > not being able to guarantee sources for the binaries in the initramfs,
>
>  If a UKI is built in koji, the initrd it embeds must also be built in koji. And
>  when the initrd is built in koji, it is just taking some binaries from rpms and
>  repacking them. We ensure that we have an srpm for any official srpm. Thus, going
>  back from the UKI, you look up the initrd, and the logs for the initrd build,
>  and get a list of rpms, and then look the appropriate srpms from that, and from
>  the srpms you get the sources. There's a few more steps, but the availability of
>  sources is guaranteed using the mechanism existing for normal rpms. 
In the past that was deemed to not be good enough. by legal as it is
too hard for the average user to do.

>  > trying to dig up the details I am having trouble finding it, but
> legal
>  > blocked it there is a reference to it in an old FESCo meeting
>  > https://lists.fedoraproject.org/pipermail/devel/2013-February/178220.html.
>  > Additionally, we should also consider
>  > https://fedoraproject.org/wiki/Features/DracutHostOnly and the size
>  > implications and why we moved to have kernel-core, kernel-modules, and
>  > kernel-modules-extra for cloud and different use cases.
>
>  Yes. That's why this proposal explicitly targets a narrow use case which doesn't
>  need many drivers and will support many different machines with the same
>  (relatively small) initrd. 
I think the proposal needs to be explicit in how other use cases and
all architectures will be handled. I think if we do not have a path
for all architectures to be supported we should spend more time
working out how to cover them all.

> Zbyszek

[Zbigniew Jędrzejewski-Szmek]

On Thu, Dec 22, 2022 at 12:42:38PM -0600, Dennis Gilmore wrote:
>  On Thu, Dec 22, 2022 at 5:25 AM Zbigniew Jędrzejewski-Szmek
>  <zbyszek(a)in.waw.pl&gt; wrote:
>  >
>  > On Wed, Dec 21, 2022 at 11:56:32AM -0600, Dennis Gilmore via devel wrote:
>  > > In my case, I have Network Manager config files included in my initrd
>  > > and bootargs to bring up the network so that I get automatic disk
>  > > decryption while on my home network, and prompted for a password when
>  > > I am not at home. I think this a reasonable enough use case it should
>  > > be considered in the long term plan. There was an effort many years
>  > > ago that built the initramfs with the kernel, it was abandoned due to
>  > > not being able to guarantee sources for the binaries in the initramfs,
>  >
>  > If a UKI is built in koji, the initrd it embeds must also be built in koji. And
>  > when the initrd is built in koji, it is just taking some binaries from rpms and
>  > repacking them. We ensure that we have an srpm for any official srpm. Thus, going
>  > back from the UKI, you look up the initrd, and the logs for the initrd build,
>  > and get a list of rpms, and then look the appropriate srpms from that, and from
>  > the srpms you get the sources. There's a few more steps, but the availability
> of
>  > sources is guaranteed using the mechanism existing for normal rpms.
>  
>  In the past that was deemed to not be good enough. by legal as it is
>  too hard for the average user to do. 
Sorry, but that doesn't make any sense.

Quoting Daniel Berrange from the other part of the thread:
>  This is the same situation we already have in Fedora with
>  libguestfs, where we're building a disk image inside Koji bundling
>  various binaries. Or for that matter, not really different from
>  building cloud disk images, or any other deliverable that bundles
>  together some binaries from other RPMs and spits out some kind of
>  image or archive. 
My understanding is that if any of those other things (including any
of our installation media that also contain an kernel and programs in
the initrd) are fine, UKIs must be fine too.

>  > > trying to dig up the details I am having trouble finding
> it, but legal
>  > > blocked it there is a reference to it in an old FESCo meeting
>  > > https://lists.fedoraproject.org/pipermail/devel/2013-February/178220.html.
>  > > Additionally, we should also consider
>  > > https://fedoraproject.org/wiki/Features/DracutHostOnly and the size
>  > > implications and why we moved to have kernel-core, kernel-modules, and
>  > > kernel-modules-extra for cloud and different use cases.
>  >
>  > Yes. That's why this proposal explicitly targets a narrow use case which
> doesn't
>  > need many drivers and will support many different machines with the same
>  > (relatively small) initrd.
>  
>  I think the proposal needs to be explicit in how other use cases and
>  all architectures will be handled. I think if we do not have a path
>  for all architectures to be supported we should spend more time
>  working out how to cover them all. 
Different architectures have different boot loaders. In particular, s390x and
ppc64el are very different. The proposal is to add support for UKIs in grub2, so
that we will cover UEFI and non-UEFI machines that use grub2. For other
architectures, we can in principle do the same thing… After all, the UKI is just
a binary with a few sections appended. But it seems to early to think such
details far ahead. As extensively discussed, the support for separate initrds is
not going anyway and the proposal only targets a small subset of the amd64 space.

Zbyszek

[Luca Boccassi]

>  On Thu, Dec 22, 2022 at 12:42:38PM -0600, Dennis Gilmore wrote:
>  Different architectures have different boot loaders. In particular, s390x and
>  ppc64el are very different. The proposal is to add support for UKIs in grub2, so
>  that we will cover UEFI and non-UEFI machines that use grub2. For other
>  architectures, we can in principle do the same thing… After all, the UKI is just
>  a binary with a few sections appended. But it seems to early to think such
>  details far ahead. As extensively discussed, the support for separate initrds is
>  not going anyway and the proposal only targets a small subset of the amd64 space. 
Also, doesn't u-boot support those architectures? The latest u-boot UEFI interface
seems to work quite well when I tried, including support for secure boot.

[Richard W.M. Jones]

On Fri, Dec 23, 2022 at 08:13:49AM +0000, Zbigniew Jędrzejewski-Szmek wrote:
>  Quoting Daniel Berrange from the other part of the thread:
>  > This is the same situation we already have in Fedora with
>  > libguestfs, where we're building a disk image inside Koji bundling
>  > various binaries. 
FWIW libguestfs does not do this.  The disk image is built on the end
user machine.

Rich.

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
virt-p2v converts physical machines to virtual machines.  Boot with a
live CD or over the network (PXE) and turn machines into KVM guests.
http://libguestfs.org/virt-v2v

[Daniel P. Berrangé]

On Wed, Dec 21, 2022 at 11:56:32AM -0600, Dennis Gilmore via devel wrote:
>  In my case, I have Network Manager config files included in my
> initrd
>  and bootargs to bring up the network so that I get automatic disk
>  decryption while on my home network, and prompted for a password when
>  I am not at home. I think this a reasonable enough use case it should
>  be considered in the long term plan. There was an effort many years
>  ago that built the initramfs with the kernel, it was abandoned due to
>  not being able to guarantee sources for the binaries in the initramfs,
>  trying to dig up the details I am having trouble finding it, but legal
>  blocked it there is a reference to it in an old FESCo meeting
>  https://lists.fedoraproject.org/pipermail/devel/2013-February/178220.html. 
I can't see any legal problem with source provision for the
binaries inside the initramfs. We're building the initrds and
UKIs inside koji, so we have a clear record of exactly what
binary RPMs went into the package, and thus have knowledge
of what sources are involved. This is the same situation we
already have in Fedora with libguestfs, where we're building
a disk image inside Koji bundling various binaries. Or for
that matter, not really different from building cloud disk
images, or any other deliverable that bundles together some
binaries from other RPMs and spits out some kind of image
or archive.

>  Additionally, we should also consider
>  https://fedoraproject.org/wiki/Features/DracutHostOnly and the size
>  implications and why we moved to have kernel-core, kernel-modules, and
>  kernel-modules-extra for cloud and different use cases. 
The UKI size for a VM should not be appreciably different from the
combination of the vmluinuz + locally generated initrd. The UKI
will contain a few more modules, as its initrd is built to cope
with Xen, VMware, HyperV + KVM[1], but this only adds a small amount
over a truly minimal initrd targetting 1 hypervisor. So I don't
expect the size of the UKI will be a problem.


With regards,
Daniel

[1]
https://gitlab.com/kraxel/kernel-ark/-/commit/4395c8f99657d14d77622a18457...
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

[Neal Gompa]

On Thu, Dec 22, 2022 at 6:40 AM Daniel P. Berrangé <berrange(a)redhat.com&gt; wrote:
>  On Wed, Dec 21, 2022 at 11:56:32AM -0600, Dennis Gilmore via devel wrote:
>  > In my case, I have Network Manager config files included in my initrd
>  > and bootargs to bring up the network so that I get automatic disk
>  > decryption while on my home network, and prompted for a password when
>  > I am not at home. I think this a reasonable enough use case it should
>  > be considered in the long term plan. There was an effort many years
>  > ago that built the initramfs with the kernel, it was abandoned due to
>  > not being able to guarantee sources for the binaries in the initramfs,
>  > trying to dig up the details I am having trouble finding it, but legal
>  > blocked it there is a reference to it in an old FESCo meeting
>  > https://lists.fedoraproject.org/pipermail/devel/2013-February/178220.html.
>
>  I can't see any legal problem with source provision for the
>  binaries inside the initramfs. We're building the initrds and
>  UKIs inside koji, so we have a clear record of exactly what
>  binary RPMs went into the package, and thus have knowledge
>  of what sources are involved. This is the same situation we
>  already have in Fedora with libguestfs, where we're building
>  a disk image inside Koji bundling various binaries. Or for
>  that matter, not really different from building cloud disk
>  images, or any other deliverable that bundles together some
>  binaries from other RPMs and spits out some kind of image
>  or archive.
>
>  > Additionally, we should also consider
>  > https://fedoraproject.org/wiki/Features/DracutHostOnly and the size
>  > implications and why we moved to have kernel-core, kernel-modules, and
>  > kernel-modules-extra for cloud and different use cases.
>
>  The UKI size for a VM should not be appreciably different from the
>  combination of the vmluinuz + locally generated initrd. The UKI
>  will contain a few more modules, as its initrd is built to cope
>  with Xen, VMware, HyperV + KVM[1], but this only adds a small amount
>  over a truly minimal initrd targetting 1 hypervisor. So I don't
>  expect the size of the UKI will be a problem.
>  
You need to add VirtualBox too. That's an incredibly common platform
for Fedora to run as a guest.



-- 
真実はいつも一つ！/ Always, there's only one truth!

[Daniel P. Berrangé]

On Thu, Dec 22, 2022 at 06:57:07AM -0500, Neal Gompa wrote:
>  On Thu, Dec 22, 2022 at 6:40 AM Daniel P. Berrangé
> <berrange(a)redhat.com&gt; wrote:
>  >
>  > On Wed, Dec 21, 2022 at 11:56:32AM -0600, Dennis Gilmore via devel wrote:
>  > > In my case, I have Network Manager config files included in my initrd
>  > > and bootargs to bring up the network so that I get automatic disk
>  > > decryption while on my home network, and prompted for a password when
>  > > I am not at home. I think this a reasonable enough use case it should
>  > > be considered in the long term plan. There was an effort many years
>  > > ago that built the initramfs with the kernel, it was abandoned due to
>  > > not being able to guarantee sources for the binaries in the initramfs,
>  > > trying to dig up the details I am having trouble finding it, but legal
>  > > blocked it there is a reference to it in an old FESCo meeting
>  > > https://lists.fedoraproject.org/pipermail/devel/2013-February/178220.html.
>  >
>  > I can't see any legal problem with source provision for the
>  > binaries inside the initramfs. We're building the initrds and
>  > UKIs inside koji, so we have a clear record of exactly what
>  > binary RPMs went into the package, and thus have knowledge
>  > of what sources are involved. This is the same situation we
>  > already have in Fedora with libguestfs, where we're building
>  > a disk image inside Koji bundling various binaries. Or for
>  > that matter, not really different from building cloud disk
>  > images, or any other deliverable that bundles together some
>  > binaries from other RPMs and spits out some kind of image
>  > or archive.
>  >
>  > > Additionally, we should also consider
>  > > https://fedoraproject.org/wiki/Features/DracutHostOnly and the size
>  > > implications and why we moved to have kernel-core, kernel-modules, and
>  > > kernel-modules-extra for cloud and different use cases.
>  >
>  > The UKI size for a VM should not be appreciably different from the
>  > combination of the vmluinuz + locally generated initrd. The UKI
>  > will contain a few more modules, as its initrd is built to cope
>  > with Xen, VMware, HyperV + KVM[1], but this only adds a small amount
>  > over a truly minimal initrd targetting 1 hypervisor. So I don't
>  > expect the size of the UKI will be a problem.
>  >
>  
>  You need to add VirtualBox too. That's an incredibly common platform
>  for Fedora to run as a guest. 
That's easy enough, what kmod is typically required for disks in
VirtualBox ?

With regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

[Neal Gompa]

On Thu, Dec 22, 2022 at 7:07 AM Daniel P. Berrangé <berrange(a)redhat.com&gt; wrote:
>  On Thu, Dec 22, 2022 at 06:57:07AM -0500, Neal Gompa wrote:
>  > On Thu, Dec 22, 2022 at 6:40 AM Daniel P. Berrangé <berrange(a)redhat.com&gt;
> wrote:
>  > >
>  > > On Wed, Dec 21, 2022 at 11:56:32AM -0600, Dennis Gilmore via devel wrote:
>  > > > In my case, I have Network Manager config files included in my initrd
>  > > > and bootargs to bring up the network so that I get automatic disk
>  > > > decryption while on my home network, and prompted for a password when
>  > > > I am not at home. I think this a reasonable enough use case it should
>  > > > be considered in the long term plan. There was an effort many years
>  > > > ago that built the initramfs with the kernel, it was abandoned due to
>  > > > not being able to guarantee sources for the binaries in the initramfs,
>  > > > trying to dig up the details I am having trouble finding it, but legal
>  > > > blocked it there is a reference to it in an old FESCo meeting
>  > > >
> https://lists.fedoraproject.org/pipermail/devel/2013-February/178220.html.
>  > >
>  > > I can't see any legal problem with source provision for the
>  > > binaries inside the initramfs. We're building the initrds and
>  > > UKIs inside koji, so we have a clear record of exactly what
>  > > binary RPMs went into the package, and thus have knowledge
>  > > of what sources are involved. This is the same situation we
>  > > already have in Fedora with libguestfs, where we're building
>  > > a disk image inside Koji bundling various binaries. Or for
>  > > that matter, not really different from building cloud disk
>  > > images, or any other deliverable that bundles together some
>  > > binaries from other RPMs and spits out some kind of image
>  > > or archive.
>  > >
>  > > > Additionally, we should also consider
>  > > > https://fedoraproject.org/wiki/Features/DracutHostOnly and the size
>  > > > implications and why we moved to have kernel-core, kernel-modules, and
>  > > > kernel-modules-extra for cloud and different use cases.
>  > >
>  > > The UKI size for a VM should not be appreciably different from the
>  > > combination of the vmluinuz + locally generated initrd. The UKI
>  > > will contain a few more modules, as its initrd is built to cope
>  > > with Xen, VMware, HyperV + KVM[1], but this only adds a small amount
>  > > over a truly minimal initrd targetting 1 hypervisor. So I don't
>  > > expect the size of the UKI will be a problem.
>  > >
>  >
>  > You need to add VirtualBox too. That's an incredibly common platform
>  > for Fedora to run as a guest.
>
>  That's easy enough, what kmod is typically required for disks in
>  VirtualBox ?
>  
I'm not sure as I don't use VirtualBox myself, but Hans de Geode would
know, since he upstreamed the guest additions some time ago...


-- 
真実はいつも一つ！/ Always, there's only one truth!

[Iñaki Ucar]

From the point of view of the workstation experience (with a laptop),
I see no discussion on how this may impact hibernation. Currently, I
have secure boot disabled essentially because I want my laptop to
automatically hibernate (and recover from that state) whenever it is
suspended for a number of hours. I would like to see a path forward
here with secure boot enabled.

Thanks,
Iñaki

On Tue, 20 Dec 2022 at 16:22, Ben Cotton <bcotton(a)redhat.com&gt; wrote:
>