On Tue, Dec 20, 2022 at 2:02 PM Daniel P. Berrangé <berrange@redhat.com> wrote:
>
> On Tue, Dec 20, 2022 at 11:28:48AM -0500, Neal Gompa wrote:
> > On Tue, Dec 20, 2022 at 10:22 AM Ben Cotton <bcotton@redhat.com> wrote:
> > >
> > > https://fedoraproject.org/wiki/Changes/Unified_Kernel_Support_Phase_1
> > >
> > > This document represents a proposed Change. As part of the Changes
> > > process, proposals are publicly announced in order to receive
> > > community feedback. This proposal will only be implemented if approved
> > > by the Fedora Engineering Steering Committee.
> > >
> > >
> > > == Summary ==
> > > Add support for unified kernels images to Fedora.
> > >
> > > == Owner ==
> > > * Name: [[User:kraxel| Gerd Hoffmann]]
> > > * Email: kraxel@redhat.com
> > >
> > >
> > > == Detailed Description ==
> > > The goal is to move away from initrd images being generated on the
> > > installed machine.  They are generated while building the kernel
> > > package instead, then shipped as part of a unified kernel image.
> > >
> > > A unified kernel image is an all-in-one efi binary containing kernel,
> > > initrd, cmdline and signature.  The secure boot signature covers
> > > everything, specifically the initrd is included which is not the case
> > > when the initrd gets loaded as separate file from /boot.
> > >
> > > Main motivation for this move is to make the distro more robust and more secure.
> > >
> > > Switching the whole distro over to unified kernels quickly is not
> > > realistic though.  Too many features are depending on the current
> > > workflow with a host-specific initrd (and host-specific kernel command
> > > line), which is fundamentally incompatible with unified kernels where
> > > everybody will have the same initrd and command line. Thats why there
> > > is 'Phase 1' in title, so we can have more Phases in future releases
> > > 😃
> > >
> > > A host-specific initrd / command line is needed today for:
> > >
> > > * features needing optional dracut modules (initrd rebuild needed to
> > > enable them).
> > > * configuration / secrets baked into the initrd (booting from iscsi
> > > for example).
> > > * configuration being specified on the kernel command line.
> > > ** root filesystem being the most important one.
> > > [https://systemd.io/DISCOVERABLE_PARTITIONS/ Discoverable partitions]
> > > allow to remove this.
> > >
> > > Phase 1 goals (high priority):
> > >
> > > * Ship a unified kernel image as (optional) kernel sub-rpm.  Users can
> > > opt-in to use that kernel by installing the sub-rpm.  Initial focus is
> > > on booting virtual machines where we have a relatively small and well
> > > defined set of drivers / features needed.  Supporting modern physical
> > > machines with standard setup (i.e. boot from local sata/nvme storage)
> > > too should be easy.
> > > * Update kernel install scripts so unified kernels are installed and
> > > updated properly.
> > > * Add bootloader support for unified kernel images.  Add
> > > [https://systemd.io/BOOT_LOADER_SPECIFICATION/#type-2-efi-unified-kernel-images
> > > unified kernel bls support] to grub2, or support using systemd-boot,
> > > or both.
> > >
> > > Phase 1 goals (lower priority, might move to Phase 2):
> > >
> > > * Add proper discoverable partitions support to installers (anaconda,
> > > image builder, ...).
> > > ** Temporary workaround possible: set types using sfdisk in %post script.
> > > ** When using btrfs: configure 'root' subvolume as default volume.
> > > * Add proper systemd-boot support to installers.
> > > ** Temporary workaround possible: run 'bootctl install' in %post script.
> > > * Better measurement and remote attestation support.
> > > ** store kernel + initrd hashes somewhere (kernel-hashes.rpm ?) to
> > > allow pre-calculate TPM PCR values.
> > > ** avoid using grub2 (measures every config file line executed which
> > > is next to impossible to pre-calculate).
> > > * Switch cloud images to use unified kernels.
> > >
> > > Phase 2/3 goals (longer-term stuff which is not realistic to complete for F38).
> > >
> > > * Move away from using the kernel command line for configuration.
> > > * Move away from storing secrets in the initrd.
> > > * Handle dracut optional modules in a different way.
>
> snip
>
> >
> > I think UKIs are fundamentally flawed and are an idea that came out of
> > a group that doesn't really interact with real users enough to
> > understand how much of a problem they actually are. I realize that
> > this Change is only about VMs, but since it explicitly talks about it
> > being "phase 1", the implication is that future Changes are coming to
> > switch fully over. Consequently, I'm going to provide much more
> > holistic feedback instead of just nitpicking on this case.
>
> The future "phase 2/3" goal are mentioned above, and don't
> mention fully switching to UKIs.
>
> I don't anticipate such a switch being realistic any time
> in the forseeable future. There are too many unknowns, in
> addition to some points you mentioned, to even consider
> that right now. The in place upgrade story gogin from
> locally generated initrd, with arbitrary user chosen
> content to a UKI doesn't even bare thinking about. I
> don't think that's even possible to achieve in a seemless
> way given the infinite number of variations. IOW, I would
> anticipate the current kernel+initrd scheme being around
> indefinitely, in parallel with any UKI support.
>

Forgive me for being pessimistic here, because Change proposals like
this are not written in such a way. You also bundled in bootloader
changes and PCR/TPM stuff too. All of which are irrelevant, not
useful, and don't actually work for Linux users.

> > In the Fedora case, things are simpler right up until we hit graphics
> > drivers. This is also a problem for VMs too, because GPU passthrough
> > is a common case for scientific and gaming workloads. As long as the
> > NVIDIA driver remains dominant in Linux, UKIs cannot work because by
> > design you cannot load anything that isn't part of the kernel image.
> > For bare metal, we *need* these drivers in early boot, though. And
> > that's another problem: no third-party early boot drivers. Even if you
> > solve the signing issue, you need to introduce some kind of two-stage
> > OS boot process so we can bring up the bare minimum to load a second
> > image containing all the remaining drivers. And at that point, you've
> > defeated the purpose of UKIs. I've heard from some people that system
> > extensions (sysexts) would be a way to solve this, and maybe it is.
>
> Yes, system extensions are one mechanism for making the initrd
> content more flexible when using UKIs. There would be one base
> layer definiing the 90% common case, and a number of add-ons
> that can cope with niche use cases. This avoids the core UKI
> having to be huge and ship with every possible feature present.
>
> This is something that might be considered at a later phase,
> but is not a priority. What's proposed in phase 1 is sufficient
> to cope with the cloud VM common case. While device passthrough
> may be common in some industries/domains usage, it is not the
> common case for cloud computing in general.
>
> > But again, we've eliminated the value of UKIs by doing so.
>
> That is not correct. There are a number of benefits of UKIs.
>
> The most critical is that the initrd content and cmdline is
> covered by the SecureBoot signature. This remains true even
> with system extensions, as such extensions would be signed
> too. They do not neccessarily need to be signed by the OS
> vendor, they could use a 3rd party SecureBoot signing key,
> or the users' own key. That is TBD and not something we're
> actively considering - its not even mentioned in phase 2/3
> ideas in the change proposal.
>

Secure Boot keys are flawed. There are many documented problems with
relying on Secure Boot keys in the real world. Redesign to not use
them, please.

> A further benefit of UKIs is supportability, since we know
> exactly what content is in the initrd that user is booting.
> With locally generated initrds there are an arbitrary
> number of combinations and many ways for the initrd creation
> process to go wrong, which are hard for Fedora maintainers
> to understand and debug.
>
> > Speaking more broadly, there are also problems that will be introduced
> > as this trickles down from Fedora into prominent downstreams (assuming
> > this is accepted). In the RHEL case, you've basically broken driver
> > disks completely. In the UKI model, there's no way to incorporate
> > early boot storage, network, and graphics drivers. This is
> > *incredibly* common for RHEL administrators because there's a general
> > acceptance of proprietary kernel modules from vendors these days. Even
> > ignoring those, Red Hat's kernel feature support mindset is completely
> > incompatible with UKIs, because RHEL does default-off rather than
> > Fedora's default-on model for kernel features. We could debate until
> > the cows come home on whether it's right or wrong, but their current
> > mindset essentially means tons of common hardware becomes unsupported
> > quite regularly. The ELRepo project is popular among RHEL folks
> > because it restores those drivers and makes it possible to use RHEL on
> > those machines through a combination of driver disks and kernel module
> > packages.
>
> Yes, this desire for UKIs in Fedora is matched by a desire to
> ship the same UKIs in RHEL too.
>
> No one is proposing to take away the existing ability to use
> locally generated initrds, either in Fedora or RHEL. Everything
> that is possible todo today will remain possible indefinitely.
> UKIs are adding a new mechanism in parallel, in order to solve
> a number of use cases that are not possible today, due to
> unsigned content being part of the boot chain when SecureBoot
> is active.
>
> > The crux of the problem here is *signing*. All of this is tied up in
> > Secure Boot and TPM, which is the wrong place to actually handle this.
> > In other operating systems (notably Windows), Secure Boot certificates
> > are not used for blocking or enabling kernel-level drivers. Instead,
> > there's a kernel-level certificate store that is used to validate and
> > enable drivers, allowing everything to be managed in a hardware
> > platform agnostic way. I've also yet to hear of a decent reason for
> > TPM measurements too. Block or filesystem verity provides similar
> > guarantees to provide tamper resistance and are both much easier to
> > debug than TPM interfaces. I am not convinced that you are providing
> > security or reliability with this and the trade-offs are *terrible*
> > for regular users.
>
> The immediate need for UKIs is indeed related to SecureBoot and
> TPMs. These are a core technology foundation of the confidential
> virtual machine stack. On Azure today, if you request an Ubuntu
> confidential VM, Azure will pre-encrypt the root filesystem and
> seal the LUKS key against predicted TPM PCR values. It guarantees
> that the root disk can only be decrypted by the specific VM
> instance that is requested, when it is running in SecureBoot
> mode with the expected measurments on AMD SEV-SNP confidential
> hardware. The Ubuntu image in Azure already uses UKIs, and boots
> them directly from shim, with no bootloader involved. The usage
> of SecureBoot and TPMs is all transparent to the user, since all
> the integration is handled by the OS vendor on their behalf.
>

Well, how is that relevant to Fedora? We don't have Azure images, and
Red Hat still blocks us from meaningfully existing in that ecosystem,
which is why we don't have official WSL images or Azure images.

I'll reiterate that we should not rely on SB signing for this.

> > With my FESCo hat on, I'm uneasy about this change. With my "Fedora
> > user and advocate" hat on, I think that the UAPI group has failed to
> > provide something useful to the Linux world here and I would be
> > extremely apprehensive about Fedora adopting any portion of this
> > stuff.
>
> As mentioned above, UKIs have provided a means to close the
> SecureBoot hole with unsigned initrd content that has existed
> in more or less every mainstream Linux distro. That is a
> clearly useful outcome, regardless of whether Fedora is interested
> in any other aspect of what that UAPI groups is proposing.
>
> In proposing the UKI support for Fedora, we're not coming at this
> as representatives of the UAPI group or its vision. We're trying
> to solve the problem of having a fully verified secureboot chain
> for VMs with no unsigned content, and to be able to use this to
> support confidential VMs with disk encryption sealed to TPMs, as
> required by Azure today, and likely KVM in future.

Yeah, I seriously doubt this. Linux's model for supporting
confidential computing is not user-friendly, so I expect low adoption
and resistance once the flaws become apparent to would-be users.