Vague proposal: ship prebuilt initramfs images

[Matthew Garrett]

Measured boot involves generating cryptographic measurements of boot 
components and configuration and using that to either control access to 
a local secret (in the case of sealing secrets to a TPM) or proving to 
another device (eg, a remote server or a local phone) what was booted. 
We're shipping most of the infrastructure to do this, but we're still 
left with a pretty fundamental problem - we need to know what the 
expected values are in order to know whether something's been tampered 
with or not.

For many components this isn't a huge problem (we build and distribute 
the files - users can extract them and calculate the appropriate 
measurements, and maybe long term we'll be able to ship the measurements 
in a digestable way), but our initramfs images are generated on the user 
system and include system-specific data. This makes it impractical to 
know the expected measurements in advance.

I've been thinking about ways to solve this for a while, and I'm coming 
to the conclusion that the best plan is probably to just ship pre-built 
initramfs images. I can think of three main reasons to want to use 
system-specific images:

1) They're smaller. By default we're already generating a generic image 
for rescue purposes, so disk space isn't the concern here - we're 
largely looking at losing boot speed. As machines have got faster this 
is probably not a huge deal.

2) They contain machine-specific configuration. Some of this can be 
passed on the kernel command line instead (eg, the machine ID), but we'd 
need answers for the rest. I can think of a couple of solutions:

 a) Stick the config in UEFI variables. It's small enough that we 
    wouldn't run out.
 b) Extend grub to read some config files and synthesise an initramfs 
    image for them. If we measure the paths that those images use then 
    we don't need to worry about the contents as long as the tools that 
    read the config can't be subverted via that configuration.

3) User customisation, such as including extra tooling. grub supports 
loading multiple initramfs images. Packages that right now install stuff 
in the initramfs could instead ship a prebuilt image that grub could 
append to the main initramfs. This would allow for things like 
overriding Plymouth themes, and we could ship the measurements for these 
pre-built images in order to allow them to be validated.

Any thoughts on this?

-- 
Matthew Garrett | mjg59(a)srcf.ucam.org

[Colin Walters]

On Mon, Jan 20, 2020, at 7:57 PM, Matthew Garrett wrote:

>  I've been thinking about ways to solve this for a while, and
> I'm coming 
>  to the conclusion that the best plan is probably to just ship pre-built 
>  initramfs images. 
rpm-ostree[1] (used for Fedora CoreOS and Silverblue and IoT among others) already works
this way by default.  (You can also enable client-side initramfs regeneration, which is
used e.g. when overlaying a custom kernel - part of being a hybrid image/package system)

For FCOS, signing the initramfs is already part of the plans; see:
https://github.com/projectatomic/rpm-ostree/issues/1883
and
https://github.com/ostreedev/ostree/pull/1959

[1] https://github.com/coreos/rpm-ostree/

[Dusty Mabe]

On 1/20/20 7:57 PM, Matthew Garrett wrote:

>  
>  I've been thinking about ways to solve this for a while, and I'm coming 
>  to the conclusion that the best plan is probably to just ship pre-built 
>  initramfs images. I can think of three main reasons to want to use 
>  system-specific images: 
For what it's worth, this is what is done for OSTree based systems (Fedora Atomic
Host, Fedora CoreOS, Fedora Silverblue). There is an option for the user to generate
an initramfs client side, but I don't think it's used very much at all.

I know in the case of OSTree based systems you are shipping more of an image than a
case like Fedora Server where you are assembling a new system from a set of defined
RPMs, but it's at least a data point.

Dusty

[Chris Murphy]

On Mon, Jan 20, 2020 at 5:58 PM Matthew Garrett <mjg59(a)srcf.ucam.org&gt; wrote:
>  I've been thinking about ways to solve this for a while, and I'm coming
>  to the conclusion that the best plan is probably to just ship pre-built
>  initramfs images. I can think of three main reasons to want to use
>  system-specific images:
>
>  1) They're smaller. By default we're already generating a generic image
>  for rescue purposes, so disk space isn't the concern here - we're
>  largely looking at losing boot speed. As machines have got faster this
>  is probably not a huge deal. 
What about the on-going cost: downloading ~80M initramfs for each kernel
update; systemd-analyze on NVMe says initrd time is  2.5s for a host-only
~25M initramfs. No-host-only initramfs is about 3x bigger. If the size to
time relationship is linear, that's a chunk of extra time. Maybe there's a
way to improve the read performance in the bootloader to compensate?

Some of the kernel and initramfs time hit comes from xz decompression. I
wonder if zstd can also compensate? Getting zstd module built into the
kernel to support zstd compressed initramfs is straightforward, but I
vaguely recall something extra is needed to support zstd compressed kernels.


>  2) They contain machine-specific configuration. Some of this can be
>  passed on the kernel command line instead (eg, the machine ID), but we'd
>  need answers for the rest. I can think of a couple of solutions:
>
>   a) Stick the config in UEFI variables. It's small enough that we
>      wouldn't run out.
>   b) Extend grub to read some config files and synthesise an initramfs
>      image for them. If we measure the paths that those images use then
>      we don't need to worry about the contents as long as the tools that
>      read the config can't be subverted via that configuration. 
Any expected hardware with TPM2 but without UEFI?

There's a systemd facility for extra boot params to contend with the arch
specific COMMAND_LINE_SIZE limit. I forget what it is, maybe it uses
efivars.


>  3) User customisation, such as including extra tooling. grub supports
>  loading multiple initramfs images. Packages that right now install stuff
>  in the initramfs could instead ship a prebuilt image that grub could
>  append to the main initramfs. This would allow for things like
>  overriding Plymouth themes, and we could ship the measurements for these
>  pre-built images in order to allow them to be validated. 
If the first initramfs contains systemd, could systemd start things in
parallel while unpacking a second initramfs?

I take it you've found some liability with measuring a locally produced
initramfs?


--
Chris Murphy

[Matthew Garrett]

On Tue, Jan 21, 2020 at 12:29:13AM -0700, Chris Murphy wrote:

>  What about the on-going cost: downloading ~80M initramfs for each
> kernel
>  update; systemd-analyze on NVMe says initrd time is  2.5s for a host-only
>  ~25M initramfs. No-host-only initramfs is about 3x bigger. If the size to
>  time relationship is linear, that's a chunk of extra time. Maybe there's a
>  way to improve the read performance in the bootloader to compensate? 
I don't see this as an obligatory choice - users should still be free to 
generate images locally instead. I'm certainly willing to do the work on 
performance calculations if there's no absolute objection to the idea.

>  Any expected hardware with TPM2 but without UEFI? 
Not on x86 - the PC client spec for TPM2 only covers UEFI.

>  If the first initramfs contains systemd, could systemd start things
> in
>  parallel while unpacking a second initramfs? 
The straightforward implementation involves the kernel unpacking all the 
initramfs archives before it starts init. In theory we could add 
functionality to the kernel to expose additional archives to userland 
rather than have the kernel unpack them, but that's not currently 
achievable - and it's one of those situations where we'd need to be very 
careful about ensuring there's no potential for races.

>  I take it you've found some liability with measuring a locally
> produced
>  initramfs? 
You'd need a trusted mechanism for passing the new initramfs 
measurements to whatever's verifying the measurements. That's not easy.

-- 
Matthew Garrett | mjg59(a)srcf.ucam.org

[Petr Pisar]

On Tue, Jan 21, 2020 at 12:57:50AM +0000, Matthew Garrett wrote:
>  2) They contain machine-specific configuration. Some of this can be 
>  passed on the kernel command line instead (eg, the machine ID), but we'd 
>  need answers for the rest. I can think of a couple of solutions:
>  
>   a) Stick the config in UEFI variables. It's small enough that we 
>      wouldn't run out.
>   b) Extend grub to read some config files and synthesise an initramfs 
>      image for them. If we measure the paths that those images use then 
>      we don't need to worry about the contents as long as the tools that 
>      read the config can't be subverted via that configuration.
>  
>  3) User customisation, such as including extra tooling. grub supports 
>  loading multiple initramfs images. Packages that right now install stuff 
>  in the initramfs could instead ship a prebuilt image that grub could 
>  append to the main initramfs. This would allow for things like 
>  overriding Plymouth themes, and we could ship the measurements for these 
>  pre-built images in order to allow them to be validated.
>  
>  Any thoughts on this?
>   
Properly measured system must measure all inputs. If you move the varying
bits from initramfs to another file, a boot loader will have to measure that
another file. At the end that's exactly what GRUB2 does. It measures any
loaded file.

In my opinon, your proposal does not solve the problem. It actually makes
things worse because the booted code would become bigger and probably slower.

-- Petr

[Matthew Garrett]

On Tue, Jan 21, 2020 at 09:09:16AM +0100, Petr Pisar wrote:
>  On Tue, Jan 21, 2020 at 12:57:50AM +0000, Matthew Garrett wrote:
>  > Any thoughts on this?
>  > 
>  Properly measured system must measure all inputs. If you move the varying
>  bits from initramfs to another file, a boot loader will have to measure that
>  another file. At the end that's exactly what GRUB2 does. It measures any
>  loaded file. 
Yes, I wrote that code. The point of measurements is to be able to make 
a policy determination. If the contents of a file aren't security 
relevant then you don't care about its contents, but you do want to 
ensure that it ends up in a position where it can't interfere with any 
other security relevant codepath. In that scenario you want to measure 
the path, not the contents (or, rather, you can measure both and the 
policy agent can ignore the contents)

>  In my opinon, your proposal does not solve the problem. It actually
> makes
>  things worse because the booted code would become bigger and probably slower. 
I'm not clear how that follows.

-- 
Matthew Garrett | mjg59(a)srcf.ucam.org

[Hans de Goede]

Hi,

On 21-01-2020 01:57, Matthew Garrett wrote:
>  Measured boot involves generating cryptographic measurements of boot
>  components and configuration and using that to either control access to
>  a local secret (in the case of sealing secrets to a TPM) or proving to
>  another device (eg, a remote server or a local phone) what was booted.
>  We're shipping most of the infrastructure to do this, but we're still
>  left with a pretty fundamental problem - we need to know what the
>  expected values are in order to know whether something's been tampered
>  with or not.
>  
>  For many components this isn't a huge problem (we build and distribute
>  the files - users can extract them and calculate the appropriate
>  measurements, and maybe long term we'll be able to ship the measurements
>  in a digestable way), but our initramfs images are generated on the user
>  system and include system-specific data. This makes it impractical to
>  know the expected measurements in advance.
>  
>  I've been thinking about ways to solve this for a while, and I'm coming
>  to the conclusion that the best plan is probably to just ship pre-built
>  initramfs images. I can think of three main reasons to want to use
>  system-specific images:
>  
>  1) They're smaller. By default we're already generating a generic image
>  for rescue purposes, so disk space isn't the concern here - we're
>  largely looking at losing boot speed. As machines have got faster this
>  is probably not a huge deal. 
Using pre-generated initrds is something which comes up from time to time
and I do believe it is something which we want to move towards.

For the size problem one approach which I think is worthwhile at least
for the x86/workstation case is making one initrd with the most common
stuff in there (say i915,nouveau and amd GPU drivers + NVME and AHCI
disk support, maybe some eMMC support) and another which contains
"everything" and then rework the dracut host-only code so that we can
leverage it to figure out which variant we want.

>  2) They contain machine-specific configuration. Some of this can be
>  passed on the kernel command line instead (eg, the machine ID), but we'd
>  need answers for the rest. I can think of a couple of solutions:
>  
>    a) Stick the config in UEFI variables. It's small enough that we
>       wouldn't run out.
>    b) Extend grub to read some config files and synthesise an initramfs
>       image for them. If we measure the paths that those images use then
>       we don't need to worry about the contents as long as the tools that
>       read the config can't be subverted via that configuration. 
We still have quite a lot of users using classic BIOS boot (and early
x86_64 hw does not do UEFI at all) so a) is not really an option as
I would like to see a common approach here for both UEFI + classic
BIOS, moreover we also have other architectures and if we change this
we should really change it everywhere rather then have multiple code
paths to support.

So b it is, I actually suggested b. on a discussion about how to set
the keyboard map for the initrd (for full disk encryption) which currently
is a problem for rpm-ostree / SilverBlue which uses pre-generated
initrds with no custom config in there.

>  3) User customisation, such as including extra tooling. grub
> supports
>  loading multiple initramfs images. Packages that right now install stuff
>  in the initramfs could instead ship a prebuilt image that grub could
>  append to the main initramfs. This would allow for things like
>  overriding Plymouth themes, and we could ship the measurements for these
>  pre-built images in order to allow them to be validated. 
Ack for using appended initrds for this, this also mixes well with my
preferred solution for 2.

Regards,

Hans

[Matthew Garrett]

On Tue, Jan 21, 2020 at 11:25:29AM +0100, Hans de Goede wrote:

>  For the size problem one approach which I think is worthwhile at
> least
>  for the x86/workstation case is making one initrd with the most common
>  stuff in there (say i915,nouveau and amd GPU drivers + NVME and AHCI
>  disk support, maybe some eMMC support) and another which contains
>  "everything" and then rework the dracut host-only code so that we can
>  leverage it to figure out which variant we want. 
Yeah, the important thing is that we have known values - there's no real 
problem with there being a large number of known values, so shipping 
multiple images (potentially to be combined if necessary) is entirely 
reasonable from that perspective.

>  So b it is, I actually suggested b. on a discussion about how to set
>  the keyboard map for the initrd (for full disk encryption) which currently
>  is a problem for rpm-ostree / SilverBlue which uses pre-generated
>  initrds with no custom config in there. 
Ok. I'll look into adding support to grub for reading some files and 
assembling them into an image. My rough idea here would be something 
like:

configinitrd file1 file2 file3
initrd initramfs1.img initramfs2.img CONFIG

where the first command generates the image and the second command 
causes it to be placed at the end of the final cpio blob?

-- 
Matthew Garrett | mjg59(a)srcf.ucam.org

[Matthew Garrett]

On Tue, Jan 21, 2020 at 12:43:47PM +0000, Matthew Garrett wrote:

>  configinitrd file1 file2 file3
>  initrd initramfs1.img initramfs2.img CONFIG 
Huh - it seems like grub may already support this? It looks like:

initrd initramfs.img newc:/etc/crypttab:/boot/crypttab

will add /boot/crypttab to the initramfs as /etc/crypttab. Haven't 
tested this yet, though, and it's entirely undocumented.

-- 
Matthew Garrett | mjg59(a)srcf.ucam.org

[Hans de Goede]

Hi,

On 22-01-2020 04:20, Matthew Garrett wrote:
>  On Tue, Jan 21, 2020 at 12:43:47PM +0000, Matthew Garrett wrote:
>  
> > configinitrd file1 file2 file3
> > initrd initramfs1.img initramfs2.img CONFIG
>  
>  Huh - it seems like grub may already support this? It looks like:
>  
>  initrd initramfs.img newc:/etc/crypttab:/boot/crypttab
>  
>  will add /boot/crypttab to the initramfs as /etc/crypttab. Haven't
>  tested this yet, though, and it's entirely undocumented. 
Oh, that is quite interesting, then just copying /etc/vconsole.conf
to /boot and using the above might be enough for the keymap issue.

Regards,

Hans

[Brian C. Lane]

On Tue, Jan 21, 2020 at 12:57:50AM +0000, Matthew Garrett wrote:
>   a) Stick the config in UEFI variables. It's small enough that we
>
>      wouldn't run out.
>   b) Extend grub to read some config files and synthesise an initramfs 
>      image for them. If we measure the paths that those images use then 
>      we don't need to worry about the contents as long as the tools that 
>      read the config can't be subverted via that configuration. 
I think a problem with (a) is that it won't work for disk images if
part of it is tied to the hardware, everything needs to be on the
image. So (b) has my vote for now.

-- 
Brian C. Lane (PST8PDT) - weldr.io - lorax - parted - pykickstart