[OS-BUILD PATCHv12 0/2] redhat/kernel.spec: add uki_addons to create UKI kernel cmdline addons

[Emanuele Giuseppe Esposito (via Email Bridge)]

From: Emanuele Giuseppe Esposito on gitlab.com
Merge Request: https://gitlab.com/cki-project/kernel-ark/-/merge_requests/2917

We want to enable kernel.spec to optionally ship UKI addons defined in a
common config file in redhat folder.
The folder redhat/addons will contain all addons configs specifying the
UKI kernel cmdline addons to be created in the next build. An addon
config is simply a .addon plain text file, where any line
is taken as kernel cmdline, except for the ones starting with '#',
which will be automatically ignored.

redhat/scripts/uki_addons.py will take care of parsing all configs and
folders in redhat/addons and call 'ukify' to create the actual addons.
The output addon filename will be a concatenation of all folders in
redhat/addons that are part of the addon config path.

For example, an addon config redhat/addons/1/2/3/test.addon will result
in generating redhat/addons_output/1-2-3-test.addon.efi.

It is also possible to add .sbat to all the generated addons, by
populating redhat/addons/sbat/sbat.conf. Syntax is same as the addons
config.

At build time, Makefile will create a tar.gz archive (addons.tar.gz)
containing all the files in redhat/addons. It will then passed to the
kernel specfile that will extract the addons from it and generate the
UKI kernel cmdline addons.

As an example of this feature, add the fips addon to optionally enable fips
(https://issues.redhat.com/browse/RHEL-23049).

---
 redhat/addons/virt/common/fips.addon |    2 +
 redhat/scripts/uki_addons.py         |  157 +++++++++++++++++++++++++++++++++++
 redhat/Makefile                      |    3 +
 redhat/kernel.spec.template          |   44 +++++++++
 4 files changed, 206 insertions(+), 0 deletions(-)

[OS-BUILD PATCHv12 1/2] redhat/kernel.spec: add uki_addons to create UKI kernel cmdline addons

[Emanuele Giuseppe Esposito (via Email Bridge)]

From: Emanuele Giuseppe Esposito <eesposit(a)redhat.com&gt;

redhat/kernel.spec: add uki_addons to create UKI kernel cmdline addons

Upstream Status: RHEL-Only

The folder redhat/addons will contain all addons configs specifying the
UKI kernel cmdline addons to be created in the next build. An addon
config is simply a .addon plain text file, where any line
is taken as kernel cmdline, except for the ones starting with '#',
which will be automatically ignored.

redhat/scripts/uki_addons.py will take care of parsing all configs and
folders in redhat/addons and call 'ukify' to create the actual addons.
The output addon filename will be a concatenation of all folders in
redhat/addons that are part of the addon config path.

The folder hierarchy inside of redhat/addons is similar to
redhat/configs: $UKI_NAME/%distro/%arch. The "common" folder
provides the default at uki, distro and arch level, and can be
overridden by specifying the same addon name in the specific
uki - distro - arch folder.

It is also possible to add .sbat to all the generated addons, by
putting it in the specified folder under sbat/sbat.conf.
Syntax is same as the addons config.

At build time, Makefile will create a tar.gz archive (addons.tar.gz)
containing all the files in redhat/addons/. It will then passed to the
kernel specfile that will extract the addons from it and generate the
UKI kernel cmdline addons.

Signed-off-by: Emanuele Giuseppe Esposito <eesposit(a)redhat.com&gt;

diff --git a/redhat/Makefile b/redhat/Makefile
index blahblah..blahblah 100644
--- a/redhat/Makefile
+++ b/redhat/Makefile
@@ -685,6 +685,7 @@ sources-rh: $(TARBALL) $(KABI_TARBALL) $(KABIDW_TARBALL)
generate-testpatch-tmp
 	@sed -e "s/%%SPECKVERSION%%/$(SPECKVERSION)/" \
 		-e "s/%%SPECKPATCHLEVEL%%/$(SPECKPATCHLEVEL)/" \
 		scripts/gating/rpminspect.yaml > $(SOURCES)/rpminspect.yaml
+	@tar -czf addons.tar.gz addons
 	@cp scripts/kernel-tools/kvm_stat.logrotate \
 		keys/rhel*.x509 \
 		keys/nvidia*.x509 \
@@ -697,6 +698,7 @@ sources-rh: $(TARBALL) $(KABI_TARBALL) $(KABIDW_TARBALL)
generate-testpatch-tmp
 		scripts/mod/mod-partner.list \
 		scripts/mod/mod-sign.sh \
 		scripts/mod/mod-kvm.list \
+		scripts/uki_addons.py \
 		configs/flavors \
 		configs/generate_all_configs.sh \
 		configs/merge.py \
@@ -705,6 +707,7 @@ sources-rh: $(TARBALL) $(KABI_TARBALL) $(KABIDW_TARBALL)
generate-testpatch-tmp
 		README.rst \
 		kernel-local \
 		dracut-virt.conf \
+		addons.tar.gz \
 		$(SOURCES)/
 	@cat $$(ls -1 $(SPECPACKAGE_NAME).changelog-* | sort -t '.' -k 3 -n -r) \
>  $(SOURCES)/kernel.changelog 
diff --git
a/redhat/kernel.spec.template b/redhat/kernel.spec.template
index blahblah..blahblah 100644
--- a/redhat/kernel.spec.template
+++ b/redhat/kernel.spec.template
@@ -788,6 +788,8 @@ BuildRequires: binutils
 BuildRequires: lvm2
 BuildRequires: systemd-boot-unsigned
 # For systemd-stub and systemd-pcrphase
+BuildRequires: systemd-ukify
+# For UKI kernel cmdline addons
 BuildRequires: systemd-udev >= 252-1
 # For TPM operations in UKI initramfs
 BuildRequires: tpm2-tools
@@ -929,6 +931,9 @@ Source86: dracut-virt.conf
 
 Source87: flavors
 
+Source151: uki_addons.py
+Source152: addons.tar.gz
+
 Source100: rheldup3.x509
 Source101: rhelkpatch1.x509
 Source102: nvidiagpuoot001.x509
@@ -1550,6 +1555,11 @@ Provides: kernel-%{?1:%{1}-}uname-r = %{KVERREL}%{uname_suffix
%{?1:+%{1}}}\
 Requires: kernel%{?1:-%{1}}-modules-core-uname-r = %{KVERREL}%{uname_suffix
%{?1:+%{1}}}\
 Requires(pre): %{kernel_prereq}\
 Requires(pre): systemd >= 254-1\
+%package %{?1:%{1}-}uki-virt-addons\
+Summary: %{variant_summary} unified kernel image addons for virtual machines\
+Provides: installonlypkg(kernel)\
+Requires: kernel%{?1:-%{1}}-uki-virt = %{specrpmversion}-%{release}\
+Requires(pre): systemd >= 254-1\
 %endif\
 %endif\
 %if %{with_gcov}\
@@ -1689,31 +1699,49 @@ input and output, etc.
 %if %{with_up} && %{with_debug} && %{with_efiuki}
 %description debug-uki-virt
 Prebuilt debug unified kernel image for virtual machines.
+
+%description debug-uki-virt-addons
+Prebuilt debug unified kernel image addons for virtual machines.
 %endif
 
 %if %{with_up_base} && %{with_efiuki}
 %description uki-virt
 Prebuilt default unified kernel image for virtual machines.
+
+%description uki-virt-addons
+Prebuilt default unified kernel image addons for virtual machines.
 %endif
 
 %if %{with_arm64_16k} && %{with_debug} && %{with_efiuki}
 %description 16k-debug-uki-virt
 Prebuilt 16k debug unified kernel image for virtual machines.
+
+%description 16k-debug-uki-virt-addons
+Prebuilt 16k debug unified kernel image addons for virtual machines.
 %endif
 
 %if %{with_arm64_16k_base} && %{with_efiuki}
 %description 16k-uki-virt
 Prebuilt 16k unified kernel image for virtual machines.
+
+%description 16k-uki-virt-addons
+Prebuilt 16k unified kernel image addons for virtual machines.
 %endif
 
 %if %{with_arm64_64k} && %{with_debug} && %{with_efiuki}
 %description 64k-debug-uki-virt
 Prebuilt 64k debug unified kernel image for virtual machines.
+
+%description 64k-debug-uki-virt-addons
+Prebuilt 64k debug unified kernel image addons for virtual machines.
 %endif
 
 %if %{with_arm64_64k_base} && %{with_efiuki}
 %description 64k-uki-virt
 Prebuilt 64k unified kernel image for virtual machines.
+
+%description 64k-uki-virt-addons
+Prebuilt 64k unified kernel image addons for virtual machines.
 %endif
 
 %if %{with_ipaclones}
@@ -2629,6 +2657,12 @@ BuildKernel() {
            --kernel-cmdline 'console=tty0 console=ttyS0' \
 	   $KernelUnifiedImage
 
+  tar -xvf %{SOURCE152} -C $KernelUnifiedImageDir
+  KernelAddonsDir="$KernelUnifiedImageDir/addons"
+  KernelAddonsDirOut="$RPM_BUILD_ROOT/usr/lib/linux/extra.d"
+  mkdir -p $KernelAddonsDirOut
+  python3 %{SOURCE151} $KernelAddonsDir $KernelAddonsDirOut virt %{primary_target}
%{_target_cpu}
+
 %if %{signkernel}
 	%{log_msg "Sign the EFI UKI kernel"}
 	%pesign -s -i $KernelUnifiedImage -o $KernelUnifiedImage.tmp -a %{secureboot_ca_0} -c
%{secureboot_key_0} -n %{pesign_name_0}
@@ -2641,9 +2675,17 @@ BuildKernel() {
     	fi
     	mv $KernelUnifiedImage.signed $KernelUnifiedImage
 
+      for addon in "$KernelAddonsDirOut"/*; do
+        %pesign -s -i $addon -o $addon.signed -a %{secureboot_ca_1} -c
%{secureboot_key_1} -n %{pesign_name_1}
+        rm -f $addon
+        mv $addon.signed $addon
+      done
+
 # signkernel
 %endif
 
+  rm -rf $KernelUnifiedImageDir/addons
+
 	pushd $RPM_BUILD_ROOT
 
 # with_efiuki
@@ -3883,6 +3925,8 @@ fi\
 /lib/modules/%{KVERREL}%{?3:+%{3}}/modules.builtin*\
 %attr(0644, root, root)
/lib/modules/%{KVERREL}%{?3:+%{3}}/%{?-k:%{-k*}}%{!?-k:vmlinuz}-virt.efi\
 %ghost
/%{image_install_path}/efi/EFI/Linux/%{?-k:%{-k*}}%{!?-k:*}-%{KVERREL}%{?3:+%{3}}.efi\
+%{expand:%%files %{?3:%{3}-}uki-virt-addons}\
+/usr/lib/linux/extra.d/*.addon.efi\
 %endif\
 %endif\
 %if %{?3:1} %{!?3:0}\
diff --git a/redhat/scripts/uki_addons.py b/redhat/scripts/uki_addons.py
new file mode 100644
index blahblah..blahblah 100644
--- /dev/null
+++ b/redhat/scripts/uki_addons.py
@@ -0,0 +1,157 @@
+#!/usr/bin/env python3
+#
+# This script recursively inspects a given uki directory, and creates an addon
+# for each file found.
+#
+# Usage: python uki_addons.py input_dir out_dir uki distro arch
+#
+# This tool requires the systemd-ukify and systemd-boot packages.
+#
+# Addon file
+#-----------
+# Each addon terminates with .addon
+# Each addon contains only two types of lines:
+# Lines beginning with '#' are description and thus ignored
+# All other lines are command line to be added.
+# The name of the end resulting addon is taken from the folder hierarchy.
+# For example, and addon in virt/rhel/x86_64/hello.addon will result in
+# an UKI addon file generated in out_dir called hello-virt.rhel.x86_64.addon.efi
+#
+# The common folder, present in any folder in redhat/addons (except the leaf folders)
+# is used as place for default addons when the same addon is not defined deep
+# in the hierarchy. For example, if we define test.addon (text: test1) in
+# redhat/addons/common and another test.addon (text: test2) in
+# redhat/addons/virt/common, any other uki except virt will have a test.addon.efi
+# with text "test1", and virt will have a test.addon.efi with
"test2"
+#
+# sbat.conf
+#----------
+# This file is containing the sbat string for *all* addons being created.
+# This file is optional, but when used has to be put in out_dir/sbat/sbat.conf.
+# It follows the same syntax as the addon files, meaning '#' is comment and
+# the rest is taken as sbat string and feed to ukify.
+#
+#
+# How to extend the script and kernel.spec with a new arch or uki or distro
+#--------------------------------------------------------------------------
+# A new distro has to be added by creating the folder in redhat/addons.
+# See above to how the directory hierarchy in redhat/addons is expected to be.
+# After that, if the distro is a different arch from the one already supported,
+# one needs to extend the %define with_efiuki in kernel.spec.template.
+# If a new UKI has to be created with a different name from the existing ones,
+# the logic to create the addons and call this script has to be implemented too
+# in kernel.spec.template. As an example, see how the 'virt' UKI addons are
+# created.
+
+import os
+import sys
+import collections
+import subprocess
+
+
+UKIFY_PATH = '/usr/lib/systemd/ukify'
+SBAT_RELATIVE_PATH = 'sbat/sbat.conf'
+
+def usage(err):
+    print(f'Usage: {os.path.basename(__file__)} input_dir output_dir uki distro
arch')
+    print(f'Error:{err}')
+    sys.exit(1)
+
+def check_clean_arguments(in_dir, out_dir):
+    # Remove end '/'
+    if in_dir[-1:] == '/':
+        in_dir = in_dir[:-1]
+    if out_dir[-1:] == '/':
+        out_dir = out_dir[:-1]
+    if not os.path.isdir(in_dir):
+        usage(f'in_dir {in_dir} is not a dir, or does not exist!')
+    if not os.path.isdir(out_dir):
+        usage(f'out_dir_dir {out_dir} is not a dir, or does not exist!')
+    return in_dir, out_dir
+
+UKICmdlineAddon = collections.namedtuple('UKICmdlineAddon', ['name',
'cmdline'])
+uki_addons_list = []
+uki_addons = {}
+addon_sbat_string = None
+
+def parse_addon_conf(path, rstrip=True):
+    with open(path, 'r') as addons:
+        lines = addons.readlines()
+    cmdline = ''
+    for l in lines:
+        l = l.lstrip()
+        if not l:
+            continue
+        if l[0] == '#':
+            continue
+        if rstrip:
+            l = l.rstrip() + ' '
+        cmdline += l
+    return cmdline
+
+def parse_all_addons_folder(in_dir):
+    global addon_sbat_string
+    sbat_loc = os.path.join(in_dir, SBAT_RELATIVE_PATH)
+
+    for el in os.listdir(in_dir):
+        el_path = os.path.join(in_dir, el)
+        # file found: save its path, replacing the old one
+        if os.path.isfile(el_path):
+            if el.endswith('.addon'):
+                uki_addons[el] = el_path
+
+    if os.path.isfile(sbat_loc):
+        # override sbat with the most specific one found
+        addon_sbat_string = parse_addon_conf(sbat_loc, rstrip=False)
+
+def recursively_find_addons(in_dir, i, folder_list):
+    # end of recursion, leaf directory. Search all addons here
+    if i == len(folder_list):
+        parse_all_addons_folder(in_dir)
+        return
+
+    # first, check for common folder
+    common_dir = os.path.join(in_dir, 'common')
+    if os.path.exists(common_dir):
+        parse_all_addons_folder(common_dir)
+
+    # second, check if there is a match with the searched folder
+    folder_dir = os.path.join(in_dir, folder_list[i])
+    if os.path.exists(folder_dir):
+        recursively_find_addons(folder_dir, i+1, folder_list)
+
+def parse_in_dir(in_dir, uki_name, distro, arch):
+    recursively_find_addons(in_dir, 0, [uki_name, distro, arch])
+
+    for addon_name, addon_path in uki_addons.items():
+        addon_name = addon_name.replace(".addon","")
+        addon_full_name = f'{addon_name}-{uki_name}.{distro}.{arch}.addon.efi'
+        cmdline = parse_addon_conf(addon_path)
+        if cmdline:
+            uki_addons_list.append(UKICmdlineAddon(addon_full_name, cmdline))
+
+def create_addons(out_dir):
+    for uki_addon in uki_addons_list:
+        out_path = os.path.join(out_dir, uki_addon.name)
+        cmd = [
+            f'{UKIFY_PATH}', 'build',
+            f'--cmdline="{uki_addon.cmdline}"',
+            f'--output={out_path}']
+        if addon_sbat_string:
+            cmd.append('--sbat="' + addon_sbat_string.rstrip()
+'"')
+        subprocess.check_call(cmd, text=True)
+
+if __name__ == "__main__":
+    argc = len(sys.argv) - 1
+    if argc != 5:
+        usage('too few or too many parameters!')
+
+    in_dir = sys.argv[1]
+    out_dir = sys.argv[2]
+    uki_name = sys.argv[3]
+    distro = sys.argv[4]
+    arch = sys.argv[5]
+
+    in_dir, out_dir = check_clean_arguments(in_dir, out_dir)
+    parse_in_dir(in_dir, uki_name, distro, arch)
+    create_addons(out_dir)

--
https://gitlab.com/cki-project/kernel-ark/-/merge_requests/2917

[OS-BUILD PATCHv12 2/2] redhat/uki_cmdline_addons.conf: add FIPS addon

[Emanuele Giuseppe Esposito (via Email Bridge)]

From: Emanuele Giuseppe Esposito <eesposit(a)redhat.com&gt;

redhat/uki_cmdline_addons.conf: add FIPS addon

Upstream Status: RHEL-Only

The fips addon simply enable fips in the kernel command line.

Signed-off-by: Emanuele Giuseppe Esposito <eesposit(a)redhat.com&gt;

diff --git a/redhat/addons/virt/common/fips.addon b/redhat/addons/virt/common/fips.addon
new file mode 100644
index blahblah..blahblah 100644
--- /dev/null
+++ b/redhat/addons/virt/common/fips.addon
@@ -0,0 +1,2 @@
+# Enable fips in RHEL (https://issues.redhat.com/browse/RHEL-23049)
+fips=1

--
https://gitlab.com/cki-project/kernel-ark/-/merge_requests/2917

[OS-BUILD PATCHv12 2/2] redhat/uki_cmdline_addons.conf: add FIPS addon

[Vitaly Kuznetsov (via Email Bridge)]

From: Vitaly Kuznetsov on gitlab.com
https://gitlab.com/cki-project/kernel-ark/-/merge_requests/2917#note_1841...

It may make sense to avoid mentioning RHEL in ARK MRs.

[Emanuele Giuseppe Esposito (via Email Bridge)]

From: Emanuele Giuseppe Esposito on gitlab.com
https://gitlab.com/cki-project/kernel-ark/-/merge_requests/2917#note_1824...

Makes sense

[Vitaly Kuznetsov (via Email Bridge)]

From: Vitaly Kuznetsov on gitlab.com
https://gitlab.com/cki-project/kernel-ark/-/merge_requests/2917#note_1841...

@eesposit I'm looking at kernel-uki-virt-
addons-6.9.0-0.rc0.a4145ce1e7bc.11.test.fc41.x86_64.rpm:

`
$ rpm -qpl /tmp/kernel-uki-virt-
addons-6.9.0-0.rc0.a4145ce1e7bc.11.test.fc41.x86_64.rpm
/usr/lib/linux/extra.d/fips-virt.fedora.x86_64.addon.efi
`
at the same time dependencies are stict:
`
$ rpm -qp --requires /tmp/kernel-uki-virt-
addons-6.9.0-0.rc0.a4145ce1e7bc.11.test.fc41.x86_64.rpm
kernel-uki-virt = 6.9.0-0.rc0.a4145ce1e7bc.11.test.fc41
...
`
so imaging I want to install two UKI versions, it'll require to install two
kernel-uki-virt-addons packages providing the same file. While it doesn't
really matter for 'FIPS' addon we have, what if we ever decide to change the
addon and thus the file won't be the same? I think we need to use a versioned
location so two installed kernel-uki-virt-addons don't clash.

[Vitaly Kuznetsov (via Email Bridge)]

From: Vitaly Kuznetsov on gitlab.com
https://gitlab.com/cki-project/kernel-ark/-/merge_requests/2917#note_1841...

@eesposit I'm looking at kernel-uki-virt-
addons-6.9.0-0.rc0.a4145ce1e7bc.11.test.fc41.x86_64.rpm:

`
$ rpm -qpl /tmp/kernel-uki-virt-
addons-6.9.0-0.rc0.a4145ce1e7bc.11.test.fc41.x86_64.rpm
/usr/lib/linux/extra.d/fips-virt.fedora.x86_64.addon.efi
`

at the same time dependencies are stict:

`
$ rpm -qp --requires /tmp/kernel-uki-virt-
addons-6.9.0-0.rc0.a4145ce1e7bc.11.test.fc41.x86_64.rpm
kernel-uki-virt = 6.9.0-0.rc0.a4145ce1e7bc.11.test.fc41
...
`

so imaging I want to install two UKI versions, it'll require to install two
kernel-uki-virt-addons packages providing the same file. While it doesn't
really matter for 'FIPS' addon we have, what if we ever decide to change the
addon and thus the file won't be the same? I think we need to use a versioned
location so two installed kernel-uki-virt-addons don't clash.

[Vitaly Kuznetsov (via Email Bridge)]

From: Vitaly Kuznetsov on gitlab.com
https://gitlab.com/cki-project/kernel-ark/-/merge_requests/2917#note_1841...

@eesposit I'm looking at kernel-uki-virt-
addons-6.9.0-0.rc0.a4145ce1e7bc.11.test.fc41.x86_64.rpm:

```
$ rpm -qpl /tmp/kernel-uki-virt-
addons-6.9.0-0.rc0.a4145ce1e7bc.11.test.fc41.x86_64.rpm
/usr/lib/linux/extra.d/fips-virt.fedora.x86_64.addon.efi
```

at the same time dependencies are stict:

```
$ rpm -qp --requires /tmp/kernel-uki-virt-
addons-6.9.0-0.rc0.a4145ce1e7bc.11.test.fc41.x86_64.rpm
kernel-uki-virt = 6.9.0-0.rc0.a4145ce1e7bc.11.test.fc41
...
```

so imaging I want to install two UKI versions, it'll require to install two
kernel-uki-virt-addons packages providing the same file. While it doesn't
really matter for 'FIPS' addon we have, what if we ever decide to change the
addon and thus the file won't be the same? I think we need to use a versioned
location so two installed kernel-uki-virt-addons don't clash.

[Vitaly Kuznetsov (via Email Bridge)]

From: Vitaly Kuznetsov on gitlab.com
https://gitlab.com/cki-project/kernel-ark/-/merge_requests/2917#note_1841...

While separating addons into `kernel-uki-virt-addons` package may make perfect
sense when we have a lot of them, we can probably postpone it and just package
them in the same package with UKI while there's just a few.

[Emanuele Giuseppe Esposito (via Email Bridge)]

From: Emanuele Giuseppe Esposito on gitlab.com
https://gitlab.com/cki-project/kernel-ark/-/merge_requests/2917#note_1843...

how do you propose to version it?

[Emanuele Giuseppe Esposito (via Email Bridge)]

From: Emanuele Giuseppe Esposito on gitlab.com
https://gitlab.com/cki-project/kernel-ark/-/merge_requests/2917#note_1843...

As long as it doesn't imply to drop all I did so far because otherwise all the
discussion done so far would be vain.

[Vitaly Kuznetsov (via Email Bridge)]

From: Vitaly Kuznetsov on gitlab.com
https://gitlab.com/cki-project/kernel-ark/-/merge_requests/2917#note_1843...

(I apologize if I've missed some parts of the discussion) Allowing to install
multiple `kernel-uki-virt` packages in parallel is a must as that's how we
upgrade kernels today. With the introduction of `kernel-uki-virt-addons` this
means several versions of `kernel-uki-virt-addons` should co-exist normally.
It may happen that addons change between versions so files should not
intersect (even if RPM allows that, not sure).

My (probably silly) suggestion would be to use directory similar to what's
going to be on the ESP and somewhat similar to /usr/lib/modules. E.g.,
```
/usr/lib/linux/extra.d/6.9.0-0.rc0.a4145ce1e7bc.11.test.fc41.x86_64/fips-
virt.fedora.x86_64.addon.efi
```
In this scheme, third party addons can live in `/usr/lib/linux/extra.d/`
without any sub-dir, e.g.
```
/usr/lib/linux/extra.d/third-party-addon-mydistro.x86_64.addon.efi
```