Fedora infra for Secure Boot components - local setup

[Kamil Aronowski]

Hello people. I need some help from the good folks who maintain the Fedora servers
responsible for building a bootchain securely, i.e. GRUB2 or the kernel. For instance, the
bkernel01.iad2.fedoraproject.org server.

Let's take a look at the build logs of a recent GRUB2 build
(https://koji.fedoraproject.org/koji/buildinfo?buildID=2185557) here:
https://kojipkgs.fedoraproject.org//packages/grub2/2.06/95.fc38/data/logs...
As far as I can see, this server has a smart-card with a private key attached and during
the building procedure, the critical components are being signed with Red Hat Bootloader
Team's `pesign` software (version +115) running in client-server mode rather than
standalone mode. By this I mean e.g. line number 7074 from the log file:
```
+ /usr/bin/pesign-client -t 'OpenSC Card (Fedora Signer)' -c '/CN=Fedora
Secure Boot Signer' -s -i grubx64.efi.orig -o grubx64.efi.onesig
```

I'd like to replicate the setup Fedora has to rebuild bootchain components on my own.
My question is: how did you make `pesign-client` work fine? Is there a procedure of some
sort that works just fine that I don't know about?

Here's what I attempted on a Fedora 38 machine:

- ensured that I have a recent version of `pesign` that Fedora Project ships:

```
$ rpm -qa --qf '%{NEVRA}\n' pesign
pesign-116-2.fc38.x86_64
```

- added myself to the `pesign` Unix group and rebooted the machine just to be sure
it's there alright:

```
$ usermod -a -G pesign user
```

- created a DNSSEC's SoftHSM token:

```
$ mkdir -p $HOME/.config/softhsm2/tokens
$ echo "directories.tokendir = $HOME/.config/softhsm2/tokens" >
$HOME/.config/softhsm2/softhsm2.conf
$ softhsm2-util --init-token --label HSM --so-pin Secret.123 --pin Secret.123 --free
```

- generated a key to be used with `pesign`:

```
$ efikeygen -d /etc/pki/pesign -n example --self-sign --common-name
"CN=example,OU=example,O=example" --kernel
```

- exported the key to a .pk12 file:

```
$ pk12util -d /etc/pki/pesign -o my-ca.pk12 -n example
Enter password for PKCS12 file:
Re-enter password:
pk12util: PKCS12 EXPORT SUCCESSFUL
```

- imported the file to a SoftHSM token:

```
$ pk12util -i my-ca.pk12 -d /etc/pki/pesign -h HSM
Enter Password or Pin for "HSM": Secret.123
Enter password for PKCS12 file:
pk12util: PKCS12 IMPORT SUCCESSFUL
```

- verified that the operation succeeded:

```
$ pkcs11-tool --module p11-kit-proxy.so -l --pin Secret.123 -O                          
Using slot 0 with a present token (0x11)
Public Key Object; RSA 2048 bits
  label:      
  ID:         3b378e192bb341ca96c2254e30948f55fe3285d0
  Usage:      encrypt, verify, wrap
  Access:     none
Certificate Object; type = X.509 cert
  label:      example
  subject:    DN: O=example, OU=example, CN=example
  serial:     2BE5E980432B464C8EE366B3F76914A4
  ID:         3b378e192bb341ca96c2254e30948f55fe3285d0
Private Key Object; RSA
  label:      example
  ID:         3b378e192bb341ca96c2254e30948f55fe3285d0
  Usage:      decrypt, sign, unwrap
  Access:     sensitive
Public Key Object; RSA 2048 bits
  label:      
  ID:         3b378e192bb341ca96c2254e30948f55fe3285d0
  Usage:      encrypt, verify, wrap
  Access:     none

$ certutil -d /etc/pki/pesign -L -h HSM

Certificate Nickname                                         Trust Attributes
                                                             SSL,S/MIME,JAR/XPI

Enter Password or Pin for "HSM":
HSM:example                                                  u,u,u
```

- removed `example` from NSS database:

```
$ certutil -d /etc/pki/pesign -D -n example
```

- ensured the building process will be using `pesign` in client-server mode rather than
standalone:

```
$ sudo systemctl enable pesign --now
Created symlink /etc/systemd/system/multi-user.target.wants/pesign.service →
/usr/lib/systemd/system/pesign.service.
```

- and at last attempted to rebuild a bootchain component like GRUB2:

```
$ dnf download --source grub2
$ rpmdev-setuptree
$ rpmdev-extract -C ~/rpmbuild/SOURCES/ grub2-2.06-95.fc38.src.rpm
[...]
$ cd ~/rpmbuild/SOURCES/ && mv */* .
$ rpmbuild -bb -D 'pe_signing_token HSM' -D 'pe_signing_cert example'
*.spec
[...]
+ /usr/bin/pesign-client -t HSM -c example -s -i grubx64.efi.orig -o grubx64.efi.onesig
pesign-client: signing failed: "pesignd starting (pid 1806)"
error: Bad exit status from /var/tmp/rpm-tmp.hha8Hd (%build)
```

So it looks like there is an error of some kind on my Fedora 38 machine. Let's try
unlocking some tokens:

```
$ pesign-client --unlock --token "NSS Certificate DB"
Enter passphrase for private key:
$ pesign-client --is-unlocked --token "NSS Certificate DB"
token "NSS Certificate DB" is unlocked
```

Good. Let's try the same with SoftHSM:

```
$ pesign-client --unlock --token "HSM"
Enter passphrase for private key: Secret.123
pesign-client: pesignd starting (pid 1806)
$ pesign-client --is-unlocked --token "HSM"
token "HSM" is locked
```

Why? Just why?

On the other hand, the **only** configuration of building GRUB2 and the kernel that worked
for me was with pesign 113 in standalone mode and only when typing the password manually
(this was some time ago on Fedora 35).
Yes, there's an issue with version +114, which I filed at
https://github.com/rhboot/pesign/issues/105, but this doesn't prevent the software
being used in noninteractive mode (by specifying the password in a `--pinfile
<FILE>` option). However, I don't know, how to accomplish this in a stock
configuration of performing a build with rpmbuild -bb -D 'pe_signing_token HSM' -D
'pe_signing_cert example' *.spec - I don't think I'm supposed to write
custom RPM macros for this. There has to be a more obvious solution.

Please, give me a helping hand with this. What procedure do I have to follow to replicate
what's on Fedora Koji instances? What is there that I'm missing?

[Kevin Fenzi]

On Wed, Jul 05, 2023 at 11:29:27AM -0000, Kamil Aronowski wrote:
>  Hello people. I need some help from the good folks who maintain the
> Fedora servers responsible for building a bootchain securely, i.e. GRUB2 or the kernel.
> For instance, the bkernel01.iad2.fedoraproject.org server. 
Happy to try and help.

>  Let's take a look at the build logs of a recent GRUB2 build
> (https://koji.fedoraproject.org/koji/buildinfo?buildID=2185557) here:
> https://kojipkgs.fedoraproject.org//packages/grub2/2.06/95.fc38/data/logs...
>  As far as I can see, this server has a smart-card with a private key attached and during
> the building procedure, the critical components are being signed with Red Hat Bootloader
> Team's `pesign` software (version +115) running in client-server mode rather than
> standalone mode. By this I mean e.g. line number 7074 from the log file:
>  ```
>  + /usr/bin/pesign-client -t 'OpenSC Card (Fedora Signer)' -c '/CN=Fedora
> Secure Boot Signer' -s -i grubx64.efi.orig -o grubx64.efi.onesig
>  ```
>  
>  I'd like to replicate the setup Fedora has to rebuild bootchain components on my own.
> My question is: how did you make `pesign-client` work fine? Is there a procedure of some
> sort that works just fine that I don't know about?
>  
>  Here's what I attempted on a Fedora 38 machine: 
...snip...
>  
>  Please, give me a helping hand with this. What procedure do I have to follow to replicate
> what's on Fedora Koji instances? What is there that I'm missing? 
So, bkernel01/02 are koji builders, so there's kojid and mock in the way
there. We have for mock:

roles/bkernel/files/bkernel-site-defaults.cfg

config_opts['plugin_conf']['bind_mount_opts']['dirs'].append(('/var/run/pesign',
'/var/run/pesign' ))
config_opts['nspawn_args'] += ['--bind=/var/run/pesign']

which bind mounts the pesign socket into the chroot.
(now of course you aren't using mock, but wanted to mention it)

Then, we have some acls on the socket and run directory:
roles/bkernel/tasks/main.yml

  acl: path=/var/run/pesign entity=kojibuilder etype=user permissions=rwx recursive=true
state=present

(and some more acls).

So, might be just being in pesign group isn't enough to connect to the
socket? Or there's some selinux denial? 

I'd try stracing it and see if you can see if it can talk to the socket
correctly?

If it's talking to the pesign-server ok, then I am not sure what the
problem is. ;( 

kevin

[Kamil Aronowski]

Thanks for the reply, Kevin. It means a lot to me, as I no longer feel alone with this
issue. I'll try the mock configuration later on, so I do not overcomplicate things
right now - once a basic config works for me, I'll then try mock.

I did try the strace method you suggested, and, as far as I can see, the socket can be
accessed since 0 is returned. This is part of my listing:

```
$ strace pesign-client --unlock --token "NSS Certificate DB"  |& grep -i
r_ok
access("/etc/ld.so.preload", R_OK)      = -1 ENOENT (No such file or directory)
access("/run//pesign/socket", R_OK)     = 0
```

I experimented a bit more, and via trial-and-error, I came to the conclusion that the
pesign suite of tools has most likely had some regressions, as it used to have these
historically. For instance, the one I mentioned earlier that I reported at:
https://github.com/rhboot/pesign/issues/105.

Why this conclusion? Let's take a deeper dive into this.

First, on April 11, I suggested some improvements to the manual page of `pesign` here:
https://github.com/rhboot/pesign/pull/106/files
I triple-checked that everything was OK on RHEL 9.1, which shipped pesign 115. However,
the command I suggested in the manual for generating a CA directly in a token no longer
works - the `-t HSM` option in `efikeygen`.

Second, I just tested `pesign` on both Fedora 38 (version 116) and CentOS 8.5 (version
112). Error handling seems different, and once I research this more, using `git bisect`,
I'll file an issue on GitHub too.

This happens on CentOS 8.5:

```
$ pesign-client --unlock --token "this token does not exist"
Enter passphrase for private key: 
pesign-client: could not find token "this token does not exist"
```

The error message is fine. And this happens on Fedora 38:

```
$ pesign-client --unlock --token "this token does not exist"
Enter passphrase for private key: 
pesign-client: pesignd starting (pid 2905)
```

The error message does not seem meaningful.

Third, I mentioned in my introductory message that `pesign` seems to work fine with the
internal NSS database, that is "NSS Certificate DB". Through some more
experiments I figured out that I can indeed build GRUB2 with `pesign` in client-server
mode with some tweaks. Here's what worked for me:

- I generated another CA directly in `pesign`'s database and named it `grub2-signer`
according to the macro in Fedora's GRUB2 SRPM (that is in the file `grub.macros`):

```
$ grep grub2-signer grub.macros 
%{expand:%%define __pesign_client_cert grub2-signer} \

$ efikeygen -d /etc/pki/pesign -n grub2-signer --self-sign --common-name "CN=test
grub2 signer CN,OU=test grub2 signer OU,O=test grub2 signer O" --kernel
```

- and ran the build just fine:

```
$ rpmbuild -bb -D 'pe_signing_token NSS Certificate DB' -D 'pe_signing_cert
grub2-signer' *.spec
[...]
+ /usr/bin/pesign-client -t 'NSS Certificate DB' -c grub2-signer -s -i
grubx64.efi.orig -o grubx64.efi.onesig          
+ [[ 2 -eq 2 ]]
[...]
```

Fourth, since I could unlock the tokens: "NSS Certificate DB" and "NSS
Generic Crypto Services", which are listed under the PKCS #11 module named "1.
NSS Internal PKCS #11 Module", but not those that either don't exist at all or
are located in the module named "2. p11-kit-proxy", I therefore suppose that
`pesign` version 116 only works with the first one properly.
Therefore, I suppose that the Fedora Project has some kind of configuration that makes the
smart card named "OpenSC Card (Fedora Signer)" be present in the slot that just
works - the most simple explanation.

So after this research, I'd like to ask the following:

- what is the output of the command `modutil -dbdir /etc/pki/pesign/ -list` ran on the
Koji build servers?
- where is the entry "token: OpenSC Card (Fedora Signer)" located? Under
"NSS Internal PKCS #11 Module" or under "p11-kit-proxy"?
- what is the output of the command `ls /usr/share/p11-kit/modules/`?
- are there any commands in the infrastructural Ansible playbooks/Salt states/shell
scripts used for provisioning Koji builders that manipulate that directory directly or
indirectly? If so, what are they?
- does a command similar to `modutil -dbdir /etc/pki/pesign/ -default p11-kit-proxy
-mechanisms
"RSA:DSA:RC2:RC4:RC5:AES:DES:DH:SHA1:SHA256:SHA512:SSL:TLS:MD5:MD2:RANDOM:FRIENDLY"`
that changes the default provider for security mechanisms run during the provisioning
stage?
- is filing issues on the `pesign` project's GitHub the proper way to keep in touch
with the developers, or is another way preferred? For instance, file them directly at
bugzilla.redhat.com.
- if it's possible to redact secrets (usernames, passwords, etc.) from the
provisioning specification (playbooks/states/scripts) Fedora Project uses for these
bootchain-related Koji servers, could these be shared with me, so I could replicate the
configuration 1:1 (apart from the physical smartcard connected to the servers)?

I appreciate your help, Kevin. Thank you for everything!

[Kevin Fenzi]

On Thu, Jul 06, 2023 at 02:18:04PM -0000, Kamil Aronowski wrote:
>  Thanks for the reply, Kevin. It means a lot to me, as I no longer
> feel alone with this issue. I'll try the mock configuration later on, so I do not
> overcomplicate things right now - once a basic config works for me, I'll then try
> mock. 
Sure.

>  I did try the strace method you suggested, and, as far as I can see,
> the socket can be accessed since 0 is returned. This is part of my listing:
>  
>  ```
>  $ strace pesign-client --unlock --token "NSS Certificate DB"  |& grep -i
> r_ok
>  access("/etc/ld.so.preload", R_OK)      = -1 ENOENT (No such file or
> directory)
>  access("/run//pesign/socket", R_OK)     = 0
>  ```
>  
>  I experimented a bit more, and via trial-and-error, I came to the conclusion that the
> pesign suite of tools has most likely had some regressions, as it used to have these
> historically. For instance, the one I mentioned earlier that I reported at:
> https://github.com/rhboot/pesign/issues/105.
>  
>  Why this conclusion? Let's take a deeper dive into this. 
...snip...

I can't really help you with upstream or RHEL versions. We run Fedora on
our builders, currently pesign-116-2.fc38.x86_64
>  
>  So after this research, I'd like to ask the following:
>  
>  - what is the output of the command `modutil -dbdir /etc/pki/pesign/ -list` ran on the
> Koji build servers? 
Listing of PKCS #11 Modules
-----------------------------------------------------------
  1. NSS Internal PKCS #11 Module
           uri:
pkcs11:library-manufacturer=Mozilla%20Foundation;library-description=NSS%20Internal%20Crypto%20Services;library-version=3.90
         slots: 2 slots attached
        status: loaded

         slot: NSS Internal Cryptographic Services
        token: NSS Generic Crypto Services
          uri:
pkcs11:token=NSS%20Generic%20Crypto%20Services;manufacturer=Mozilla%20Foundation;serial=0000000000000000;model=NSS%203

         slot: NSS User Private Key and Certificate Services
        token: NSS Certificate DB
          uri:
pkcs11:token=NSS%20Certificate%20DB;manufacturer=Mozilla%20Foundation;serial=0000000000000000;model=NSS%203

  2. p11-kit-proxy
        library name: p11-kit-proxy.so
           uri:
pkcs11:library-manufacturer=PKCS%2311%20Kit;library-description=PKCS%2311%20Kit%20Proxy%20Module;library-version=1.1
         slots: 1 slot attached
        status: loaded

         slot: Alcor Micro AU9520 00 00
        token: OpenSC Card (Fedora Signer)
          uri:
pkcs11:token=OpenSC%20Card%20(Fedora%20Signer);manufacturer=OpenSC%20Project;serial=25b585160722;model=PKCS%2315
-----------------------------------------------------------

>  - where is the entry "token: OpenSC Card (Fedora Signer)"
> located? Under "NSS Internal PKCS #11 Module" or under
> "p11-kit-proxy"? 
The latter.

>  - what is the output of the command `ls /usr/share/p11-kit/modules/`?
opensc.module  p11-kit-trust.module

>  - are there any commands in the infrastructural Ansible
> playbooks/Salt states/shell scripts used for provisioning Koji builders that manipulate
> that directory directly or indirectly? If so, what are they? 
All our ansible content is available at
https://pagure.io/fedora-infra/ansible

Nothing touches the p11-kit dir that I can see.

>  - does a command similar to `modutil -dbdir /etc/pki/pesign/ -default
> p11-kit-proxy -mechanisms
> "RSA:DSA:RC2:RC4:RC5:AES:DES:DH:SHA1:SHA256:SHA512:SSL:TLS:MD5:MD2:RANDOM:FRIENDLY"`
> that changes the default provider for security mechanisms run during the provisioning
> stage? 
no

>  - is filing issues on the `pesign` project's GitHub the proper
> way to keep in touch with the developers, or is another way preferred? For instance, file
> them directly at bugzilla.redhat.com. 
I don't know. I would think github.

>  - if it's possible to redact secrets (usernames, passwords, etc.)
> from the provisioning specification (playbooks/states/scripts) Fedora Project uses for
> these bootchain-related Koji servers, could these be shared with me, so I could replicate
> the configuration 1:1 (apart from the physical smartcard connected to the servers)?
See above. Do note that our builders are Fedora, not RHEL.

>  I appreciate your help, Kevin. Thank you for everything! 
Good luck! Sorry it's being such a pain...

kevin