Hello people. I need some help from the good folks who maintain the Fedora servers responsible for building a bootchain securely, i.e. GRUB2 or the kernel. For instance, the bkernel01.iad2.fedoraproject.org server.
Let's take a look at the build logs of a recent GRUB2 build (https://koji.fedoraproject.org/koji/buildinfo?buildID=2185557) here: https://kojipkgs.fedoraproject.org//packages/grub2/2.06/95.fc38/data/logs/x8... As far as I can see, this server has a smart-card with a private key attached and during the building procedure, the critical components are being signed with Red Hat Bootloader Team's `pesign` software (version +115) running in client-server mode rather than standalone mode. By this I mean e.g. line number 7074 from the log file: ``` + /usr/bin/pesign-client -t 'OpenSC Card (Fedora Signer)' -c '/CN=Fedora Secure Boot Signer' -s -i grubx64.efi.orig -o grubx64.efi.onesig ```
I'd like to replicate the setup Fedora has to rebuild bootchain components on my own. My question is: how did you make `pesign-client` work fine? Is there a procedure of some sort that works just fine that I don't know about?
Here's what I attempted on a Fedora 38 machine:
- ensured that I have a recent version of `pesign` that Fedora Project ships:
``` $ rpm -qa --qf '%{NEVRA}\n' pesign pesign-116-2.fc38.x86_64 ```
- added myself to the `pesign` Unix group and rebooted the machine just to be sure it's there alright:
``` $ usermod -a -G pesign user ```
- created a DNSSEC's SoftHSM token:
``` $ mkdir -p $HOME/.config/softhsm2/tokens $ echo "directories.tokendir = $HOME/.config/softhsm2/tokens" > $HOME/.config/softhsm2/softhsm2.conf $ softhsm2-util --init-token --label HSM --so-pin Secret.123 --pin Secret.123 --free ```
- generated a key to be used with `pesign`:
``` $ efikeygen -d /etc/pki/pesign -n example --self-sign --common-name "CN=example,OU=example,O=example" --kernel ```
- exported the key to a .pk12 file:
``` $ pk12util -d /etc/pki/pesign -o my-ca.pk12 -n example Enter password for PKCS12 file: Re-enter password: pk12util: PKCS12 EXPORT SUCCESSFUL ```
- imported the file to a SoftHSM token:
``` $ pk12util -i my-ca.pk12 -d /etc/pki/pesign -h HSM Enter Password or Pin for "HSM": Secret.123 Enter password for PKCS12 file: pk12util: PKCS12 IMPORT SUCCESSFUL ```
- verified that the operation succeeded:
``` $ pkcs11-tool --module p11-kit-proxy.so -l --pin Secret.123 -O Using slot 0 with a present token (0x11) Public Key Object; RSA 2048 bits label: ID: 3b378e192bb341ca96c2254e30948f55fe3285d0 Usage: encrypt, verify, wrap Access: none Certificate Object; type = X.509 cert label: example subject: DN: O=example, OU=example, CN=example serial: 2BE5E980432B464C8EE366B3F76914A4 ID: 3b378e192bb341ca96c2254e30948f55fe3285d0 Private Key Object; RSA label: example ID: 3b378e192bb341ca96c2254e30948f55fe3285d0 Usage: decrypt, sign, unwrap Access: sensitive Public Key Object; RSA 2048 bits label: ID: 3b378e192bb341ca96c2254e30948f55fe3285d0 Usage: encrypt, verify, wrap Access: none
$ certutil -d /etc/pki/pesign -L -h HSM
Certificate Nickname Trust Attributes SSL,S/MIME,JAR/XPI
Enter Password or Pin for "HSM": HSM:example u,u,u ```
- removed `example` from NSS database:
``` $ certutil -d /etc/pki/pesign -D -n example ```
- ensured the building process will be using `pesign` in client-server mode rather than standalone:
``` $ sudo systemctl enable pesign --now Created symlink /etc/systemd/system/multi-user.target.wants/pesign.service → /usr/lib/systemd/system/pesign.service. ```
- and at last attempted to rebuild a bootchain component like GRUB2:
``` $ dnf download --source grub2 $ rpmdev-setuptree $ rpmdev-extract -C ~/rpmbuild/SOURCES/ grub2-2.06-95.fc38.src.rpm [...] $ cd ~/rpmbuild/SOURCES/ && mv */* . $ rpmbuild -bb -D 'pe_signing_token HSM' -D 'pe_signing_cert example' *.spec [...] + /usr/bin/pesign-client -t HSM -c example -s -i grubx64.efi.orig -o grubx64.efi.onesig pesign-client: signing failed: "pesignd starting (pid 1806)" error: Bad exit status from /var/tmp/rpm-tmp.hha8Hd (%build) ```
So it looks like there is an error of some kind on my Fedora 38 machine. Let's try unlocking some tokens:
``` $ pesign-client --unlock --token "NSS Certificate DB" Enter passphrase for private key: $ pesign-client --is-unlocked --token "NSS Certificate DB" token "NSS Certificate DB" is unlocked ```
Good. Let's try the same with SoftHSM:
``` $ pesign-client --unlock --token "HSM" Enter passphrase for private key: Secret.123 pesign-client: pesignd starting (pid 1806) $ pesign-client --is-unlocked --token "HSM" token "HSM" is locked ```
Why? Just why?
On the other hand, the **only** configuration of building GRUB2 and the kernel that worked for me was with pesign 113 in standalone mode and only when typing the password manually (this was some time ago on Fedora 35). Yes, there's an issue with version +114, which I filed at https://github.com/rhboot/pesign/issues/105, but this doesn't prevent the software being used in noninteractive mode (by specifying the password in a `--pinfile <FILE>` option). However, I don't know, how to accomplish this in a stock configuration of performing a build with rpmbuild -bb -D 'pe_signing_token HSM' -D 'pe_signing_cert example' *.spec - I don't think I'm supposed to write custom RPM macros for this. There has to be a more obvious solution.
Please, give me a helping hand with this. What procedure do I have to follow to replicate what's on Fedora Koji instances? What is there that I'm missing?
On Wed, Jul 05, 2023 at 11:29:27AM -0000, Kamil Aronowski wrote:
Hello people. I need some help from the good folks who maintain the Fedora servers responsible for building a bootchain securely, i.e. GRUB2 or the kernel. For instance, the bkernel01.iad2.fedoraproject.org server.
Happy to try and help.
Let's take a look at the build logs of a recent GRUB2 build (https://koji.fedoraproject.org/koji/buildinfo?buildID=2185557) here: https://kojipkgs.fedoraproject.org//packages/grub2/2.06/95.fc38/data/logs/x8... As far as I can see, this server has a smart-card with a private key attached and during the building procedure, the critical components are being signed with Red Hat Bootloader Team's `pesign` software (version +115) running in client-server mode rather than standalone mode. By this I mean e.g. line number 7074 from the log file:
+ /usr/bin/pesign-client -t 'OpenSC Card (Fedora Signer)' -c '/CN=Fedora Secure Boot Signer' -s -i grubx64.efi.orig -o grubx64.efi.onesigI'd like to replicate the setup Fedora has to rebuild bootchain components on my own. My question is: how did you make `pesign-client` work fine? Is there a procedure of some sort that works just fine that I don't know about?
Here's what I attempted on a Fedora 38 machine:
...snip...
Please, give me a helping hand with this. What procedure do I have to follow to replicate what's on Fedora Koji instances? What is there that I'm missing?
So, bkernel01/02 are koji builders, so there's kojid and mock in the way there. We have for mock:
roles/bkernel/files/bkernel-site-defaults.cfg
config_opts['plugin_conf']['bind_mount_opts']['dirs'].append(('/var/run/pesign', '/var/run/pesign' )) config_opts['nspawn_args'] += ['--bind=/var/run/pesign']
which bind mounts the pesign socket into the chroot. (now of course you aren't using mock, but wanted to mention it)
Then, we have some acls on the socket and run directory: roles/bkernel/tasks/main.yml
acl: path=/var/run/pesign entity=kojibuilder etype=user permissions=rwx recursive=true state=present
(and some more acls).
So, might be just being in pesign group isn't enough to connect to the socket? Or there's some selinux denial?
I'd try stracing it and see if you can see if it can talk to the socket correctly?
If it's talking to the pesign-server ok, then I am not sure what the problem is. ;(
kevin
Thanks for the reply, Kevin. It means a lot to me, as I no longer feel alone with this issue. I'll try the mock configuration later on, so I do not overcomplicate things right now - once a basic config works for me, I'll then try mock.
I did try the strace method you suggested, and, as far as I can see, the socket can be accessed since 0 is returned. This is part of my listing:
``` $ strace pesign-client --unlock --token "NSS Certificate DB" |& grep -i r_ok access("/etc/ld.so.preload", R_OK) = -1 ENOENT (No such file or directory) access("/run//pesign/socket", R_OK) = 0 ```
I experimented a bit more, and via trial-and-error, I came to the conclusion that the pesign suite of tools has most likely had some regressions, as it used to have these historically. For instance, the one I mentioned earlier that I reported at: https://github.com/rhboot/pesign/issues/105.
Why this conclusion? Let's take a deeper dive into this.
First, on April 11, I suggested some improvements to the manual page of `pesign` here: https://github.com/rhboot/pesign/pull/106/files I triple-checked that everything was OK on RHEL 9.1, which shipped pesign 115. However, the command I suggested in the manual for generating a CA directly in a token no longer works - the `-t HSM` option in `efikeygen`.
Second, I just tested `pesign` on both Fedora 38 (version 116) and CentOS 8.5 (version 112). Error handling seems different, and once I research this more, using `git bisect`, I'll file an issue on GitHub too.
This happens on CentOS 8.5:
``` $ pesign-client --unlock --token "this token does not exist" Enter passphrase for private key: pesign-client: could not find token "this token does not exist" ```
The error message is fine. And this happens on Fedora 38:
``` $ pesign-client --unlock --token "this token does not exist" Enter passphrase for private key: pesign-client: pesignd starting (pid 2905) ```
The error message does not seem meaningful.
Third, I mentioned in my introductory message that `pesign` seems to work fine with the internal NSS database, that is "NSS Certificate DB". Through some more experiments I figured out that I can indeed build GRUB2 with `pesign` in client-server mode with some tweaks. Here's what worked for me:
- I generated another CA directly in `pesign`'s database and named it `grub2-signer` according to the macro in Fedora's GRUB2 SRPM (that is in the file `grub.macros`):
``` $ grep grub2-signer grub.macros %{expand:%%define __pesign_client_cert grub2-signer} \
$ efikeygen -d /etc/pki/pesign -n grub2-signer --self-sign --common-name "CN=test grub2 signer CN,OU=test grub2 signer OU,O=test grub2 signer O" --kernel ```
- and ran the build just fine:
``` $ rpmbuild -bb -D 'pe_signing_token NSS Certificate DB' -D 'pe_signing_cert grub2-signer' *.spec [...] + /usr/bin/pesign-client -t 'NSS Certificate DB' -c grub2-signer -s -i grubx64.efi.orig -o grubx64.efi.onesig + [[ 2 -eq 2 ]] [...] ```
Fourth, since I could unlock the tokens: "NSS Certificate DB" and "NSS Generic Crypto Services", which are listed under the PKCS #11 module named "1. NSS Internal PKCS #11 Module", but not those that either don't exist at all or are located in the module named "2. p11-kit-proxy", I therefore suppose that `pesign` version 116 only works with the first one properly. Therefore, I suppose that the Fedora Project has some kind of configuration that makes the smart card named "OpenSC Card (Fedora Signer)" be present in the slot that just works - the most simple explanation.
So after this research, I'd like to ask the following:
- what is the output of the command `modutil -dbdir /etc/pki/pesign/ -list` ran on the Koji build servers? - where is the entry "token: OpenSC Card (Fedora Signer)" located? Under "NSS Internal PKCS #11 Module" or under "p11-kit-proxy"? - what is the output of the command `ls /usr/share/p11-kit/modules/`? - are there any commands in the infrastructural Ansible playbooks/Salt states/shell scripts used for provisioning Koji builders that manipulate that directory directly or indirectly? If so, what are they? - does a command similar to `modutil -dbdir /etc/pki/pesign/ -default p11-kit-proxy -mechanisms "RSA:DSA:RC2:RC4:RC5:AES:DES:DH:SHA1:SHA256:SHA512:SSL:TLS:MD5:MD2:RANDOM:FRIENDLY"` that changes the default provider for security mechanisms run during the provisioning stage? - is filing issues on the `pesign` project's GitHub the proper way to keep in touch with the developers, or is another way preferred? For instance, file them directly at bugzilla.redhat.com. - if it's possible to redact secrets (usernames, passwords, etc.) from the provisioning specification (playbooks/states/scripts) Fedora Project uses for these bootchain-related Koji servers, could these be shared with me, so I could replicate the configuration 1:1 (apart from the physical smartcard connected to the servers)?
I appreciate your help, Kevin. Thank you for everything!
On Thu, Jul 06, 2023 at 02:18:04PM -0000, Kamil Aronowski wrote:
Thanks for the reply, Kevin. It means a lot to me, as I no longer feel alone with this issue. I'll try the mock configuration later on, so I do not overcomplicate things right now - once a basic config works for me, I'll then try mock.
Sure.
I did try the strace method you suggested, and, as far as I can see, the socket can be accessed since 0 is returned. This is part of my listing:
$ strace pesign-client --unlock --token "NSS Certificate DB" |& grep -i r_ok access("/etc/ld.so.preload", R_OK) = -1 ENOENT (No such file or directory) access("/run//pesign/socket", R_OK) = 0I experimented a bit more, and via trial-and-error, I came to the conclusion that the pesign suite of tools has most likely had some regressions, as it used to have these historically. For instance, the one I mentioned earlier that I reported at: https://github.com/rhboot/pesign/issues/105.
Why this conclusion? Let's take a deeper dive into this.
...snip...
I can't really help you with upstream or RHEL versions. We run Fedora on our builders, currently pesign-116-2.fc38.x86_64
So after this research, I'd like to ask the following:
- what is the output of the command `modutil -dbdir /etc/pki/pesign/ -list` ran on the Koji build servers?
Listing of PKCS #11 Modules ----------------------------------------------------------- 1. NSS Internal PKCS #11 Module uri: pkcs11:library-manufacturer=Mozilla%20Foundation;library-description=NSS%20Internal%20Crypto%20Services;library-version=3.90 slots: 2 slots attached status: loaded
slot: NSS Internal Cryptographic Services token: NSS Generic Crypto Services uri: pkcs11:token=NSS%20Generic%20Crypto%20Services;manufacturer=Mozilla%20Foundation;serial=0000000000000000;model=NSS%203
slot: NSS User Private Key and Certificate Services token: NSS Certificate DB uri: pkcs11:token=NSS%20Certificate%20DB;manufacturer=Mozilla%20Foundation;serial=0000000000000000;model=NSS%203
2. p11-kit-proxy library name: p11-kit-proxy.so uri: pkcs11:library-manufacturer=PKCS%2311%20Kit;library-description=PKCS%2311%20Kit%20Proxy%20Module;library-version=1.1 slots: 1 slot attached status: loaded
slot: Alcor Micro AU9520 00 00 token: OpenSC Card (Fedora Signer) uri: pkcs11:token=OpenSC%20Card%20(Fedora%20Signer);manufacturer=OpenSC%20Project;serial=25b585160722;model=PKCS%2315 -----------------------------------------------------------
- where is the entry "token: OpenSC Card (Fedora Signer)" located? Under "NSS Internal PKCS #11 Module" or under "p11-kit-proxy"?
The latter.
- what is the output of the command `ls /usr/share/p11-kit/modules/`?
opensc.module p11-kit-trust.module
- are there any commands in the infrastructural Ansible playbooks/Salt states/shell scripts used for provisioning Koji builders that manipulate that directory directly or indirectly? If so, what are they?
All our ansible content is available at https://pagure.io/fedora-infra/ansible
Nothing touches the p11-kit dir that I can see.
- does a command similar to `modutil -dbdir /etc/pki/pesign/ -default p11-kit-proxy -mechanisms "RSA:DSA:RC2:RC4:RC5:AES:DES:DH:SHA1:SHA256:SHA512:SSL:TLS:MD5:MD2:RANDOM:FRIENDLY"` that changes the default provider for security mechanisms run during the provisioning stage?
no
- is filing issues on the `pesign` project's GitHub the proper way to keep in touch with the developers, or is another way preferred? For instance, file them directly at bugzilla.redhat.com.
I don't know. I would think github.
- if it's possible to redact secrets (usernames, passwords, etc.) from the provisioning specification (playbooks/states/scripts) Fedora Project uses for these bootchain-related Koji servers, could these be shared with me, so I could replicate the configuration 1:1 (apart from the physical smartcard connected to the servers)?
See above. Do note that our builders are Fedora, not RHEL.
I appreciate your help, Kevin. Thank you for everything!
Good luck! Sorry it's being such a pain...
kevin
infrastructure@lists.fedoraproject.org
-
Kamil Aronowski -
Kevin Fenzi