AWS gp2 -> gp3
by Miroslav Suchý
tl;dr I want to change **all** storage types in AWS from gp2 to gp3 next week.
The recent Change proposal
https://fedoraproject.org/wiki/Changes/CloudEC2gp3
brought to my attention that gp3 is actually faster and cheaper than gp2. The comparision is as follows:
gp2:
* $0.10 per GB-month
* IOPS: 100 IOPB, burstable to 3000 IOPS
* Throughput: Throughput limit is between 128 MiB/s and 250 MiB/s, depending on the volume size.
gp3:
* IOPS: 3000 for free, addional IOPS can be bought
* Throughput 125 MiB/s for free, addional throughput can be bought
For more information see
https://aws.amazon.com/blogs/storage/migrate-your-amazon-ebs-volumes-from...
We have about 150 volumes of gp2 type in AWS. Copr is the most prominent user, but there are other: libravatar, taiga,
TF, and other...
The type of the storage can be changed on the fly - even for attached and volumes in-use. I already tried that with my
private server. Unless somebody will have objections I plan to change volume types to gp3 for Copr next week and for all
other services in week following Easter.
I also modified fedimg template to allow gp3
https://pagure.io/fedora-infra/ansible/c/034cf953db5d0519eb327136abd72a2e...
If you have scripts outside of fedora-infra/ansible to provision instances that uses gp2 for volumes, it will be great
if you can move it to gp3.
Miroslav
1 week, 1 day
congrats to our new sysadmin-mainer
by Kevin Fenzi
I'm happy to announce that We have approved a new member in our
sysadmin-main group:
darknao - Francois Andrieu
This is the core group of trusted folks that high level access to most
everything in fedora infrastructure.
Francois has done of ton of things around Fedora infrastructure. From
helping manage our OpenShift clusters, to revamping how our docs and
websites are built and deployed, to just helping anyone with issues all
around.
He has proved his dedication, trustworthiness, and ability.
Congrats!
Use your powers for good! :)
kevin
1 month, 3 weeks
Meeting Agenda Item: (Re-)Introduction Patrick Creech
by Patrick Creech
Hi there! I popped into the meeting last week and said "Hi", but
figured I should also send this out as well.
IRC: pcreech
Since my last introduction[0] the primary thing that has changed is that
I moved into doing release engineering for pulp, which I then moved into
doing release engineering for the Red Hat Satellite product
(Foreman/Katello upstream). I've had the joy of all manner of release
engineering related work, as well as helping manage release engineering
infrastructure, and constantly improving the quality and cadence of our
ability to deliver bits to be tested.
Recently, my interest in contributing to Fedora has grown again, and I
decided to get involved again, especially since life derailed me for a
while last time. I'm looking forward to helping out!
[0] Previous introduction:
https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedora...
2 months
batcave01 move to rhel9 - 2023-07-13 21UTC
by Kevin Fenzi
Hey folks. I keep putting it off, but I need to move it forward, so I am
proposing to move batcave01 to rhel9 tomorrow, starting at 21UTC.
My plan is roughly:
- sync data from batcave01 to batcave02
- at 21UTC take down services on batcave01
- sync data again
- take batcave01 down.
- rename batcave02 to batcave01 and re-ip it to the right IP
- reboot it back up as batcave01
- Fix any issues that come up.
This means that the ssh host key for batcave01 WILL CHANGE!
If you are using https://admin.fedoraproject.org/ssh_known_hosts
in your ~/.ssh/known_hosts it shouldn't matter. If you aren't you may
have to accept the new changed host key.
For the time when I am switching things, EPEL builds may fail.
This is because koji pulls RHEL packages from batcave01.
I would expect this to be a very short window.
The playbook completes fine on batcave02 and as far as I can see things
should work fine there, but there may be some broken items.
Please file tickets/let me know if you find anything like that.
In the event that things go very poorly for some reason, I can always
bring the old vm back up and we can revert to it.
Thanks,
kevin
2 months, 1 week
I'm enabling topic authorization on the production bus
by Aurelien Bompard
Hey folks!
After a lot of asking around and quite a bit of testing on staging, I'm
ready to enable topic authorization on production. If your application does
not declare the `sent_topics` regexp in Ansible, it will only be allowed to
publish on topics that start with
`org.fedoraproject.{env}.{rabbit_username}.`.
This is just a heads-up. I'll watch the rabbitmq logs closely to see if
there are permission refused errors, in which case I'll try to contact the
app owner.
Fingers crossed! In theory all goes well :-D
Aurélien
2 months, 1 week
Fedora infra for Secure Boot components - local setup
by Kamil Aronowski
Hello people. I need some help from the good folks who maintain the Fedora servers responsible for building a bootchain securely, i.e. GRUB2 or the kernel. For instance, the bkernel01.iad2.fedoraproject.org server.
Let's take a look at the build logs of a recent GRUB2 build (https://koji.fedoraproject.org/koji/buildinfo?buildID=2185557) here: https://kojipkgs.fedoraproject.org//packages/grub2/2.06/95.fc38/data/logs...
As far as I can see, this server has a smart-card with a private key attached and during the building procedure, the critical components are being signed with Red Hat Bootloader Team's `pesign` software (version +115) running in client-server mode rather than standalone mode. By this I mean e.g. line number 7074 from the log file:
```
+ /usr/bin/pesign-client -t 'OpenSC Card (Fedora Signer)' -c '/CN=Fedora Secure Boot Signer' -s -i grubx64.efi.orig -o grubx64.efi.onesig
```
I'd like to replicate the setup Fedora has to rebuild bootchain components on my own. My question is: how did you make `pesign-client` work fine? Is there a procedure of some sort that works just fine that I don't know about?
Here's what I attempted on a Fedora 38 machine:
- ensured that I have a recent version of `pesign` that Fedora Project ships:
```
$ rpm -qa --qf '%{NEVRA}\n' pesign
pesign-116-2.fc38.x86_64
```
- added myself to the `pesign` Unix group and rebooted the machine just to be sure it's there alright:
```
$ usermod -a -G pesign user
```
- created a DNSSEC's SoftHSM token:
```
$ mkdir -p $HOME/.config/softhsm2/tokens
$ echo "directories.tokendir = $HOME/.config/softhsm2/tokens" > $HOME/.config/softhsm2/softhsm2.conf
$ softhsm2-util --init-token --label HSM --so-pin Secret.123 --pin Secret.123 --free
```
- generated a key to be used with `pesign`:
```
$ efikeygen -d /etc/pki/pesign -n example --self-sign --common-name "CN=example,OU=example,O=example" --kernel
```
- exported the key to a .pk12 file:
```
$ pk12util -d /etc/pki/pesign -o my-ca.pk12 -n example
Enter password for PKCS12 file:
Re-enter password:
pk12util: PKCS12 EXPORT SUCCESSFUL
```
- imported the file to a SoftHSM token:
```
$ pk12util -i my-ca.pk12 -d /etc/pki/pesign -h HSM
Enter Password or Pin for "HSM": Secret.123
Enter password for PKCS12 file:
pk12util: PKCS12 IMPORT SUCCESSFUL
```
- verified that the operation succeeded:
```
$ pkcs11-tool --module p11-kit-proxy.so -l --pin Secret.123 -O
Using slot 0 with a present token (0x11)
Public Key Object; RSA 2048 bits
label:
ID: 3b378e192bb341ca96c2254e30948f55fe3285d0
Usage: encrypt, verify, wrap
Access: none
Certificate Object; type = X.509 cert
label: example
subject: DN: O=example, OU=example, CN=example
serial: 2BE5E980432B464C8EE366B3F76914A4
ID: 3b378e192bb341ca96c2254e30948f55fe3285d0
Private Key Object; RSA
label: example
ID: 3b378e192bb341ca96c2254e30948f55fe3285d0
Usage: decrypt, sign, unwrap
Access: sensitive
Public Key Object; RSA 2048 bits
label:
ID: 3b378e192bb341ca96c2254e30948f55fe3285d0
Usage: encrypt, verify, wrap
Access: none
$ certutil -d /etc/pki/pesign -L -h HSM
Certificate Nickname Trust Attributes
SSL,S/MIME,JAR/XPI
Enter Password or Pin for "HSM":
HSM:example u,u,u
```
- removed `example` from NSS database:
```
$ certutil -d /etc/pki/pesign -D -n example
```
- ensured the building process will be using `pesign` in client-server mode rather than standalone:
```
$ sudo systemctl enable pesign --now
Created symlink /etc/systemd/system/multi-user.target.wants/pesign.service → /usr/lib/systemd/system/pesign.service.
```
- and at last attempted to rebuild a bootchain component like GRUB2:
```
$ dnf download --source grub2
$ rpmdev-setuptree
$ rpmdev-extract -C ~/rpmbuild/SOURCES/ grub2-2.06-95.fc38.src.rpm
[...]
$ cd ~/rpmbuild/SOURCES/ && mv */* .
$ rpmbuild -bb -D 'pe_signing_token HSM' -D 'pe_signing_cert example' *.spec
[...]
+ /usr/bin/pesign-client -t HSM -c example -s -i grubx64.efi.orig -o grubx64.efi.onesig
pesign-client: signing failed: "pesignd starting (pid 1806)"
error: Bad exit status from /var/tmp/rpm-tmp.hha8Hd (%build)
```
So it looks like there is an error of some kind on my Fedora 38 machine. Let's try unlocking some tokens:
```
$ pesign-client --unlock --token "NSS Certificate DB"
Enter passphrase for private key:
$ pesign-client --is-unlocked --token "NSS Certificate DB"
token "NSS Certificate DB" is unlocked
```
Good. Let's try the same with SoftHSM:
```
$ pesign-client --unlock --token "HSM"
Enter passphrase for private key: Secret.123
pesign-client: pesignd starting (pid 1806)
$ pesign-client --is-unlocked --token "HSM"
token "HSM" is locked
```
Why? Just why?
On the other hand, the **only** configuration of building GRUB2 and the kernel that worked for me was with pesign 113 in standalone mode and only when typing the password manually (this was some time ago on Fedora 35).
Yes, there's an issue with version +114, which I filed at https://github.com/rhboot/pesign/issues/105, but this doesn't prevent the software being used in noninteractive mode (by specifying the password in a `--pinfile <FILE>` option). However, I don't know, how to accomplish this in a stock configuration of performing a build with rpmbuild -bb -D 'pe_signing_token HSM' -D 'pe_signing_cert example' *.spec - I don't think I'm supposed to write custom RPM macros for this. There has to be a more obvious solution.
Please, give me a helping hand with this. What procedure do I have to follow to replicate what's on Fedora Koji instances? What is there that I'm missing?
2 months, 1 week
Fedora infra development Streaming session
by Aurelien Bompard
Hey folks!
This Friday at 13:00 UTC I'll be steaming on Twitch[1] about the
development of Fedora infrastructure apps. I'll start on a clean env,
checkout one of our apps, setup a dev env, fix a small bug, test it, and
create a PR.
[1] https://twitch.tv/ohwellien
I haven't decided which app it'll be yet, but It's going to be a simple bug
so that I can do all that in 1h30 max.
Come and ask any questions! :-)
Aurélien
P.S.: I'm not tied to Twitch in any way, it'll be the second time only I do
this sort of thing, and I'm happy to switch to a more appropriate platform
if needed as soon as I'm more comfortable generally streaming stuff :-)
2 months, 2 weeks
