infrastructure October 2023

infrastructure@lists.fedoraproject.org

10 participants
8 discussions

VDO
by Miroslav Suchý 04 Jul '26

04 Jul '26

After I seen the talk about VDO: https://www.youtube.com/watch?v=7CGr5LEAfRY I went ahead and tried it. I tried it on small (12GB) sample of Copr data and I saved 20-30% data. I then deployed it on production server retrace.fedoraproject.org and saved there 15% out of 2TB. Several notes: The RPM packages are only available for RHEL. Recently the public github has been updated with version of VDO, which should work on Fedora. No RPM available for Fedora yet though. Here are my notes: # vdo create --name=vdo1 --device=/dev/vdb --vdoLogicalSize=1T # mkfs.ext4 /dev/mapper/vdo1 Some slides suggest to use -E discard. This is not needed for new volumes. See vdo-devel mailing list for reasoning. # mount -o discard /dev/mapper/vdo1 /mnt/a Probably best scenario is to create vdo on whole disk and then make PV from whole VDO disk and put LVM on top of that. When you already have data, you cannot convert your disk to VDO. You need create new VDO disk, format it, mount it and copy the data. In my case, due limited space for migration I created a small LVM volume "vdosrv": # lvcreate -L 80G -n vdosrv vol0 I converted it to VDO which claims to have 2TB of vitual total size # vdo create --name=srv18 --device=/dev/vol0/vdosrv --vdoLogicalSize=2T # mkfs.ext4 /dev/mapper/srv18 See how many space is *actually* free. # vdostats # mkdir /mnt/srv18 # mount -o discard /dev/mapper/srv18 /mnt/srv18 Now I moved some data from old volume to /mnt/srv18. I shrank the FS of old volume, I shrank the old LV. And then I enlarged the new LV. # lvresize -L +400G /dev/vol0/vdosrv and let know vdo that it can use the new space: # vdo growPhysical -n srv18 I repeated this several times. I have to say that my experience is so far good. I hope that my notes help someone. Miroslav

10 14

hyperkitty/mailman3 now updated to the latest
by Reon Beon 13 Nov '23

13 Nov '23

Nice.

5 7

Fedora 39 Final freeze now in effect!
by Kevin Fenzi 08 Nov '23

08 Nov '23

Greetings. we are now in the infrastructure freeze leading up to the Fedora 39 Final release. This is a final release freeze. We do this to ensure that our infrastructure is stable and ready to release Fedora 39 when it's available. You can see a list of hosts that do not freeze by checking out the ansible repo and running the freezelist script: git clone https://infrastructure.fedoraproject.org/infra/ansible.git ansible/scripts/freezelist -i inventory Any hosts listed as freezes is frozen until 2023-10-18 (or later if release slips). Frozen hosts should have no changes made to them without a sign-off on the change from at least 2 sysadmin-main or rel-eng members, along with (in most cases) a patch of the exact change to be made to this list and/or a pull-request to the infra/ansible repo. Thanks, kevin

1 1

AWS usage per group
by Miroslav Suchý 31 Oct '23

31 Oct '23

Here comes October edition of resources running in AWS. It snapshot of resources running today's morning. Some TestingFarm resources are not included because they still do not tag properly all resources. Few notes: http://debuginfod1.aws.centos.org/ uses io1 type volume. This is suboptimal as io2 is better. But from monitoring the machine does not utilize the IOPS and the price for this volume is ten times more than gp3 which would provide equivalent performance There are some gp2 volumes. Gp3 is always better, but this is residuum of rootfs volumes that are by default still gp2. I see some old intels in the list. The AMD instance are AFAIK always cheaper at the same power: e.g., t3large ($0.0832) vs. t3a.large ($0.0752) Full report: FedoraGroup: copr Region: us-east-1 Instance Type: t3a.small - Count: 1 Instance Type: t3a.medium - Count: 5 Instance Type: t3a.xlarge - Count: 3 Instance Type: m5a.4xlarge - Count: 1 Instance Type: c7g.xlarge - Count: 103 Instance Type: c7i.xlarge - Count: 25 Instance Type: t2.medium - Count: 1 Volume Type: st1 - Total Size: 6000 GiB Volume Type: gp3 - Total Size: 5154 GiB Volume Type: io2 - Total Size: 20 GiB Volume Type: sc1 - Total Size: 81652 GiB FedoraGroup: ci Region: us-east-1 Instance Type: c5.2xlarge - Count: 6 Volume Type: gp3 - Total Size: 3056 GiB Region: us-east-2 Instance Type: m5a.4xlarge - Count: 1 Instance Type: r5.large - Count: 4 Instance Type: i3.2xlarge - Count: 7 Instance Type: r5.xlarge - Count: 2 Instance Type: c5.2xlarge - Count: 2 Instance Type: c6a.large - Count: 294 Instance Type: c6g.large - Count: 24 Instance Type: c6g.xlarge - Count: 2 Volume Type: gp3 - Total Size: 16238 GiB Volume Type: gp2 - Total Size: 722 GiB FedoraGroup: abrt Region: us-east-1 Instance Type: t3a.2xlarge - Count: 1 Volume Type: gp3 - Total Size: 10 GiB FedoraGroup: centos Region: ap-south-1 Instance Type: t3.2xlarge - Count: 3 Instance Type: t3.large - Count: 1 Instance Type: m6i.2xlarge - Count: 1 Volume Type: gp3 - Total Size: 4570 GiB Region: eu-central-1 Instance Type: t2.small - Count: 1 Instance Type: t2.large - Count: 2 Instance Type: r5b.8xlarge - Count: 1 Instance Type: m6i.2xlarge - Count: 1 Instance Type: t3.xlarge - Count: 1 Volume Type: gp3 - Total Size: 13200 GiB Region: us-west-2 Instance Type: m6i.2xlarge - Count: 2 Instance Type: t3.small - Count: 1 Instance Type: t3.xlarge - Count: 1 Volume Type: gp3 - Total Size: 13145 GiB Region: af-south-1 Instance Type: m6i.2xlarge - Count: 1 Volume Type: gp3 - Total Size: 4050 GiB Region: eu-west-3 Instance Type: t2.large - Count: 2 Instance Type: t2.2xlarge - Count: 1 Instance Type: t3a.xlarge - Count: 1 Instance Type: t2.small - Count: 2 Instance Type: t3.xlarge - Count: 1 Instance Type: t3.large - Count: 1 Volume Type: gp3 - Total Size: 2917 GiB Region: eu-west-2 Instance Type: t2.small - Count: 1 Instance Type: t2.large - Count: 1 Instance Type: t3a.large - Count: 2 Instance Type: r5a.8xlarge - Count: 1 Instance Type: t3.large - Count: 2 Instance Type: t3.xlarge - Count: 1 Instance Type: t2.medium - Count: 1 Instance Type: m6i.2xlarge - Count: 1 Volume Type: gp3 - Total Size: 22420 GiB Region: eu-west-1 Instance Type: t2.medium - Count: 4 Instance Type: t2.xlarge - Count: 3 Instance Type: t2.large - Count: 1 Instance Type: t2.small - Count: 2 Instance Type: t3.large - Count: 1 Instance Type: t3.medium - Count: 2 Volume Type: gp3 - Total Size: 584 GiB Region: ap-northeast-1 Instance Type: c6g.2xlarge - Count: 1 Instance Type: m6i.2xlarge - Count: 1 Volume Type: gp3 - Total Size: 8070 GiB Region: sa-east-1 Instance Type: c6g.4xlarge - Count: 1 Instance Type: m6i.2xlarge - Count: 1 Volume Type: gp3 - Total Size: 4250 GiB Region: ap-southeast-1 Instance Type: m6i.2xlarge - Count: 1 Volume Type: gp3 - Total Size: 4050 GiB Region: ap-southeast-2 Instance Type: m6i.2xlarge - Count: 2 Volume Type: gp3 - Total Size: 8100 GiB Region: us-east-1 Instance Type: t3.xlarge - Count: 2 Volume Type: gp3 - Total Size: 24350 GiB Region: us-east-2 Instance Type: t2.2xlarge - Count: 1 Instance Type: m5a.2xlarge - Count: 1 Instance Type: t3.xlarge - Count: 4 Instance Type: t3a.large - Count: 1 Instance Type: m6i.2xlarge - Count: 2 Instance Type: t2.large - Count: 2 Instance Type: t3.large - Count: 1 Instance Type: m4.2xlarge - Count: 1 Instance Type: t3.small - Count: 1 Instance Type: t3.medium - Count: 1 Volume Type: gp3 - Total Size: 44890 GiB Volume Type: io1 - Total Size: 200 GiB FedoraGroup: respins Region: us-east-1 Volume Type: gp3 - Total Size: 500 GiB FedoraGroup: centos-stream-build Region: us-east-2 Instance Type: t2.micro - Count: 1 FedoraGroup: garbage-collector Region: ap-southeast-1 Volume Type: standard - Total Size: 6 GiB FedoraGroup: min Region: eu-central-1 Instance Type: t3.2xlarge - Count: 1 Volume Type: gp3 - Total Size: 1000 GiB FedoraGroup: centos-stream-osci Region: ca-central-1 Instance Type: m5d.large - Count: 6 Instance Type: t2.micro - Count: 1 Instance Type: m5d.xlarge - Count: 3 Volume Type: gp3 - Total Size: 36 GiB Volume Type: gp2 - Total Size: 12 GiB FedoraGroup: infra Region: ap-south-1 Instance Type: c5d.xlarge - Count: 1 Volume Type: gp3 - Total Size: 8 GiB Region: eu-central-1 Instance Type: c5d.xlarge - Count: 1 Instance Type: c5n.2xlarge - Count: 1 Instance Type: m5.xlarge - Count: 1 Instance Type: m6gd.4xlarge - Count: 1 Volume Type: gp3 - Total Size: 16223 GiB Region: us-west-1 Instance Type: t3.medium - Count: 1 Volume Type: gp3 - Total Size: 40 GiB Region: us-west-2 Instance Type: m5.large - Count: 4 Instance Type: m6g.large - Count: 1 Instance Type: t3.large - Count: 1 Instance Type: c5n.2xlarge - Count: 1 Instance Type: c6gd.xlarge - Count: 1 Volume Type: standard - Total Size: 100 GiB Volume Type: gp3 - Total Size: 900 GiB Region: af-south-1 Instance Type: c5d.xlarge - Count: 2 Volume Type: gp3 - Total Size: 25 GiB Region: eu-west-2 Instance Type: c5d.xlarge - Count: 1 Volume Type: gp3 - Total Size: 20 GiB Region: ap-northeast-2 Instance Type: c5n.2xlarge - Count: 1 Volume Type: gp3 - Total Size: 196 GiB Region: sa-east-1 Instance Type: c5.2xlarge - Count: 1 Instance Type: c5n.2xlarge - Count: 1 Volume Type: gp3 - Total Size: 200 GiB Region: ap-southeast-1 Instance Type: c5d.xlarge - Count: 1 Instance Type: c5n.2xlarge - Count: 1 Volume Type: gp3 - Total Size: 108 GiB Region: us-east-1 Instance Type: t3.medium - Count: 1 Instance Type: t3.small - Count: 1 Instance Type: c5.xlarge - Count: 2 Instance Type: t2.medium - Count: 2 Instance Type: c5.2xlarge - Count: 1 Volume Type: gp3 - Total Size: 330 GiB Region: us-east-2 Instance Type: c5d.large - Count: 2 Instance Type: t2.micro - Count: 1 Instance Type: t3.medium - Count: 1 Volume Type: gp3 - Total Size: 113 GiB -- Miroslav Suchy, RHCA Red Hat, Manager, Packit and CPT, #brno, #fedora-buildsys

2 2

AWS discontinue IAM...
by Miroslav Suchý 30 Oct '23

30 Oct '23

From https://us-east-1.console.aws.amazon.com/iamv2/home?region=us-east-2#/home Attention: Extended Deadline for Updating Your Access Permissions - December 11, 2023 AWS has discontinued use of old IAM permissions for Billing, Cost Management, and Accounts consoles. If you do not take action, you may lose access to these consoles. We are granting an extension until December 11, 2023 for you to update your IAM policies.Update your policies or contact your organization's access administrator.For assistance, please visit the blog ,or review the policy migration utilities- Bulk Policy Migrator ,old to granular action mapping guide . The Billing does not worry us. But Cost Management and Account consoles? Is this something we should handle. Or we have everything handled? -- Miroslav Suchy, RHCA Red Hat, Manager, Packit and CPT, #brno, #fedora-buildsys

2 1

Dist-git decoupling investigation
by Michal Konecny 23 Oct '23

23 Oct '23

Hi everyone, We would like to inform you that Advanced Reconnaissance Crew (ARC) [0] from the CPE team is currently conducting research into decoupling dist-git from its pagure-related dependencies. This investigation has come about as part of discussions that arose from https://pagure.io/cpe/initiatives-proposal/issue/26 [1]. The investigation has the following objectives: 1. Enumerate and write down a description of all the integrations we rely on in our current git forge (Pagure) 2. Create a list of integrations to continue after moving to different git forge 3. Write a recommendation plan on how we can loosely couple these integrations generically with another git forge (make it git forge agnostic as possible) 4. Add the list and descriptions of integrations to the right place in our documentation This will then inform future design requirements and the next set of objectives that looks like this: * Map API calls that need to remain * Removing the Pagure from dist-git * Base git as backend ** What functionality needs to move to dist-git when git is used as base backend You can find a list of existing work items here: https://pagure.io/fedora-infra/arc/issues [2] You can also follow along with the findings on our HackMD: https://hackmd.io/@fnF231raRruGKZbPBWKsHQ/SJceJYaxa [3] On behalf of ARC, Michal P.S.: You can also discuss this on discourse: https://discussion.fedoraproject.org/t/dist-git-decoupling-investigation/93… [4] [0] https://fedora-arc.readthedocs.io/en/latest/workflow.html [1] https://pagure.io/cpe/initiatives-proposal/issue/26 [2] https://pagure.io/fedora-infra/arc/issues [3] https://hackmd.io/@fnF231raRruGKZbPBWKsHQ/SJceJYaxa [4] https://discussion.fedoraproject.org/t/dist-git-decoupling-investigation/93…

1 0

Meeting Agenda Item: Introduction Blake Ridgway
by Blake Ridgway 11 Oct '23

11 Oct '23

Good day Fedora Infrastructure team, I'm Blake Ridgway. My IRC alias is /Zormak/ and my Matrix alias is /@blakeridgway:fedora.im. /I am reaching out to introduce myself to the Fedora Infrastructure team and express my enthusiasm for collaboration. I serve as a System Administrator in the Agribusiness sector, where my role is diverse and vital for the organization's operational efficiency. My responsibilities encompass the maintenance of our Windows Server, Office 365 environment, Asterisk PBX system running on CentOS, and Ubuntu Server infrastructure. Beyond my professional duties, I'm actively engaged in crafting a custom application suite for a non-profit organization based in Oklahoma. I run Fedora on all of my hardware, which includes two servers, my desktop, and a variety of laptops. This comprehensive setup enables me to thoroughly test and verify the Operating System's compatibility across diverse hardware configurations. While I've been quietly observing and familiarizing myself with how the team operates and communicates, I'm not approaching this with a blank slate. I plan to explore the open issues and Easyfix tasks to identify areas where I can make a valuable contribution before tackling more longstanding challenges. I'm genuinely eager to contribute to the Fedora community. I am excited to both help offer my assistance and learn from the experienced members of the Fedora Infrastructure team. I look forward to hearing from you all. Thanks, Blake Ridgway

3 2

Freeze Break request: update koji-flatpak on builders
by Kevin Fenzi 05 Oct '23

05 Oct '23

We tried to get everything working with the new flatpak building setup before freeze, but at the last minute we ran into some issues between bodhi and the metadata that flatpak builds store in koji. We got several things sorted out yesterday before freeze, but there's still an issue with the metadata we need to fix. See: https://pagure.io/fedora-infrastructure/issue/11557 So, I'd like to tag that build into f38-infra and update it on all the builders and reload kojid. +1s? kevin

5 5

2026

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

infrastructure October 2023