infrastructure November 2020

infrastructure@lists.fedoraproject.org

23 participants
20 discussions

VDO
by Miroslav Suchý 04 Jul '26

04 Jul '26

After I seen the talk about VDO: https://www.youtube.com/watch?v=7CGr5LEAfRY I went ahead and tried it. I tried it on small (12GB) sample of Copr data and I saved 20-30% data. I then deployed it on production server retrace.fedoraproject.org and saved there 15% out of 2TB. Several notes: The RPM packages are only available for RHEL. Recently the public github has been updated with version of VDO, which should work on Fedora. No RPM available for Fedora yet though. Here are my notes: # vdo create --name=vdo1 --device=/dev/vdb --vdoLogicalSize=1T # mkfs.ext4 /dev/mapper/vdo1 Some slides suggest to use -E discard. This is not needed for new volumes. See vdo-devel mailing list for reasoning. # mount -o discard /dev/mapper/vdo1 /mnt/a Probably best scenario is to create vdo on whole disk and then make PV from whole VDO disk and put LVM on top of that. When you already have data, you cannot convert your disk to VDO. You need create new VDO disk, format it, mount it and copy the data. In my case, due limited space for migration I created a small LVM volume "vdosrv": # lvcreate -L 80G -n vdosrv vol0 I converted it to VDO which claims to have 2TB of vitual total size # vdo create --name=srv18 --device=/dev/vol0/vdosrv --vdoLogicalSize=2T # mkfs.ext4 /dev/mapper/srv18 See how many space is *actually* free. # vdostats # mkdir /mnt/srv18 # mount -o discard /dev/mapper/srv18 /mnt/srv18 Now I moved some data from old volume to /mnt/srv18. I shrank the FS of old volume, I shrank the old LV. And then I enlarged the new LV. # lvresize -L +400G /dev/vol0/vdosrv and let know vdo that it can use the new space: # vdo growPhysical -n srv18 I repeated this several times. I have to say that my experience is so far good. I hope that my notes help someone. Miroslav

10 14

About JS framework
by Pierre-Yves Chibon 17 Oct '22

17 Oct '22

Good Morning Everyone, Our infrastructure is mostly a python store, meaning almost all our apps are written in python and most using wsgi. However in python we are using a number of framework: * flask for most * pyramid for some of the biggest (bodhi, FAS3) * Django (askbot, Hyperkitty) * TurboGears2 (fedora-packages) * aiohttp (python3, async app: mdapi) While this makes sometime things difficult, these are fairly standard framework and most of our developers are able to help on all. However, as I see us starting to look at JS for some of our apps (fedora-hubs, wartaa...), I wonder if we could start the discussion early about the different framework and eventually see if we can unify around one. This would also allow those of us not familiar with any JS framework to look at the recommended one instead of picking one up semi-randomly. So has anyone experience with one or more JS framework? Do you have one that would you recommend? Why? Thanks for your inputs, Pierre

10 12

[PATCH 0/1] Add a flatpak-indexer openshift service
by Owen Taylor 01 Dec '20

01 Dec '20

From: "Owen W. Taylor" <otaylor(a)fishsoup.net> This is an initial attempt to create a configuration for flatpak-indexer to replace regindexer and add an image delta capability. The config here is derived from a working openshift configuration, but is untested in this form. See: https://pagure.io/fedora-infrastructure/issue/9272 Open questions: How to propagate content to the registry.fedoraproject.org reverse proxy ======================================================================== Currently the regindexer-generated content is rsync'ed from sundries to fedora-web/registry. How should this be done with flatpak-indexer running as an openshift app? Some possibilities that come to mind: - Run a rsyncd within the openshift app (either as a separate deploymentconfig or as a sidecar to the indexer) and expose a route to it internally in Fedora infrastructure. - Run a web server within the openshift app, expose a route to it internally in Fedora infrasturcture, and reverse proxy the content on fedora-web/registry instead of rsync'ing it. - Write the content onto a netapp volume, and mount that volume RO either on a host running rsyncd or directly on fedora-web/registry. What to use for a redis image ============================= Redis is used for caching and communication between the components. What redis image should be used? registry.redhat.io/rhel8/redis-5 needs configuration of a subscription docker.io/library/redis:5 centos/redis-5-centos7 don't rely on such images currently Custom Dockerfile image built from fedora:32 how would rebuilds be triggered? For the two other images needed here, I used ubi8 images - which aren't currently used elsewhere, but are presumably ok. How to handle identifying versions to build for staging/production ================================================================== I see that most openshift applications simply use 'staging'/'production' tags in the upstream repo, while a few take the approach of having specific hashes checked into the infrastructure ansible repository. Is the upstream tag approach considered sufficiently secure? (Making the service write a malicious index could allow causing users to upgrade to arbitrary application binaries.) Owen W. Taylor (1): Add a flatpak-indexer openshift service playbooks/openshift-apps/flatpak-indexer.yml | 56 +++++ .../reversepassproxy.registry-generic.conf | 34 ++- .../flatpak-indexer/files/imagestream.yml | 52 +++++ .../flatpak-indexer/files/service.yml | 16 ++ .../flatpak-indexer/files/storage.yml | 24 ++ .../flatpak-indexer/templates/buildconfig.yml | 84 +++++++ .../flatpak-indexer/templates/configmap.yml | 98 ++++++++ .../templates/deploymentconfig.yml | 221 ++++++++++++++++++ .../flatpak-indexer/templates/secret.yml | 11 + roles/regindexer/build/tasks/main.yml | 21 -- roles/regindexer/build/templates/config.yaml | 74 ------ 11 files changed, 584 insertions(+), 107 deletions(-) create mode 100644 playbooks/openshift-apps/flatpak-indexer.yml create mode 100644 roles/openshift-apps/flatpak-indexer/files/imagestream.yml create mode 100644 roles/openshift-apps/flatpak-indexer/files/service.yml create mode 100644 roles/openshift-apps/flatpak-indexer/files/storage.yml create mode 100644 roles/openshift-apps/flatpak-indexer/templates/buildconfig.yml create mode 100644 roles/openshift-apps/flatpak-indexer/templates/configmap.yml create mode 100644 roles/openshift-apps/flatpak-indexer/templates/deploymentconfig.yml create mode 100644 roles/openshift-apps/flatpak-indexer/templates/secret.yml delete mode 100644 roles/regindexer/build/tasks/main.yml delete mode 100644 roles/regindexer/build/templates/config.yaml -- 2.28.0

2 3

Is it possible delete my FAS account?
by Qian Cai 28 Nov '20

28 Nov '20

Hi there, in the process to delete my FAS account (caiqian), it looks like I can only mark it inactive. However, the docs said 'to remove all personal information, and mark the account as "Inactive."', so I "accidentally" also changed the email address to something totally random like xxx@xxx (I forgot the exactly). Later, I want to make sure that email is totally non-existent, so that nobody could access the FAS account anymore (it has some privileges). The bad news is that even though I can still remember the password, it now will ask me to enter the email address to reset the password since it was marked inactive. Since I don't remember the email and have no access to it, I can't reset it. Is there any way to remove the FAS account entirely?

2 3

Fedora packages site down
by Phil Clayton 27 Nov '20

27 Nov '20

For a while now https://apps.fedoraproject.org/packages/ has been reporting Service Unavailable The server is temporarily unable to service your request due to maintenance downtime or capacity problems. Please try again later. Has this packages site been retired or does a server just need a boot? :)

8 14

Potential service degradation - containers and flatpaks builds (OSBS) - 2020-11-26 09:30 UTC
by David Kirwan 25 Nov '20

25 Nov '20

There will be potential service degradation for OSBS starting at 2020-11-26 09:30 UTC Work is being carried out to add an aarch64 build cluster to production OSBS. While this work is being carried out you may experience some containers or flatpaks build failures. Please follow the outage ticket https://pagure.io/fedora-infrastructure/issue/9489 to keep track of its status. Thanks, Fedora OSBS Initiative -- David Kirwan Software Engineer Community Platform Engineering @ Red Hat T: +(353) 86-8624108 IM: @dkirwan

1 0

[Fedocal] Reminder meeting : Fedora Infrastructure
by smooge＠gmail.com 25 Nov '20

25 Nov '20

Dear all, You are kindly invited to the meeting: Fedora Infrastructure on 2020-11-26 from 08:00:00 to 09:00:00 America/Los_Angeles At fedora-meeting-1(a)irc.freenode.net The meeting will be about: Weekly Fedora Infrastructure meeting. See infrastructure list for agenda a day before. Source: https://apps.fedoraproject.org/calendar/meeting/9675/

2 1

CPE Weekly: 2020-11-22
by Aoife Moloney 23 Nov '20

23 Nov '20

Hi Everyone, Below is this week's CPE weekly for week ending 2020-11-22 for both Fedora & CentOS, and if you want to visit the hackmd link https://hackmd.io/8iV7PilARSG68Tqv8CzKOQ?view you can then use the header bar on your left to skip to Fedora or CentOS updates that interest you. ## General Project Updates Final project submission date for consideration in Q1 is Friday 27th November. If you have an initiative that may take weeks/months and multiple people to work on and want to request it to CPE, please follow the steps outlined in our initiatives repo and create your issue before 27th November https://pagure.io/cpe/initiatives-proposal Our updated initative timetable can be viewed here for 2021 https://docs.fedoraproject.org/en-US/cpe/time_tables/ Below are the projects the CPE team are currently working on for the months of October, November & December: * CentOS Stream Phase 4 - Build system services * Noggin Phase 4 - Data Migration of Fedora & CentOS Accounts, Community testing * OSBS for aarch64 - this will begin in November * Fedora Messaging Schemas - this work is continuing from Q3 and is being worked on part-time ### Misc #### GitLab New GitLab topic sent to devel-announce(a)lists.fedoraproject.org & centos-devel(a)centos.org on Message Bus is out. See email in hackmd here https://hackmd.io/oZrDwbSeSWO-l_X65A1ndg?view ## Project Updates *The below updates are pulled directly from our CPE team call we have every week.* ### Fedora ### Staging Environment * Completed - any issues you find please report them in fedora infra https://pagure.io/fedora-infrastructure/issues ### Noggin/AAA * Testing team owned apps in staging with Noggin * We will be requesting community member testing before December so keep an eye out for an announcement! * The teams kanban board where they track their work can be found here https://github.com/orgs/fedora-infra/projects/6 * And we have a project tracker available to be viewed here https://github.com/fedora-infra/aaa-tracker ### OSBS for aarch64 * Basic OKD 3.11 working on aarm64 with F31 * Working on repeating that install with F33 * Next step will be to ### Fedora Messaging Schemas * This project is worked on on a part time basis as we are prioritizing completing Noggin first before fully committing to its completion * There is a list of applications that require messaging schemas can be found here https://hackmd.io/@nilsph/H1i8CAbkP/edit * There is a readme which contains documentation on messaging schemas, a cookie-cutter template to create the schema and a definition of Done for writing a schemas https://github.com/fedora-infra/fedora-messaging-schemas-issues * The board they are working from can be viewed here https://github.com/orgs/fedora-infra/projects/7 ## CentOS Updates ### CentOS * CentOS 6 is EOL 30th November * CFP for FOSDEM Dojo - https://wiki.centos.org/Events/Dojo/FOSDEM2021 * Updated CentOS CI Openshift staging cluster to latest 4.6.4, Waiting for stable release in the 4.6 branch before rolling out to production. * CentOS 7.9.2009 is released! (for x86_64, i386, ppc64, ppc64le, armhfp and aarch64 architectures) * Lot of work done for Noggin/AAA ### CentOS Stream * Use centos-stream-release package to convert from CentOS 6 to CentOS Stream before 30th November * Working on integrating ODCS in Stream * Curating out t_functional suite https://github.com/centos/sig-core-t_functional * Refining our testing for finding issues at distro-level ## Team Info ### CPE Product Owner Office Hours IRC office hours are now once per month.Below are the logs from the most recent meetings and dates for the next ones. #### #fedora-meeting-1 * Next Meeting: 2020-12-17 @ 1300 UTC on #fedora-meeting-1 #### #centos-meeting * Next Meeting: 2020-12-15 @ 1500 UTC on #centos-meeting ## Background: The Community Platform Engineering group, or CPE for short, is the Red Hat team combining IT and release engineering from Fedora and CentOS. Our goal is to keep core servers and services running and maintained, build releases, and other strategic tasks that need more dedicated time than volunteers can give. See our wiki page here for more information:https://docs.fedoraproject.org/en-US/cpe/ As always, feedback is welcome, and we will continue to look at ways to improve the delivery and readability of this weekly report. Have a great week! Aoife Source: https://hackmd.io/8iV7PilARSG68Tqv8CzKOQ?view -- Aoife Moloney Product Owner Community Platform Engineering Team Red Hat EMEA Communications House Cork Road Waterford

1 0

[Fedocal] Reminder meeting : Fedora Infrastructure
by smooge＠gmail.com 18 Nov '20

18 Nov '20

Dear all, You are kindly invited to the meeting: Fedora Infrastructure on 2020-11-19 from 08:00:00 to 09:00:00 America/Los_Angeles At fedora-meeting-1(a)irc.freenode.net The meeting will be about: Weekly Fedora Infrastructure meeting. See infrastructure list for agenda a day before. Source: https://apps.fedoraproject.org/calendar/meeting/9675/

1 0

ancient backups of the moinmoin wiki?
by Matthew Miller 17 Nov '20

17 Nov '20

This is a long shot, I know. When we converted to MediaWiki from MoinMoin in 2008, pages were imported, but history was lost. I often like to go spelunking in the past, and it's often frustrating to hit this barrier. As I recall, MoinMoin stored everything in flat files, so it wouldn't even really need to be stood up again to get a look at the history. If someone happened to keep it, that is. Is there any chance we have a tarball snapshot of the wiki circa 2008? -- Matthew Miller <mattdm(a)fedoraproject.org> Fedora Project Leader

5 5

dnf-automatic role: dnf makecache masked through timer
by Pavel Raiskup 16 Nov '20

16 Nov '20

JFTR, I applied https://pagure.io/fedora-infra/ansible/c/cc79e327dc74e615e3c008a811c09950c7… which affects everyone I guess. One may need to run `systemctl reset-failed` too manually if the timer happened to fail already. Pavel

2 1

Request for Feedback/Resources for Packager Dashboard/Oraculum
by Frantisek Zatloukal 16 Nov '20

16 Nov '20

Hello, many of you already heard about Fedora Packager Dashboard [0]. In short, for those who have not, it's a web application and backend aiming to make the life of Fedora packagers easier. It combines data from multiple sources (Pagure, Bugzilla, Bodhi, Koschei,...) relevant to the maintainers of Fedora packages. Tracking all these sites can be time consuming, especially if you maintain dozens of different packages, so the Dashboard provides everything a packager might need (or at least what we've thought of) - condensed, cached, searchable and filterable on one page. You can check it out/play with it even if you don't maintain any Fedora packages, there is no authentication needed, just enter any packager’s username. Feature-wise, it's pretty robust already, and there are more things to come (like allowing users to authenticate and see private bugs and more), but the original planned feature set is complete. Currently, it's processing only publicly available data. When we'll start implementing the ability to authenticate and process any private data, we'll work closely with the Infra team to make sure there are no open holes. Since the announcement of the testing phase, it has been running on a temporary server which is barely keeping up, so I'd like to open up a conversation about migration to the Infra OpenShift cluster. Here is a brief overview of it's internals (I can elaborate more if anybody needs me to): Backend is a Flask/Python application leveraging Celery for planning and executing the cache refreshes. The API is striving to be as non-blocking as possible, using asynchronous-inspired behaviour. The client is given the data currently available in cache, and advised about the completeness (complete/partial cache misses) via HTTP status code. The parameters for cache refreshes can be customized, depending on the resources we have/can get. Currently, it's retrieving data for most of the items every 2 hours (with exceptions like package versions which run daily and are terribly slow). Backend is caching data for PRs, bugs, and pre-calculated updates/overrides data for users visiting the app at least once in two weeks. The main storage is a PostgreSQL database, and optionally, if RAM is not an issue, we have a local memory-cached layer that can be enabled (we find it not necessary ATM). Apart from storing the pre-crunched information from public sources, we keep timestamps of the last visit for each user. Frontend is a React app fetching and displaying data from the backend (really nothing to add here :) ). Based on the OpenShift testing, I've come to the following schema of pods: - Redis pod (Celery backend) - PostgreSQL pod (cache and watchdog data storage) - Beat pod (scheduling tasks for data refresh) - Flower pod (backend monitoring; nice to have, not absolutely necessary) - Gunicorn/NGINX pod (Oraculum backend) - NGINX pod (Packager Dashboard front-end). On top of that, we need a number of worker pods completing the scheduled Celery tasks. I am not sure how much RAM each pod in Infra OpenShift has, currently the workers seem to be performing best with about 512 MB memory limit. Ideally we’d like to have at least 12-16 Celery workers (more workers can, of course, run on a single pod). Resource-wise it's not a small application (at least from my perspective :D ), but we believe it's a great value application which saves time for Red Hat and community package maintainers. I'd like to hear your feedback, questions, requests for changes if there is anything preventing it from deployment in Infra OpenShift (architecture/code wise). And of course opinions on the feasibility of moving Oraculum and Packager Dashboard into the Infra OpenShift cluster. Thanks! [0] https://packager.fedorainfracloud.org/

4 11

DKIM & fedora lists
by Dominique Martinet 15 Nov '20

15 Nov '20

Hi, I just enabled dkim/dmarc on my domain last week, so sent my first email with that setup to devel@ just earlier today and got a few "invalid dkim signature" return emails... So I was wondering how do people deal with that? Normally lists have two ways of handling dkim: - either they don't mangle the subject/signed headers (for me: h=Date:From:To:Cc:Subject:References:In-Reply-To:From). In that case, just leave the original dkim header and things should just work™. That's what e.g. kernel lists do and worked well for me. - either they DO mangle headers, often adding a [tag] to the subject line; in which case the From is also updated to be the list address with the original sender name (e.g. Bob <whateverlist@somewhere>) and the original mail is eventually appended to the Reply-To addresses, with the original dkim header stripped off. As far as I can see devel(a)lists.fedoraproject.org doesn't mangle anything so should fall into the first category of "doing nothing just works" -- but it stripped my original dkim header, hence the failures. I'm pretty sure mailman can deal with this, is that on purpose? Or is it just a mishap? my dmarc policy says to ignore dkim failures (for now) so I could just ignore this but it's a bit annoying that I had setup dmarc/dkim because my mails often get treated as spam for some reason and such errors won't be helping... Thanks! PS: I subscribed to the infra list just for this, but did look through the archives quickly and didn't find much, sorry if this had been brought up before. I'll likely unsubscribe back once that's answered one way or another. -- Dominique

4 5

best place to run an email-forwarding bot account?
by Matthew Miller 12 Nov '20

12 Nov '20

The Fedora Council is moving from our mailing list to discussion.fedoraproject.org. However, I want to have a digest of discussion go to the old list, for people who prefer to follow a summary that way. Discourse has a digest feature, but no "automatically send a digest to this address" ability separate from individual user accounts. So, I was thinking I could create a "fedora-council-discuss-bot" FAS account, and have that email go to somewhere with a procmail filter which forwards the digests (and only the digests, obviously) to the council list. Then I'd set the list to only allow posts from that bot address. 1. Does this plan seem reasonable? 2. Is there a place in Fedora Infrastructure that can receive and bounce mail like that? -- Matthew Miller <mattdm(a)fedoraproject.org> Fedora Project Leader

3 4

[Fedocal] Reminder meeting : Fedora Infrastructure
by smooge＠gmail.com 11 Nov '20

11 Nov '20

Dear all, You are kindly invited to the meeting: Fedora Infrastructure on 2020-11-12 from 08:00:00 to 09:00:00 America/Los_Angeles At fedora-meeting-1(a)irc.freenode.net The meeting will be about: Weekly Fedora Infrastructure meeting. See infrastructure list for agenda a day before. Source: https://apps.fedoraproject.org/calendar/meeting/9675/

1 0

Planned Outage: 2020-11-11 21:00 UTC Fedora Services
by Stephen John Smoogen 09 Nov '20

09 Nov '20

There will be an outage starting at 2020-11-11 21:00UTC, which will last approximately 6 hours. To convert UTC to your local time, take a look at http://fedoraproject.org/wiki/Infrastructure/UTCHowto or run: date -d '2017-05-03 21:00UTC' Reason for outage: Apply updates and new kernels for all systems running RHEL7/RHEL8 and Fedora 32 Affected Services: Most services in IAD2 will see downtime as we run updates and reboot systems. Ticket Link: https://pagure.io/fedora-infrastructure/issue/NNNN Please join #fedora-admin or #fedora-noc on irc.freenode.net or add comments to the ticket for this outage above. -- Stephen J Smoogen.

1 1

CPE Weekly: 2020-11-08
by Aoife Moloney 08 Nov '20

08 Nov '20

Hi Everyone, Below is this week's CPE weekly for week ending 2020-10-25 for both Fedora & CentOS, and if you want to visit the hackmd link https://hackmd.io/8iV7PilARSG68Tqv8CzKOQ?view you can then use the header bar on your left to skip to Fedora or CentOS updates that interest you. ## General Project Updates We've updated our How to work with us section on our wiki https://docs.fedoraproject.org/en-US/cpe/initiatives/ Updated initative timetable can be viewed here https://docs.fedoraproject.org/en-US/cpe/time_tables/ And we have a new initiative repo live on Pagure https://pagure.io/cpe/initiatives-proposal Below are the scheduled projects the CPE team are currently working on for the months of October, November & December: * CentOS Stream Phase 4 - Build system services * Noggin Phase 4 - Data Migration of Fedora & CentOS Accounts, Community testing * OSBS for aarch64 - this will begin in November * Fedora Messaging Schemas - this work is continuing from Q3 and is being worked on part-time ### Misc #### GitLab New GitLab topic sent to devel-announce(a)lists.fedoraproject.org & centos-devel(a)centos.org on Message Bus is out. See email in hackmd here https://hackmd.io/tfOqCXNEQtqsGNLAEfZ2zg?view ## Project Updates *The below updates are pulled directly from our CPE team call we have every week.* ### Fedora * F33 is released!!!! * F33 arm images are re-created and pushed as a point release * Testdays app up in openshift stg: https://testdays.stg.fedoraproject.org/events and may even be in production at the time of this email! * Openqa has all hardware up and running except one broken power9 box - working on resolving this ### Staging Environment * Build system nearly done - waiting on a firewall change ### Noggin/AAA * If you maintain an application that uses FAS, please test your app authentication in staging and report any issues you may find to the team in #fedora-aaa. * The team are particularly interested in if you cannot login or your user does not have the right groups/permissions/attributes, eg [cla_done] * The teams kanban board where they track their work can be found here https://github.com/orgs/fedora-infra/projects/6 * And we have a project tracker available to be viewed here https://github.com/fedora-infra/aaa-tracker ### OSBS for aarch64 * The team are tracking their work here https://pagure.io/cpe/osbs-project/issues * This project started on November 4th and the team are investigating technical approaches and requirements such as permissions & access to images. ### Fedora Messaging Schemas * This project is worked on on a part time basis as we are prioritizing completing Noggin first before fully committing to its completion * There is a list of applications that require messaging schemas can be found here https://hackmd.io/@nilsph/H1i8CAbkP/edit * There is a readme which contains documentation on messaging schemas, a cookie-cutter template to create the schema and a definition of Done for writing a schemas https://github.com/fedora-infra/fedora-messaging-schemas-issues * The board they are working from can be viewed here https://github.com/orgs/fedora-infra/projects/7 ## CentOS Updates ### CentOS * Update CentOS CI Openshift clusters to latest 4.5.16 * Storage resync completed this week. ### CentOS Stream * The team are working on developing ODCS service in the Stream build infra * Developing scripts to help automate adding artifacts to module builds * Check out the CentOS Stream page on centos.org/centos-stream for information on the latest download information and how to convert from CentOS Linux to Stream - especially if you are running on CentOS Linux 6 which is EOL on November 30th :) ## Team Info ### CPE Product Owner Office Hours IRC office hours are now once per month.Below are the logs from the most recent meetings and dates for the next ones. #### #fedora-meeting-1 * Next Meeting: 2020-11-12 @ 1300 UTC on #fedora-meeting-1 #### #centos-meeting * Next Meeting: 2020-11-10 @ 1500 UTC on #centos-meeting ## Background: The Community Platform Engineering group, or CPE for short, is the Red Hat team combining IT and release engineering from Fedora and CentOS. Our goal is to keep core servers and services running and maintained, build releases, and other strategic tasks that need more dedicated time than volunteers can give. See our wiki page here for more information:https://docs.fedoraproject.org/en-US/cpe/ As always, feedback is welcome, and we will continue to look at ways to improve the delivery and readability of this weekly report. Have a great week! Aoife Source: https://hackmd.io/8iV7PilARSG68Tqv8CzKOQ?view -- Aoife Moloney Product Owner Community Platform Engineering Team Red Hat EMEA Communications House Cork Road Waterford

1 0

Possible outdated Jenkins jobs
by Leonardo Rossetti 05 Nov '20

05 Nov '20

4 12

[Fedocal] Reminder meeting : Fedora Infrastructure
by smooge＠gmail.com 04 Nov '20

04 Nov '20

Dear all, You are kindly invited to the meeting: Fedora Infrastructure on 2020-11-05 from 15:00:00 to 16:00:00 UTC At fedora-meeting-1(a)irc.freenode.net The meeting will be about: Weekly Fedora Infrastructure meeting. See infrastructure list for agenda a day before. Source: https://apps.fedoraproject.org/calendar/meeting/9675/

1 0

Another Rust MirrorManager experiment
by Adrian Reber 03 Nov '20

03 Nov '20

Our MirrorManager setup exports the current state of all mirrors every hour at :30 to a protobuf based file which is then used by the mirrorlist servers to answer the requests from yum and dnf. The Python script requires up to 10GB of memory and takes between 35 and 50 minutes. The script does a lot of SQL queries and also some really big SQL queries joining up to 6 large MirrorManager tables. I have rewritten this Python script in Rust and now it only needs around 1 minute instead of 35 to 50 minutes and only 600MB instead of 10GB. I think the biggest difference is that I am almost not doing any joins in my SQL request. I download all the tables once and then I do a lot of loops over the downloaded tables and this seems to be massively faster. As the mirrorlist-server in Rust has proven to be extremely stable over the last months we have been using it I would also like to replace the mirrorlist protbuf input generation with my new Rust based code. I am planing to try out the new protobuf file in staging in the next days and would then try to get my new protobuf generation program into Fedora. Once it is packaged I would discuss here how and if we want to deploy in Fedora's infrastructure. Having the possibility to generate the mirrorlist input data in about a minute would significantly reduce the load on the database server and enable us to react much faster if broken protobuf data has been synced to the mirrorlist servers on the proxies. Adrian

4 17

2026

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

infrastructure November 2020