infrastructure April 2023

infrastructure@lists.fedoraproject.org

23 participants
17 discussions

AWS gp2 -> gp3
by Miroslav Suchý 13 Sep '23

13 Sep '23

tl;dr I want to change **all** storage types in AWS from gp2 to gp3 next week. The recent Change proposal https://fedoraproject.org/wiki/Changes/CloudEC2gp3 brought to my attention that gp3 is actually faster and cheaper than gp2. The comparision is as follows: gp2: * $0.10 per GB-month * IOPS: 100 IOPB, burstable to 3000 IOPS * Throughput: Throughput limit is between 128 MiB/s and 250 MiB/s, depending on the volume size. gp3: * IOPS: 3000 for free, addional IOPS can be bought * Throughput 125 MiB/s for free, addional throughput can be bought For more information see https://aws.amazon.com/blogs/storage/migrate-your-amazon-ebs-volumes-from-g… We have about 150 volumes of gp2 type in AWS. Copr is the most prominent user, but there are other: libravatar, taiga, TF, and other... The type of the storage can be changed on the fly - even for attached and volumes in-use. I already tried that with my private server. Unless somebody will have objections I plan to change volume types to gp3 for Copr next week and for all other services in week following Easter. I also modified fedimg template to allow gp3 https://pagure.io/fedora-infra/ansible/c/034cf953db5d0519eb327136abd72a2ee6… If you have scripts outside of fedora-infra/ansible to provision instances that uses gp2 for volumes, it will be great if you can move it to gp3. Miroslav

4 18

This list and discussion.fedoraproject.org
by Kevin Fenzi 04 May '23

04 May '23

As some of you who follow the devel list may know, Matthew put forward the idea of starting to move some discussion over to discussion.fedoraproject.org (our hosted dicourse instance) instead of the mailing list. For those of you who don't know: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org… (and the long thread after it). Please do read that. I'll wait. ok. So, my question is: Should we look at moving to discussion.fedoraproject.org instead of this list? We have a tag there already. I've personally setup discourse to email me things, so it actually ends up being very similar to a list, with a few corner cases. I'm happy to help anyone who has trouble with that setup. Thoughts? kevin

7 7

clean up on aisle pagure.io
by Kevin Fenzi 27 Apr '23

27 Apr '23

Hey folks. So, a few weeks back I noticed some spam projects on pagure.io. So, I cleaned up about 165 of them and deactivated 165 spam users. Now, I see there's another pile of them. :( So, I started to look at cleaning things up again, but I think we need a better solution that doesn't involve admins manually cleaning things up. ;( Additionally, the clean up is not really very scripted and takes a long time to do. So, thoughts on a longer term solution? I can think of a few: 1. only allow fedora 'contributors' to make new projects. (ie, people in at least one non cla/non base ipa group Pros: - Would very likely cut off the spam or at least cut it way down. - Might be easy to implement? (not sure tho!) Cons: - Would block legit people who aren't fedora contributors. 2 Some kind of moderation for new projects Pros: - Would let non fedora folks make new projects. - Would likely cut spam Cons: - Would require someone to moderate things - Would requite us to make some kind of moderation code 3. A script to do all the cleanup so we could do it easier and some kind of 'bad words' blocklist we could put in place to stop obvious spammers (most of these are bogus "exam answers" ones) Pros: - Will cut down on spam some, but not fully. Cons: - Will have to write the script and implement the blocklist - It's likely spammers will use different words over time and avoid the block. Or perhaps someone has further clever ideas? Happy to hear em. ;) kevin

8 11

awscli2 and Fedora sso
by Miroslav Suchý 26 Apr '23

26 Apr '23

Has anyone use awscli2? How did you configure it to use Fedora SSO? I played with it, but with not luck. BTW awcli2 is now in package review process and is available at https://copr.fedorainfracloud.org/coprs/g/fedora-review/fedora-review-21894… Miroslav

2 1

AWS volume cleanup
by Miroslav Suchý 26 Apr '23

26 Apr '23

When I have been working in the AWS console I noticed we have lots of volumes that are: not described, not owned and not attached. Sometimes even combination of all of these. And some of such volumes have been created in 2020. Here is screenshot of suspicious volumes: https://k00.fr/1cn48fze This link to console with Name = Not tagged https://us-east-1.console.aws.amazon.com/ec2/home?region=us-east-1#Volumes:… Please check your volumes and add these tags: * Owner: * FedoraGroup * Name In a week I will write longer and detailed mail and everything that is not properly tagged (in a reasonable timeframe) will be proposed for deletion. Miroslav

2 3

Orphaning python-simplemediawiki
by Mattia Verga 25 Apr '23

25 Apr '23

I'm going to orphan python-simplemediawiki. Upstream is long time dead and the github repository was archived in 2018. It was used by Bodhi backend to fetch TestCases information from Fedora Wiki, but Bodhi 7.1 switched to python-pymediawiki. I cannot find any other use in fedora infrastructure. infra-sig is listed as co-maintainer, let me know if you want me to remove the group before orphaning the package. Mattia

2 1

Re: planet
by Pedro Moura 25 Apr '23

25 Apr '23

> On 2023-04-18 15:07, Pedro Moura wrote: > > Random thought: does the http server have any metrics or logs that could > be counted to give an indication of how many requests there are for each > feed (including the main one) over a period of time? For each feed, I don't think so About the survey results: 100% agreed to drop subplanets [image: Screenshot from 2023-04-25 14-46-49.png] I made a PR on noggin creating a new rss field https://github.com/fedora-infra/noggin/pull/1216 This is a work in progress, because Im having this error "Unknown option: fasrssurl" So If any of you knows how to fix it, the help is appreciated regards, Pedro Moura he/him/his Software Engineer Red Hat <https://www.redhat.com> pmoura(a)redhat.com <https://www.redhat.com> On Wed, Apr 19, 2023 at 5:59 PM Gerard Ryan <gerard(a)ryan.lt> wrote: > On 2023-04-18 15:07, Pedro Moura wrote: > > Thanks for the suggestions :) > > > > Just sent to announce mailing list. > > We could get planet users and send to them directly, but them would > > have an opinion only of those who are posting their blogs there and > > sending in announce we might have an opinion of those who are using > > planet also to read? > > Random thought: does the http server have any metrics or logs that could > be counted to give an indication of how many requests there are for each > feed (including the main one) over a period of time? > >

1 0

FMN replacement deployment
by Aurelien Bompard 21 Apr '23

21 Apr '23

Hey folks! I have a few questions about the final deployment of the FMN replacement: - There's been a request to handle a user being disabled in IPA, which should trigger their rules being disabled (FMN#826). We can do that but we have questions about re-enablement: should the rules be auto-enabled when the user is re-enabled? What about rules that the user may have disabled previously? Should we just leave things disabled? - How do you see the transition to the new system? We were thinking: - move the current FMN to a different URL, such as notifications-old.fp.o. It will still be processing messages and sending notifications - run the new system in notifications.fp.o (in place of the old) - add a small banner to the new system to point people to the old in case they want to change their rules there Is it too quick? Should we deploy the new one to a notifications-new URL first? That's one more step to get to the final setup, so more work for you. You get to decide :-) → Michal & Kevin are fine with this plan. - IRC account: we can't connect to libera.chat with the same account twice, we'll need a second account. I can create it, but do you prefer that we: 1. run the old FMN on the new account (fedora-notifs-old) and the new FMN on the usual account 2. run the old FMN on the usual account and the new FMN on the new account (fedora-notifs-new) and then switch back to the usual one when we retire the old FMN 3. run the old FMN on the usual account and the new FMN on a new account (fedora-fmn) and just drop the old account when we retire the old FMN (I guess that's connected to the URL transition issue) → Kevin prefers option 3. - Let's do an "ask us anything" session for infra people. We can do that on IRC or you can ask all your questions in this email thread. - When should we do the switch? → Kevin suggests next week, with an email to devel-announce. What do you think? I've had a few answers from Michal & Kevin already and added them inline. Kevin also notes that we may need a RHIT ticket to allow IRC out from OpenShift. I'll test it as soon as I've created the new IRC account. He also suggests a sunset date for the old FMN after the F39 release. Thanks! Aurélien

4 6

Failure to download metadata for “CentOS-PowerTools” repo
by Laura Flores 21 Apr '23

21 Apr '23

Hello, Around two weeks ago, I started seeing this error after running “sudo yum install -y kernel”. It seems to be an issue with the CentOS-PowerTools repo. Is this a known issue, or does anyone have any suggestions? This is happening on Centos Stream 8 and RHEL 8.6. Multiple mirrors are tried with no success. ``` Curl error (7): Couldn't connect to server for https://download-cc-rdu01.fedoraproject.org/pub/centos/8-stream/PowerTools/… [Failed to connect to download-cc-rdu01.fedoraproject.org port 443: Connection timed out] 2023-04-20T13:06:39.243 ``` Let me know if any additional information is needed. Thanks, Laura Flores

4 4

planet
by Kevin Fenzi 19 Apr '23

19 Apr '23

Hey folks. I thought I would open a discussion about fedoraplanet and possibly some plans for it. Right now: fedoraplanet.org runs on people02.fedoraproject.org (aka fedorapeople). To add a blog/rss feed you have to login there and edit your .planet file, then scripting pulls all those .planet files and tries to fetch all the feeds and then serves them up at http://fedoraplanet.org. It uses a app called 'venus' to do this. venus is written in very old python2 and very very dead upstream. We run into the following problems with it: * Sometimes it gets stuck and just stops processing until it's killed. * It's serving on a http site, which causes people to ask us to make it https, but that would just change the errors because many feeds it pulls are still http since they were added back before letsencrypt existed. * We have a handy 'website' field in our new account system, but aren't using it at all. * The .planet parsing is poor, any number of things can cause it to break. We have two open tickets on it: - https://pagure.io/fedora-infrastructure/issue/10383 (upgrade to pluto, a ruby based, but maintained thing) - https://pagure.io/fedora-infrastructure/issue/10490 ( planet not served via ssl) Which I am just going to close now. So, I can think of a number of options and would love everyone who has thoughts on it to chime in: 1. Do nothing. Venus "works" and .planet files are cool and retro. 2. Switch to pluto and use account system 'website' fields of contributors. We could likely shove it in openshift and serve it directly from there to avoid fedorapeople entirely. (This would likely break anyone who has multiple feeds in there) 3. Switch to something better/bigger. I would think (although I don't know) that there might be something that would not only aggregate rss feeds for contributors, but perhaps mastodon/twitter/whatever also. 4. Planets are old and tired, just drop the entire thing. People can maintain their own rss lists. 5. Planets are old and tired, just drop the entire thing. But also, get our social media people to maintain contributor / interesting lists. ie, the fedoraproject twitter account could maintain a list of 'fedora contributors' and 'fedora packagers' or whatever. 6. Switch to pluto as in 2, but also setup some curators. Have a 'firehose' of all feeds, but the main fedora planet would be just curated things that are known to be related to fedora and not off topic or unrelated. 6. Get someones (not it!) to take in all the twitter/facebook/mastodon/blog posts/rss feeds and post some kind of curated round up every week or something. 7. Your brilliant idea here! So, thoughts? this is not at all urgent, but we should end up doing something with it sometime. :) kevin

13 44

Fedora 38 Final freeze now in effect!
by Kevin Fenzi 19 Apr '23

19 Apr '23

Greetings. we are now in the infrastructure freeze leading up to the Fedora 38 Final release. This is a final release freeze. We do this to ensure that our infrastructure is stable and ready to release Fedora 38 when it's available. You can see a list of hosts that do not freeze by checking out the ansible repo and running the freezelist script: git clone https://infrastructure.fedoraproject.org/infra/ansible.git ansible/scripts/freezelist -i inventory Any hosts listed as freezes is frozen until 2023-04-18 (or later if release slips). Frozen hosts should have no changes made to them without a sign-off on the change from at least 2 sysadmin-main or rel-eng members, along with (in most cases) a patch of the exact change to be made to this list and/or a pull-request to the infra/ansible repo. Thanks, kevin

1 1

Meeting Agenda Item: Introduction Leon
by Leon Khan 17 Apr '23

17 Apr '23

Dear Fedora Infrastructure team, My name is Leon (*IRC handle : leonkhan, FAS user name leon82 *), and I am writing to introduce myself and express my interest in contributing to the Fedora Infrastructure team. I have a few years of experience as an M365 application support specialist. One of my areas of expertise is effective communication with people. Besides that, I am a hobbyist Fedora user, and I always find it very interesting how things work in a Linux environment. I am keen to dedicate myself to learning Linux system administration skills. I have my own testing machine and will be happy to offer at least 20 hours per week of my time to contribute to the Fedora Infrastructure team. However, as a new contributor, I am not sure about the first few steps that I need to take to get started. Therefore, I would like to politely ask if any group member could guide me on the first few steps of open-source contribution in the Fedora Infrastructure team. I am eager to contribute to the community and will try my best to make meaningful contributions to the Fedora Infrastructure team. I look forward to hearing from you! Thanks Leon

3 6

Not sure if this is the correct list but getting errors trying to do yum updates
by James Stroehmann 15 Apr '23

15 Apr '23

Getting 403 forbidden when talking to dl.fedoraproject.org at this IP address: 38.145.60.23 HTTP/1.1 403 Forbidden the other two IP addresses seem fine: 38.145.60.22 38.145.60.24

2 2

FBR add getfedora.org redirects
by darknao 14 Apr '23

14 Apr '23

Hello folks! I would like to submit a Freeze Break Request to add redirects[1] from getfedora.org to the new website[2] on fedoraproject.org. This will need to be applied during F38 launch on Tuesday. Can I get some +1s? [1]: https://pagure.io/fedora-infra/ansible/pull-request/1372 [2]: https://fedoraproject.org/ Thanks! -darknao

2 1

2 freeze breaks
by Kevin Fenzi 11 Apr '23

11 Apr '23

Hey folks. Two new freeze breaks I could use review on: 1. Adding a bot alias: https://pagure.io/fedora-infra/ansible/pull-request/1368 for https://pagure.io/fedora-infrastructure/issue/11232 2. Adding a new data-reports01 vm: https://pagure.io/fedora-infra/ansible/pull-request/1369 This is to allow Akash to delop some reports that the Fedora council would like to have for determining how many contributors we have, etc. This should be short lived, its just needed for developing things and once they are well established we can look at where best to run them and publish the data out. I think both of them should be pretty non distruptive. Also, my previous freeze break is still needing another +1: https://pagure.io/fedora-infra/ansible/pull-request/1366 (fixing rsync on download servers and others). Thanks! kevin

2 2

2 Freeze Break requests
by Andrew Heath 11 Apr '23

11 Apr '23

Hey all, I have two freeze break requests I would like reviewed. 1. Removing greenwave checks per ticket 11090 <https://pagure.io/fedora-infrastructure/issue/11090>Removed the NRPE check for greenwave as it is no longer needed according to the ticket. 2. Adding sysadmin-noc to busgateway to clean alerts Adding the sysadmin-noc group to buggateway to help troubleshoot and resolve the other alerts in ticket 11090 <https://pagure.io/fedora-infrastructure/issue/11090> Both requests are part of PR https://pagure.io/fedora-infra/ansible/pull-request/1367 Please let me know if you have any questions. -- Sincerely, Andrew Heath aheath1992(a)gmail.com

2 1

Freeze break request: fix up rsyncd role
by Kevin Fenzi 06 Apr '23

06 Apr '23

Right now our rsyncd role isn't enabling the systemd service on rhel9 hosts. So, I fixed that and also cleaned up the logic since we don't have any Fedora instances anymore than have xinetd. Only rhel7 and rhel8 have xinetd anymore. https://pagure.io/fedora-infra/ansible/pull-request/1366 Please check my logic in case I made any mistakes. :) +1s from any sysadmin-main/releng folks welcome. kevin

1 0

2026

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

infrastructure April 2023