infrastructure June 2020

infrastructure@lists.fedoraproject.org

33 participants
31 discussions

OIDC testing
by Benson Muite 12 Jun '20

12 Jun '20

Hi, Does one need to request anything other than client secrets through an infrastructure Pagure issue to be able to test application authentication with https://iddev.fedorainfracloud.org/ Regards, Benson

4 6

Re: Datacenter move day 2
by Kevin Fenzi 11 Jun '20

11 Jun '20

On Wed, Jun 10, 2020 at 04:31:39PM +0200, Dominik 'Rathann' Mierzejewski wrote: > On Wednesday, 10 June 2020 at 08:24, Kevin Fenzi wrote: > > Greetings everyone. > > > > Just a quick recap of our day 2 datacenter move. > > Thanks for all your hard work and for providing these (not so little) > updates! You're welcome. > > I hope you get some well-deserved rest after all is done. Me too. ;) I look forward to getting back to helping people get cool things done in our community. kevin

1 0

[Fedocal] Reminder meeting : Fedora Infrastructure
by smooge＠gmail.com 10 Jun '20

10 Jun '20

Dear all, You are kindly invited to the meeting: Fedora Infrastructure on 2020-06-11 from 15:00:00 to 16:00:00 UTC At fedora-meeting-1(a)irc.freenode.net The meeting will be about: Weekly Fedora Infrastructure meeting. See infrastructure list for agenda a day before. Source: https://apps.fedoraproject.org/calendar/meeting/9675/

1 0

Datacenter move day 2
by Kevin Fenzi 10 Jun '20

10 Jun '20

Greetings everyone. Just a quick recap of our day 2 datacenter move. Today was the buildsystem and all it's assocated things. * src.fedoraproject.org took longer than anticipated due to some repeated syncs to make sure we got all the content bit for bit. It's up now and should be working mostly normally. There's an issue with fedora-messaging bus that we hope to fix in the morning. * koji took a long time to move. Almost all of it in a database dump/restore. We actually did this instead of a replication or the like because we wanted to get out of the rhel7 postgresql 9.2 to a more recent version. We are now on rhel8 with postgresql 12.2. This gives us ability moving forward to do nice things like partitioning the large tables we have and in general I hope it will be faster. koji is also now on 1.21.0 release and all the builders (and now the hubs too) are Fedora 32. As for builders in the new datacenter, we have 32 buildvm's, 40 buildvm-ppc64's, the same number of s390x (15) and 15 each of buildvm-a64 (aarch6) and buildvm-a32 (armv7/armhfp). So, likely arm builds are going to be the last to finish for many. Luckily this only needs to last us a few weeks until we can get the rest of our arm builders shipped and re-added. Our signing infrastructure is up and tested, but we have a few tweaks to make to autosigning tomorrow morning. So, some builds may wait to be signed for a bit until thats processing away again. We have not yet does updates pushes or rawhide compose, it's 11pm now for me, so I think tomorrow is the better time to try those and clean up anything we hit. Additionally, adding new packages seems to mostly work, but it's emitting a error also, so we want to check that in the morning before processing down the existing queue of requests. ODCS, mbs, osbs and resultsdb were all moved over, but there's various issues bringing them up, so we will be doing that tomorrow. Tomorrow is fixing those things above, and moving mailman and datagrepper. After today we are on the downhill side I think. Thanks everyone for being patient while we get everything back to fully working. kevin

1 0

Day one of the datacenter service migration
by Kevin Fenzi 09 Jun '20

09 Jun '20

Greetings everyone. I thought I'd share with everyone how things went today and where we are at on the datacenter service migration. :) We did get everything we planns to migrated today: * staging services shutdown and machines with those resources readied to be shipped out. * fedora-messaging/fedmsg buses/clusters. We ran into a brief hiccup here as I hadn't properly issued the rabbitmq cluster certs with an alternative name for 'rabbitmq.fedoraproject.org', but we found it and fixed it. All consumers and producers should be connecting to the new rabbitmq cluster in the new datacenter now. * notifications service. Since this service is brittle and ancient we elected to just copy the vm over and adjust its settings for the new dc after. I am not 100% sure it's functioning properly yet (as it takes a while to start up), but it seems to be close. * pdc. This one was much longer than I expected, and I am sorry about that as it prevents people from committing. It turns out the pdc database is pretty gigantic, so it took a few hours to dump it out and load it in the new db server. ;( It's up and working now. * mirrormanager. There's some stats reporting and a few crons that may need twekaing, but I think the basic service is working. * Authenitcation stack (ipa, fas, ipsilon). I ran into a few snags here with routing and vpns, but everything should be moved over and working normally. On the unplanned side it turned out to be more complex than I had though to just move some of our openshift apps and not the others. Because of that I made the decision to just move all the (user facing) ones today. That includes: fas, ipsilon, bodhi, compose-tracker, elections, greenwave, waiverdb, mdapi and a few more non user facing ones. The openshift apps move caused a outage for the elections app (both a short one while it was entirely down, and another short time when authentication wasn't yet working), and additionally when bodhi was moved it was inadvertently restarted before it's database was synced over, so if you say a bunch of bodhi actions today where it said it was pushing things to stable that were already stable, that was the cause. ;( I quicked stopped the app and resynced the db, and hopefully not much damage was done. Tomorrow is the big day for koji and all it's assosicated services. This work will start at 15UTC, so if you have any builds to do, make sure they finish before then. Any in progress builds will be canceled at around 15UTC, you can resubmit them once things are back up. Thanks again for everyones patience with this move and hopefully we will survive the week. :) kevin

1 0

CPE Weekly: 2020-06-07
by Aoife Moloney 08 Jun '20

08 Jun '20

--- title: CPE Weekly status email tags: CPE Weekly, email --- # CPE Weekly: 2020-06-07 Background: The Community Platform Engineering group is the Red Hat team combining IT and release engineering from Fedora and CentOS. Our goal is to keep core servers and services running and maintained, build releases, and other strategic tasks that need more dedicated time than volunteers can give. See our wiki page here for more information:https://docs.fedoraproject.org/en-US/cpe/ ## General Project Updates Please check out our updated initiative timetable for briefing in new projects to our team here:https://docs.fedoraproject.org/en-US/cpe/time_tables/ *Note: Initiatives are large pieces of work that require a team of people and weeks/months to complete. Please continue to open tickets in the normal way for bugs, issues, etc. Dont forget to view our taiga board to see the projects we are currently working on, what we have scoped and whats in our backlog https://tree.taiga.io/project/amoloney1-cpe-team-projects/kanban?epic=null CPE Product Owner Office Hours: Thursdays @ 1300 UTC on #fedora-meeting-1 ## Fedora Updates ### Data Centre Move * A reduced services offering of Fedora will begin tomorrow, June 8th until July 28th, est. * This is to complete the final shipment of hardware from Phoenix to Washington, so please be patient and understanding during this timeframe as some services will be off and the rest, much slower. * Please read the below email sent by kfenzi if you have not already done so: https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedorapr… * Details on what this move may mean for you can be found here https://lists.fedoraproject.org/archives/list/devel-announce@lists.fedorapr… * If an application is not working correctly at all, please check this list https://hackmd.io/hpYYJQRjQy-oHxUS7IonIA?view before opening a ticket to make sure its not listed as being moved. If it is being moved, please wait a day or two, then try again. * Similarly, please be patient when opening tickets for service issues in general as we have now reached the critical point in this move and all of our sys-admins and wider teams will be assisting in the successful bringup of the reduced Fedora service and facilitation of the final hardware shipment and move. ### AAA Replacement * The team are working on implementing an aspect of the service that will allow users to select their applicable contributor agreement(s) as we are merging both Fedora and CentOS authentication system under the Noggin' solution. * The team have also added a blog pref to the feature set in Noggin * They have added pagination to the solution * And are working through redirecting applications to interface with the new API * Please feel free to check out the team kanban board for more information on the features the team are working on and have already completed here https://github.com/orgs/fedora-infra/projects/6 ### Mbbox * Project Dashboard here https://github.com/fedora-infra/mbbox/projects/1 * Sprint 4 is underway with the following work being addressed: * Kojira CRD & Documentation completed * MBS Backend CRD is almost done * And the team are now waiting on a staging environment to deploy and test in. We hope to have this in place by the end of this week ### Gitforge Good discussion on the CPE PO Office Hours meeting this Thursday, 4th June around the possibility of scheduling an AMA session/technical panel session with some folks from GitLab to allow the Fedora community a direct line to discuss services, potential blockers, etc. General feedback is that this will be welcome so I will work with GitLab and cverna, who is still leading the technical side of the project (link to ticket below) to plan the best time for this session & work with you all to set an agenda/discussion points for the meeting. NOTE: Due to the data centre move, this session will not happen before late August or early September at a minimum. Thank you for your engagement with us on this! Link to public issue ticket: https://gitlab.com/gitlab-org/gitlab/-/issues/217350 Meeting minutes log https://meetbot.fedoraproject.org/fedora-meeting-1/2020-06-04/cpe_po_office… ## CentOS Updates ### CentOS * CentOS Linux 8.2.2004 RC composes are with QA ### CentOS Stream * The team resolved all of the 8.2 build failures in CentOS Stream in their recent sprint (May 21st - June 5th) * An issue where a SIG was unable to push new content to git.centos.org was resolved, and is currently in staging * The team are also investigating how best to separate CentOS Linux branding from CentOS Stream * We are also focusing on building more packages in Stream for our current sprint (June 5th - 19th) and working on more automation and variance to help improve the current process and time it takes for our team to build packages As always, feedback is welcome, and we will continue to look at ways to improve the delivery and readability of this weekly report. Have a great week ahead! Aoife Source: https://hackmd.io/8iV7PilARSG68Tqv8CzKOQ?view -- Aoife Moloney Product Owner Community Platform Engineering Team Red Hat EMEA Communications House Cork Road Waterford

1 0

batcave moved to iad2
by Kevin Fenzi 05 Jun '20

05 Jun '20

Eariler today we moved things over to batcave01.iad2.fedoraproject.org batcave01.phx2.fedoraproject.org is still up, but all it's services are disabled and it has a /etc/nologin to keep anyone from using it. If you need to run playbooks or anything else you normally did on batcave01.phx2.fedoraproject.org, please do that not on batcave01.iad2.fedoraproject.org. ;) If something is broken, please file a ticket and we will look into it as soon as we are able. kevin

2 1

Upcoming fedoraproject Datacenter move reminder and plans
by Kevin Fenzi 05 Jun '20

05 Jun '20

Greetings. As previously announced, fedoraproject is moving many of it's servers from one datacenter (phx2 near phoenix, arizona, usa) to another (iad2: near arlington, virginia, usa). As we move from the old datacenter to the new, we will have a temporary reduction in capacity. The new datacenter has a smaller, less-redundant, lower-capacity version of our infrastructure. Over the next two weeks, we will migrate services to it so that we can finish moving out of the old datacenter. After everything is moved from the old datacenter, many of the servers there will be shipped to the new datacenter and then re-added to bring us back to full redundency and capacity. Out detailed checklist for these migrations is available at https://hackmd.io/@fedorainfra2020/rJpsA4FLL To summarize what we are moving when: 2020-06-03 wed: The fedoraproject master mirrors will move to IAD2. A very small outage may be noticed as dns changes. There may be some mirroring slowdowns as we work out bugs. 2020-06-04 thu: Our internal ansible control host and the fedoraproject wiki will move. The wiki will be down for a few hours. 2020-06-05 fri: Our meeting minutes archive (https://meetbot.fedoraproject.org) and our freenode irc bot (zodbot). These two services will see a hour outage or less. 2020-06-07 sun: We will pause for the next week adding new packages and unretiring packages to avoid problems. 2020-06-08 mon: Our fedora-messaging bus and gateways to it (github2fedmsg, bugzilla2fedmsg), mirrormanager, product definition center (pdc), and our identity and authentication systems. Messages over our message bus may be slow or missing and users may be unable to login at various times as we migrate services over. Additionally, we will be stopping services that will not be back until later in the month. These include: * Fedocal * Badges * Nuancier * koschei * simple-koji-ci * All staging services (*.stg.fedoraproject.org) 2020-06-09 tue: The build and packaging ecosystem. This includes koji, src.fedoraproject.org, osbs, odcs, container registries, bodhi (updates system). During this day maintainers should avoid builds/updates if at all possible as they may or may not work at various times. 2020-06-10 wed: Various small apps (mdapi, anitya, waiverdb, greenwave, etc), mailman/lists.fedoraproject.org, and our datagrepper/datanommer services. Mailing lists will be down for several hours as data is migrated. Datagrepper will be down for most of the day as it's database is moved. Other services will be down for short amounts of time while they are moved. 2020-06-11 thu: Various small site building apps (docs building, fedora websites building, reviewstats, blockerbugs) and elections will be moved. elections will be up until the currently running elections complete. (GO VOTE! https://elections.fedoraproject.org) 2020-06-12 fri: Catch up and fix issues day, along with re-enabling package unretirements/new packages, and other 'paused' items. The week after this servers will be shipped and the week after that we expect to start setting them up and getting them re-added. During this time, we may have to make further changes to what services are available in order to deal with load changes. If you have any questions or concerns, please file an infrastructure ticket ( https://pagure.io/fedora-infrastructure) or come talk to us in #fedora-admin on irc.freenode.net. Finally, I'd like to ask everyone to be patient as we do this move. I know that it's painful when you are unable to contibute something when you have time to do so, but rest assured that we are trying to migrate things as quickly and smoothly as we can. Thanks. kevin

2 4

Updating nauncier to work with Python 3
by Jonathan Trossbach 04 Jun '20

04 Jun '20

Hi everyone! I am a first time contributor to the Fedora project. As my first contribution, I am going to try and update nauncier to work with Python 3. I started this thread for two reasons: (1) to make sure that nobody has updated it already and (2) to ask questions should I get stuck somewhere. If you are aware of anyone doing this work already please let me know by responding here or if you have any suggestion on how to complete this work more efficiently also respond here. Thank you, Jon Trossbach

2 2

Updating fedocal to work with Python 3
by Jonathan Trossbach 03 Jun '20

03 Jun '20

Hi everyone! My name is Jon Trossbach and I am a first time contributor to the Fedora project. My first goal is to get fedocal working with Python 3. I am currently trying to use this set of Ansible playbooks (https://pagure.io/fedora-infra/ansible) to deploy fedocal in a CentOS 7 virtual machine using VirtualBox and Vagrant but I can't get the playbook to finish without running into some errors. So I guess my first question is: am I using these Ansible playbooks for their intended purpose in trying to set up an individual development environment for fedocal? Or should I just follow the README in its github repository (https://github.com/fedora-infra/fedocal)? I've attempted both ways of setting up a development environment and can't seem to get either to work. Once I know which way I should set up my development environment for fedocal I can share the spot where I'm getting stuck using that method. Thanks for your time in reading this. sincerely, Jon Trossbach

3 3

2026

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

infrastructure June 2020