COVID-19 contingency plans for infra and release engineering
by Ben Cotton
Hi teams,
As many of you already know, Red Hat has asked all employees to work
from home. I know that many (most?) of you don't work from a Red Hat
office normally (particularly if you're not a Red Hat employee!), but
I want to check on the impact that an extended shutdown might have on
the infrastructure, particularly the release engineering activities as
we approach the Fedora 32 GA (currently scheduled for 21 April).
In addition, what are our personnel risks? For example, if Mohan gets
too sick to work, do we have enough shared knowledge and access to cut
a release?
Obviously, everyone is going to be impacted in some way, even if none
of us get sick. This is a stressful and uncertain time. I've made it
clear to Fedora and Red Hat leadership that we will continue with the
schedule, but that the priority is the health of our community
members. I don't want anyone working beyond what's reasonable for
their personal situation. I just need to understand the risks so we
can make plans to mitigate them or adjust the release schedule.
Thanks,
BC
--
Ben Cotton
He / Him / His
Senior Program Manager, Fedora & CentOS Stream
Red Hat
TZ=America/Indiana/Indianapolis
3 years
CPE WEekly: 2020-03-14
by Aoife Moloney
---
title: CPE Weekly status email
tags: CPE Weekly, email
---
# CPE Weekly: 2020-03-06
Background:
The Community Platform Engineering group is the Red Hat team combining
IT and release engineering from Fedora and CentOS. Our goal is to keep
core servers and services running and maintained, build releases, and
other strategic tasks that need more dedicated time than volunteers
can give.
For better communication, we will be giving weekly reports to the
CentOS and Fedora communities about the general tasks and work being
done. Also for better communication between our groups we have
created #redhat-cpe on Freenode IRC! Please feel free to catch us
there, a mail has landed on both the CentOS and Fedora devel lists
with context here.
## Fedora Updates
* Fedora 32 Beta work is underway
* Fedora Minimal Compose is being worked on with a dedicated ODCS
backend running being investigated
* Check out the Fedora Council video meeting to find out more about
our team here https://www.youtube.com/watch?v=QN8HqBeoozo
* Take a look at our (work in progress!) project taiga board to see
what we're working on now, next, and still to do!
https://tree.taiga.io/project/amoloney1-cpe-team-projects/kanban?epic=null
* Have a read of one of our teams article in Fedora magazine:
https://fedoramagazine.org/fish-a-friendly-interactive-shell
### Data Centre Move
* Communishift will be unavailable from 2020-04-13 until 2020-05-08
* Check out our detailed move shedule here
https://hackmd.io/R3EkjzVyTG2TYwQvkfzYrA?sync=&type=
### AAA Replacement
* OTP & 2FA authentication is now working!
* Fedora messaging is also integrated in the project
* As always, check out our progress on github here
https://github.com/orgs/fedora-infra/projects/6
### CI/CD
* Progress on tagging package history: tests, robustness
* Progress on koji plugin to run the rpmautospec
* Most work in this repo: https://pagure.io/Fedora-Infra/rpmautospec
### Sustaining Team
* Mbbox upgrade work underway with the team looking at operators to
deploy and maintain the different services
* Infra tickets are starting to be prioritized
https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedora...
* Looking at the ansible operator https://github.com/fedora-infra/mbbox/issues/5
* Creating mbbox sustaining project
https://github.com/fedora-infra/mbbox/projects/1
* Vagrant OpenShift 3.11 for mbbox https://github.com/fedora-infra/mbbox/pull/1
## Docs
### Misc Updates/Review Requests
* PR Reviews needed for the following:
* Vagrant Operator SDK for mbbox review
https://github.com/fedora-infra/mbbox/pull/9
* Rebase Silverblue to Fedora 32 - multi monitor setup issue
https://bugzilla.redhat.com/show_bug.cgi?id=1810070
* Fedora magazine: Article proposals about Silverblue rebase to F32
https://pagure.io/fedora-magazine-proposals/issue/60
https://pagure.io/fedora-magazine-proposals/issue/59
* Packit integration in the-new-hotness
https://github.com/packit-service/packit/issues/689
* KeepassXC flatpak issue https://pagure.io/flatpak-module-tools/issue/6
* Jms-messaging-plugin reviews
https://github.com/jenkinsci/jms-messaging-plugin/pull/162
* Misc Updates:
* Helped firefox maintainer sort out koji auth
* Gathered logs for qgis maintainer to debug why builds hang
* Checked over rawhide compose after open-h264 repo was enabled by
default to make sure no content from it got in.
* PR sent to fix f32 media with rawhide kickstarts
* Migrated fedora-comps to weblate.
* Upgraded koji to 1.20.1 via freeze break
* PR's Created this week:
* Bodhi 5.2 PR reviews
https://github.com/fedora-infra/bodhi/pull/3945
https://github.com/fedora-infra/bodhi/pull/3940
https://github.com/fedora-infra/bodhi/pull/3933
https://github.com/fedora-infra/bodhi/pull/3931
https://github.com/fedora-infra/bodhi/pull/3910
https://github.com/fedora-infra/bodhi/pull/3903
https://github.com/fedora-infra/bodhi/pull/3798
* Infra tickets
https://pagure.io/fedora-infrastructure/issue/8684
https://pagure.io/Fedora-Infra/review_stats/issue/2
* Fedmsg-koji-plugin
https://pagure.io/koji-fedmsg-plugin/pull-request/12
https://pagure.io/koji-fedmsg-plugin/pull-request/13
* Fedora-messaging
https://github.com/fedora-infra/fedora-messaging/pull/163
## CentOS Updates
### CentOS
* CentOS CI uptime is 100%
* Kickstarter for CentOS 8 arch ppc64le
* Check out the mail thread about dealing with unshipped -devel
packages for CentOS Linux & Stream too
https://lists.centos.org/pipermail/centos-devel/2020-March/036644.html
### CentOS Stream
* Python38 module is running in c8S!
* More work being done on the contributor workflow too with help from
the Packit Team
As always, feedback is welcome, and we will continue to look at ways
to improve the delivery and readability of this weekly report.
Have a great weekend!
Aoife
Source: https://hackmd.io/8iV7PilARSG68Tqv8CzKOQ
--
Aoife Moloney
Product Owner
Community Platform Engineering Team
Red Hat EMEA
Communications House
Cork Road
Waterford
3 years
Freeze Break Request
by Mohan Boddu
Hi,
I need to install compose-utils on bodhi-backend01.phx2.fp.o to use
compose-partial-copy which will rsync the compose to the /pub/.
+1's please.
Thanks.
3 years
[kevin@scrye.com: NOTICE: Upcoming extended downtime for Fedora
Project Communishift]
by Kevin Fenzi
Just forwarding on this to the infrastructure list too
in case anyone else needs to know. Spread the word. :)
kevin
--
----- Forwarded message from Kevin Fenzi <kevin(a)scrye.com> -----
Date: Fri, 13 Mar 2020 12:55:36 -0700
From: Kevin Fenzi <kevin(a)scrye.com>
To: communishift-members(a)fedoraproject.org
Cc: kevin(a)scrye.com
Subject: NOTICE: Upcoming extended downtime for Fedora Project Communishift
Greetings,
You are receiving this email because you are in the Fedora Project
Communishift system group that tracks users who have been
granted provisioning ability in Communishift.
We would like to notify you of some upcoming extended downtime
for the communishift cluster. As some of you may know, later
this year the Fedora Project is moving entirely out of our main
datacenter in Phoenix, AZ, USA (where the communishift cluster
is currently located).
We explored various options to migrate the cluster to minimize
downtime, but we just don't have extra hardware or resources to
setup a duplicate cluster anywhere currently. So, unfortunately,
that means we will need to power off the cluster, ship it to a
new datacenter and bring it back up there.
The estimated timeline for this looks like:
2020-04-13 - power off, derack, pack and ship
2020-04-20 - Arrival at new DC (or before)
2020-04-23 - Unpack, re-rack, cable, network
2020-04-30 - Reconfigure/reinstall
2020-05-08 - fully back in service
Note that these dates are subject to shipping and availability of
people to rack and setup and other items beyond our control.
We will update the following ticket with progress/status:
https://pagure.io/fedora-infrastructure/issue//8748
To be ready for this outage, we recommend you back up all your
application configuration and data. Due to network changes we may
have to reinstall the cluster and thus you may need to reload your
applications/data.
You can find information on this at:
https://docs.openshift.com/dedicated/3/admin_guide/assembly_backing-up-re...
Please let us know if there is anything we can do to assist
you during this move or if you have any questions for us.
kevin
----- End forwarded message -----
3 years