CPE PO Office Hours Cancellation Today, 18th June 2020
by Aoife Moloney
Hi All,
Due to a personal conflict the CPE PO Office Hours is cancelled today on
#fedora-meeting-1 @ 1300 UTC.
As Fedocal is down due to the datacentre move I am unable to remove the
slot, so please disregard the calendar meeting today.
The office hours will resume next week as scheduled and I look forward to
talking to you all then!
Kindest regards,
Aoife
--
Aoife Moloney
Product Owner
Community Platform Engineering Team
Red Hat EMEA <https://www.redhat.com>
Communications House
Cork Road
Waterford
<https://www.redhat.com>
3 years, 3 months
Staging
by Kevin Fenzi
Greetings everyone.
As you hopefully know, we took down our old staging env in phx2 as part
of the datacenter move. Once machines from phx2 are shipped to iad2 and
racked and installed and setup we can look at re-enabling our staging
env.
However, I'd like to ask everyone about how we want that to look.
Some questions:
* Before we had a staging openshift with staging applications in it.
This is sort of not how openshift is designed to work. In the ideal
openshift world you don't need staging, you just have enough tests and
CI and gradual rollout of new versions so everything just works.
Granted a staging openshift cluster is useful to ops folks to test
upgrades and try out things, and it's useful for developers in our case
to get all the parts setup right in ansible to deploy their application.
So, what do you think? should we setup a staging cluster as before?
Or shall we try and just use the one production cluster for staging and
prod?
* Another question is openshift 4. Openshift 3.11 is supported until
june of 2022, so we have some time, but do we want to or need to look at
moving to openshift 4 for our clusters? One thing I hate about this is
that you must have 3 master nodes, and the only machines we have are big
powerfull virthost servers, so it's very wastefull of resources to
deploy a openshift 4 cluster (with the machines we have currently
anyhow).
* In our old staging env we had a subset of things. Some of them we used
the staging instances all the time, others we almost never did. I'm not
sure we have the resources to deploy a 100% copy of our prod env, but
assuming we did, where should we shoot for on the line? ie, between 100%
duplicate of prod or nothing?
* We came up with a pretty elaborate koji sync from prod->staging. There
were lots of reasons we got to that, but I suppose if someone wants to
propose another method of doing this we could revisit that.
* Any other things we definitely want from a staging env?
* Has staging been helpful to you?
* Is there anything we could do to make it better?
Thoughts?
kevin
3 years, 3 months
Introduction
by johe＠fedoraproject.org
Morning all,
after being away from the Fedoraproject for quite some time I thought it might be a good idea to get involved again.
So my name is Joerg Stephan, 40y old and living in Amsterdam. I have a system administrator background up until ~6y ago, since then I am working in the security field. Part of this job I am snooping into different cloud stuff and Ansible in the mix. Not an expert but getting (back) there. My Fedora profile can be found here https://fedoraproject.org/wiki/User:Johe
I am around on slack as @jstephan
Hope to have a conversation soon
Cheers
3 years, 3 months
Meeting Agenda Item: Introduction Kwame Asante
by KwameKert
IRC handle: kwamekert
Skills: Python, Java, Javascript, PHP.
I will like to learn infrastructure developments. Im new to this
project and I will do my best to learn as fast as I can so I can
contribute efficiently to the community.
I hope to hear from the community soon. Thank you.
3 years, 3 months
Meeting Agenda Item: Introduction Nikolay Udovik
by Nikolay Udovik
Hello Infrastructure Team,
My IRC handle: Nikolay
Skills to offer:
* I am currently working as Network and Security Tier 2 Specialist at ISP/MSP the company, so basically I can highlight the following skills:
* Network design and troubleshoot: Switching/Routing/Firewalls, TCP/IP, DNS, IPSec, SSL\TLS, Redundancy solutions using BGP, FHRP and other.
* Well-versed in Networking in Linux
* Network Automation tasks using Python
* Management and troubleshoot BIND/Unbound/Postfix/Exim
* CCNA certified
* I finished RHSCA/RHCE courses in Academy and currently preparing for the exam:
* So you can suspect that I know everything that required for RHCE in the best way
* Some things I know even more
What I would like to learn/work on:
* I am looking to implement and strengthen my hand zone skills as Linux Administrator
* I am looking to learn new technologies
* Improve my BASH/Python skill
* I would like to care on every sysAdmin task
Best Regards,
Nikolay Udovik
3 years, 3 months
CPE Weekly: 2020-06-14
by Aoife Moloney
---
title: CPE Weekly status email
tags: CPE Weekly, email
---
# CPE Weekly: 2020-06-14
Background:
The Community Platform Engineering group is the Red Hat team combining
IT and release engineering from Fedora and CentOS. Our goal is to keep
core servers and services running and maintained, build releases, and
other strategic tasks that need more dedicated time than volunteers
can give.
See our wiki page here for more
information:https://docs.fedoraproject.org/en-US/cpe/
## General Project Updates
Please check out our updated initiative timetable for briefing in new
projects to our team
here:https://docs.fedoraproject.org/en-US/cpe/time_tables/
*Note: Initiatives are large pieces of work that require a team of
people and weeks/months to complete. Please continue to open tickets
in the normal way for bugs, issues, etc.
Dont forget to view our taiga board to see the projects we are
currently working on, what we have scoped and whats in our backlog
https://tree.taiga.io/project/amoloney1-cpe-team-projects/kanban?epic=null
CPE Product Owner Office Hours
* Weekly on Thursdays @ 1300 UTC on #fedora-meeting-1
* Every second Tuesday @ 1500 UTC on #centos-meeting (next meeting 23rd June)
* Meeting Logs
* Fedora: https://meetbot.fedoraproject.org/teams/cpe_product_owner_office_hours/cp...
* CentOS: https://www.centos.org/minutes/2020/June/centos-meeting.2020-06-09-15.00....
## Fedora Updates
### Data Centre Move
* The final Fedora hardware shipment is due to happen on Tuesday 16th June.
* We expect the shipment to arrive in the new data centre the week
beginning 22nd June and the team will begin bringing up services that
are affected by the move.
* A list of affected services is available here
https://hackmd.io/hpYYJQRjQy-oHxUS7IonIA?view
* Please read the below email sent by kfenzi if you have not already
done so: https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedora...
* Details on what this move may mean for you can be found here
https://lists.fedoraproject.org/archives/list/devel-announce@lists.fedora...
* If an application is not working correctly at all, please check this
list https://hackmd.io/hpYYJQRjQy-oHxUS7IonIA?view before opening a
ticket to make sure its not listed as being moved. If it is being
moved, please wait a day or two, then try again.
* Similarly, please be patient when opening tickets for service issues
in general as we have now reached the critical point in this move and
all of our sys-admins and wider teams will be assisting in the
successful bringup of the reduced Fedora service and facilitation of
the final hardware shipment and move.
### AAA Replacement
* The team are working on the changing application code bases to use
the new solution
* They are also working on the CentOS account integration and the
solution will see users be able to select the contributor agreements
that are relative to their account.
* They are also testing a script to use in data migration from the
current FAS system to Noggin to allow for very little, if any, changes
to the user
* The team are also completing user documentation that includes a
how-to migration guide for maintainers and a guide to help users
understand the differences between Noggin and the current FAS option
(issue #246 on board)
* Please feel free to check out the team kanban board for more
information on the features the team are working on and have already
completed here https://github.com/orgs/fedora-infra/projects/6
### Mbbox
* Project Dashboard here https://github.com/fedora-infra/mbbox/projects/1
* Tasks completed in the project currently
* MBS Backend CRD + documentation
* MBS Frontend CRD - configuration and certificates
* Draft of blog post about MBBox
* Tasks underway currently
* MBS Frontend CRD - deployment config and service
* Staging environment required
* MBox shared CRD
### Gitforge
With the data centre move taking most of the teams focus this week,
discussions with gitlab have been quiet. We are still discussing
technical aspects of the project and these are tracked here:
https://gitlab.com/gitlab-org/gitlab/-/issues/217350
We will keep you up to date with the developments as and when we have
information to share and thank you for your patience.
## CentOS Updates
### CentOS
* OCP4 staging cluster in progress
* CentOS Linux 8.2.2004 content is in pre-release with more artifacts to come
* There was also a recent CentOS AMA on Reddit, logs are here so check
it out if you missed it
https://www.reddit.com/r/CentOS/comments/gwnvd2/centos_ama/
### CentOS Stream
* The team have been working really hard to get package sources pushed
to keep caught up with RHEL
* We are also working on tooling for more push automation
* And tooling for RealTime and other Variants are coming in Stream soon
As always, feedback is welcome, and we will continue to look at ways
to improve the delivery and readability of this weekly report.
Have a great week ahead!
Aoife
Source: https://hackmd.io/8iV7PilARSG68Tqv8CzKOQ?view
--
Aoife Moloney
Product Owner
Community Platform Engineering Team
Red Hat EMEA
Communications House
Cork Road
Waterford
3 years, 3 months
Meeting Agenda Item: Introduction Perry Aganad
by Perry Aganad
Hello Everyone,
My name is Perry, I am a brand new python developer looking to learn and work on real world projects. I would like to gain experience in programming python and working with a group. Id like to soak up as much knowledge as possible with the fedora infrastructure team and be an active contributor to this awesome linux OS that I have stumbled on to.
Perry Aganad
gogoperry(a)aol.comhexchat IRC - goyoperry
3 years, 3 months
The future of loopabull
by Pierre-Yves Chibon
Good Morning Everyone,
I know this question has already been raised a few times, but I think we should
raise it once more: what do we see as future for loopabull?
It is currently triggered on 4 topics (3 from prod and 1 from stg) to do basically
three actions:
- Flag commit successfully built in koji, in other words it adds these flags
to dist-git:
https://src.fedoraproject.org/rpms/mingw-filesystem/c/717f2a929bd25b62a04...
- Flag when the Fedora CI start testing a PR
- Flag when the Fedora CI finished testing a PR (and thus reports Pass/Fail)
Upstream released yesterday a 0.0.7 release which brings supports for
fedora-messaging (contributed by your servitor).
Looking at the code, it should be python3 compatible, but it doesn't say
specifically in the setup.py and I honestly don't remember if I've tested that
or not.
The package has been orphaned in Fedora for over 10 months and has thus been
retired.
I've had a chat with upstream yesterday and they are still interested in the
project but more as a pet project and their time is just like the rest of us,
limited for pet projects these days.
That being said the code base is really quite small and involves technologies
we're already using in other places (python-pika, celery, rabbitmq, ansible...)
so there isn't really anything new there.
One of its limitation currently is with secrets, how to pass/specify them.
This is something we could circumvent via ansible-vault or so, but it needs a
little investigation.
I basically see three ways forward with this:
- We continue with loopabull and we need to check its python3 support, how to
deal with secrets, if we can get it to run in openshift & so on.
- We look for something else, similar. The requirements being:
- Run a task when seeing a message in our message bus
- Handle secrets
- Scalable up/down
- Runnable in openshift is a bonus
- Preferably in a language we can debug (python++, potentially rust)
- We write something that fits our needs and requirements
Would you know something that fits our requirements and that we could just run?
If not, do you have a preferred way between options #1 and #3?
Thanks for your thoughts,
Pierre
3 years, 3 months
Datacenter move days 3 and 4
by Kevin Fenzi
Greetings all.
This email will cover days 3 and 4, as by the time I was going to send
yesterdays it was late and mailman was still down anyhow. :)
So, yesterday started out seeming like a pretty simple day, but didn't
turn out that way. We planned to move only two things and work on fixing
issues from the buildsystem and other moves in the first two days.
* datagrepper / datanommer. This took until this morning as the database
is really gigantic. Again, we wanted to load it into a more modern
postgres. Now that it's moved and on postgres 12.2, we will be looking
into partitioning the data (perhaps by month? quarter?) so queries for
anything recent are much faster.
* mailman / lists: This turned out to be our biggest problem of the
move. :( We are working on getting this install moved over to recent
fedora or rhel, but for now it's rhel7 and python34. Because of that we
decided to just copy the instance over entire and adjust it over a fresh
install. The copy ran most of the day, and was nearing completion but
then we acidentally resized the orig instance. :( We resized it back,
but the filesystem was messed up and the instance would no longer boot.
It was at this point we decided that lack of sleep could leed to poor
decisions and mistakes and we started a copy off of the data on the copy
to another freshly installed instance and went and got some sleep.
The next day, in a stroke of luck, the copy we were doing had already copied
all the disk that had data on it, so we were able to fsck it and resize
it and we were back in business. mailman/lists was back up this morning
and happily processing away.
Today, in addition to finishing the above two migrations from yesterday,
we moved:
* openqa. Right now it doesn't have any arm or power workers, but we
have some almost ready to go there that we should have in place next
week.
* Various openshift apps (docsbuilding, websites building, cron jobs,
etc). We even have release-monitoring and the new hotness up and
running. I am trying to bring koschei up as well, but it needs some more
work.
* Some small misc apps: blockerbugs, kerneltest, etc.
* We also fixed tons and tons of issues all over the map. Mostly around
things reaching other things or something not running for some
configuration reason.
At this point everything we planned to be in the minimal fedora should
be up and working. We do have a more capacity than we need, so if things
go smoothly without too many more things to fix, I'd like to see about
bringing up badges as it's a popular app and if we have capacity and can
easily do it we can bring it up.
Tomorrow and this weekend we are going to work on taking things down in
the old datacenter and get them ready for shipping next week. They will
be in transit next week, then we hopefully can get them racked and built
and start adding capacity back the week after.
So, if you notice something not working now, please do look to see if
there's already a ticket on it, and if not please file one.
( https://pagure.io/fedora-infrastructure/issues ).
Overall things went pretty good from my view, and I would really like to
thank the awesome fedora community for being patient with us. I was
pretty surprised how few people asked why things were down and when they
did other community memebers were quick to tell them.
kevin
3 years, 3 months
