About JS framework
by Pierre-Yves Chibon
Good Morning Everyone,
Our infrastructure is mostly a python store, meaning almost all our apps are
written in python and most using wsgi.
However in python we are using a number of framework:
* flask for most
* pyramid for some of the biggest (bodhi, FAS3)
* Django (askbot, Hyperkitty)
* TurboGears2 (fedora-packages)
* aiohttp (python3, async app: mdapi)
While this makes sometime things difficult, these are fairly standard framework
and most of our developers are able to help on all.
However, as I see us starting to look at JS for some of our apps (fedora-hubs,
wartaa...), I wonder if we could start the discussion early about the different
framework and eventually see if we can unify around one.
This would also allow those of us not familiar with any JS framework to look at
the recommended one instead of picking one up semi-randomly.
So has anyone experience with one or more JS framework? Do you have one that
would you recommend? Why?
Thanks for your inputs,
Pierre
7 months, 2 weeks
Another Rust MirrorManager experiment
by Adrian Reber
Our MirrorManager setup exports the current state of all mirrors every
hour at :30 to a protobuf based file which is then used by the
mirrorlist servers to answer the requests from yum and dnf.
The Python script requires up to 10GB of memory and takes between 35 and
50 minutes. The script does a lot of SQL queries and also some really
big SQL queries joining up to 6 large MirrorManager tables.
I have rewritten this Python script in Rust and now it only needs around
1 minute instead of 35 to 50 minutes and only 600MB instead of 10GB.
I think the biggest difference is that I am almost not doing any joins
in my SQL request. I download all the tables once and then I do a lot of
loops over the downloaded tables and this seems to be massively faster.
As the mirrorlist-server in Rust has proven to be extremely stable over
the last months we have been using it I would also like to replace the
mirrorlist protbuf input generation with my new Rust based code.
I am planing to try out the new protobuf file in staging in the next
days and would then try to get my new protobuf generation program into
Fedora. Once it is packaged I would discuss here how and if we want to
deploy in Fedora's infrastructure.
Having the possibility to generate the mirrorlist input data in about a
minute would significantly reduce the load on the database server and
enable us to react much faster if broken protobuf data has been synced
to the mirrorlist servers on the proxies.
Adrian
2 years, 6 months
What is our technical debt?
by Pierre-Yves Chibon
Good Morning Everyone,
Just like every team we have technical debt in our work.
I would like your help to try to define what it is for us.
So far, I've come up with the following:
- python3 support/migration
- fedora-messaging
- fedora-messaging schema
- documentation
- (unit-)tests
- OpenID Connect
What else would we want in there?
Looking forward to your thoughts,
Pierre
2 years, 8 months
Tech debt analysis tool
by James Richardson
Hi All,
Vipul and I are looking into several different tools that will allow us to
better analyze our tech debt with any new code that is merged into apps in
http://github.com/fedora-infra.
Currently, we have looked at the tools below, but we would love any and all
input from the team and community on this.
SonarCloud
LGTM
ShiftLeft
Our goal is to find an open source tool that is easy to integrate as well
as providing useful and timely feedback.
So far, SonarCloud has proved to be the one that looks best, but again, we
are very open to any and all suggestions, and at this early stage, a good
conversation to arrive at the best solution.
Regards,
James
--
James Richardson
Engineering Intern
He | Him | His
Red Hat Waterford <https://www.redhat.com/>
Communications House
Cork Road, Waterford City
jamricha(a)redhat.com <lgriffin(a)redhat.com>
M: +353851970521 <+353877545162> IM: jamricha
@redhatjobs <https://twitter.com/redhatjobs> redhatjobs
<https://www.facebook.com/redhatjobs> @redhatjobs
<https://instagram.com/redhatjobs>
<https://red.ht/sig>
2 years, 9 months
s390x for EPEL
by Miroslav Suchý
What is using Koji to build EPEL builds for s390x.
We want to enable s390x in Copr. It is without problem for Fedora, but we cannot do that for EPEL, because there is no
CentOS for s390x. Do we have entitlements for RHEL I can use? Or we have some local repo? Something completely different?
--
Miroslav Suchy, RHCA
Red Hat, Associate Manager ABRT/Copr, #brno, #fedora-buildsys
2 years, 10 months
packages app
by Kevin Fenzi
So, I think we have two groups approaching a replacement app for
packages?
Can we perhaps decide which way we want to go and look at deploying
something soon? This has been more important since we didn't move the
old broken packages app from the old datacenter and people keep asking
about it.
We could wait until we have staging back, but I don't think we need to
wait for that to decide what we want to do here?
Thoughts?
kevin
2 years, 10 months
CPE Weekly: 2020-07-25
by Aoife Moloney
---
title: CPE Weekly status email
tags: CPE Weekly, email
---
# CPE Weekly: 2020-07-25
Background:
The Community Platform Engineering group is the Red Hat team combining
IT and release engineering from Fedora and CentOS. Our goal is to keep
core servers and services running and maintained, build releases, and
other strategic tasks that need more dedicated time than volunteers
can give.
See our wiki page here for more
information:https://docs.fedoraproject.org/en-US/cpe/
## General Project Updates
The CPE team are working on the following projects for Quarter 3,
which is the months of July, August & September:
* Data Centre Move - Final Works
* CentOS Stream Phase 3
* Noggin Phase 3
* Packager Workflow Healthcare
* Fedora Messaging Schemas
Details of the above projects, and of projects currently in progress,
done and what projects are in our backlog, can be found on our taiga
board per project card:
https://tree.taiga.io/project/amoloney1-cpe-team-projects/kanban?epic=null
We also have an updated initiative timetable for briefing in new
projects to our team & key dates
here:https://docs.fedoraproject.org/en-US/cpe/time_tables/
*Note: Initiatives are large pieces of work that require a team of
people and weeks/months to complete. Please continue to open tickets
in the normal way for bugs, issues, etc.
### CPE Product Owner Office Hours
#### #fedora-meeting-1
* Weekly on Thursdays @ 1300 UTC on #fedora-meeting-1
* Next Meeting: 2020-07-30
#### #centos-meeting
* Every second Tuesday @ 1500 UTC on #centos-meeting
* Next Meeting: 2020-08-04
### Misc
* CFP for Nest with Fedora submissions are closing on Monday 27th July
- dont forget to submit your ideas/talks here https://pagure.io/flock
* CPE Engagement email was sent a few weeks ago, please read if you
have not already and feel free to send me your feedback on or off list
before 31st July. A follow up email will be sent with feedback
received and actions taken where it makes sense then afterwards.
Thanks for your
engagement!https://lists.fedoraproject.org/archives/list/devel-announce@l...
## Fedora Updates
### Data Centre Move
* The team have been working incredibly hard over the last few weeks
and are now working on reinstalling the staging environment.
* We estimate this work to be completed by July 31st, however things
happen so we thank you in advance for your patience
* The project card on taiga for builder - staging - remaining app
bringup dates here
https://tree.taiga.io/project/amoloney1-cpe-team-projects/us/42?kanban-st...
* A list of affected services is available here
https://hackmd.io/hpYYJQRjQy-oHxUS7IonIA?view
* Details on what this move may mean for you can be found here
https://lists.fedoraproject.org/archives/list/devel-announce@lists.fedora...
* If an application is not working correctly at all, please check this
list https://hackmd.io/hpYYJQRjQy-oHxUS7IonIA?view before opening a
ticket to make sure its not listed as being moved. If it is being
moved, please wait a day or two, then try again.
* And finally a reminder to please be patient when opening tickets for
service issues in general as we are in the critical point in this move
and all of our sys-admins and wider teams are assisting in the
successful bringup of the reduced Fedora service and facilitation of
the final hardware shipment and move.
### AAA Replacement
* The team are focusing work on this ticket and collaborating with the
Sustaining team for the next sprint, with the goal to have noggin
installed & deployed to staging
https://pagure.io/fedora-infrastructure/issue/9152
* The have also been addressing how to curtail spam email and more
information on that effort can be found here
https://github.com/fedora-infra/noggin/issues/27
* And have added the feature to allow groups with a prefix for groups
to import to avoid overlaps between CentOS and Fedora
https://github.com/fedora-infra/fas2ipa/issues/5
* Please feel free to check out the team kanban board for more
information on the features the team are working on and have already
completed here https://github.com/orgs/fedora-infra/projects/6
### Fedora Messaging Schemas
* The team are using a kanban approach for this project as Noggin
takes priority until fully complete
* The team have already built a list of applications that require
messaging schemas, list can be found here
https://hackmd.io/@nilsph/H1i8CAbkP/edit
* They also have completed a readme which contains documentation on
messaging schemas, a cookie-cutter template to create the schema and a
definition of Done for writing a schemas
https://github.com/fedora-infra/fedora-messaging-schemas-issues
* The board they are working from can be viewed here
https://github.com/orgs/fedora-infra/projects/7
### Packager Workflow Healthcare
* Project information: This is an investigative project that aims to
look at the entire packager workflow as a single piece of tooling to
identify where failures happen, and try to identify when and why
packages fail at different points within the workflow. We hope to have
two possible outcomes from this project at the end of the quarter
(September):
* The workflow breaks at X point and we will work on a solution to fix
* OR
* The workflow works fine, but we will need better monitoring on
the pipeline so we will work on a solution for this
* The team are using this Monitor-Script
https://pagure.io/fedora-ci/monitor-gating/
* Current status: failing consistently because of CI test timeouts
* The jenkins instance being used is under-resourced, jobs waiting
up to 40 minutes for a node
* They are using a workable dump of historical datanommer/datagrepper
data for queries to pull metrics on the historical/aggregate Packager
Workflow from as a baseline
* And are working on an outline of the workflow steps (from packager
PoV) and systems involved (CPE team PoV), identifying metrics to be
measured
* The teams work is being tracked here
https://teams.fedoraproject.org/project/cpe-cicd/kanban
## CentOS Updates
### CentOS
* CentOS-infra project created https://pagure.io/centos-infra/
* Day to Day working with CentOS-infra being documented at:
https://pagure.io/cpe/docs/pull-request/10
* New website is live https://www.centos.org
### CentOS Stream
* Business as usual, the team are churning through building packages in Stream
* They are also working on tooling to bring back some rpkg features
As always, feedback is welcome, and we will continue to look at ways
to improve the delivery and readability of this weekly report.
Have a great week!
Aoife
Source: https://hackmd.io/8iV7PilARSG68Tqv8CzKOQ?view
--
Aoife Moloney
Product Owner
Community Platform Engineering Team
Red Hat EMEA
Communications House
Cork Road
Waterford
2 years, 10 months
The state of the planet (the app)
by Pierre-Yves Chibon
Good Morning Everyone,
Running a personal planet and having upgraded the box running it to a recent OS
(Fedora in this case), I have found out that the underlying application does not
support python3.
To be precise, the main script says:
Requires Python 2.1, recommends 2.3.
Needless to say, this doesn't work anymore with python3.
So I went looking a bit around to see what is available.
There is Venus which we run, which has this problem ^ and hasn't been touched in
9 years:
https://github.com/rubys/venus
There are a few java-based, php-based, or ruby-based, with different level of
activity (though to be fair, RSS and planets have been there for a while now, so
a working application likely does not need much maintenance anymore).
However, nothing jumped at me as a clear successor for our setup.
Do people have some experience with planet? Would you recommend us something?
At this stage, it is very unclear to me if the best approach is to switch
application (new stack, new UI to do, new configuration file to do/adjust...) or
port venus to python3 (potentially a 1 time effort but what about the long term
maintenance?).
Looking forward to hear your thoughts,
Pierre
2 years, 10 months
