About JS framework
by Pierre-Yves Chibon
Good Morning Everyone,
Our infrastructure is mostly a python store, meaning almost all our apps are
written in python and most using wsgi.
However in python we are using a number of framework:
* flask for most
* pyramid for some of the biggest (bodhi, FAS3)
* Django (askbot, Hyperkitty)
* TurboGears2 (fedora-packages)
* aiohttp (python3, async app: mdapi)
While this makes sometime things difficult, these are fairly standard framework
and most of our developers are able to help on all.
However, as I see us starting to look at JS for some of our apps (fedora-hubs,
wartaa...), I wonder if we could start the discussion early about the different
framework and eventually see if we can unify around one.
This would also allow those of us not familiar with any JS framework to look at
the recommended one instead of picking one up semi-randomly.
So has anyone experience with one or more JS framework? Do you have one that
would you recommend? Why?
Thanks for your inputs,
Pierre
11 months, 3 weeks
Fedora Infrastructure Authentication Roadmap
by Kevin Fenzi
Greetings.
Sorry it took so long to write this up and send it out, but here's our
proposed plan for authentication moving forward.
Please do feel free to comment or suggest changes/improvements here. Any
mistakes are mine alone. :)
Fedora Project Auth roadmap
Background/History
The Fedora project created its own authentication/user/group
management system at nearly the beginning of its existance. FAS (Fedora
Account System) (version 1) and then a rewrite (FAS2). At each of these
points other solutions were investigated and found unacceptable for
various reasons. Over the last few years, several additional
applications have been added next to FAS2 to provide additional
functionality: ipisilon has been added as a identity provider, and
FreeIPA has been added for kerberos authentication. FAS2 is still the
authoritative source of authentication data. FAS2 is currently deployed
on RHEL6 servers and won't run on RHEL7.
Also during the last few years, a new FAS re-write has been slowly in
the works. FAS3 is written in a modern framework and has a number of
functionality and interface improvements over FAS2. Additionally it can
run on RHEL7.
Goals and Critera
Maintaining authentication applications is difficult and time
consuming work, and it has always been a goal to try and move to more
industry standard applications as much as possible given our goals and
critera. The last time we looked, Some of those goals/critera include:
* User self service registration
* User self service password reset
* FPCA acceptance requirement
* Basset integration (may not be needed anymore)
* Allow Self Service groups with their own sponsors/admins
* Allow group requirements (other group first, etc)
Plan:
On discussion with FreeIPA developers and looking at how things are
setup now, we came up with a plan to get what we need, but reduce the
footprint and maintance we need to do. Many of the features we were
hoping to use in FAS3 have now been implemented upstream in
FreeIPA (2fa, fasClient syncing better, etc).
Basically we will:
* A new small wrapper type project is written (Community Account
Information API) or CAIAPI.
This small app provides the Critera listed above, talking at first to
FAS2 on the backend then, later switching to talking to FreeIPA on the
backend and providing a json API to consumers.
* Switch anything we have using the direct FAS api to use CAIAPI instead.
* Move to FreeIPA being the canonical source for authentication data.
This should just be a switch to CAIAPI, and no consumers should even
notice.
* FreeIPA still provides kerberos auth.
Note that kerberos will remain limited to use at ipsilon and koji.
* Ipsilon provides identity auth for applications, preferably with OIDC
(still provides others)
* A new small website that uses the CAIAPI json to provide end user
access / management. This thing would be in flask and needs a name still.
Since https://fedoraproject.org/wiki/Infrastructure/fas_freeipa FreeIPA
has matured and our understanding of the work required to make CAIAPI
and a small web consumer has clarified.
Pros:
* IPA handles all the storing of credentials, replication and such and
we just use it.
* Our maint goes from needing to maintain FAS2 or FAS3 to just CAIAPI
(a much smaller api) and a
very small web application.
* Easier to audit 2 small apps.
* We can try and share the CAIAPI with other open source communities
(Gnome? CentOS? others?)
Open Source Communities already using FreeIPA would be easy to add
this to.
* We can stop using fasClient in favor of ipa-client setup (no more
heavy syncing)
* The heavy security aspects will be handled by upstreams we don't
need to fully maintain
(FreeIPA, sssd, ipsilon, etc).
Cons:
* We still need to write the CAIAPI/webapp, although Patrick may have
CAIAPI already somewhat implemented.
* It feels very sad to have spent so long on FAS3 and never deploy it,
but sometimes
thats just the way things go. ;(
5 years, 3 months
Infrastructure office hours
by Clement Verna
Dear all,
I would like to get feedback about starting a weekly infrastructure office
hours which would be used to help anyone with contributing to Fedora's
infrastructure.
This would be a dedicated IRC meeting where people would be available to
answer questions, help setting up development environment or even maybe
work together on an issue. It could also be used to add more details to the
issues we have in our backlog to help new contributor to get started.
Note that we already have a general "ask whatever, whenever" on
#fedora-apps and #fedora-admin where anybody can "ask whatever, whenever"
:-), but this does not seems to work since we have very few requests.
So do you think that having a dedicated meeting where you know that people
are available would improve the situation ?
Thanks
Clément
5 years, 3 months
Fedora and PDC
by Pierre-Yves Chibon
Good Morning Everyone,
We have been informed thst week at the upstream devs working on the
product-definition-center (PDC) are moving away from the project and are going
to leave it without a maintainer. Since we adopted PDC for a variety of data
flows, this puts us in an awkward position. :(
Ralph and I met up on Tuesday to brainstorm the list of things we actively use
PDC for today and to come up with a contingency plan for how to handle them. One
overarching option is for us (fedora-infra) to take ownership of the PDC
codebase as a whole. We didn't fully explore this option, figuring that the
codebase is large and contains lots of tables, endpoints, and codepaths that we
didn't use nor which we plan to use.
Instead, below we've got the four things we use PDC for and some options for
what to do with each.
With the exception of /modules/, one common pattern that we like is to
investigate splitting out the "django apps" that make up PDC into their own
projects. We're calling these "pdc-lite", for fun. See more below.
* Modules
The data in the /modules/ PDC endpoint is *also* in the MBS db. Ralph's
team is going to just use that and stop using pdc anything for modules.
We're going to need to patch pungi, mbs for local builds, and a few other
places. This should be a relatively low-pain transition.
* Stream branches, branch ownership, retirement dates?
- SLA/EOL are currently stored in PDC.
- Queried by releng scripts for retirement, fedrepo-req for new branches,
etc..
option #1
git repo full of yaml file similar to the override repo
compiled into a single JSON blob
Single place for all retired packages
This feels like the lowest tech option.
git gives us change control for free... but people easily get lost in the
"UX" of navigating a gigantic git repo full of plaintext files.
option #2
pagure's DB/API
pagure knows nothing about branches currently, so that would be bigger
lift
option #3
PDC internally is composed of ~20 "django apps"
https://github.com/product-definition-center/product-definition-center/tr...
We could pick the 2 or 3 that comprise the branches feature, copy them
out, and turn them into their own service: the "branch definition center":
BDC.
That would be the "pdc-lite" approach mentionned above, ie: PDC with only
the "app" of interest
* release/life-circle tracking?
option #1:
PDC lite with just that app of interest
option #2:
JSON/yaml file on the proxies
option #3:
pkgdb-lite
option #4:
???
compose tracking?
impacted: fedfind
option #1:
PDC-lite with just that app of interest
option #2:
Drop this entirely?
Adam probably really wants to keep the record of composes.
option #3:
???
The "pdc-lite" options are attractive, across the board. One thing we get from
this is greater clarity when discussing things formerly in PDC. If something is
in the branch-definition-center, the compose-definition-center, or the
release-definition-center.. you know what you're talking about. Today, when
talking about whether or not something should be or is in "PDC", it is easy to
get confused.
I propose we start the discussion on the list and plan for a meeting sometime
late next week to discuss it further with the interested parties (please signal
yourself)
What do you think?
Pierre and Ralph
5 years, 3 months
Migrating fedmsg to AMQP: a proposal
by Aurelien Bompard
Hey folks!
Jeremy and I have been working on a proposal to migrate fedmsg from our
current brokerless architecture to a broker-based architecture.
The overview and reasons for the migration are described on this page:
https://fedmsg-migration-tools.readthedocs.io/en/latest/migration/overvie...
Head there if you want the details. The plan has the following requirements:
* No flag day.
* Don't disrupt any services or applications.
* Don't break any services outside of Fedora's infrastructure relying on
these messages.
The first step is to deploy a broker in Fedora to use. In order to avoid a
flag day, bridges from AMQP to ZeroMQ and ZeroMQ to AMQP have been
implemented and will be deployed.
In order to validate that the bridges are functioning, a small service will
be run during the transition period that connects to fedmsg and to the AMQP
queues to compare messages.
After the bridges are running, applications are free to migrate. There are
several options when migrating and each has advantages and disadvantages.
We have written a new library called fedora-messaging that has the
following features:
* A method to define message schemas and offer automatic validation of
messages using those schemas.
* Boilerplate for typical publishers and consumers.
Head over to the document for a demo!
What do you think of this proposal? Any blind spots?
Thanks!
Aurélien
5 years, 4 months
Looking up module information via Koji
by Owen Taylor
My understanding is that with the planned retirement of the PDC:
https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedora...
Querying for module information should be done using the MBS and/or Koji
APIs.
Various code that I maintain (in OSBS, fedmod, and random tooling) wants to
do module build lookups - different variations of "look up the modulemd
for latest build of a NAME:STREAM[:VERSION]". Variations generally being
exactly what "latest" means here.
The code generally already is using Koji and the MBS api is quite limited,
so I've chosen to do the lookups via Koji.
https://fishsoup.net/misc/get-module-builds
Is a test tool that incorporates most of the capability that I needed
across my uses. It's distinctly more than a couple of lines of code - I can
cut-and-paste it for now, but what's the right long-term home? Is there a
simpler way?
My best idea right now is that if the 'base_version' and'status' part of my
code was simplified to simply be "tag" and avoid reliance on the tag
structure of Fedora, then this might make a reasonable addition to the Koji
CLI and API - there are some things that using raw tags for the query makes
trickier, but it's probably workable.
Thanks for any input!
Owen
5 years, 4 months
Librepo/dnf zchunk integration question
by Jonathan Dieter
Zchunk works by comparing an old version of the file with the one you
want to download, but when dnf refreshes a repository, it downloads the
new file into a temporary directory with no information passed to the
handle about where the old files are.
I've been trying to keep my code changes in libsolv and librepo to make
zchunk integration as universal as possible. Up until now, I have
managed to do so without changing librepo's API, but I don't see any
way to fix this except to have dnf pass information about the old
directory (or, even better, the cache directory) to the handle, which
will mean an API change.
It would also mean that other utilities would probably need to do the
same. Is there something I'm missing in dnf's interaction with librepo
that would allow me to work around this, or do I just need to bite the
bullet and propose a librepo API change?
Jonathan
5 years, 4 months
Infrastructure Meeting 2018-05-31 14:00 UTC
by Stephen John Smoogen
= Preamble =
The infrastructure team will be having its weekly meeting tomorrow,
2018-05-31 at 14:00 UTC in #fedora-meeting-1 on the freenode network.
We have a gobby document
(see: https://fedoraproject.org/wiki/Gobby )
fedora-infrastructure-meeting-next is the document.
Please try and review and edit that document before the meeting and we
will use it to have our agenda of things to discuss. A copy as of today
is included in this email.
If you have something to discuss, add the topic to the discussion area
with your name. If you would like to teach other folks about some
application or setup in our infrastructure, please add that topic and
your name to the learn about section.
= Introduction =
We will use it over the week before the meeting to gather status and info and
discussion items and so forth, then use it in the irc meeting to transfer
information to the meetbot logs.
= Meeting start stuff =
#startmeeting Infrastructure (2018-05-31)
#meetingname infrastructure
#topic aloha
#chair smooge relrod nirik pingou puiterwijk tflink
= Let new people say hello =
#topic New folks introductions
#info This is a place where people who are interested in Fedora
Infrastructure can introduce themselves
= Status / Information / Trivia / Announcements =
(We put things here we want others on the team to know, but don't need
to discuss)
(Please use #info <the thing> - your name)
#topic announcements and information
#info relrod PTO 9 Jun - 19 Jun
#info smooge PTO 8 Jun, 15 Jun -> 18 Jun
#info bodhi-3.8.0 deployed
= Things we should discuss =
We use this section to bring up discussion topics. Things we want to talk about
as a group and come up with some consensus /suor decision or just brainstorm a
problem or issue. If there are none of these we skip this section.
(Use #topic your discussion topic - your username)
#topic Oncall
#info Smooge is on call from 2018-05-29->2018-06-04
#info Nirik is on call from 2018-06-05->2018-06-11
#info Smooge is on call from 2018-06-11->2018-06-15
#info (!Smooge|!Relrod) is on call from 2018-06-15->2018-06-18
#topic Fedora web dev clinic resurrection
(https://communityblog.fedoraproject.org/friday-fedora-web-dev-clinic/)
- cverna
#topic Tickets discussion
#info https://pagure.io/fedora-infrastructure/issues?status=Open&priority=2
Go thru each ticket one by one
= Apprentice office hours =
#topic Apprentice Open office minutes
#info A time where apprentices may ask for help or look at problems.
Here we will discuss any apprentice questions, try and match up people looking
for things to do with things to do, progress, testing anything like that.
= Learn about some application or setup in infrastructure =
(This section, each week we get 1 person to talk about an application or setup
that we have. Just going over what it is, how to contribute, ideas for
improvement,
etc. Whoever would like to do this, just add the i/nfo in this section. In the
event we don't find someone to teach about something, we skip this section
and just move on to open floor.)
#info none this week
= Meeting end stuff =
#topic Open Floor
#endmeeting
--
Stephen J Smoogen.
5 years, 4 months
