About JS framework
by Pierre-Yves Chibon
Good Morning Everyone,
Our infrastructure is mostly a python store, meaning almost all our apps are
written in python and most using wsgi.
However in python we are using a number of framework:
* flask for most
* pyramid for some of the biggest (bodhi, FAS3)
* Django (askbot, Hyperkitty)
* TurboGears2 (fedora-packages)
* aiohttp (python3, async app: mdapi)
While this makes sometime things difficult, these are fairly standard framework
and most of our developers are able to help on all.
However, as I see us starting to look at JS for some of our apps (fedora-hubs,
wartaa...), I wonder if we could start the discussion early about the different
framework and eventually see if we can unify around one.
This would also allow those of us not familiar with any JS framework to look at
the recommended one instead of picking one up semi-randomly.
So has anyone experience with one or more JS framework? Do you have one that
would you recommend? Why?
Thanks for your inputs,
Pierre
7 months, 3 weeks
[PATCH 0/1] Add a flatpak-indexer openshift service
by Owen Taylor
From: "Owen W. Taylor" <otaylor(a)fishsoup.net>
This is an initial attempt to create a configuration for flatpak-indexer to replace
regindexer and add an image delta capability. The config here is derived from
a working openshift configuration, but is untested in this form.
See: https://pagure.io/fedora-infrastructure/issue/9272
Open questions:
How to propagate content to the registry.fedoraproject.org reverse proxy
========================================================================
Currently the regindexer-generated content is rsync'ed from sundries to
fedora-web/registry. How should this be done with flatpak-indexer running as
an openshift app? Some possibilities that come to mind:
- Run a rsyncd within the openshift app (either as a separate deploymentconfig
or as a sidecar to the indexer) and expose a route to it internally in
Fedora infrastructure.
- Run a web server within the openshift app, expose a route to it internally
in Fedora infrasturcture, and reverse proxy the content on fedora-web/registry
instead of rsync'ing it.
- Write the content onto a netapp volume, and mount that volume RO either
on a host running rsyncd or directly on fedora-web/registry.
What to use for a redis image
=============================
Redis is used for caching and communication between the components. What redis
image should be used?
registry.redhat.io/rhel8/redis-5
needs configuration of a subscription
docker.io/library/redis:5
centos/redis-5-centos7
don't rely on such images currently
Custom Dockerfile image built from fedora:32
how would rebuilds be triggered?
For the two other images needed here, I used ubi8 images - which aren't currently
used elsewhere, but are presumably ok.
How to handle identifying versions to build for staging/production
==================================================================
I see that most openshift applications simply use 'staging'/'production' tags
in the upstream repo, while a few take the approach of having specific hashes
checked into the infrastructure ansible repository.
Is the upstream tag approach considered sufficiently secure? (Making the service
write a malicious index could allow causing users to upgrade to arbitrary
application binaries.)
Owen W. Taylor (1):
Add a flatpak-indexer openshift service
playbooks/openshift-apps/flatpak-indexer.yml | 56 +++++
.../reversepassproxy.registry-generic.conf | 34 ++-
.../flatpak-indexer/files/imagestream.yml | 52 +++++
.../flatpak-indexer/files/service.yml | 16 ++
.../flatpak-indexer/files/storage.yml | 24 ++
.../flatpak-indexer/templates/buildconfig.yml | 84 +++++++
.../flatpak-indexer/templates/configmap.yml | 98 ++++++++
.../templates/deploymentconfig.yml | 221 ++++++++++++++++++
.../flatpak-indexer/templates/secret.yml | 11 +
roles/regindexer/build/tasks/main.yml | 21 --
roles/regindexer/build/templates/config.yaml | 74 ------
11 files changed, 584 insertions(+), 107 deletions(-)
create mode 100644 playbooks/openshift-apps/flatpak-indexer.yml
create mode 100644 roles/openshift-apps/flatpak-indexer/files/imagestream.yml
create mode 100644 roles/openshift-apps/flatpak-indexer/files/service.yml
create mode 100644 roles/openshift-apps/flatpak-indexer/files/storage.yml
create mode 100644 roles/openshift-apps/flatpak-indexer/templates/buildconfig.yml
create mode 100644 roles/openshift-apps/flatpak-indexer/templates/configmap.yml
create mode 100644 roles/openshift-apps/flatpak-indexer/templates/deploymentconfig.yml
create mode 100644 roles/openshift-apps/flatpak-indexer/templates/secret.yml
delete mode 100644 roles/regindexer/build/tasks/main.yml
delete mode 100644 roles/regindexer/build/templates/config.yaml
--
2.28.0
2 years, 6 months
Another Rust MirrorManager experiment
by Adrian Reber
Our MirrorManager setup exports the current state of all mirrors every
hour at :30 to a protobuf based file which is then used by the
mirrorlist servers to answer the requests from yum and dnf.
The Python script requires up to 10GB of memory and takes between 35 and
50 minutes. The script does a lot of SQL queries and also some really
big SQL queries joining up to 6 large MirrorManager tables.
I have rewritten this Python script in Rust and now it only needs around
1 minute instead of 35 to 50 minutes and only 600MB instead of 10GB.
I think the biggest difference is that I am almost not doing any joins
in my SQL request. I download all the tables once and then I do a lot of
loops over the downloaded tables and this seems to be massively faster.
As the mirrorlist-server in Rust has proven to be extremely stable over
the last months we have been using it I would also like to replace the
mirrorlist protbuf input generation with my new Rust based code.
I am planing to try out the new protobuf file in staging in the next
days and would then try to get my new protobuf generation program into
Fedora. Once it is packaged I would discuss here how and if we want to
deploy in Fedora's infrastructure.
Having the possibility to generate the mirrorlist input data in about a
minute would significantly reduce the load on the database server and
enable us to react much faster if broken protobuf data has been synced
to the mirrorlist servers on the proxies.
Adrian
2 years, 7 months
Fedora 33 Final Freeze now in effect!
by Kevin Fenzi
Greetings.
we are now in the infrastructure freeze leading up to the Fedora 33
Final release. This is a final release freeze.
We do this to ensure that our infrastructure is stable and ready to
release Fedora 33 when it's available.
You can see a list of hosts that do not freeze by checking out the
ansible repo and running the freezelist script:
git clone
https://infrastructure.fedoraproject.org/infra/ansible.git
ansible/scripts/freezelist -i inventory
Any hosts listed as freezes is frozen until 2020-10-21 (or later if
release slips). Frozen hosts should have no changes made to them without
a sign-off on the change from at least 2 sysadmin-main or rel-eng
members, along with (in most cases) a patch of the exact change to be
made to this list.
Thanks,
kevin
2 years, 7 months
[FBR] Bump the IoT stable keys to fedora-33
by Patrick Uiterwijk
Hi,
Can I get acks for the following patch to move the stable IoT composes
to the Fedora 33 key?
(Note: there's a single "empty" line change as well, that was a line
with purely whitespace, which my IDE automatically cleans up.)
Thanks,
Patrick
commit 3ee8dd3f9e883414c7846d115d2926f73ec9ea99 (HEAD -> master)
Author: Patrick Uiterwijk <patrick(a)puiterwijk.org>
Date: Wed Oct 28 11:26:44 2020 +0100
Bump the IoT stable keys to fedora-33
Signed-off-by: Patrick Uiterwijk <patrick(a)puiterwijk.org>
diff --git a/roles/robosignatory/templates/robosignatory.toml.j2
b/roles/robosignatory/templates/robosignatory.toml.j2
index 6a933bda9..70fde0d55 100644
--- a/roles/robosignatory/templates/robosignatory.toml.j2
+++ b/roles/robosignatory/templates/robosignatory.toml.j2
@@ -380,7 +380,7 @@ handlers = ["console"]
key = "{{ (env == 'production')|ternary('fedora-34', 'testkey') }}"
keyid = "{{ (env == 'production')|ternary('45719a39',
'd300e724') }}"
type = "modular"
-
+
# ELN Mass Rebuild
[[consumer_config.koji_instances.primary.tags]]
@@ -445,13 +445,13 @@ handlers = ["console"]
[consumer_config.ostree_refs."fedora/stable/x86_64/iot"]
directory = "/mnt/fedora_koji/koji/compose/iot/repo/"
- key = "{{ (env == 'production')|ternary('fedora-32', 'testkey') }}"
+ key = "{{ (env == 'production')|ternary('fedora-33', 'testkey') }}"
[consumer_config.ostree_refs."fedora/stable/aarch64/iot"]
directory = "/mnt/fedora_koji/koji/compose/iot/repo/"
- key = "{{ (env == 'production')|ternary('fedora-32', 'testkey') }}"
+ key = "{{ (env == 'production')|ternary('fedora-33', 'testkey') }}"
[consumer_config.ostree_refs."fedora/stable/armhfp/iot"]
directory = "/mnt/fedora_koji/koji/compose/iot/repo/"
- key = "{{ (env == 'production')|ternary('fedora-32', 'testkey') }}"
+ key = "{{ (env == 'production')|ternary('fedora-33', 'testkey') }}"
[consumer_config.ostree_refs."fedora/29/x86_64/atomic-host"]
directory = "/mnt/fedora_koji/koji/compose/ostree/repo/"
2 years, 7 months
Auth test apps in staging
by Aurelien Bompard
Hey folks,
To test authentication with the new AAA system I'd like to deploy a couple
very basic apps that do nothing but auth in staging's openshift. It
shouldn't touch any configuration besides the reverse proxies and the new
project in openshift. And it's staging only.
Is it OK?
Thanks.
Aurélien
2 years, 7 months
Re: master mirror slow sync
by Kevin Fenzi
On Mon, Oct 26, 2020 at 04:00:06PM +0100, Jan Kasprzak wrote:
> Kevin Fenzi wrote:
> : On Mon, Oct 26, 2020 at 11:12:32AM +0100, Jan Kasprzak wrote:
> : > Hello,
> : >
> : >
> : > Kevin Fenzi wrote:
> : > : As some of you may know, there's been reports of slow sync times for
> : > : content from the master mirrors last week.
> : > : (see: https://pagure.io/fedora-infrastructure/issue/9392 )
> : >
> : > ftp.linux.cz Tier1 mirror admin here. Thanks for the heads up.
> : >
> : > : * Our two mirrors located in other datacenters:
> : > : download-ib01.fedoraproject.org
> : > : and
> : > : download-cc-rdu01.fedoraproject.org
> : >
> : > None of the above contain the fedora-enchilada0 rsync package,
> : > and in the fedora-releases package there is no 33 subdirectory.
> : > Which package should I use for rsyncing?
> :
> : Odd. They do have it defined. Just like the main master mirrors. ;(
> :
> : It could be something to do with them having ipv6 addresses and you
> : coming from a different ipv6 address? We have
> : ftp.linux.cz
> : in rsyncd.conf... so it should reverse lookup the ip and allow it, just
> : like the iad2 masters. ;(
>
> OK, the reverse lookup would be a problem then. Adding -4 to the
> rsync command line fixes the problem - I can now download from
> download-ib101.fp.o.
>
> Can you add 2001:718:801:230::cd to the ACL? Unfortunately, I have
> many different FQDNs pointing to my server, so PTRs can be inconsistent
> (my fault, I know).
We can. Would it be ok to add it in a few days?
Or do you need it now?
Thanks!
kevin
2 years, 7 months
Re: master mirror slow sync
by Kevin Fenzi
On Mon, Oct 26, 2020 at 11:12:32AM +0100, Jan Kasprzak wrote:
> Hello,
>
>
> Kevin Fenzi wrote:
> : As some of you may know, there's been reports of slow sync times for
> : content from the master mirrors last week.
> : (see: https://pagure.io/fedora-infrastructure/issue/9392 )
>
> ftp.linux.cz Tier1 mirror admin here. Thanks for the heads up.
>
> : * Our two mirrors located in other datacenters:
> : download-ib01.fedoraproject.org
> : and
> : download-cc-rdu01.fedoraproject.org
>
> None of the above contain the fedora-enchilada0 rsync package,
> and in the fedora-releases package there is no 33 subdirectory.
> Which package should I use for rsyncing?
Odd. They do have it defined. Just like the main master mirrors. ;(
It could be something to do with them having ipv6 addresses and you
coming from a different ipv6 address? We have
ftp.linux.cz
in rsyncd.conf... so it should reverse lookup the ip and allow it, just
like the iad2 masters. ;(
If you can drop by #fedora-admin on irc.freenode.net we can try and
debug it? Or I can just add some hard coded IPv4/IPv6 addresses for you?
> : may provide faster speeds than the dl-tier01.fedoraproject.org mirrors
> : right now depending on your location/routing, so you may want to at
> : least temporarily sync against them.
>
> So far I have manually hardlinked releases/test/33_Beta tree to
> releases/33, and I am running rsync of
> rsync://dl-tier1.fedoraproject.org/fedora-enchilada0/linux/releases/33/
> on top of it. The download speed around 50 Mbit/s.
>
> Is there a faster way how to get my mirror in sync? Thanks,
Well, fedora-quick-mirror will only copy those specific files you still
need and won't cause our end to have to stat the entire tree so you can
compare it (like bare rsync does). But otherwise that should work...
kevin
2 years, 7 months
