Deploying Hubs to our Openshift instance
by Aurelien Bompard
Hey folks!
I would like to try and do the Fedora Hubs deployment in our Openshift
instance.
The thing is, I have never deployed anything in Openshift, much less using
our ansible playbooks.
Do you know of a documentation I could read up on to understand what our
`openshift/project`, `openshift/secret-file` and `openshift/*` roles work,
and how I could use them?
I'm trying to read the playbooks for the apps we already have in openshift,
but there's too much magic going on for me.
Any introduction document would help. I did try to read the F docs, but did
not find anything in the SOPs or on the wiki.
Thanks!
Aurélien
6 years, 2 months
Unable to push changes (access denied)
by Antonio Trande
Hello.
I can't push changes via GIT for 'coin-or-CoinUtils' package:
$ git push
Counting objects: 3, done.
Delta compression using up to 2 threads.
Compressing objects: 100% (3/3), done.
Writing objects: 100% (3/3), 418 bytes | 418.00 KiB/s, done.
Total 3 (delta 2), reused 0 (delta 0)
remote: FATAL: W refs/heads/master rpms/coin-or-CoinUtils sagitter
DENIED by fallthru
remote: error: hook declined to update refs/heads/master
To ssh://pkgs.fedoraproject.org/rpms/coin-or-CoinUtils
! [remote rejected] master -> master (hook declined)
error: failed to push some refs to
'ssh://sagitter@pkgs.fedoraproject.org/rpms/coin-or-CoinUtils'
--
---
Antonio Trande
Fedora Project
mailto 'sagitter at fedoraproject dot org'
GPG key: 0x5E212EE1D35568BE
GPG key server: https://keys.fedoraproject.org/
6 years, 2 months
A proof-of-concept for delta'ing repodata
by Jonathan Dieter
<tl;dr>
I've come up with a method of splitting repodata into chunks that can
be downloaded and combined with chunks that are already on the local
system to create a byte-for-byte copy of the compressed repodata.
Tools and scripts are at:
https://www.jdieter.net/downloads/
</tl;dr>
Background:
With DNF, we're currently downloading ~20MB of repository data every
time the updates repository changes.
When casync was released, I wondered if we could use it to only
download the deltas for the repodata. At Flock last summer, I ran some
tests against the uncompressed repodata and saw a reduction of 30-40%
from one day to the next, which seemed low, but was a good starting
point.
Unfortunately, due to the way casync separates each file into thousands
of compressed chunks, building each file required thousands of (serial)
downloads which, even on a decent internet connection, took *forever*.
When I talked through the idea with Kevin and Patrick, they also
pointed out that our mirrors might not be too keen on the idea of
adding thousands of tiny files that change every day.
The Solution(?):
One potential solution to the "multitude of files" problem is to merge
the chunks back into a single file, and use HTTP ranges to only
download the parts of the file we want. An added bonus is that most
web servers are configured to support hundreds of ranges in one
request, which greatly reduces the number of requests we have to make.
The other problem with casync is that it's chunk separation is naïve,
which is why we were only achieving 30-40% savings. But we know what
the XML file is supposed to look like, so we can separate the chunks on
the tag boundaries in the XML.
So I've ditched casync altogether and put together a proof-of-concept
(tentatively named zchunk) that takes an XML file, compresses each tag
separately, and then concatenates all of them into one file. The tool
also creates an index file that tells you the sha256sum for each
compressed chunk and the location of the chunk in the file.
I've also written a small script that will download a zchunk off the
internet. If you don't specify an old file, it will just download
everything, but if you specify an old file, it will download the index
of the new file and compare the sha256sums of each chunk. Any
checksums that match will be taken from the old file, and the rest will
be downloaded.
In testing, I've seen savings ranging from 10% (December 17 to today)
to 95% (yesterday to today).
Remaining problems:
* Zchunk files are bigger than their gzip equivalents. This ranges
from 5% larger for filelists.xml to 300% larger for primary.xml.
This can be greatly reduced by chunking primary.xml based on srpm
rather than rpm, which brings the size increase for primary.xml down
to roughly 30%.
* Many changes to the metadata can mean a large number of ranges
requested. I ran a check on our mirrors, and three (out of around
150 that had the file I was testing) don't honor range requests at
all, and three others only honor a small number in a single request.
A further seven didn't respond at all (not sure if that had
anything to do with the range requests), and the rest supported
between 256 and 512 ranges in a single request. We can reduce the
number of ranges requested by always ordering our packages by date.
This would ensure that new packages are grouped at the end of the
xml where they will be grabbed in one contiguous range.
* Zchunk files use zlib (it gives better compression than xz with such
small chunks), but, because they use a custom zdict, they are not gz
files. This means that we'll need new tools to read and write them.
(And I am volunteering to do the work here)
The tools:
The proof-of-concept tools are all sitting in
https://www.jdieter.net/downloads/zchunk-scripts/
They are full of ugly hacks, especially when it comes to parsing the
XML, there's little to no error reporting, and I didn't comment them
well at all, but they should work.
If all you want to do is download zchunks, you need to run dl_zchunk.py
with the url you want to download (ending in .zck) as the first
parameter. Repodata for various days over the last few weeks is at:
https://www.jdieter.net/downloads/zchunk-test/ You may need to hover
over the links to see which is which. The downloads directory is also
available over rsync at rsync://jdieter.net/downloads/zchunk-test.
dl_zchunk.py doesn't show anything if you download the full file, but
if you run the command with an old file as the second parameter, it
will show four numbers: bytes taken from the old file, bytes downloaded
from the new, total downloaded bytes and total uploaded bytes.
zchunk.py creates a .zck file. To group chunks by source rpm in
primary.xml, run
./zchunk.py <file> rpm:sourcerpm
unzchunk.py decompresses a .zck file to stdout
I realize that there's a lot to digest here, and it's late, so I know I
missed something. Please let me know if you have any suggestions,
criticisms or flames, though it might be a few hours before I respond.
Jonathan
6 years, 2 months
Meeting Agenda Item: Introduction Travis Newton
by Travis Newton
Hello everyone!
I've been working as a network admin in the financial industry. I've been
in the Red Hat family of distros ever since I discovered Linux (introduced
to Fedora Core 2 back in 2004). I've always been comfortable with Fedora
and wanted to volunteer and give back to the distro that taught me so
much.
I was a DevOps Engineer in a full CentOS environment which helped me learn
a little bit of Bash, HAProxy, Varnish, RabbitMQ, and a few other tools. I
also have a couple of dedicated servers that I manage and learn off of to
better myself as a Linux admin. All of my professional background has been
with Microsoft products, but I've been bit by the open source bug. I'm
hoping to learn as much as I can about better ways to do things and larger
infrastructures like Fedora such as HA, build systems, and more. I'm just
an information sponge. I have a home lab so I'm really looking towards
sharpening my skills.
On my todo list are a few items like learning Python and C++ and expanding
my networking skills.
My IRC handle is greyaxe90 and look forward with chatting and learning
with everyone!
Travis
6 years, 2 months
Re: [Rpm-ecosystem] A proof-of-concept for delta'ing repodata
by Igor Gnatenko
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256
On Tue, 2018-02-13 at 12:09 +0000, Michael Schroeder wrote:
> On Tue, Feb 13, 2018 at 10:52:14AM +0100, Igor Gnatenko wrote:
> > This would "break" DNF, because libsolv is assigning Id's by the order of
> > packages in metadata. So if something requires "webserver" and there is
> > "nginx"
> > and "httpd" providing it (without versions), then lowest Id is picked up
> > (not
> > going into details of this).
>
> No, this is not correct. Libsolv doesn't use the Id to pick a package,
> exactly to be independent on the package order of the repository.
Oh, could you link me source code for this part? I tried to look it up, but
haven't found. Thanks!
- --
- -Igor Gnatenko
-----BEGIN PGP SIGNATURE-----
iQIzBAEBCAAdFiEEhLFO09aHZVqO+CM6aVcUvRu8X0wFAlqEgP8ACgkQaVcUvRu8
X0z1zw//Vpu6NgZEHZUyes7Bjb5ABmIlP5b9XCo5d0GHAWAfq761eFV4qot/y8H2
ErLDA4+OttOgag+PWNLo5Jpx2pN39r0ccqTWpISfb7lMpumKhlqBAvojSu6tM1nr
9Z3FtBm4EG7pSyjzEQW6Z2BmLovLQOUUncoCR0hbntOB5XK3X/96lkfxmW/8UqRE
62g2SCJHNjz4w74Xc23/ZAkCZtAby6Dw7KJ0DwyPEkl4bRM2GAJhZsnTw8EVHgnH
q4v1aSmYezG4GvKNAF/mepMrKJVxMD1JdkaOsFpzaKqFU/QjNpfwDlEGPzMvnCgA
ZmVQ7SYMFPGg8KiTUfsIWExm5qTEfmrN0e7RmZgHtFmS/H+h1H40ZUGFfd60k+Q/
pr5z1phc2bZ8iew2SscORj8IYe4xT3oOWbcabLGL1w1aM6QobFckavSL39K1Rpfz
YyKGqYOKhENlduhghYvB8+aBM357MVZTrJM5+x9u9NmMiuvH7e/bXQPxhd7LzTCS
ESsO7ryWh5kypccHgizrVLKliz+S3Nl+kf9oQLY98gVyxbMNkhfYDkp0gUY46Zvo
0LoTZ8QcajYiryJzOfi4C+7sJpozowxmasHYb15miCspnA1O3PIa82MgrlS77E8B
8gkbCckeGCxaAa3dXa2JgpENBznQ7DgAZaraQT+ZSxb1jF7QPW8=
=O1D0
-----END PGP SIGNATURE-----
6 years, 2 months
Meeting Agenda Item: Introduction Dennis Otugo
by Wesley Otugo
Hello World! My name is Dennis W. Otugo. I am from Lagos, Nigeria. My IRC Handle is wesleyotugo.
I am familiar with Python & Javascript. I do not have any certification yet, I am currently training for one. I am physically involved in various tech communities but not nothing open source as there is no available community here. But, I am involved with CommOps.
I want to learn as much as I possibly can. I would like to work on any as to gain experience as much as I possibly can.
Thanks.
6 years, 2 months
[release] mdapi: 2.10.4 and 2.10.5
by Pierre-Yves Chibon
Good Morning Everyone,
I cut two new releases of mdapi this morning: 2.10.4 and 2.10.5 the last one
fixing an issue for fedora-packages that Clément ran into.
Here are the corresponding changelogs:
* Fri Feb 09 2018 Pierre-Yves Chibon <pingou(a)pingoured.fr> - 2.10.5-1
- Update to 2.10.5
- Fix the srcpkg endpoint by making it search for a direct match before trying
the sub-packages
* Fri Feb 09 2018 Pierre-Yves Chibon <pingou(a)pingoured.fr> - 2.10.4-1
- Update to 2.10.4
- Fix the content_type header on JSONP output
This is currently happily running in stg.
Happy querying!
Pierre
6 years, 2 months
[PATCH] - Fix fedora-packages Sources tab
by Clement Verna
Hi all,
I have attached a patch to allow httpd to clone the git repo of
package so that fedora-packages can display the spec file and patches.
Thanks
Clément
--- PATCH -----
From 5b818a229d3dfdcfc0d880ce45b82ed093eba73d Mon Sep 17 00:00:00 2001
From: Clement Verna <cverna(a)tutanota.com>
Date: Fri, 9 Feb 2018 17:16:08 +0100
Subject: [PATCH 1/1] Allow httpd to write the git repo used by the source tab
The Source tab is using git repository that are cloned under
/var/cache/fedoracommunity/git.fp.o.
This commit allow httpd to access this folder.
Signed-off-by: Clement Verna <cverna(a)tutanota.com>
---
roles/packages3/web/files/packages-httpd.conf | 4 ++++
1 file changed, 4 insertions(+)
diff --git a/roles/packages3/web/files/packages-httpd.conf
b/roles/packages3/web/files/packages-httpd.conf
index 4d255a9bc..ae873c3ac 100644
--- a/roles/packages3/web/files/packages-httpd.conf
+++ b/roles/packages3/web/files/packages-httpd.conf
@@ -61,6 +61,10 @@ WSGIScriptAlias /packages
/usr/share/fedoracommunity/production/apache/fedoracom
WSGIProcessGroup fedoracommunity
</Directory>
+<Directory /var/cache/fedoracommunity/git.fedoraproject.org>
+ Require all granted
+</Directory>
+
<Directory /var/cache/fedoracommunity/packages/icons/>
# If someone tries to access an icon that doesn't exist,
# then send them to the default icon. This is used by
--
2.14.3
6 years, 2 months
