Celery in Fedora Infra (was: RFC: Fedora Scale-Out Docker Registry
by Pierre-Yves Chibon
Proposal)
Reply-To:
On Mon, May 09, 2016 at 01:49:26PM -0600, Kevin Fenzi wrote:
> We have tried to avoid celery serveral times in the past and always
> managed to, but perhaps we can't this time. Is there any alternative to
> the celery use?
Outside of the topic at hand, do you remember the reasons why we tried to avoid
celery until now?
Was this something specific to celery? Or AMQP in general? Or?
Since I'm investigating changing the architecture of FMN's backend in such a way
that it would use both redis and AMQP (rabbitmq to be precise), I'm curious if
there is anything I should be aware of before going to far down this road :)
Thanks,
Pierre
8 years
Plan for tomorrow's Fedora Infrastructure meeting - 2016-05-12
by Kevin Fenzi
The infrastructure team will be having it's weekly meeting tomorrow,
2016-05-12 at 18:00 UTC in #fedora-meeting on the freenode network.
We have a gobby document
(see: https://fedoraproject.org/wiki/Gobby )
fedora-infrastructure-meeting-next is the document.
Please try and review and edit that document before the meeting and we
will use it to have our agenda of things to discuss. A copy as of today
is included in this email.
If you have something to discuss, add the topic to the discussion area
with your name. If you would like to teach other folks about some
application or setup in our infrastructure, please add that topic and
your name to the learn about section.
kevin
--
= Introduction =
This shared document is for the next fedora infrastructure meeting.
We will use it over the week before the meeting to gather status and info and
discussion items and so forth, then use it in the irc meeting to transfer
information to the meetbot logs.
= Meeting start stuff =
#startmeeting Infrastructure (2016-05-12)
#meetingname infrastructure
#topic aloha
#chair smooge relrod nirik abadger1999 lmacken dgilmore threebean pingou puiterwijk pbrobinson
#topic New folks introductions / Apprentice feedback
= Status / information / Trivia / Announcements =
(We put things here we want others on the team to know, but don't need to discuss)
(Please use #info <the thing> - your name)
#topic announcements and information
#info Fedora 24 Beta is out the door! - everyone
#info Thus we are no longer in freeze - everyone
#info Added memory to mailman01 to hopefully prevent alerts - kevin
#info Going to migrate the last of the lists to mailman01 this week - abompard
#info most NFS netapp mounts are now moved to nfsv4 - kevin
#info Planning to migrate comps and spin-kickstarts to pagure on friday - kevin
#info
= Things we should discuss =
We use this section to bring up discussion topics. Things we want to talk about
as a group and come up with some consensus or decision or just brainstorm a
problem or issue. If there are none of these we skip this section.
(Use #topic your discussion topic - your username)
#topic Beta release day issues - kevin
#topic Fedorahosted migration / EOL - kevin
#topic
= Apprentice office hours =
Here we will discuss any apprentice questions, try and match up people looking
for things to do with things to do, progress, testing anything like that.
#topic Apprentice office hours
= Learn about some application or setup in infrastructure =
(This section, each week we get 1 person to talk about an application or setup
that we have. Just going over what it is, how to contribute, ideas for improvement,
etc. Whoever would like to do this, just add the info in this section. In the
event we don't find someone to teach about something, we skip this section
and just move on to open floor.)
#topic Learn about: Our various datacenters - kevin
= Meeting end stuff =
#topic Open Floor
#endmeeting
8 years
Freeze Break: fullefiletime list
by Kevin Fenzi
Greetings.
There was some talk on the mirror lists about making a fullfilelist
version that also had timestamps in it so mirrors could look for all
the files that changed since the last time they synced and only sync
those specific files. This would be MUCH faster for them as they don't
need to pull the entire metadata over. This would be MUCH better for us
as they don't have to hammer our netapps to get all the metadata.
Of course this is not a 100% solution, as they won't see deleted files
or hardlinks. So, they will likely still need to do a real full sync
every once in a while (monthly?) to get those things synced up.
I'd like to push the following change to create the file and then we
can look at writing up some simple scripting for people to use and see
if we can get people to use it and lower our netapp metadata hammering.
+1s?
kevin
--
diff --git a/roles/bodhi2/backend/files/update-fullfilelist b/roles/bodhi2/backend/files/update-fullfilelist
index 0302c6a..bac3f9c 100755
--- a/roles/bodhi2/backend/files/update-fullfilelist
+++ b/roles/bodhi2/backend/files/update-fullfilelist
@@ -1,6 +1,6 @@
#!/bin/bash
-# currently runs on releng2.fedora.phx.redhat.com
+# currently runs on bodhi-backend01 after updates pushes
MOD=$1
[ -z "$MOD" ] && {
@@ -8,6 +8,8 @@ MOD=$1
exit 1
}
+# This is the old traditional fullfilelist with no timestamps
+
TMPFILE=$(mktemp -p /tmp/)
pushd /pub/$MOD > /dev/null
find * -print > $TMPFILE
@@ -18,3 +20,16 @@ else
fi
chmod 0644 fullfilelist
popd > /dev/null
+
+# This is the new list with timestamps
+
+TMPFILE=$(mktemp -p /tmp/)
+pushd /pub/$MOD > /dev/null
+/usr/bin/rsync --no-h --list-only -r . > $TMPFILE
+if diff $TMPFILE fullfiletimelist > /dev/null; then
+ rm -f $TMPFILE
+else
+ mv $TMPFILE fullfiletimelist
+fi
+chmod 0644 fullfiletimelist
+popd > /dev/null
diff --git a/roles/releng/files/update-fullfilelist b/roles/releng/files/update-fullfilelist
index 0302c6a..bac3f9c 100755
--- a/roles/releng/files/update-fullfilelist
+++ b/roles/releng/files/update-fullfilelist
@@ -1,6 +1,6 @@
#!/bin/bash
-# currently runs on releng2.fedora.phx.redhat.com
+# currently runs on bodhi-backend01 after updates pushes
MOD=$1
[ -z "$MOD" ] && {
@@ -8,6 +8,8 @@ MOD=$1
exit 1
}
+# This is the old traditional fullfilelist with no timestamps
+
TMPFILE=$(mktemp -p /tmp/)
pushd /pub/$MOD > /dev/null
find * -print > $TMPFILE
@@ -18,3 +20,16 @@ else
fi
chmod 0644 fullfilelist
popd > /dev/null
+
+# This is the new list with timestamps
+
+TMPFILE=$(mktemp -p /tmp/)
+pushd /pub/$MOD > /dev/null
+/usr/bin/rsync --no-h --list-only -r . > $TMPFILE
+if diff $TMPFILE fullfiletimelist > /dev/null; then
+ rm -f $TMPFILE
+else
+ mv $TMPFILE fullfiletimelist
+fi
+chmod 0644 fullfiletimelist
+popd > /dev/null
8 years
Freeze break: increase memory on mailman01
by Kevin Fenzi
Greetings.
Over this month so far, we have had mailman01 alert 27 times about swap
space being low. When we look, it is indeed low, but there is still
lots of memory available, so a swapoff -a && swapon -a "fixes" it.
My theory as to what is happening is that from time to time there's a
Django thread that takes up lots and lots of memory, causing the
machine to swap out other things, then it completes, leaving lots of
things swapped out, but memory available. I have in fact seen the
web-ui take up a ton of memory.
Right now we have the webui set to:
maximum-requests=1000 processes=4 threads=4
We may want to adjust that? Perhaps less processes and more threads?
However, one simple thing we can do right now is to just throw more
memory at it. The virthost it's on has a lot of memory to spare, so it
would be trivial to just double it's memory and see if the issue goes
away. (from 16GB to 32GB). This would require a quick reboot, but
downtime should be very short.
Thoughts? +1s?
kevin
8 years
Fedora 24 Beta freeze now in effect
by Kevin Fenzi
Greetings.
we are now in the infrastructure freeze leading up to the Fedora 24
Beta release. This is a pre-release freeze.
We do this to ensure that our infrastructure is stable and ready to
release the Fedora 24 Beta when it's available.
You can see a list of hosts that do not freeze by checking out the
ansible repo and running the freezelist script:
git clone https://infrastructure.fedoraproject.org/infra/ansible.git
scripts/freezelist -i inventory
Any hosts listed as freezes is frozen until 2016-05-03. (or later if
Beta slips). Frozen hosts should have no changes made to them
without a sign-off on the change from at least 2 sysadmin-main or
rel-eng members, along with (in most cases) a patch of the exact
change to be made to this list.
Thanks,
kevin
8 years
Backups for piwik cloud instance
by Ricky Elrod
As some of you know, some of us (mostly smooge and I) have been working
on statistics gathering for Fedora web infra as of late.
We are working on setting up a piwik instance to be used on our
websites. However, this instance lives in the cloud and thus can't hit
the VPN.
A question I ran into and was told to email the list about and/or bring
up at the meeting, is: What should our backups story look like for such
cases? Mostly, we have a MySQL database on this node that will get
really big fairly quickly (right now /var/lib/mysql is about 1.1GB and
piwik only has one site added, and it's only been there for a few weeks).
Are other cloud instances backed up in any way, and if so how was it
done? If not, how do we go about coming up with a plan for doing this,
for this instance and other instances in the future that end up having
similar requirements?
(Smooge can weigh in if I'm forgetting any important details here).
-Ricky
8 years
OAuth2 + OpenID Connect: patches incoming
by Patrick Uiterwijk
Hi all,
As some of you might know, Ipsilon, the identity provider that we use
for id.fedoraproject.org, has just merged its OAuth2 + OpenID connect
support.
I will be rolling a new Ipsilon release in the forseeable future, and
I hope to get this deployed into at least staging between Beta F24 and
Final freeze.
This email is mostly meant to inform people that I will be sending
patches to various projects in use at Fedora Infrastructure to add
support for this
for authentication to the API and web interface.
While I have a set of patches ready for certain applications, others
might take some time to write, so if you would like to get started
with the migration and
don't want to wait for me to port, feel free to get in touch with me
and I can help you get started.
I will send more information as I get further in the process.
Regards,
Patrick Uiterwijk
8 years
Last of mailing lists migration
by Kevin Fenzi
Greetings.
I thought I would start a thread here about the end part of our
mailman3 migration plans.
There's currently just 4 lists left in mailman2.
One of those is package-announce that uses 'topics' pretty heavily and
unfortunately, mailman3 still lacks topics support. I don't want to
wait until it's implemented since we have no idea when that might be.
So, I have a blog post ready that discusses topics support and
alternatives to get that information, as well as a bit of stuff added
to the wiki: https://fedoraproject.org/wiki/Mailman3_Migration
I can post that blog post a few days before the migration?
Or just publish it now?
I'd like to complete the migration after Beta is out the door and we
are out of freeze. Either May 11th or May 13th would be fine with me.
We would migrate the last lists, then we need to change things to
proxy everything to mailman01 and take collab03 out of the mix
entirely. If all looks well after a week or so, we can retire collab03.
Thoughts? (especially from Aurélien :)
kevin
8 years
