Plan for tomorrow's Fedora Infrastructure meeting - 2016-03-17
by Kevin Fenzi
The infrastructure team will be having it's weekly meeting tomorrow,
2016-03-17 at 18:00 UTC in #fedora-meeting on the freenode network.
We have a gobby document
(see: https://fedoraproject.org/wiki/Gobby )
fedora-infrastructure-meeting-next is the document.
Please try and review and edit that document before the meeting and we
will use it to have our agenda of things to discuss. A copy as of today
is included in this email.
If you have something to discuss, add the topic to the discussion area
with your name. If you would like to teach other folks about some
application or setup in our infrastructure, please add that topic and
your name to the learn about section.
kevin
--
= Introduction =
This shared document is for the next fedora infrastructure meeting.
We will use it over the week before the meeting to gather status and info and
discussion items and so forth, then use it in the irc meeting to transfer
information to the meetbot logs.
= Meeting start stuff =
#startmeeting Infrastructure (2016-03-17)
#meetingname infrastructure
#topic aloha
#chair smooge relrod nirik abadger1999 lmacken dgilmore threebean pingou puiterwijk pbrobinson
#topic New folks introductions / Apprentice feedback
= Status / information / Trivia / Announcements =
(We put things here we want others on the team to know, but don't need to discuss)
(Please use #info <the thing> - your name)
#topic announcements and information
#info F24 alpha freeze continues - everyone
#info db-koji01/db01 postgresql adjustments - kevin
#info copr RFR has been filed and in progress - msuchy
#info
= Things we should discuss =
We use this section to bring up discussion topics. Things we want to talk about
as a group and come up with some consensus or decision or just brainstorm a
problem or issue. If there are none of these we skip this section.
(Use #topic your discussion topic - your username)
#topic Assessing the Tech Debt Week - ralph
#link https://fedoraproject.org/wiki/Infrastructure/Debt/2016/March
#link https://fedorahosted.org/fedora-infrastructure/search?q=SOP
#topic Getting updated hawkey/libsolv on koschei machine - msimacek
#topic
= Learn about some application or setup in infrastructure =
(This section, each week we get 1 person to talk about an application or setup
that we have. Just going over what it is, how to contribute, ideas for improvement,
etc. Whoever would like to do this, just add the info in this section. In the
event we don't find someone to teach about something, we skip this section
and just move on to open floor.)
Schedule:
2016-03-17 - RFR - patrick
#topic Learn about:
= Meeting end stuff =
#topic Open Floor
#endmeeting
8 years, 1 month
Meeting Agenda Item: Introduction Dimitar Ivanov
by Dimitar Ivanov
Hello everyone,
Allow me to express how honored I feel for being able to write this, I hope
the following e-mail is structured properly and I've added all the
information you need.
Name - Dimitar Ivanov
Time Zone / Country - Bulgaria, GMT +2
Basic skills and experiences
QA guy by profession, going through a re-specialization as part of the
Operations team at the company I work for;
System administrator by hobby and passion. A bit of Bash knowledge,
currently doing some Python courses to enhance my skills for automating
tasks.
Why are you joining?
I've always been a fan of the Open Source idea and community, Linux in
particular. While by no means I'd call myself a power user, I'd say I have
a little bit of a knowledge about this and that.
I've been hopping among different distros through the years, mainly *ubuntu
but I only felt home when I switched to Fedora from CentOS 7. Since I get
such a great OS, and all of its perks, for free, I felt a desire to give
back and hopefully contribute to it and help with its growth.
Reliability and infrastructure has been something I really like working on
so I decided to give it a try and apply for the Fedora Infrastructure team.
As my career is slowly taking a turn in that direction, I hope I will be
able to apply everything I learn, both at work and and in my free time, and
my skills will prove valuable to this project.
What are you looking to do?
Help out with the maintenance of the infrastructure in the face of keeping
servers and services up and running, building tools that will make
everyone's life easier, help with documentation and hopefully be
knowledgeable and skillful enough to be on the front line in case of
outages or critical issues that need attention.
How much time can you contribute?
15 to 20 hours, maybe more depending on my work schedule.
Your IRC handle - winterchillz
What skills you have to offer and which you would like to learn:
Basic knowledge of Bash, MySQL, git, subversion, Python, Nagios, Apache;
Saddened to say I don't feel an expert in any those.
If you decide I might be a valuable asset to the team, I think it'd be good
to start off with maybe getting familiar with the infrastructure, services
and the related documentation.
Thank you for taking your time to read through my e-mail. Feel free to ask
any questions and I'll gladly answer. Any criticism is also more than
welcome.
Of course, if you believe I'm not suitable for this responsibility, I'll
understand and hopefully we'll meet again when I improve my skills. :)
With best regards,
Dimitar
8 years, 1 month
Freeze break: link on pkgs lookaside for fedpkg namespace changes
by Kevin Fenzi
Greetings.
In recent fedpkg/rpkg versions (at least the ones in rawhide now) it's
incorrectly looking for a namespaced (rpms vs anything else) lookaside
instead of the non namespaced one we have.
So, this gives:
% fedpkg -v srpm
Creating repo object from /home/kevin/git/pkgs/midori/master
Downloading midori_0.5.11_all_.tar.bz2
Full url: http://pkgs.fedoraproject.org/repo/pkgs/rpms/midori/midori_0.5.11_all_.ta...
######################################################################## 100.0%
Could not execute srpm: Server returned status code 404
Traceback (most recent call last):
File "/usr/bin/fedpkg", line 16, in <module>
main()
File "/usr/lib/python2.7/site-packages/fedpkg/__main__.py", line 69, in main
sys.exit(client.args.command())
File "/usr/lib/python2.7/site-packages/pyrpkg/cli.py", line 1234, in srpm
self.cmd.sources()
File "/usr/lib/python2.7/site-packages/pyrpkg/__init__.py", line 1696, in sources
hashtype=entry.hashtype, branch=self.branch_merge)
File "/usr/lib/python2.7/site-packages/pyrpkg/lookaside.py", line 174, in download
raise DownloadError('Server returned status code %d' % status)
pyrpkg.errors.DownloadError: Server returned status code 404
And on upload it puts it in a rpms dir thats not correct.
We can see 3 packages already have this:
-bash-4.2$ pwd
/srv/cache/lookaside/pkgs/rpms
-bash-4.2$ ls -l
total 12
drwxr-sr-x. 3 apache apache 4096 Mar 15 12:21 compose-utils
drwxr-sr-x. 3 apache apache 4096 Mar 16 11:06 dbus
drwxr-sr-x. 3 apache apache 4096 Mar 16 11:00 xdg-app
So, I would like to do the following:
* Move all the /srv/cache/lookaside/pkgs/rpms/ to
their /src/cache/lookaside/pkgs/ versions.
* create a link from /srv/cache/lookaside/pkgs/rpms
to /srv/cache/lookaside/pkgs
This should fix this issue until we push out a fedpkg/rpkg fix or want
to namespace lookaside.
+1s?
kevin
8 years, 1 month
Freeze Break Request: Turn on fedmsg for PDC
by Ralph Bean
We currently use a fedmsg-driven daemon to update PDC about stuff that
gets composed by pungi.
This patch turns on a new layer that lets PDC publish its own messages
when it gets updated (the software comes with fedmsg as one of four
different message-publication methods it can use, which is cool!)
adamw wants to use these new messages to drive the validation event
creation project.
The iptables rules and the fedmsg.d/ endpoints and the certs were all
set up when I first stood up PDC.. this patch just turns on the
plugin, and adds the one topic PDC produces to our fedmsg policy.
I'll have to do a master.yml playbook run of the 'fedmsgdconfig' tag,
which will touch all hosts.. but I expect this will be "fine".
Can I get two +1's to push this out?
diff --git a/inventory/group_vars/pdc-web b/inventory/group_vars/pdc-web
index f07deb7..59073da 100644
--- a/inventory/group_vars/pdc-web
+++ b/inventory/group_vars/pdc-web
@@ -29,6 +29,5 @@ fedmsg_certs:
- service: pdc
owner: root
group: apache
- # We don't have notifications from PDC yet, but when we do, add them here.
- #can_send:
- #- pdc.somethingorother
+ can_send:
+ - pdc.compose
diff --git a/inventory/group_vars/pdc-web-stg b/inventory/group_vars/pdc-web-stg
index 1c55f07..a7e45f9 100644
--- a/inventory/group_vars/pdc-web-stg
+++ b/inventory/group_vars/pdc-web-stg
@@ -29,6 +29,5 @@ fedmsg_certs:
- service: pdc
owner: root
group: apache
- # We don't have notifications from PDC yet, but when we do, add them here.
- #can_send:
- #- pdc.somethingorother
+ can_send:
+ - pdc.compose
diff --git a/roles/pdc/frontend/templates/settings_local.py b/roles/pdc/frontend/templates/settings_local.py
index 2d021a5..256260c 100644
--- a/roles/pdc/frontend/templates/settings_local.py
+++ b/roles/pdc/frontend/templates/settings_local.py
@@ -10,6 +10,11 @@
# settings, please remember to update your settings_local.py
# when the items you extended got updated in settings.py.
+# Turn on the fedmsg publishing plugin.
+MESSAGE_BUS = {
+ 'MLP': 'fedmsg', # MLP: Messaging Library Package
+}
+
REST_FRAMEWORK = {
'DEFAULT_AUTHENTICATION_CLASSES': (
'pdc.apps.auth.authentication.TokenAuthenticationWithChangeSet',
8 years, 1 month
Freeze break: db01 tuning
by Kevin Fenzi
Greetings.
I was going to wait until after freeze to mess with the other
postgresql hosts, but then mattdm pointed out this morning that
hyperkitty is really slow currently. I'm not sure this will help, but
it really make db-koji01 a great deal happier and it should be pretty
easy to revert if needed.
So, I'd like to make the same changes that made db-koji01 happy to
db01:
diff --git a/roles/postgresql_server/templates/postgresql.conf b/roles/postgresql_server/templates/postgresql.conf
index 4a8bdfb..de9f1f0 100644
--- a/roles/postgresql_server/templates/postgresql.conf
+++ b/roles/postgresql_server/templates/postgresql.conf
@@ -124,6 +124,9 @@ shared_buffers = {{ shared_buffers }} # min 128kB or max_connections*16kB
{% if ansible_hostname.startswith("db-koji01") %}
work_mem = 4MB # min 64kB
maintenance_work_mem = 16MB # min 1MB
+{% elif ansible_hostname.startswith("db01") %}
+work_mem = 4MB # min 64kB
+maintenance_work_mem = 1024MB # min 1MB
{% else %}
work_mem = 2MB # min 64kB
maintenance_work_mem = 1024MB # min 1MB
@@ -222,8 +225,10 @@ random_page_cost = 3.0 # same scale as above
#cpu_operator_cost = 0.0025 # same scale as above
{% if ansible_hostname.startswith("db-koji01") %}
effective_cache_size = 24GB
+{% elif ansible_hostname.startswith("db01") %}
+effective_cache_size = 12GB
{% else %}
-effective_cache_size = 5GB
+effective_cache_size = 5GB
{% endif %}
# - Genetic Query Optimizer -
+1s?
kevin
8 years, 1 month
Freeze break: db-koji01 tuning
by Kevin Fenzi
Greetings.
Over the last few weeks, db-koji01's load has been climbing.
We have been getting a lot of noise from our monitoring when the query
we use to monitor doesn't return fast enough.
https://admin.fedoraproject.org/collectd/bin/index.cgi?hostname=db-koji01...
There's very likely some upstream work to be doing adding indexes in
the right places, etc, but I think we can tune things a bit now to make
it better in the short term.
I looked over
https://wiki.postgresql.org/wiki/Tuning_Your_PostgreSQL_Server and saw
that some of our settings were out of wack.
I'd like to apply the below patch and restart postgresql later tonight
when things aren't as busy. If it causes some issue we can back it out,
but I hope it might improve things.
I tried to setup the changes so they only apply to db-koji01 for now,
but after the freeze we might adjust other postgresql servers.
+1s?
kevin
--
diff --git a/inventory/host_vars/db-koji01.phx2.fedoraproject.org b/inventory/host_vars/db-koji01.phx2.fedoraproject.org
index 41f3677..855a220 100644
--- a/inventory/host_vars/db-koji01.phx2.fedoraproject.org
+++ b/inventory/host_vars/db-koji01.phx2.fedoraproject.org
@@ -39,4 +39,4 @@ nrpe_procs_warn: 600
nrpe_procs_crit: 700
host_backup_targets: ['/backups']
-shared_buffers: "4GB"
+shared_buffers: "8GB"
diff --git a/roles/postgresql_server/templates/postgresql.conf b/roles/postgresql_server/templates/postgresql.conf
index 603f9ea..4a8bdfb 100644
--- a/roles/postgresql_server/templates/postgresql.conf
+++ b/roles/postgresql_server/templates/postgresql.conf
@@ -121,8 +121,13 @@ shared_buffers = {{ shared_buffers }} # min 128kB or max_connections*16kB
# 8 MB is probably on the high side. We can probably do with 4MB. But we
# were seeing a problem and we have the RAM so we're going to try this.
-work_mem = 2MB # min 64kB
-maintenance_work_mem = 1024MB # min 1MB
+{% if ansible_hostname.startswith("db-koji01") %}
+work_mem = 4MB # min 64kB
+maintenance_work_mem = 16MB # min 1MB
+{% else %}
+work_mem = 2MB # min 64kB
+maintenance_work_mem = 1024MB # min 1MB
+{% endif %}
#max_stack_depth = 2MB # min 100kB
# - Free Space Map -
@@ -215,7 +220,11 @@ random_page_cost = 3.0 # same scale as above
#cpu_tuple_cost = 0.01 # same scale as above
#cpu_index_tuple_cost = 0.005 # same scale as above
#cpu_operator_cost = 0.0025 # same scale as above
+{% if ansible_hostname.startswith("db-koji01") %}
+effective_cache_size = 24GB
+{% else %}
effective_cache_size = 5GB
+{% endif %}
# - Genetic Query Optimizer -
8 years, 1 month