infrastructure November 2014

infrastructure@lists.fedoraproject.org

36 participants
52 discussions

firewall rules on builders (iptables, firewalld, libvirt...)
by Matthew Miller 30 Apr '15

30 Apr '15

It's my understanding (Dennis please correct if I'm wrong) that the problem with cloud image creation was due to libvirt iptables rules being lost when iptables was restarted. This is a fundamental known issue (see last paragraph of <http://libvirt.org/firewall.html>), and one of the things firewalld was meant to solve. Dennis says that there are lot of complicated rules on the builders making switching to firewalld difficult. One possibility might be to move those complicated rules from the builders to a network firewall, and keep the host rules simple and functional. But that's probably a big undertaking. In the meantime, any time iptables is restarted or reloaded, libvirt needs a SIGHUP. (I suppose this means: ansible playbooks and also added to any manual procedures.) [cc rel-eng, reply-to infrastructure] -- Matthew Miller <mattdm(a)fedoraproject.org> Fedora Project Leader

6 8

Gitolite3 on pkgs01.stg
by Pierre-Yves Chibon 17 Dec '14

17 Dec '14

Dear all, This morning Mathieu (bochecha) and I spent some time trying to get gitolite3 work on our pkgs01.stg. This is an overview of the situation. By design gitolite is meant to be installed with a dedicated user. Everyone uses that user. gitolite then maps the public ssh key to its user database and its configuration file to handle who is allowed to do what on which git repo. Our setup is quite different from this. We have a single configuration file at /etc/gitolite/conf/gitolite.conf that contains all the information about who is allowed to do what on which git repos. In addition we have a single /etc/gitolite/gitolite.rc configuration file that specifies to gitolite where are the git repositories located. Each user has then an account on the machine but by default no shell access. The access is restricted via ssh with in ~/.ssh/authorized_keys the instructions: `command="/usr/bin/gl-auth-command"` (people with shell access have a different command). That commands combines the info from /etc/gitolite/gitolite.rc and /etc/gitolite/conf/gitolite.conf and allow or deny the access/action. Now gitolite3 has changed quite a bit compared to our setup. Basically, gitolite3 expects the gitolite.rc file to be at ~/.gitolite.rc and the repositories to be at ~/repositories. Having symlink to these locations are fine, but they should be there. What we can do also is play on the $HOME variable. In pkgs01.stg we kept the configurations files (gitolite.conf and gitolite.rc) in /etc/gitolite as before, but we created a couple of symlink in /srv/git/ .gitolite -> /etc/gitolite/ repositories -> /srv/git/rpms/ The idea is to be able to set HOME=/srv/git and run the gitolite command. This way we managed to get the genacls.sh script working. It is the script that generates the gitolite.conf file based on the pkgdb information and compile that configuration file so that gitolite can use it. gitolite3 also changes the command in the ~/.ssh/authorized_keys file to: command="/usr/share/gitolite3/gitolite-shell user" That command is set in the sources in `src/triggers/post-compile/ssh-authkeys` and put in the file by (I believe) `gitolite compile` that is ran by genacls.sh. In order to test, I (manually) changed Mathieu's authorized_keys to look like: command="HOME=/srv/git/ /usr/share/gitolite3/gitolite-shell user" We then tried to get Mathieu to clone a repo and we found out: - the ~/.gitolite folder, that gitolite3 requires, needs to have a `logs/` subfolder, and that folder needs to be accessible read and write to everyone so `chmod 777 -R /srv/git/.gitolite/logs` (ie: chmod 777 /etc/gitolite/logs since it's a symlink) - the /etc/gitolite/conf/gitolite.conf-compiled.pm needs to be readable by everyone since it's the file gitolite uses to see who can do what on which repo so `chmod o+r conf/gitolite.conf-compiled.pm - Mathieu was finally able to clone and push in a repo he has access to and could not on the kernel repo (as supposed to since he does not have commit there), but the error returned is just plain ugly So to conclude: - we managed to get it to work - its needs chmod 777 on /srv/git/.gitolite/logs - its needs chmod o+r on conf/gitolite.conf-compiled.pm (and that after each time the file is generated) - we need to find a way to fix the command in the user's ~/.ssh/authorized_keys so that they specify the HOME variable to use when running the gitolite command -> Might very well require to patch gitolite itself for this (and thus keep the patch in our package ad vitam aeternam) That being said, I believe our options are: 1) Talk with upstream, in the past I believe he was quite reactive and willing to help us. We are the largest public deployment of gitolite maybe he'll still be willing to help us to discuss: - setting HOME in the authorized_keys - writing logs - accessing gitolite.conf-compiled.pm 2) Stick with gitolite2 -> We need to find out how easy it is to maintain it on epel7 and we need someone willing to maintain it (knowing that it's perl) -> Upstream said gitolite2 is still maintain wrt security fixes but I guess only for some time 3) Move to the model gitolite relies on (1 specific gitolite user everyone accesses the repo through). + No more need to create account on the machine for each and every packager + Inline with gitolite's design - Breaking all existing git clone around (as the push/pull urls change) --> URLs change could be handled at the fedpkg level + We are already forcing an update of fedpkg to our packager with the change from md5 to sha for the lookaside cache, so we could couple both change in the same update and preventing forcing two fedpkg updates. - Huge number of keys to handle for that user ? Performance impact of this number of users? ? Dennis mentionned logging at the meeting yesterday, but it looks like gitolite does provide logging itself 4) Move away from gitolite. I did not check what else would be available/suitable for us, so no ideas there. Suggestions welcome I guess :) I think that covers most if not all our findings. Questions and suggestions welcome :) Pierre and Mathieu

7 11

[release] pkgdb2 1.21
by Pierre-Yves Chibon 12 Dec '14

12 Dec '14

Hi all, I have just released and pushed to staging a new version of pkgdb2, here is the changelog: * Fri Nov 21 2014 Pierre-Yves Chibon <pingou(a)pingoured.fr> - 1.21-1 - Update to 1.21 - DB optimization: do not use LIKE in queries where there is no '%' - Update the layout of the user list page - Add flag to set/update the monitoring flag on packages. This flag tells the-new-hotness whether the maintainers are interested in getting bugzilla tickets about updates available on their package (updates being monitored via anitya: release-monitoring.org) - Clean(er) logic to mark the fields mandatory in the forms - Fix indentation in the mail_logging module - Branch orphaned packages as well as we branch for new releases (Subho-bcrec) - Auto select packagers search in packager details page. (Ratnadeep Debnath) - Obsolete all ACLs on a package that is retired - Nicer docstring and API documentation by using textwrap.dedent() - Add a `former_poc` argument to the orphan API endpoint allowing to restrict the orphan to only the package of a certain packager - Add the former_poc keyword argument to api_acl_reassign, same principle as when orphaning a package - DB optimization: optimize the API endpoint returning the packager's ACLs - Include the monitoring state in the GET API. (Ralph Bean) - Include the api_groups and api_monitored in the API documentation - Update the update_package_info script to include other releases than rawhide The biggest change being likely the introduction of the monitoring flag to integrate pkgdb, anitya and the-new-hotness. Testing welcome :) Thanks, Pierre

2 5

Fedora 21 Final Freeze now in effect
by Kevin Fenzi 10 Dec '14

10 Dec '14

Greetings. we are now in the infrastructure freeze leading up to the Fedora 21 Final release. This is a final release freeze. You can see a list of hosts that do not freeze by checking out the ansible repo and running the freezelist script: git clone http://infrastructure.fedoraproject.org/infra/ansible.git scripts/freezelist -i inventory/inventory Anything listed as freezes is frozen until 2014-12-09 (or later if the release slips). Frozen hosts should have no changes made to them without a signoff on the change from at least 2 sysadmin-main or rel-eng members. Thanks, kevin

1 1

OOM killer on bapp02
by Adrian Reber 09 Dec '14

09 Dec '14

The OOM killer on bapp02 has terminated a few mirrormanager crawler processes. It seems it needs more memory or the number of parallel crawlers has to be further limited. Another good idea would be to limit the duration of the rsync crawls in /etc/mirrormanager/prod.cfg to maybe one day (--timeout=86400) to avoid stale rsync processes. Adrian

3 4

freeze break request: delete .tx/config to pull po files automatically
by Robert Mayr 04 Dec '14

04 Dec '14

Hi all, we need to delete the /var/lib/transifex-client/.tx/config file on bapp, which should execute directly syncTranslations.sh on puppet (fedora-web/files), if not it gets executed at next cron-job. Doing so re-creates all transifex resources, included the new getfedora.org resource, in /var/tmp/po_dir/, needed to pull pos automatically once a day. The procedure has been already tested on app01.stg and worked as expected. Actually we are pulling the pos manually, but we'd need to be sure this automatic process works smooth before releasing F21. Any +1's or concerns? Thank you. -- Robert Mayr (robyduck)

3 7

Code Search for Fedora
by Michael Stapelberg 01 Dec '14

01 Dec '14

Hey, Recently I’ve been talking to Hannes (cc'ed) about whether Fedora would be interested in having the equivalent of http://codesearch.debian.net/¹ The project came to live as my Bachelor of Science Thesis² and aims to provide fast regular expression search over a big corpus, in this case 140 GB of source code of all software included in the Debian main distribution (as opposed to non-free or contrib, which we excluded because of licensing concerns). It is based on the work Russ Cox published, which in turn resembles the work he did on Google Code Search when he was an intern there in 2006. So, what’s this discussion about? What I’m offering is setting up/running a public version of Code Search for Fedora. It needs to be public because I want the open source community as a whole profit from it, and also I’m told you have somewhat comparable tools internally anyway :). My motivation comes from multiple places: 1) I’m fairly sure Fedora packages a slightly different set of software than Debian, so running both DCS (Debian Code Search) and FCS (Fedora Code Search) would enlarge the amount of searchable software. 2) I’m interested in my work having a positive effect on the world (or at least the open source community), and running multiple instances of Code Search reduces its dependency on any single distribution, thereby increasing its reliability and scope. 3) Last but not least, I intend to try Fedora on one of my computers to broaden my horizons. I figured getting in contact with some of you while working on this project may be a good way to set a foot into the community and see whether I like it around here. In terms of what I’d need in order to make this project a success, there are some hardware requirements (aside from, of course, time and motivation): The in-memory index and searchable source code can be sharded on an almost arbitrary number of different computers, which is necessary to some extent, due to maximum size limitations for the index of a single shard to be < 2 GB. At the moment, we are running 6 different index-backend VMs, each serving 1.8G in-memory indexes and about 40G of source code (including partial indexes). In order to grep through the source quickly, the source is stored on local SSDs (as opposed to a network block storage volume, or even regular HDDs). In addition to the actual data, we also need a web frontend to serve and combine this data, and we have one more VM which scrapes monitoring information and shows nice graphs about how the whole system behaves. So, in total, we run 8 VMs, of which 6 are equipped with 4 cores, 4G of RAM (for 2G of index + 2G page cache for grepping files) and 40G SSD volumes each. The web frontend uses 4 cores and 2G of RAM, and also an SSD for caching entire query results. The monitoring VM needs just one core and 2G of RAM. Does that sound reasonable and feasible? I’m not sure what kind of hardware you have available for projects like this one, and currently we’re sponsored by Rackspace because Debian doesn’t have that sort of hardware easily available. I feel like this email is long enough already, so I’ll just ask a general: what do you think? Do you need any more information? Please just ask, and keep me CC'ed, since I’m not subscribed to this list. Thanks in advance, Best regards, Michael Stapelberg ¹ Note that there is a rather big redesign in progress, both architecturally and visually: https://people.debian.org/~stapelberg//2014/11/09/upcoming-debian-codesearc… So, in case you browse around on the current version and conclude that it sucks, just wait for the update and everything will be awesome ;). ² http://codesearch.debian.net/research/

6 11

Freeze break: fas2.py hotfix
by Kevin Fenzi 01 Dec '14

01 Dec '14

We have one user (lsm5) that had a override in python-fedora to map their bugzilla address to their @fedoraproject.org address. They have just changed this and want us to remove the override. We also have currently a locally modified fas2.py file on pkgdb02 with these overrides, so I'd like to pull it into an ansible hotfix. The first part of this is adding the hotfix, the second part is the changes from the stock python-fedora fas2.py file (one override thats not in the package currently). +1s? kevin -- diff --git a/roles/pkgdb2/tasks/main.yml b/roles/pkgdb2/tasks/main.yml index 0185c27..7e2c3e0 100644 --- a/roles/pkgdb2/tasks/main.yml +++ b/roles/pkgdb2/tasks/main.yml @@ -19,6 +19,12 @@ tags: - packages +# HOTFIX: adjust bugzilla overrides +- name: HOTFIX: adjust bugzilla overrides + copy: src=fas2.py dest=/usr/lib/python2.7/site-packages/fedora/client/fas2.py + tags: + - config + - name: copy sundry pkgdb configuration template: src={{ item.file }} dest={{ item.location }}/{{ item.dest }} --- roles/pkgdb2/files/fas2.py.orig 2014-11-30 17:33:54.350767000 +0000 +++ roles/pkgdb2/files/fas2.py 2014-11-30 17:32:37.557767000 +0000 @@ -255,6 +255,8 @@ 103551: 'fschwarz(a)fedoraproject.org', # Martin Holec: martix(a)martix.names 137561: 'mholec(a)redhat.com', + # John Dulaney: j_dulaney(a)live.com + 149140: 'jdulaney(a)fedoraproject.org', } # A few people have an email account that is used in owners.list but # have setup a bugzilla account for their primary account system email

4 3

Freeze exception request: Mediawiki upgrade
by Patrick Uiterwijk 28 Nov '14

28 Nov '14

Hi all, Yesterday, MediaWiki 1.19.22 has been release which fixes some security issues. This is a code-only change, and there's no database or configuration changes. They did add a new permission flag to change another user's common.js, but as far as I have been able to see, nobody in our instance used that. This version is now running in stg (https://stg.fedoraproject.org/wiki/Special:Version) Could I get +1s to upgrade production? With kind regards, Patrick Uiterwijk Associate Software Engineer, Red Hat

3 2

Freeze exception request: nagios contact info
by Patrick Uiterwijk 28 Nov '14

28 Nov '14

Hi, I want to update my contact information for nagios alerts, any +1s to push this? >From bbf537503698e109c1304b1deba2820753c3e531 Mon Sep 17 00:00:00 2001 From: Patrick Uiterwijk <puiterwijk(a)redhat.com> Date: Fri, 28 Nov 2014 16:11:32 +0000 Subject: [PATCH] Update puiterwijk nagios contact info --- .../contactgroups/fedora-sysadmin-email.cfg | 2 +- .../files/nagios-external/contacts/puiterwijk.cfg | 27 ++++++++++++++++++- .../nagios/contactgroups/fedora-sysadmin-pager.cfg | 2 +- .../files/nagios/contacts/puiterwijk.cfg | 27 ++++++++++++++++++- 4 files changed, 52 insertions(+), 6 deletions(-) diff --git a/roles/nagios_server/files/nagios-external/contactgroups/fedora-sysadmin-email.cfg b/roles/nagios_server/files/nagios-external/contactgroups/fedora-sysadmin-email.cfg index e099113..d1bc113 100644 --- a/roles/nagios_server/files/nagios-external/contactgroups/fedora-sysadmin-email.cfg +++ b/roles/nagios_server/files/nagios-external/contactgroups/fedora-sysadmin-email.cfg @@ -1,5 +1,5 @@ define contactgroup{ contactgroup_name fedora-sysadmin-email alias Fedora Sysadmin Email Contacts - members mmcgrath,ausil,admin,nigelj,ricky,jcollie,jmtaylor,jstanley,smooge,nb,rigeld2,codeblock,kevin,hvivani,puiterwijkp + members mmcgrath,ausil,admin,nigelj,ricky,jcollie,jmtaylor,jstanley,smooge,nb,rigeld2,codeblock,kevin,hvivani,puiterwijk } diff --git a/roles/nagios_server/files/nagios-external/contacts/puiterwijk.cfg b/roles/nagios_server/files/nagios-external/contacts/puiterwijk.cfg index 4772e4c..910d020 100644 --- a/roles/nagios_server/files/nagios-external/contacts/puiterwijk.cfg +++ b/roles/nagios_server/files/nagios-external/contacts/puiterwijk.cfg @@ -1,4 +1,16 @@ define contact{ + contact_name puiterwijk + alias Patrick Uiterwijk + service_notification_period 24x7 + host_notification_period 24x7 + service_notification_options w,u,c,r + host_notification_options d,u,r + service_notification_commands notify-by-epager + host_notification_commands host-notify-by-epager + email patrick(a)puiterwijk.org +} + +define contact{ contact_name puiterwijkp alias Patrick Uiterwijk service_notification_period 24x7 @@ -7,6 +19,17 @@ define contact{ host_notification_options d,u,r service_notification_commands notify-by-epager host_notification_commands host-notify-by-epager - email puiterwijk(a)gmail.com - pager puiterwijk(a)gmail.com + email patrick-pager(a)puiterwijk.org +} + +define contact{ + contact_name puiterwijk-emergency + alias Patrick Uiterwijk + service_notification_period 24x7 + host_notification_period 24x7 + service_notification_options w,u,c,r + host_notification_options d,u,r + service_notification_commands notify-by-epager + host_notification_commands host-notify-by-epager + email +31681462748(a)gin.nl } diff --git a/roles/nagios_server/files/nagios/contactgroups/fedora-sysadmin-pager.cfg b/roles/nagios_server/files/nagios/contactgroups/fedora-sysadmin-pager.cfg index 526aa04..1a8d2ba 100644 --- a/roles/nagios_server/files/nagios/contactgroups/fedora-sysadmin-pager.cfg +++ b/roles/nagios_server/files/nagios/contactgroups/fedora-sysadmin-pager.cfg @@ -6,5 +6,5 @@ define contactgroup{ define contactgroup{ contactgroup_name fedora-sysadmin-emergency alias Fedora Sysadmin Pager Contacts - members mmcgrath-emergency,ricky-emergency,jstanley-emergency,smooge-emergency,kevin-emergency,puiterwijkp + members mmcgrath-emergency,ricky-emergency,jstanley-emergency,smooge-emergency,kevin-emergency,puiterwijk-emergency } diff --git a/roles/nagios_server/files/nagios/contacts/puiterwijk.cfg b/roles/nagios_server/files/nagios/contacts/puiterwijk.cfg index 38b8392..910d020 100644 --- a/roles/nagios_server/files/nagios/contacts/puiterwijk.cfg +++ b/roles/nagios_server/files/nagios/contacts/puiterwijk.cfg @@ -1,4 +1,16 @@ define contact{ + contact_name puiterwijk + alias Patrick Uiterwijk + service_notification_period 24x7 + host_notification_period 24x7 + service_notification_options w,u,c,r + host_notification_options d,u,r + service_notification_commands notify-by-epager + host_notification_commands host-notify-by-epager + email patrick(a)puiterwijk.org +} + +define contact{ contact_name puiterwijkp alias Patrick Uiterwijk service_notification_period 24x7 @@ -7,6 +19,17 @@ define contact{ host_notification_options d,u,r service_notification_commands notify-by-epager host_notification_commands host-notify-by-epager - email puiterwijk(a)gmail.com - pager puiterwijk(a)gmail.com + email patrick-pager(a)puiterwijk.org +} + +define contact{ + contact_name puiterwijk-emergency + alias Patrick Uiterwijk + service_notification_period 24x7 + host_notification_period 24x7 + service_notification_options w,u,c,r + host_notification_options d,u,r + service_notification_commands notify-by-epager + host_notification_commands host-notify-by-epager + email +31681462748(a)gin.nl } -- 1.7.2.1 With kind regards, Patrick Uiterwijk Associate Software Engineer, Red Hat

3 2

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

infrastructure November 2014