firewall rules on builders (iptables, firewalld, libvirt...)
by Matthew Miller
It's my understanding (Dennis please correct if I'm wrong) that the
problem with cloud image creation was due to libvirt iptables rules
being lost when iptables was restarted. This is a fundamental known
issue (see last paragraph of <http://libvirt.org/firewall.html>), and
one of the things firewalld was meant to solve.
Dennis says that there are lot of complicated rules on the builders
making switching to firewalld difficult. One possibility might be to
move those complicated rules from the builders to a network firewall,
and keep the host rules simple and functional. But that's probably a
big undertaking.
In the meantime, any time iptables is restarted or reloaded, libvirt
needs a SIGHUP. (I suppose this means: ansible playbooks and also added
to any manual procedures.)
[cc rel-eng, reply-to infrastructure]
--
Matthew Miller
<mattdm(a)fedoraproject.org>
Fedora Project Leader
8 years, 7 months
Gitolite3 on pkgs01.stg
by Pierre-Yves Chibon
Dear all,
This morning Mathieu (bochecha) and I spent some time trying to get gitolite3
work on our pkgs01.stg.
This is an overview of the situation.
By design gitolite is meant to be installed with a dedicated user. Everyone uses
that user. gitolite then maps the public ssh key to its user database and its
configuration file to handle who is allowed to do what on which git repo.
Our setup is quite different from this. We have a single configuration file at
/etc/gitolite/conf/gitolite.conf that contains all the information about who
is allowed to do what on which git repos. In addition we have a single
/etc/gitolite/gitolite.rc configuration file that specifies to gitolite where
are the git repositories located.
Each user has then an account on the machine but by default no shell access.
The access is restricted via ssh with in ~/.ssh/authorized_keys the instructions:
`command="/usr/bin/gl-auth-command"` (people with shell access have a different
command).
That commands combines the info from /etc/gitolite/gitolite.rc and
/etc/gitolite/conf/gitolite.conf and allow or deny the access/action.
Now gitolite3 has changed quite a bit compared to our setup.
Basically, gitolite3 expects the gitolite.rc file to be at ~/.gitolite.rc and
the repositories to be at ~/repositories. Having symlink to these locations
are fine, but they should be there.
What we can do also is play on the $HOME variable.
In pkgs01.stg we kept the configurations files (gitolite.conf and gitolite.rc)
in /etc/gitolite as before, but we created a couple of symlink in /srv/git/
.gitolite -> /etc/gitolite/
repositories -> /srv/git/rpms/
The idea is to be able to set HOME=/srv/git and run the gitolite command.
This way we managed to get the genacls.sh script working. It is the script
that generates the gitolite.conf file based on the pkgdb information and compile
that configuration file so that gitolite can use it.
gitolite3 also changes the command in the ~/.ssh/authorized_keys file to:
command="/usr/share/gitolite3/gitolite-shell user"
That command is set in the sources in `src/triggers/post-compile/ssh-authkeys`
and put in the file by (I believe) `gitolite compile` that is ran by genacls.sh.
In order to test, I (manually) changed Mathieu's authorized_keys to look like:
command="HOME=/srv/git/ /usr/share/gitolite3/gitolite-shell user"
We then tried to get Mathieu to clone a repo and we found out:
- the ~/.gitolite folder, that gitolite3 requires, needs to have a `logs/`
subfolder, and that folder needs to be accessible read and write to everyone
so `chmod 777 -R /srv/git/.gitolite/logs` (ie: chmod 777 /etc/gitolite/logs
since it's a symlink)
- the /etc/gitolite/conf/gitolite.conf-compiled.pm needs to be readable by
everyone since it's the file gitolite uses to see who can do what on which
repo
so `chmod o+r conf/gitolite.conf-compiled.pm
- Mathieu was finally able to clone and push in a repo he has access to and
could not on the kernel repo (as supposed to since he does not have commit
there), but the error returned is just plain ugly
So to conclude:
- we managed to get it to work
- its needs chmod 777 on /srv/git/.gitolite/logs
- its needs chmod o+r on conf/gitolite.conf-compiled.pm (and that after each
time the file is generated)
- we need to find a way to fix the command in the user's ~/.ssh/authorized_keys
so that they specify the HOME variable to use when running the gitolite
command
-> Might very well require to patch gitolite itself for this (and thus keep
the patch in our package ad vitam aeternam)
That being said, I believe our options are:
1) Talk with upstream, in the past I believe he was quite reactive and willing
to help us. We are the largest public deployment of gitolite maybe he'll
still be willing to help us
to discuss:
- setting HOME in the authorized_keys
- writing logs
- accessing gitolite.conf-compiled.pm
2) Stick with gitolite2
-> We need to find out how easy it is to maintain it on epel7 and we need
someone willing to maintain it (knowing that it's perl)
-> Upstream said gitolite2 is still maintain wrt security fixes but I guess
only for some time
3) Move to the model gitolite relies on (1 specific gitolite user everyone
accesses the repo through).
+ No more need to create account on the machine for each and every packager
+ Inline with gitolite's design
- Breaking all existing git clone around (as the push/pull urls change)
--> URLs change could be handled at the fedpkg level
+ We are already forcing an update of fedpkg to our packager with the change
from md5 to sha for the lookaside cache, so we could couple both change in
the same update and preventing forcing two fedpkg updates.
- Huge number of keys to handle for that user
? Performance impact of this number of users?
? Dennis mentionned logging at the meeting yesterday, but it looks like
gitolite does provide logging itself
4) Move away from gitolite.
I did not check what else would be available/suitable for us, so no ideas
there. Suggestions welcome I guess :)
I think that covers most if not all our findings.
Questions and suggestions welcome :)
Pierre and Mathieu
8 years, 11 months
[release] pkgdb2 1.21
by Pierre-Yves Chibon
Hi all,
I have just released and pushed to staging a new version of pkgdb2, here is the
changelog:
* Fri Nov 21 2014 Pierre-Yves Chibon <pingou(a)pingoured.fr> - 1.21-1
- Update to 1.21
- DB optimization: do not use LIKE in queries where there is no '%'
- Update the layout of the user list page
- Add flag to set/update the monitoring flag on packages. This flag tells
the-new-hotness whether the maintainers are interested in getting bugzilla
tickets about updates available on their package (updates being monitored
via anitya: release-monitoring.org)
- Clean(er) logic to mark the fields mandatory in the forms
- Fix indentation in the mail_logging module
- Branch orphaned packages as well as we branch for new releases (Subho-bcrec)
- Auto select packagers search in packager details page. (Ratnadeep Debnath)
- Obsolete all ACLs on a package that is retired
- Nicer docstring and API documentation by using textwrap.dedent()
- Add a `former_poc` argument to the orphan API endpoint allowing to restrict
the orphan to only the package of a certain packager
- Add the former_poc keyword argument to api_acl_reassign, same principle as
when orphaning a package
- DB optimization: optimize the API endpoint returning the packager's ACLs
- Include the monitoring state in the GET API. (Ralph Bean)
- Include the api_groups and api_monitored in the API documentation
- Update the update_package_info script to include other releases than rawhide
The biggest change being likely the introduction of the monitoring flag to
integrate pkgdb, anitya and the-new-hotness.
Testing welcome :)
Thanks,
Pierre
9 years
Fedora 21 Final Freeze now in effect
by Kevin Fenzi
Greetings.
we are now in the infrastructure freeze leading up to the Fedora 21
Final release. This is a final release freeze.
You can see a list of hosts that do not freeze by checking out the
ansible repo and running the freezelist script:
git clone http://infrastructure.fedoraproject.org/infra/ansible.git
scripts/freezelist -i inventory/inventory
Anything listed as freezes is frozen until 2014-12-09 (or later if the
release slips). Frozen hosts should have no changes made to them without
a signoff on the change from at least 2 sysadmin-main or rel-eng
members.
Thanks,
kevin
9 years
OOM killer on bapp02
by Adrian Reber
The OOM killer on bapp02 has terminated a few mirrormanager crawler
processes. It seems it needs more memory or the number of parallel
crawlers has to be further limited.
Another good idea would be to limit the duration of the rsync crawls in
/etc/mirrormanager/prod.cfg to maybe one day (--timeout=86400) to avoid
stale rsync processes.
Adrian
9 years
freeze break request: delete .tx/config to pull po files automatically
by Robert Mayr
Hi all,
we need to delete the /var/lib/transifex-client/.tx/config file on bapp,
which should execute directly syncTranslations.sh on puppet
(fedora-web/files), if not it gets executed at next cron-job.
Doing so re-creates all transifex resources, included the new getfedora.org
resource, in /var/tmp/po_dir/, needed to pull pos automatically once a day.
The procedure has been already tested on app01.stg and worked as expected.
Actually we are pulling the pos manually, but we'd need to be sure this
automatic process works smooth before releasing F21.
Any +1's or concerns?
Thank you.
--
Robert Mayr
(robyduck)
9 years
Code Search for Fedora
by Michael Stapelberg
Hey,
Recently I’ve been talking to Hannes (cc'ed) about whether Fedora
would be interested in having the equivalent of
http://codesearch.debian.net/¹
The project came to live as my Bachelor of Science Thesis² and aims to
provide fast regular expression search over a big corpus, in this case
140 GB of source code of all software included in the Debian main
distribution (as opposed to non-free or contrib, which we excluded
because of licensing concerns). It is based on the work Russ Cox
published, which in turn resembles the work he did on Google Code
Search when he was an intern there in 2006.
So, what’s this discussion about?
What I’m offering is setting up/running a public version of Code
Search for Fedora. It needs to be public because I want the open
source community as a whole profit from it, and also I’m told you have
somewhat comparable tools internally anyway :).
My motivation comes from multiple places:
1) I’m fairly sure Fedora packages a slightly different set of
software than Debian, so running both DCS (Debian Code Search) and FCS
(Fedora Code Search) would enlarge the amount of searchable software.
2) I’m interested in my work having a positive effect on the world (or
at least the open source community), and running multiple instances of
Code Search reduces its dependency on any single distribution, thereby
increasing its reliability and scope.
3) Last but not least, I intend to try Fedora on one of my computers
to broaden my horizons. I figured getting in contact with some of you
while working on this project may be a good way to set a foot into the
community and see whether I like it around here.
In terms of what I’d need in order to make this project a success,
there are some hardware requirements (aside from, of course, time and
motivation):
The in-memory index and searchable source code can be sharded on an
almost arbitrary number of different computers, which is necessary to
some extent, due to maximum size limitations for the index of a single
shard to be < 2 GB. At the moment, we are running 6 different
index-backend VMs, each serving 1.8G in-memory indexes and about 40G
of source code (including partial indexes). In order to grep through
the source quickly, the source is stored on local SSDs (as opposed to
a network block storage volume, or even regular HDDs).
In addition to the actual data, we also need a web frontend to serve
and combine this data, and we have one more VM which scrapes
monitoring information and shows nice graphs about how the whole
system behaves.
So, in total, we run 8 VMs, of which 6 are equipped with 4 cores, 4G
of RAM (for 2G of index + 2G page cache for grepping files) and 40G
SSD volumes each. The web frontend uses 4 cores and 2G of RAM, and
also an SSD for caching entire query results. The monitoring VM needs
just one core and 2G of RAM.
Does that sound reasonable and feasible? I’m not sure what kind of
hardware you have available for projects like this one, and currently
we’re sponsored by Rackspace because Debian doesn’t have that sort of
hardware easily available.
I feel like this email is long enough already, so I’ll just ask a
general: what do you think? Do you need any more information? Please
just ask, and keep me CC'ed, since I’m not subscribed to this list.
Thanks in advance,
Best regards,
Michael Stapelberg
¹ Note that there is a rather big redesign in progress, both
architecturally and visually:
https://people.debian.org/~stapelberg//2014/11/09/upcoming-debian-codesea...
So, in case you browse around on the current version and conclude that
it sucks, just wait for the update and everything will be awesome ;).
² http://codesearch.debian.net/research/
9 years
Freeze break: fas2.py hotfix
by Kevin Fenzi
We have one user (lsm5) that had a override in python-fedora to map
their bugzilla address to their @fedoraproject.org address. They have
just changed this and want us to remove the override.
We also have currently a locally modified fas2.py file on pkgdb02 with
these overrides, so I'd like to pull it into an ansible hotfix.
The first part of this is adding the hotfix, the second part is the
changes from the stock python-fedora fas2.py file (one override thats
not in the package currently).
+1s?
kevin
--
diff --git a/roles/pkgdb2/tasks/main.yml b/roles/pkgdb2/tasks/main.yml
index 0185c27..7e2c3e0 100644
--- a/roles/pkgdb2/tasks/main.yml
+++ b/roles/pkgdb2/tasks/main.yml
@@ -19,6 +19,12 @@
tags:
- packages
+# HOTFIX: adjust bugzilla overrides
+- name: HOTFIX: adjust bugzilla overrides
+ copy: src=fas2.py dest=/usr/lib/python2.7/site-packages/fedora/client/fas2.py
+ tags:
+ - config
+
- name: copy sundry pkgdb configuration
template: src={{ item.file }}
dest={{ item.location }}/{{ item.dest }}
--- roles/pkgdb2/files/fas2.py.orig 2014-11-30 17:33:54.350767000
+0000 +++ roles/pkgdb2/files/fas2.py 2014-11-30 17:32:37.557767000
+0000 @@ -255,6 +255,8 @@
103551: 'fschwarz(a)fedoraproject.org',
# Martin Holec: martix(a)martix.names
137561: 'mholec(a)redhat.com',
+ # John Dulaney: j_dulaney(a)live.com
+ 149140: 'jdulaney(a)fedoraproject.org',
}
# A few people have an email account that is used in
owners.list but # have setup a bugzilla account for their primary
account system email
9 years
Freeze exception request: Mediawiki upgrade
by Patrick Uiterwijk
Hi all,
Yesterday, MediaWiki 1.19.22 has been release which fixes some security issues.
This is a code-only change, and there's no database or configuration changes.
They did add a new permission flag to change another user's common.js, but as far as I have been able to see, nobody in our instance used that.
This version is now running in stg (https://stg.fedoraproject.org/wiki/Special:Version).
Could I get +1s to upgrade production?
With kind regards,
Patrick Uiterwijk
Associate Software Engineer, Red Hat
9 years
Freeze exception request: nagios contact info
by Patrick Uiterwijk
Hi,
I want to update my contact information for nagios alerts, any +1s to push this?
>From bbf537503698e109c1304b1deba2820753c3e531 Mon Sep 17 00:00:00 2001
From: Patrick Uiterwijk <puiterwijk(a)redhat.com>
Date: Fri, 28 Nov 2014 16:11:32 +0000
Subject: [PATCH] Update puiterwijk nagios contact info
---
.../contactgroups/fedora-sysadmin-email.cfg | 2 +-
.../files/nagios-external/contacts/puiterwijk.cfg | 27 ++++++++++++++++++-
.../nagios/contactgroups/fedora-sysadmin-pager.cfg | 2 +-
.../files/nagios/contacts/puiterwijk.cfg | 27 ++++++++++++++++++-
4 files changed, 52 insertions(+), 6 deletions(-)
diff --git a/roles/nagios_server/files/nagios-external/contactgroups/fedora-sysadmin-email.cfg b/roles/nagios_server/files/nagios-external/contactgroups/fedora-sysadmin-email.cfg
index e099113..d1bc113 100644
--- a/roles/nagios_server/files/nagios-external/contactgroups/fedora-sysadmin-email.cfg
+++ b/roles/nagios_server/files/nagios-external/contactgroups/fedora-sysadmin-email.cfg
@@ -1,5 +1,5 @@
define contactgroup{
contactgroup_name fedora-sysadmin-email
alias Fedora Sysadmin Email Contacts
- members mmcgrath,ausil,admin,nigelj,ricky,jcollie,jmtaylor,jstanley,smooge,nb,rigeld2,codeblock,kevin,hvivani,puiterwijkp
+ members mmcgrath,ausil,admin,nigelj,ricky,jcollie,jmtaylor,jstanley,smooge,nb,rigeld2,codeblock,kevin,hvivani,puiterwijk
}
diff --git a/roles/nagios_server/files/nagios-external/contacts/puiterwijk.cfg b/roles/nagios_server/files/nagios-external/contacts/puiterwijk.cfg
index 4772e4c..910d020 100644
--- a/roles/nagios_server/files/nagios-external/contacts/puiterwijk.cfg
+++ b/roles/nagios_server/files/nagios-external/contacts/puiterwijk.cfg
@@ -1,4 +1,16 @@
define contact{
+ contact_name puiterwijk
+ alias Patrick Uiterwijk
+ service_notification_period 24x7
+ host_notification_period 24x7
+ service_notification_options w,u,c,r
+ host_notification_options d,u,r
+ service_notification_commands notify-by-epager
+ host_notification_commands host-notify-by-epager
+ email patrick(a)puiterwijk.org
+}
+
+define contact{
contact_name puiterwijkp
alias Patrick Uiterwijk
service_notification_period 24x7
@@ -7,6 +19,17 @@ define contact{
host_notification_options d,u,r
service_notification_commands notify-by-epager
host_notification_commands host-notify-by-epager
- email puiterwijk(a)gmail.com
- pager puiterwijk(a)gmail.com
+ email patrick-pager(a)puiterwijk.org
+}
+
+define contact{
+ contact_name puiterwijk-emergency
+ alias Patrick Uiterwijk
+ service_notification_period 24x7
+ host_notification_period 24x7
+ service_notification_options w,u,c,r
+ host_notification_options d,u,r
+ service_notification_commands notify-by-epager
+ host_notification_commands host-notify-by-epager
+ email +31681462748(a)gin.nl
}
diff --git a/roles/nagios_server/files/nagios/contactgroups/fedora-sysadmin-pager.cfg b/roles/nagios_server/files/nagios/contactgroups/fedora-sysadmin-pager.cfg
index 526aa04..1a8d2ba 100644
--- a/roles/nagios_server/files/nagios/contactgroups/fedora-sysadmin-pager.cfg
+++ b/roles/nagios_server/files/nagios/contactgroups/fedora-sysadmin-pager.cfg
@@ -6,5 +6,5 @@ define contactgroup{
define contactgroup{
contactgroup_name fedora-sysadmin-emergency
alias Fedora Sysadmin Pager Contacts
- members mmcgrath-emergency,ricky-emergency,jstanley-emergency,smooge-emergency,kevin-emergency,puiterwijkp
+ members mmcgrath-emergency,ricky-emergency,jstanley-emergency,smooge-emergency,kevin-emergency,puiterwijk-emergency
}
diff --git a/roles/nagios_server/files/nagios/contacts/puiterwijk.cfg b/roles/nagios_server/files/nagios/contacts/puiterwijk.cfg
index 38b8392..910d020 100644
--- a/roles/nagios_server/files/nagios/contacts/puiterwijk.cfg
+++ b/roles/nagios_server/files/nagios/contacts/puiterwijk.cfg
@@ -1,4 +1,16 @@
define contact{
+ contact_name puiterwijk
+ alias Patrick Uiterwijk
+ service_notification_period 24x7
+ host_notification_period 24x7
+ service_notification_options w,u,c,r
+ host_notification_options d,u,r
+ service_notification_commands notify-by-epager
+ host_notification_commands host-notify-by-epager
+ email patrick(a)puiterwijk.org
+}
+
+define contact{
contact_name puiterwijkp
alias Patrick Uiterwijk
service_notification_period 24x7
@@ -7,6 +19,17 @@ define contact{
host_notification_options d,u,r
service_notification_commands notify-by-epager
host_notification_commands host-notify-by-epager
- email puiterwijk(a)gmail.com
- pager puiterwijk(a)gmail.com
+ email patrick-pager(a)puiterwijk.org
+}
+
+define contact{
+ contact_name puiterwijk-emergency
+ alias Patrick Uiterwijk
+ service_notification_period 24x7
+ host_notification_period 24x7
+ service_notification_options w,u,c,r
+ host_notification_options d,u,r
+ service_notification_commands notify-by-epager
+ host_notification_commands host-notify-by-epager
+ email +31681462748(a)gin.nl
}
--
1.7.2.1
With kind regards,
Patrick Uiterwijk
Associate Software Engineer, Red Hat
9 years
