firewall rules on builders (iptables, firewalld, libvirt...)
by Matthew Miller
It's my understanding (Dennis please correct if I'm wrong) that the
problem with cloud image creation was due to libvirt iptables rules
being lost when iptables was restarted. This is a fundamental known
issue (see last paragraph of <http://libvirt.org/firewall.html>), and
one of the things firewalld was meant to solve.
Dennis says that there are lot of complicated rules on the builders
making switching to firewalld difficult. One possibility might be to
move those complicated rules from the builders to a network firewall,
and keep the host rules simple and functional. But that's probably a
big undertaking.
In the meantime, any time iptables is restarted or reloaded, libvirt
needs a SIGHUP. (I suppose this means: ansible playbooks and also added
to any manual procedures.)
[cc rel-eng, reply-to infrastructure]
--
Matthew Miller
<mattdm(a)fedoraproject.org>
Fedora Project Leader
8 years, 5 months
Fedora 22 Alpha Freeze now in effect
by Kevin Fenzi
Greetings.
we are now in the infrastructure freeze leading up to the Fedora 22
Alpha release. This is a pre-release freeze.
We do this to ensure that our infrastructure is stable and ready to
release the Fedora 22 Alpha when it's available.
You can see a list of hosts that do not freeze by checking out the
ansible repo and running the freezelist script:
git clone https://infrastructure.fedoraproject.org/infra/ansible.git
scripts/freezelist -i inventory
Any hosts listed as freezes is frozen until 2015-03-10. (or later if
Alpha slips). Frozen hosts should have no changes made to them
without a sign-off on the change from at least 2 sysadmin-main or
rel-eng members, along with (in most cases) a patch of the exact
change to be made to this list.
Thanks,
kevin
8 years, 6 months
New OpenStack instance - status
by Miroslav Suchý
Since I'm leaving for one week vacation, I think I may write down current status of our new OpenStack instance and write
down TODO list. Just in case someone is desperate enough to do some fixes.
I updated docs.git/cloud.txt - mainly which playbooks we use right now and where to write down IP, when you add new
compute node.
Controller - should be OK. At least I see no problems there right now. Network is stable. I can log to EqualLogic
(credentials are at bottom of cinder.conf). Volumes are created correctly. I can reach compute nodes. AMQP works and is
reachable from Compute nodes (do not try to play with SSL&RabbitMQ it will never work on RHEL7). Horizon works (over https).
Compute nodes - it looks good until you try to start VM. :) I fixed several problems, but new ones still pop ups.
If you want to debug it, just go to dashboard and start new VM (note that m1.tiny is too small for Fedora image) and on
controller do:
tail -f /var/log/nova/nova-scheduler.log
And look for something like:
Choosing host WeighedHost [host: fed-cloud13.cloud.fedoraproject.org, weight: 1.0] for instance
75f1b5ca-88d5-4e57-8c18-8d6554e1f2bc
then log to that instance (right now root@fed-cloud09 can ssh directly as root@fed-cloudXX) and
tail -f /var/log/nova/nova-compute.log /var/log/neutron/openvswitch-agent.log
When spin up of VM fail, then controller try 2 next machines before giving up.
Right now there is some error:
TypeError: unsupported operand type(s) for +: 'NoneType' and 'str'\n"
which is new to me and which I will not manage to fix before I will leave today.
It may be last one problem or they may be dozen other still waiting in queue. It's hard to tell.
Smaller fixes to do:
* playbook hosts/fed-cloud09.cloud.fedoraproject.org.yml can be enhanced that after packstack execution the machine
should be restarted. Right now I am waiting for first error after packstack and then I restart the machine manualy and
re-run playbook again. This is last manual workaround. Everything else was already automated.
* routing between compute nodes and controller using public IP does not work. Not fatal right now, but nice to have.
--
Miroslav Suchy, RHCE, RHCDS
Red Hat, Senior Software Engineer, #brno, #devexp, #fedora-buildsys
8 years, 6 months
Freeze break request: Make setup_git_package also handle recreating the repo
by Patrick マルタインアンドレアス Uiterwijk
>From 6e98a2dc667365633e47c63730feefc8d752886c Mon Sep 17 00:00:00 2001
From: Patrick Uiterwijk <puiterwijk(a)redhat.com>
Date: Fri, 27 Feb 2015 23:39:59 +0000
Subject: [PATCH] Fix setup_git_package to handle setting up after gitolite
setup a broken repo
setup_git_package did the same check as pkgdb_sync did [1].
Also, if gitolite already came by, it will have created an update hook,
which blocks us from pushing the initial commit (since gitolite has no idea
who we are).
So in that case, we will just remove the update hook: gitolite will
recreate it when it comes by the repo with genacls.sh.
The change to mkdir -p is just syntax to make it not print an error in case
the directory already existed.
[1]:
https://lists.fedoraproject.org/pipermail/infrastructure/2015-February/01...
Signed-off-by: Patrick Uiterwijk <puiterwijk(a)redhat.com>
---
roles/distgit/files/setup_git_package | 10 ++++++++--
1 files changed, 8 insertions(+), 2 deletions(-)
diff --git a/roles/distgit/files/setup_git_package
b/roles/distgit/files/setup_git_package
index b22b31f..67103ec 100644
--- a/roles/distgit/files/setup_git_package
+++ b/roles/distgit/files/setup_git_package
@@ -68,11 +68,17 @@ fi
# Sanity checks before we start doing damage
[ $VERBOSE -gt 1 ] && echo "Checking package $PACKAGE..."
-if [ -d $GITROOT/$PACKAGE.git ] ; then
+if [ -f $GITROOT/$PACKAGE.git/refs/heads/master ] ; then
echo "ERROR: Package module $PACKAGE already exists!" >&2
exit -1
fi
+# A cleanup in case gitolite came by this repo
+if [ -f $GITROOT/$PACKAGE.git/hooks/update ] ; then
+ echo "Gitolite already initialized this repo. Will remove its hooks"
+ rm -f $GITROOT/$PACKAGE.git/hooks/update
+fi
+
# "global" permissions check
if [ ! -w $GITROOT ] ; then
echo "ERROR: You can not write to $GITROOT"
@@ -85,7 +91,7 @@ fi
TMPDIR=$(mktemp -d /tmp/tmpXXXXXX)
# First create the master repo
-mkdir $GITROOT/$PACKAGE.git
+mkdir -p $GITROOT/$PACKAGE.git
pushd $GITROOT/$PACKAGE.git >/dev/null
git init -q --shared --bare
echo "$PACKAGE" > description # This is used to figure out who to send
mail to.
--
1.7.2.1
8 years, 7 months
Freeze break request: fixing race condition breakage for pkgdb_sync
by Patrick Uiterwijk
Any +1s?
Explanation in the commit message.
Note: we should work on improving the script in general, but that is not something we should
consider pushing during freeze, so this is a patch that hopefully gets us through the freeze.
commit 2d98b6748663747698cd6a0bfc23d05bbc371a01
Author: Patrick Uiterwijk <puiterwijk(a)redhat.com>
Date: Fri Feb 27 08:44:57 2015 +0000
Make pkgdb_sync more resilient by always calling SETUP_PACKAGE if master does not exist
This should make it recover in case there are any repos in a weird state (aka, no master branch).
This could for example happen if the package was created in pkgdb after pkgdb_sync ran, but before
genacls.pkgdb runs, because then gitolite will see it as a new repo, and create a totally blank repo.
Especially since pkgdb_sync will abort after the first error, it would then leave all later created
repos broken as well.
Calling setup_git_package in a repo without master branch has no ill effect:
- Running git init on a pre-initialized repo is a no-op (or at least doesn't destroy current git data)
- Since it doesn't have master, it didn't run the script, so adding the first commit and hooks etc are
what we want the script to do
Signed-off-by: Patrick Uiterwijk <puiterwijk(a)redhat.com>
diff --git a/roles/distgit/templates/pkgdb_sync_git_branches.py b/roles/distgit/templates/pkgdb_sync_git_branches.py
index ced139b..8643165 100644
--- a/roles/distgit/templates/pkgdb_sync_git_branches.py
+++ b/roles/distgit/templates/pkgdb_sync_git_branches.py
@@ -187,7 +187,7 @@ def branch_package(pkgname, branches):
# Create the devel branch if necessary
if not os.path.exists(
- os.path.join(GIT_FOLDER, '%s.git' % pkgname)):
+ os.path.join(GIT_FOLDER, '%s.git/refs/heads/master' % pkgname)):
_invoke(SETUP_PACKAGE, [pkgname])
if 'master' in branches:
branches.remove('master') # SETUP_PACKAGE creates master
8 years, 7 months
retroactive freeze break request: kojipkgs01
by Kevin Fenzi
We have been having some persistent issues with kojipkgs01 lately.
kojipkgs01 is our squid proxy in front of koji builds. It allows users
and builders to get fast access to packages. (When it's working).
Lately, it's been working fine at first, then in a few days or so it
starts getting really slow. Downloads go from 25M/s to 200k/sec and
sometimes things even just timeout.
Restarting squid seems to fix this... for a few more days.
There is never any errors on the box, i/o, load and everything is fine.
I looked this morning a bunch at options and adjusted the memory cache
down in case we were hitting some kind of issue with memory cache.
I'd like +1's for that change, and also to solicit ideas for what we
can do to fix this once and for all (if these changes don't do so).
diff --git a/roles/kojipkgs/files/squid.conf b/roles/kojipkgs/files/squid.conf
index b011143..a0d5312 100644
--- a/roles/kojipkgs/files/squid.conf
+++ b/roles/kojipkgs/files/squid.conf
@@ -6,8 +6,8 @@ hierarchy_stoplist cgi-bin ?
cache_swap_low 98
cache_swap_high 99
-cache_mem 50 GB
-maximum_object_size 700 MB
+cache_mem 10 GB
+maximum_object_size 200 MB
minimum_object_size 0 KB
cache_replacement_policy heap LFUDA
maximum_object_size_in_memory 100 MB
8 years, 7 months
Next weeks meeting - shared document info
by Kevin Fenzi
Greetings.
In the meeting today I talked about restructuring how we do our
meetings some. The goal would be to make the meetings more about things
we want to discuss than gathering status and such, as well as letting
us remember better what we wanted to discuss or note.
So, I have made a document in our gobby instance:
https://fedoraproject.org/wiki/Gobby
Called fedora-infrastructure-meeting-next
If you don't have access to fedorapeople to get the super sekret
password, just ping someone who does. :)
Basically over the next week if people could login and add things to
that document as they go we can use it for the next weeks meeting.
I've attached the current contents here, and when I send out the
meeting reminder next wed, I will copy out the document then too.
Hopefully it makes sense and helps us out. ;)
ideas and changes welcome.
kevin
--
= Introduction =
This shared document is for the next fedora infrastructure meeting.
We will use it over the week before the meeting to gather status and info and
discussion items and so forth, then use it in the irc meeting to transfer
information to the meetbot logs.
= Meeting start stuff =
#startmeeting Infrastructure (2015-03-05)
#meetingname infrastructure
#topic aloha
#chair smooge relrod nirik abadger1999 lmacken dgilmore mdomsch threebean pingou puiterwijk
#topic New folks introductions / Apprentice feedback
= Status / information / Trivia / Announcements =
(We put things here we want others on the team to know, but don't need to discuss)
(Please use #info <the thing> - your name)
#info Trying new meeting process this week with shared document - kevin
= Things we should discuss =
1. kevin - new meeting process
#topic New meeting process
2.
= Learn about some application or setup in infrastructure =
(This section, each week we get 1 person to talk about an application or setup
that we have. Just going over what it is, how to contribute, ideas for improvement,
etc. Whoever would like to do this, just add the info in this section)
#topic
= Action Items / todos =
(Prefer probibly to make a ticket or track these outside the meeting process)
1.
= Meeting end stuff =
#topic Open Floor
#endmeeting
8 years, 7 months
Plan for tomorrow's Fedora Infrastructure meeting (2015-02-26)
by Kevin Fenzi
The infrastructure team will be having it's weekly meeting tomorrow,
2015-02-26 at 18:00 UTC in #fedora-meeting on the freenode network.
Suggested topics:
#topic New folks introductions and Apprentice tasks.
If any new folks want to give a quick one line bio or any apprentices
would like to ask general questions, they can do so in this part of the
meeting. Don't be shy!
#topic Applications status / discussion
Check in on status of our applications: pkgdb, fas, bodhi, koji,
community, voting, tagger, packager, dpsearch, etc.
If there's new releases, bugs we need to work around or things to note.
#topic Sysadmin status / discussion
Here we talk about sysadmin related happenings from the previous week,
or things that are upcoming.
#topic nagios/alerts recap
Here we go over the last weeks alerts and see if we can find ways to
make it so they don't happen again.
http://ur1.ca/jr7j4
#topic Upcoming Tasks/Items
https://apps.fedoraproject.org/calendar/list/infrastructure/
#topic Open Floor
Submit your agenda items, as tickets in the trac instance and send a
note replying to this thread.
More info here:
https://fedoraproject.org/wiki/Infrastructure/Meetings#Meetings
Thanks
kevin
8 years, 7 months
Fwd: To Whomever: You Made My Day
by Stephen John Smoogen
The little things hidden in plain view make the best of a bad day. Thank
you to the people involved.
---------- Forwarded message ----------
From: John Florian <john.florian(a)dart.biz>
Date: 26 February 2015 at 06:31
Subject: To Whomever: You Made My Day
To: devel <devel(a)lists.fedoraproject.org>
Yesterday I was browsing the Fedora pkgdb/git/bohdi pages and this morning
I returned to go backwards thru my web browser history when I stumbled upon
a real hidden gem for a HTTP 500 response. Our hot dog armed with a ray
gun against a nuclear panda… oh man that was great. Best 500 I’ve seen yet.
--
John Florian
--
devel mailing list
devel(a)lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel
Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct
--
Stephen J Smoogen.
8 years, 7 months
[release] anitya 0.2.0
by Pierre-Yves Chibon
Hi all,
Since anitya is not frozen and I had a bunch of changes merged, I just cut a new
release and push it to prod.
Here is the changelog:
* Thu Feb 26 2015 Pierre-Yves Chibon <pingou(a)pingoured.fr> - 0.2.0
- Fix doc typos (reported by tibbs, fixed by Ralph Bean)
- Fix typo when unable to retrieve the latest version (Thomas Spura)
- Autocomplete distribution name when mapping a project
- Dynamically check existing projects while adding a new one
- Check now button available after an edit
- More information for the custom backend
- Fix the drupal backends to make them suppor project named 'drupal6: foo'
or 'drupal7: bar'
- Fix pagination on quite some pages
- Allow users to browse their own log
- On postgresql '_' is a special character that we should escape
- User friendly url for example: /project/guake
- More info when mapping a project fails (including a link to the project
causing the failure to map)
Enjoy!
Pierre
8 years, 7 months
