infrastructure April 2015

infrastructure@lists.fedoraproject.org

46 participants
79 discussions

Cantas open source card app?
by Paul W. Frields 21 Apr '15

21 Apr '15

I separately had both Adam Miller and Stephen Smoogen ask me about "card" based project tracking applications. I responded to both that there's a 100% open source app for that called Cantas: https://github.com/onepiecejs/nodejs-cantas I believe the major portion of its dependencies are all in Fedora already, probably thanks in part to Ralph Bean sorting through a ton of nodejs stuff earlier. I've seen numerous folks using apps like this, or Trello, or other similar, to get development done faster and with good accountability. Sometimes they're using Agile with a big A, sometimes they're just trying to be more agile with a little a. I think Adam's intent was to try to use this kind of system for upcoming projects that involve Fedora rel-eng and infrastructure. If Fedora data like that is in it, I'd love to see that eventually be tracked in Fedora itself. While it doesn't have to be a blocker to moving forward, it could end up being useful to more people. Is this of interest to anyone else in Fedora Infrastructure? -- Paul W. Frields http://paul.frields.org/ gpg fingerprint: 3DA6 A0AC 6D58 FEC4 0233 5906 ACDB C937 BD11 3717 http://redhat.com/ - - - - http://pfrields.fedorapeople.org/ The open source story continues to grow: http://opensource.com

1 0

Upstream for dist-git [RFC]
by Miroslav Suchý 18 Apr '15

18 Apr '15

Hi, Adam Šamalík took dist-git files from fedora-infra ansible.git. He separated what belongs to dist-git itself and what is Fedora specific and with cooperation of Dan Mach and Palo Babinčák he created upstream for dist-git: https://github.com/release-engineering/dist-git This is first attempt and request for comments. The changes from ansible.git version are described here: https://github.com/release-engineering/dist-git/blob/master/changes.txt and he extracted some code to be configuration driven: https://github.com/release-engineering/dist-git/blob/master/configs/dist-gi… Feel free to experiment with this project and we are looking for your questions and comments. I have one question thou: There is no license information in files header but two files: scripts/httpd/upload.cgi - GPLv1 scripts/dist-git/pkgdb_sync_git_branches.py - GPLv2+ Everything else is without license. Can I assume that we can realease the code under GPLv2+? The author of upload.cgi seems to be Kevin F. - Kevin, are you willing to change license your file to GPLv2+ so we have uniform license across all files? Future plans: 1) Listen to your initial feedback and do alternations according to your feedback 2) After license clarification, announce this project to Red Hat and CentOS rel-engs and ask them to merge their changes of dist-git to this upstream. 3) Get this package into Fedora distribution 4) Change Fedora dist-git server to use this package. .... 10) Enjoy the benefits of common upstream. -- Miroslav Suchy, RHCA Red Hat, Senior Software Engineer, #brno, #devexp, #fedora-buildsys

4 4

Freeze Break: increase bodhi timeout on releng04/relepel01
by Kevin Fenzi 17 Apr '15

17 Apr '15

I'm really not sure why they have become so slow, but releng04 and relepel01 are now timing out when trying to push updates. :( Until Luke is back and able to take a closer look, I just upped the timeouts on both of them from 300 to 900 seconds so we could get updates out. I can put this in as a hotfix, or we can see if Luke can figure out the slowdown first. Anyhow, retroactive +1s? kevin -- --- /usr/bin/bodhi 2015-04-17 14:18:24.611000665 +0000 +++ /home/fedora/kevin/bodhi.orig 2015-04-17 15:10:16.917000664 +0000 @@ -192,7 +192,7 @@ setup_logger(opts.verbose) bodhi = BodhiClient(opts.bodhi_url, username=opts.username, debug=opts.verbose) - bodhi.timeout = 900 + bodhi.timeout = 300 def verify_args(args): if not args and len(args) != 1:

3 2

Fedora Cloud questions and proposal
by Miroslav Suchý 17 Apr '15

17 Apr '15

I have few questions regarding how we plan to use our new Fedora Cloud: 1) Do we want to use Swift there? How much space we should allocate for Swift storage? 2) How should I configure /var/lib/nova/instances? This is directory where RootFS, swap and ephemeral storages of instances are stored. Right now fed-cloud09 have this path on root mount point where is 15 GB available. But it can be extended to whole size of "vg_server" volume group, which is 383GB. fed-cloud1X have now: /dev/mapper/vg_guests-nova 3.2T 32G 3.0T 2% /var/lib/nova This gives us accumulated (over controller and 6 compute nodes) storage of 18.4 TB storage. However this setup does not allows live migration, because /var/lib/nova is not on shared FS. So I propose that /var/lib/nova/instances is NFS exported from fed-cloud09 to all compute nodes. This will allow live migrations of VMs. However this means that /var/lib/nova/instances on fed-cloud09 should be big enough to host all rootfs, swaps and ephemeral devices for all VMs in Fedora Cloud. This is 3-20 GB per instance. so 383 GB is not enough. Other disadvantage is that ephemeral devices will be on network storage and not so fast as local device. However I suppose we do not use ephemeral storage too much (maybe not at all). I would use /dev/md127 from fed-cloud09 (2.89 TB) for /var/lib/nova. This should be enough for approximately 287 instances. Which is cca accumulated capacity of our compute nodes. However it may slightly limit us in future if we ever add more compute nodes. I would suggest to use /dev/md127 (2.89TB raid6 storage) on fed-cloud1X(s) as storage for Swift. And configure Glance to store images in Swift. This have advantage that Swift can be set up to store more replicas. On the other hand this is little bit overkill. As we would have 3TB+ (it depend how much replicas we will use) storage for Swift. However biggest consumer would be Glance. And Glance in fed-cloud02 only consume 100GB currently. What you like most, hove more space for /var/lib/nova as in current setup but without live migrations? Or smaller storage, which on the other hand allows live migration? Or you have better idea how to split disks and machines to different services? P.S. I am putting aside normal volumes for data, which will reside in DellEqualogic which is 20TB. And it is perfectly fine. -- Miroslav Suchy, RHCE, RHCDS Red Hat, Senior Software Engineer, #brno, #devexp, #fedora-buildsys

2 5

Retrospective freeze break: Block riddler.io
by Patrick Uiterwijk 17 Apr '15

17 Apr '15

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA512 Hi, Earlier today, the riddler.io bot pulled Bodhi under by performing a lot of heavy requests. I know from experience that this bot does NOT listen to robots.txt, even though the website says it does (it hit my personal webserver as well at an earlier time), so I have currently blocked it by user agent. We probably need to fix this better soon, but for the time being I put this in to make Bodhi behave again. Any comments? 1 files changed, 3 insertions(+), 1 deletions(-) [puiterwijk@lockbox01 files]$ git show HEAD commit a635e91849f29dd3c146b8067f602871ff959f00 Author: Patrick Uiterwijk <puiterwijk(a)redhat.com> Date: Fri Apr 17 11:29:48 2015 +0000 Block the Riddler.io bot from accessing bodhi This bot does NOT follow robots.txt, even though it announces that it does, and it hits Bodhi so much that it pulls the bodhi servers under. An email has been sent to the maintainer of the bot, but for the time being, let's block it from using any bodhi resources. Signed-off-by: Patrick Uiterwijk <puiterwijk(a)redhat.com> diff --git a/roles/bodhi/base/files/bodhi-app.conf b/roles/bodhi/base/files/bodhi-app.conf index 3e10a59..3f44964 100644 - --- a/roles/bodhi/base/files/bodhi-app.conf +++ b/roles/bodhi/base/files/bodhi-app.conf @@ -13,6 +13,8 @@ Alias /updates/tg_widgets/tgmochikit/packed/MochiKit/MochiKit.js /usr/lib/python <Directory /usr/share/bodhi> WSGIProcessGroup bodhi - - Order deny,allow + SetEnvIf User-Agent Riddler GoAway=1 + Order allow,deny Allow from all + Deny from env=GoAway </Directory> -----BEGIN PGP SIGNATURE----- Version: GnuPG v1 iQIcBAEBCgAGBQJVMPBRAAoJEIZXmA2atR5QBUAP/3+GOLJKRJ3hPB+GBBnx9Dg/ KDEdK6gCtOZUP0sYOPhEVmwTSHXjz/Z51Gvj9EO8FIFQFtnivA80tS2hiSp4x+gR K35z6cxIGbtlG7INLIBoyqTOD6bzvRe0mUP2cKL0+bgFxTzwbZx9HUrQd3PGwA4N 2W2SXROoVv2FVB4rpHTpQouY2VYdcyCDBsibjs2Y37Cxah/ltQC2FHaGlEMJSVCx DAQMAkxdGE0ww2AzRXysHDs46mHtpVFIOHgvtjd6A4NfMuB3gEGaJbcOj28AGmlt RIGU/arVp3OaNFwjAHyrHRCP56i68c7X/N0NwA/hWXLgOm3AJ4AbFoPPNSC0IwYo DVXvJBwRZp5hiuBED0T6/A6j2gVE52n1ytIcAmMznnw7qiupNxKLLpwow9T0RbYi 468aG97vvrFhr7Zj23i/yYsTXJvaMygHlG+nH82k/pdd93tXc5XVLL6MxjKAO53v gaHuvPcHhQuqXus/vv6nNXf/fRhtH3DlnTQRxMz5GZlXVdE9UCaROKNFcISgkCPR 62RYEBG4ZPPM6T9/Ps7CUtSVaCJnfZe1CSUhDlyA3JeCLizmLlAwUJXh3ysS1ZwX iGGkNjj/EdoJZdXz2941LurG2Q03o7cbcqGpZofJvRFDGdFhRdD5ePtqq1+/E0uw A0lPKspP13sayVSaqdBQ =QgK+ -----END PGP SIGNATURE-----

3 3

db-koji01 slowness
by Kevin Fenzi 17 Apr '15

17 Apr '15

So, last weekend we rebooted bvirthost09 and db-koji01. It helped somewhat. Database dumps are back to a reasonable few hours. However, it's still got high load and occasionally alerts and also now it's sometimes causing builders to stop talking to the hub. (They timeout and just stop checking in). I've asked netapp folks to look and see if they can see any problems with the iscsi lun that guest is on, but they say they are not aware of any issues. I do see some packet dropping on db-koji01: 2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP mode DEFAULT qlen 1000 link/ether 52:54:00:06:90:a4 brd ff:ff:ff:ff:ff:ff RX: bytes packets errors dropped overrun mcast 65269562146 369724151 0 128685 0 0 TX: bytes packets errors dropped carrier collsns 395224051163 377221287 0 0 0 0 My only ideas at this point: a) run another postgresql vacuum analyze. Perhaps the first one made some poor choices and another one would make things happier. In any case that shouldn't make things any worse. b) Switch the network card on db-koji01 to e1000 instead of virtio-net. This really shouldn't be needed, but perhaps we are hitting some weird virtio-net bug. This would require a short outage. c) Some other brilliant idea. ;) kevin

4 7

Updating puppet.txt with Ansible workflow
by mhurron 17 Apr '15

17 Apr '15

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 http://paste.fedoraproject.org/211971/ The working document is above, at least to let it be looked over. Any feedback is appreciated. I'll set up a gobby session for it later so edits and additions can be made to the document instead of emailing changes back and forth. -----BEGIN PGP SIGNATURE----- Version: GnuPG v1 iQIcBAEBAgAGBQJVL+AJAAoJEMmBfkn7a3w1l4IP/1vHsPUjXeDEUJy1FkzeLfoM PlxniJYDpwZ8/f/QZ1BHPKLd1+k/X5wL15mWhfaJzgekm7gG1DXK7dE4XeMKh0l0 mnMCrGPKuDpcEi93sUKw01OlzRJiRGTIbqHn8BptPChB5ba4/sk+lhpKCj7c7hEL K77kjgRk1LZg9e3PZm5bgQiuIAdXdP02SJ/L4x44BBNQ5TLwPGDCu4jjIAtCVjyo xiZJ9W9GUusxotIvF/1PqQKjRkHU2yVUs/PWmwzqFdNZMPN2/epT+XsxdB1ijv1t shlg+8Re+L6odM4tEDw1eqjkTsQNgeuODHvyajpwIceVbbFo+sG8Y66EQ09r987C k/PqeKyLVWjoUtFZoWI10Mox8aM68i2pztSvDf0rL2dcerfL/G1KV7g2jrygUkQ1 kzNFpPo2tmdiWoc0ilrF2a+urVsrUK/HDsyn22JQKURjO+vGJET8xS1WJPqXyL2Y O2GsX9FCPCr5/XBiRUKw5lj6vr6zFaq+xsBlEF6onGgyp5Nyh8KmrBpjUI8cFO5G JJZ8zPpXAOa49duhXOfIK9D2pwitpBDp6sFiYpgifZDGxjMR84CbQaGAncuIowZ9 +V8o6EkdvXeFCWa9ce7huV/2Rz8YX1JJYwd4vrnjQ9VJCkTXdYvafkBnRQPIyw25 IGRIzPuJqyb7roVLhxnd =1Ufs -----END PGP SIGNATURE-----

2 1

Cleanup of directories on lockbox01
by Stephen John Smoogen 16 Apr '15

16 Apr '15

We nearly ran out of disk space on lockbox01 today so with Kevin's approval, I removed the following directories from all homes ~/*/dns/ This directory is backed up so if we lost something you were working on we should be able to get yesterdays version of it. The DNS git repository is growing exponentially due to dnssec keys causing every file to change largely. So at the moment the workflow to work on dns should be: 1) git clone /git/dns 2) cd ./dns 3) make the fixes and changes you need. 4) git add . 5) ./do-domains 6) git add . 7) git commit -a -m 'I made the following changes' 8) git push 9) make sure changes worked. 10) cd .. ; rm -rf ./dns -- Stephen J Smoogen.

3 2

Plan for tomorrow's Fedora Infrastructure meeting (2015-04-16)
by Kevin Fenzi 16 Apr '15

16 Apr '15

The infrastructure team will be having it's weekly meeting tomorrow, 2015-04-16 at 18:00 UTC in #fedora-meeting on the freenode network. This week we are continuing to try something new. We have a gobby document (see: https://fedoraproject.org/wiki/Gobby ) fedora-infrastructure-meeting-next is the document. Please try and review and edit that document before the meeting and we will use it to have our agenda of things to discuss. A copy as of this morning is included in this email. kevin -- = Introduction = This shared document is for the next fedora infrastructure meeting. We will use it over the week before the meeting to gather status and info and discussion items and so forth, then use it in the irc meeting to transfer information to the meetbot logs. = Meeting start stuff = #startmeeting Infrastructure (2015-04-16) #meetingname infrastructure #topic aloha #chair smooge relrod nirik abadger1999 lmacken dgilmore mdomsch threebean pingou puiterwijk #topic New folks introductions / Apprentice feedback = Status / information / Trivia / Announcements = (We put things here we want others on the team to know, but don't need to discuss) (Please use #info <the thing> - your name) #topic announcements and information #info rebooted db-koji01 to help performance issues - kevin and smooge #info Lots more various things to get performance on db-koji01 happy - kevin #info Inactive apprentices removed for the month - kevin #info Network outage in phx2 tuesday night, brought everything back - kevin and smooge #info Lots of folks out at pycon this week #info = Things we should discuss = We use this section to bring up discussion topics. Things we want to talk about as a group and come up with some consensus or decision or just brainstorm a problem or issue. If there are none of these we skip this section. (Use #topic your discussion topic - your username) #topic = Learn about some application or setup in infrastructure = (This section, each week we get 1 person to talk about an application or setup that we have. Just going over what it is, how to contribute, ideas for improvement, etc. Whoever would like to do this, just add the info in this section. In the event we don't find someone to teach about something, we skip this section and just end the meeting.) #topic Learn about fpaste - our pastebin application = Meeting end stuff = #topic Open Floor #endmeeting

1 0

Infra docs formatting
by Pete Travis 16 Apr '15

16 Apr '15

Hi there, I'm working on a project[0] to build a documentation website using buildbot. We have plenty of publican books for me to play with, but I'd also like to enable other formats. ReStructuredText[1] seems like a good first choice, as it's easy to learn and use (just squint and pretend it's markdown, if you aren't familiar), and there are processors for it at hand. Anerist is centered around consuming git repositories. Ultimately, I'd like the site to offer the both user and contributor facing documentation. I also need a repo full of RST content to bang out implementation details. infra-docs.git seems ideally positioned to help out here, and any time invested would carry forward to the end product. Each article would need some kind of header[2] to work at scale, though. So, what does the infrastructure team think about writing in RST? If I'm willing to convert everything over, are you willing to sustain it? --Pete [0] slowly, anyway... https://github.com/immanetize/anerist [1] http://docutils.sourceforge.net/docs/ref/rst/restructuredtext.html [1b] more digestable: http://www.getnikola.com/quickstart.html [2] Probably something like https://lists.fedoraproject.org/pipermail/docs/2015-March/016101.html, discussion encouraged.

3 3

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

infrastructure April 2015