12:45 p.m.
============================================
#fedora-meeting: Infrastructure (2014-11-20)
============================================
Meeting started by nirik at 18:00:01 UTC. The full logs are available at
http://meetbot.fedoraproject.org/fedora-meeting/2014-11-20/infrastructure...
.
Meeting summary
---------------
* aloha (nirik, 18:00:01)
* New folks introductions and Apprentice tasks (nirik, 18:01:42)
* Freeze reminder (nirik, 18:03:32)
* Applications status / discussion (nirik, 18:04:45)
* elections update possibly on the 26th. (nirik, 18:09:27)
* fedoauth/ipsilon merge is done. Staging soon. (nirik, 18:09:51)
* the-new-hotness (filing bugs on updated package versions) is moving
along. (nirik, 18:11:21)
* New folks introductions and Apprentice tasks (part 2) (nirik,
18:14:13)
* LINK: https://fedoraproject.org/wiki/Infrastructure_Apprentice
(nirik, 18:18:05)
* LINK:
https://fedoraproject.org/easyfix/https://fedoraproject.org/easyfix/
(nirik, 18:18:16)
* Sysadmin status / discussion (nirik, 18:20:11)
* bodhost01 up after failed disk (nirik, 18:20:47)
* host1plus01 site up and running (it has a mirrorlist server there)
(nirik, 18:21:18)
* dedicated-solutions site in process of being added/setup. (nirik,
18:21:38)
* memcached failures recently. Bug filed, setup to autorestart,
collectd added, monitoring added. (nirik, 18:22:42)
* some issues installing new cloud. Hopefully progress next week.
(nirik, 18:23:59)
* nagios/alerts recap (nirik, 18:25:35)
* Upcoming Tasks/Items (nirik, 18:29:23)
* LINK: https://apps.fedoraproject.org/calendar/list/infrastructure/
(nirik, 18:29:23)
* Open Floor (nirik, 18:37:42)
Meeting ended at 18:44:18 UTC.
Action Items
------------
Action Items, by person
-----------------------
* **UNASSIGNED**
* (none)
People Present (lines said)
---------------------------
* nirik (87)
* pingou (28)
* threebean (20)
* puiterwijk (12)
* zodbot (7)
* oddshocks (7)
* tflink (6)
* sayhar (5)
* alexove (4)
* danielbruno (4)
* danofsatx-work (3)
* lmacken (2)
* mirek-hm (2)
* relrod (1)
* janeznemanic (1)
* abadger1999 (0)
* smooge (0)
* mdomsch (0)
* dgilmore (0)
--
18:00:01 <nirik> #startmeeting Infrastructure (2014-11-20)
18:00:01 <zodbot> Meeting started Thu Nov 20 18:00:01 2014 UTC. The chair is nirik.
Information about MeetBot at http://wiki.debian.org/MeetBot.
18:00:01 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
18:00:01 <nirik> #meetingname infrastructure
18:00:01 <zodbot> The meeting name has been set to 'infrastructure'
18:00:01 <nirik> #topic aloha
18:00:01 <nirik> #chair smooge relrod nirik abadger1999 lmacken dgilmore mdomsch
threebean pingou puiterwijk
18:00:01 <zodbot> Current chairs: abadger1999 dgilmore lmacken mdomsch nirik pingou
puiterwijk relrod smooge threebean
18:00:08 <relrod> here
18:00:12 * pingou here
18:00:13 * puiterwijk here
18:00:15 * threebean here
18:00:17 <janeznemanic> hi
18:00:21 * danielbruno here
18:01:42 <nirik> #topic New folks introductions and Apprentice tasks
18:01:53 <nirik> any new folks around who would like to introduce themselves?
18:02:00 <nirik> or apprentices with questions or comments?
18:03:26 <nirik> ok then... moving along. ;)
18:03:32 <nirik> #topic Freeze reminder
18:03:40 <nirik> Just a reminder that we are in F21 final freeze.
18:03:40 <danielbruno> after the work that I did on my first ticket on easyfix, I
should look for another ticket there, or there is some specific tasks?
18:03:50 <danielbruno> sorry for the late ask :)
18:04:11 <nirik> danielbruno: no worries. Sure, you can look for more easyfixes...
or other tickets.
18:04:29 <danielbruno> nirik, great :)
18:04:45 <nirik> #topic Applications status / discussion
18:04:59 <nirik> anything this week for applications?
18:05:15 <pingou> I've updated elections to 2.4 and 2.4.1 in stg
18:05:31 <nirik> is the plan then to update prod after freeze?
18:05:33 <pingou> it has a couple of new features and some bugfixes that should have
been released before the current election :(
18:05:43 <nirik> yeah, oh well. it happens. ;)
18:05:55 <pingou> nirik: actually I would like to ask for a freeze-break to update
it on November 26th
18:06:01 <pingou> so the day after the current election ends
18:06:18 <nirik> ok.
18:06:19 <pingou> this way we can benefit from the new features and bugfixes for the
next elections
18:06:29 <threebean> pingou: that might be right around holiday time for NA people..
not sure if that helps or hurts.
18:06:40 <nirik> right, thats right before thanksgiving holidays...
18:06:54 <nirik> but thats ok as long as there aren't any active elections I
don't think it's a big deal.
18:07:12 <pingou> I was hoping to do it just between the end of the election and the
announce of the result as there is a feature for this in the new release
18:07:57 <pingou> threebean: it is mostly some UI changes, a new election type, the
text version of the result page and the memcached integration
18:07:58 <nirik> ok. I think next elections were going to be fesco/etc in january...
or the env-and-stacks folks asked about doing one... not sure when that would be
18:08:06 * pingou is optimist :)
18:08:31 <threebean> cool. +1 here especially if it's all happy in staging
first.
18:08:42 * nirik nods.
18:08:52 <pingou> so far so good :)
18:08:56 <threebean> I've been having fun working on the-new-hotness over in
staging, too.
18:09:01 <threebean> it basically works -> http://fpaste.org/152521/50363114/
18:09:06 * lmacken has been doing a bunch of hacking on the atomic compose process, and
will be hooking it into bodhi.stg for testing shortly
https://github.com/fedora-infra/fedmsg-atomic-composer
18:09:12 <threebean> I just need to add some fedmsg messages its going to publish,
etc.
18:09:14 <puiterwijk> As announced
(http://patrick.uiterwijk.org/2014/11/14/fedoauth-ipsilon-merge-complete/), Ipsilon and
FedOAuth merge has completed. A release is scheduled for tomorrow. I'll be putting it
into staging pretty soon after.
18:09:16 <pingou> on the app note, I'm going to try to get a new pkgdb-cli
release out of the door, testing welcome :)
18:09:24 <lmacken> threebean: nice!
18:09:27 <nirik> #info elections update possibly on the 26th.
18:09:49 <pingou> puiterwijk: cool!
18:09:51 <nirik> #info fedoauth/ipsilon merge is done. Staging soon.
18:10:04 <alexove> Hello
18:10:12 <threebean> puiterwijk: congrats :)
18:10:24 <puiterwijk> threebean: thanks :)
18:10:51 <nirik> cool.
18:11:21 <nirik> #info the-new-hotness (filing bugs on updated package versions) is
moving along.
18:11:27 <nirik> welcome alexove.
18:12:04 <sayhar> (Hi all I’m (also?) new but missed the “new person shoutout” at
the beginning
18:12:04 <sayhar> )
18:12:32 <threebean> sayhar: hi there :)
18:12:37 <nirik> sayhar / alexove: lets finish this section of the meeting, then we
can do a new intros section. :)
18:12:51 <nirik> any other application news?
18:13:38 <alexove> Understood :-)
18:14:13 <nirik> #topic New folks introductions and Apprentice tasks (part 2)
18:14:36 <nirik> alexove / sayhar: care to give us a one line introduction? are you
all more interested in development or sysadmin type tasks?
18:16:31 <sayhar> nirik et all: Heyo. I’m excited to help the cause of truly libre
software / libre humans. Threebean brought me. I’m here to learn-by-doing.
18:16:43 <alexove> Hello World, I'm from Perú (latam) and fedora Ambassador too
. I'm interested in the sysadmin task.
18:16:54 <nirik> excellent. Welcome to you both! :)
18:17:00 <threebean> :)
18:17:11 <alexove> Thanks all guys
18:17:30 <nirik> if you are interested in our sysadmin apprentice program I can add
you after the meeting to look around. we also have a list of easyfix tickets to look at
and see if something interests you.
18:17:52 * pingou waves welcome
18:18:05 <nirik> https://fedoraproject.org/wiki/Infrastructure_Apprentice
18:18:16 <nirik>
https://fedoraproject.org/easyfix/https://fedoraproject.org/easyfix/
18:18:22 <nirik> oops. https://fedoraproject.org/easyfix/
18:19:38 <sayhar> :-)
18:19:59 <nirik> anyhow, see me in #fedora-admin after the meeting if interested and
do chime in with questions anytime. ;)
18:20:11 <nirik> #topic Sysadmin status / discussion
18:20:18 <sayhar> nirik: Thanks I’ll take a look at both. I’m more into dev than
systadmin but happy to check out whatever
18:20:31 <nirik> not too much on the sysadmin front to report... we are in freeze,
so not making too many changes.
18:20:34 <puiterwijk> We've had some discussions with bodhost, but bodhost01 is
back up and disks synced.
18:20:47 <nirik> #info bodhost01 up after failed disk
18:20:58 <puiterwijk> Still some trouble with proxy07's varnish, but I'm
looking at that
18:21:03 <nirik> We also have several new sites on line or coming on line soon...
18:21:18 <nirik> #info host1plus01 site up and running (it has a mirrorlist server
there)
18:21:38 <nirik> #info dedicated-solutions site in process of being added/setup.
18:21:42 <threebean> memcached failed a number of times this week - saturday,
monday, and wednesday mornings.
18:21:48 <nirik> we may have one more new machine soon too.
18:21:53 <threebean> we got some monitoring installed to see if we can figure out
why.
18:22:02 <threebean> .tiny
https://admin.fedoraproject.org/collectd/bin/index.cgi?hostname=memcached...
18:22:02 <zodbot> threebean: http://tinyurl.com/o9k8tt3
18:22:19 <mirek-hm> nirik: update on fed-cloud09 - I have problems with installer, I
get past one problem, but hit another, I wanted to dicuss it mmagr from our office, but he
is in US this week, so I'm leaving that for next week
18:22:23 <nirik> #memcached failures recently. Bug filed, setup to autorestart,
collectd added, monitoring added.
18:22:42 <nirik> #info memcached failures recently. Bug filed, setup to autorestart,
collectd added, monitoring added.
18:23:07 <nirik> mirek-hm: ok. ;( Hope we can get it soon. I'd love to migrate
to new cloud by the end of the year... but we will see.
18:23:16 <mirek-hm> me too
18:23:59 <nirik> #info some issues installing new cloud. Hopefully progress next
week.
18:24:31 <nirik> oh also on memcached, I have abrt setup on memcached02. If it
crashes again we should have a stack trace for the developers.
18:24:44 <threebean> cool
18:24:58 <nirik> so, I think we piled nicely on to that problem as much as we can.
;)
18:25:13 <threebean> yup :p
18:25:35 <nirik> #topic nagios/alerts recap
18:25:40 * nirik digs up the url again.
18:26:45 <nirik> .tiny
https://admin.fedoraproject.org/nagios/cgi-bin//summary.cgi?report=1&...
18:26:46 <zodbot> nirik: http://tinyurl.com/k3fznlm
18:26:57 <puiterwijk> .tiny
https://admin.fedoraproject.org/nagios/cgi-bin//summary.cgi?report=1&...
18:26:57 <zodbot> puiterwijk: http://tinyurl.com/l3vjae8
18:27:13 <nirik> so, badges-backend01 was swapping... did we bump memory on it? oh,
threebean reduced threads on it.
18:27:22 <nirik> proxy07 is the varnish issue puiterwijk mentioned
18:27:34 <nirik> badges-web was the memcached thing
18:27:51 * threebean nods
18:27:56 <nirik> I think the packages0* ones were when they were re-installed?
18:28:26 <nirik> proxy01 and mirrorlist-phx2 have been hitting process limits, we
might need to just bump those.
18:28:32 <threebean> not sure about those.. I don't think they were reinstalled
recently (packages0*...)
18:28:47 <threebean> ah - they were memcached also, actually.
18:28:54 <nirik> ah right.
18:29:20 <nirik> so, nothing we already haven't started addressing really.
Hopefully things will go more quiet
18:29:23 <nirik> #topic Upcoming Tasks/Items
18:29:23 <nirik> https://apps.fedoraproject.org/calendar/list/infrastructure/
18:29:33 <nirik> anything upcoming that anyone would like to note or schedule?
18:29:39 <nirik> I'll note a few things:
18:29:55 <nirik> Next thursday/friday are holidays in the US. So, no meeting next
week.
18:30:06 <puiterwijk> the MirrorManager2/Ansible-Puppet FAD on 2-9 dec
18:30:17 <nirik> yep. Then that. ;)
18:30:23 <threebean> exciting :)
18:30:27 <pingou> \ó/
18:30:31 <nirik> and f21 release is hopefully the 9th also
18:30:44 <puiterwijk> Red Hat closure late december (I think 26-30?)
18:30:50 <threebean> (I'll be on PTO/vacation a day early for thanksgiving, so
no threebean next wednesday.)
18:31:11 <nirik> 2014-12-24 to 2014-12-26 RH holidays
18:31:11 <nirik> 2014-12-29 to 2014-12-31 RH holidays
18:31:11 <nirik> 2015-01-01 RH new years day holiday
18:31:13 * danofsatx-work is present, troubleshooting (still) an RDO openstack deployment
that doesn't want to play
18:31:35 * tflink is on PTO tomorrow and pretty much all of next week with the US
holidays
18:31:52 <pingou> tflink: enjoy :)
18:32:15 <nirik> pingou: for the fad, two things: 1) would it make sense for us to
get together and try and come up with questions or areas to ask mdomsch about? or just go
over things when we are there?
18:32:24 <tflink> pingou: well, I'm attempting to finish up my MS thesis work so
I can graduate this semester so I'm not sure it's going to be all that much fun
:)
18:32:53 <pingou> nirik: I'd love that
18:33:00 <puiterwijk> tflink: good luck then :)
18:33:03 <tflink> I'll be checking email and IRC from time to time, though since
I'm going to be sitting in front of the computer
18:33:09 <tflink> puiterwijk: thanks
18:33:13 <nirik> and 2) looks like on the 4th we may be able to meet with some RHIT
folks and take a tour of RDU2. So, I figure we can break away for that midday and
otherwise go to/come back to mm fad?
18:33:34 <pingou> tflink: beware of IRC while writing thesis, I can speak of
experience :D
18:33:35 <nirik> pingou: it might save us time...
18:33:58 <nirik> (the pre gathering questions for mdomsch thing)
18:34:16 <pingou> nirik: +1
18:34:32 <pingou> nirik: for the tour, it sounds nice, but Matt will only be there
for two days
18:34:49 <pingou> so in a way we should also optimize that time
18:34:55 <tflink> pingou: yeah, I'm planning to turn off my client most of the
time. the looming deadline is proving to be quite a motivator, though :)
18:35:07 <nirik> pingou: yeah, so I was thinking it might make sense for me and
smooge to go do that (and puiterwijk if he wants) and leave the rest of you to hack away
18:35:14 <pingou> tflink: let us know if you need a /kick ;-)
18:35:24 <pingou> nirik: also possible sure
18:35:59 <puiterwijk> nirik: yeah, I was planning to come allong
18:36:26 <nirik> pingou: also, hopefully won't be too long... 1hr meeting, 1hr
tour type thing... :)
18:36:46 * oddshocks here way late
18:36:55 * oddshocks failing on the notifications lately >.<
18:36:58 <pingou> nirik: feel free to add it to the wiki
18:37:09 <tflink> pingou: I suppose getting /kick-ed would be one way to increase
productivity :)
18:37:26 <nirik> pingou: good idea. will do that when it becomes more clear.
18:37:32 <pingou> cool :)
18:37:32 <nirik> hey oddshocks. :)
18:37:42 <nirik> #topic Open Floor
18:37:54 <nirik> ok, anything for open floor? questions, ideas, comments, beer
recipies?
18:38:55 <pingou> might take a while for beer recipies :)
18:39:33 <oddshocks> 1) apply bottle opener to sealed beer 2) consume
18:39:37 <oddshocks> #oddshocksbrewery
18:40:24 <nirik> ha. ;)
18:40:33 <nirik> ok, if nothing else will close out in a minute...
18:41:04 <puiterwijk> nirik: "nothing else" :-)
18:41:20 <nirik> hurray, now we can close out. :)
18:41:28 <oddshocks> in seriousness I have a new big version of fedimg that I'd
like to push out here. trying to pack in as many fixes as possible before I do the fedora
release but we really need it in place at this point so I'm going to send out the
update today at the latest
18:41:55 <nirik> oddshocks: ok, we are in freeze, so you will need to send a freeze
exception to the list...
18:41:58 <pingou> puiterwijk: buzz, incorrect, it was
18:42:00 <pingou> nirik: nothing
18:42:12 <pingou> the `else` is already part of the second condition :)
18:42:13 <oddshocks> nirik: right. cool if I send it to at least staging, right?
18:42:16 <threebean> me nods
18:42:17 <danofsatx-work> do we as in infrastructure run the fedora openstack
instance?
18:42:29 <nirik> oddshocks: staging is always open. ;)
18:42:30 <puiterwijk> pingou: hah, bummer. so now I don't win the toaster? :(
18:42:33 <oddshocks> groovy
18:42:49 <nirik> danofsatx-work: yes, we do have a private cloud running openstack
(ancient folsom)
18:43:09 <danofsatx-work> darn, I was hoping to pick some brains on neutron
18:43:28 <nirik> we are working on setting up a new one, but it's not there
yet.
18:44:15 <nirik> anyhow, thanks for coming everyone! Do continue in #fedora-admin,
#fedora-noc and #fedora-apps. :)
18:44:18 <nirik> #endmeeting