Summary/Minutes from today's Fedora Infrastructure meeting (2014-12-11)

Thu Dec 11 18:43:28 UTC 2014

============================================
#fedora-meeting: Infrastructure (2014-12-11)
============================================

Meeting started by nirik at 18:00:36 UTC. The full logs are available at
http://meetbot.fedoraproject.org/fedora-meeting/2014-12-11/infrastructure.2014-12-11-18.00.log.html
.

Meeting summary
---------------
* aloha  (nirik, 18:00:37)

* New folks introductions and Apprentice tasks  (nirik, 18:02:05)

* FAD recap  (nirik, 18:06:48)
  * got lots of details about mirrormanager 1 from Matt Domsch  (nirik,
    18:07:31)
  * got Mirrormanager 2 pretty much all implemented  (nirik, 18:08:38)
  * got some test coverage added to Mirrormanager 2  (nirik, 18:08:50)
  * got Mirrormanager2 mostly added to ansible and deployed in staging
    (nirik, 18:09:09)
  * LINK: https://admin.stg.fedoraproject.org/mirrormanager2/   (pingou,
    18:09:19)
  * targeting mid january for production deployment. Might do some kind
    of phased approach with mirrorlist vs the rest.  (nirik, 18:12:01)
  * ansible migrations took up the rest of the FAD time.  (nirik,
    18:13:01)
  * puiterwijk got nameservers in ansible and all redeployed with it.
    (nirik, 18:13:34)
  * various other playbooks written, just need to finish things up.
    (nirik, 18:14:13)
  * down to 57 machines in puppet.  (nirik, 18:14:21)

* Applications status / discussion  (nirik, 18:15:38)
  * puiterwijk got single login in badges, and spec for single logout
    (more to come)  (nirik, 18:17:00)
  * proxy playbooks are about 75% done. TODO list is in
    playbooks/groups/proxies-reverseproxy.yml  (nirik, 18:20:05)
  * hopefully roll out a bodhi2 instance in stg asap and see what all we
    need for production  (nirik, 18:27:35)

* Sysadmin status / discussion  (nirik, 18:28:28)
  * monday outage - updates/reboots  (nirik, 18:29:59)
  * tuesday outage - database server migrations  (nirik, 18:30:08)
  * wed outage - more ansible/rhel7 migrations  (nirik, 18:30:19)

* nagios recap  (nirik, 18:35:31)

* Upcoming Tasks/Items  (nirik, 18:37:28)
  * LINK: https://apps.fedoraproject.org/calendar/list/infrastructure/
    (nirik, 18:37:28)
  * we will meet next week as normal, but not the following 2 weeks
    (nirik, 18:38:34)
  * ACTION: nirik to send list of ansible work we still need done to
    list.  (nirik, 18:40:33)

* Open Floor  (nirik, 18:40:39)
  * talk to puiterwijk if you want single login on your fedora app
    (nirik, 18:41:43)

Meeting ended at 18:43:04 UTC.

Action Items
------------
* nirik to send list of ansible work we still need done to list.

Action Items, by person
-----------------------
* nirik
  * nirik to send list of ansible work we still need done to list.
* **UNASSIGNED**
  * (none)

People Present (lines said)
---------------------------
* nirik (102)
* pingou (20)
* puiterwijk (16)
* lmacken (13)
* wallison (11)
* tflink (9)
* threebean (8)
* zodbot (5)
* adimania (3)
* mirek-hm (3)
* Khaytsus (1)
* bwood09 (1)
* danielbruno (1)
* relrod (1)
* abadger1999 (0)
* smooge (0)
* mdomsch (0)
* dgilmore (0

--
18:00:36 <nirik> #startmeeting Infrastructure (2014-12-11)
18:00:37 <zodbot> Meeting started Thu Dec 11 18:00:36 2014 UTC.  The chair is nirik. Information about MeetBot at http://wiki.debian.org/MeetBot.
18:00:37 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
18:00:37 <nirik> #meetingname infrastructure
18:00:37 <zodbot> The meeting name has been set to 'infrastructure'
18:00:37 <nirik> #topic aloha
18:00:37 <nirik> #chair smooge relrod nirik abadger1999 lmacken dgilmore mdomsch threebean pingou puiterwijk
18:00:37 <zodbot> Current chairs: abadger1999 dgilmore lmacken mdomsch nirik pingou puiterwijk relrod smooge threebean
18:00:42 * lmacken 
18:00:42 * threebean waves
18:00:45 * puiterwijk is here
18:00:48 * danielbruno is here
18:00:52 * mirek-hm is here
18:01:00 * adimania is here
18:01:03 * pingou 
18:01:09 * bwood09 here
18:02:05 <nirik> #topic New folks introductions and Apprentice tasks
18:02:20 <nirik> any new folks like to provide a short introduction of themselves?
18:02:28 <nirik> or apprentices with questions or comments ?
18:02:48 <wallison> Guess I'll go...
18:03:23 <wallison> My name is Will, I'm still working through the Primary Steps of Getting Started
18:03:33 <nirik> cool. welcome wallison
18:03:44 <wallison> I'm not an IT pro, but have been using Linux at home for at least 15 years
18:03:53 <nirik> are you primarilly interested in sysadmin type things, or application development, or both?
18:04:00 <wallison> and want to contribute (finally) to the FOSS community
18:04:10 <wallison> Fedora seems to make that easy!
18:04:15 <threebean> :)
18:04:18 <nirik> we try. ;)
18:04:24 <wallison> nirik sysadmin stuff
18:04:40 <wallison> i'd like to do that as a second career after college
18:04:46 <nirik> cool. See me after the meeting over in #fedora-admin and I can get you added to our apprentice group...
18:04:49 <wallison> threebean: nice vids by the way
18:04:58 <nirik> and you can look around and look at easyfix tickets
18:05:18 <wallison> nirik: sounds awesome, I'll probably need some hand holding!
18:05:37 * relrod late, sorry
18:05:38 <nirik> sure, there are usually folks around in #fedora-admin or #fedora-noc that can help you out.
18:05:42 <wallison> Thanks for allowing my to introduce myself...
18:05:46 <threebean> wallison: thanks :) lemme know if you find holes in the information or important stuff left out.  could maybe use an updated sequel..
18:06:24 <nirik> any other new folks? or apprentices with questions?
18:06:24 <wallison> will do threebean
18:06:38 <pingou> welcome wallison
18:06:48 <nirik> #topic FAD recap
18:07:06 <nirik> So, we recently had a FAD (fedora activity days) in raleigh...
18:07:13 <pingou> it was nice :)
18:07:13 <nirik> I think we got some great stuff done, wanted to recap. ;)
18:07:31 <nirik> #info got lots of details about mirrormanager 1 from Matt Domsch
18:07:49 <nirik> did we actually get mm2 all implemented?
18:07:58 <nirik> or were there still some minor gaps?
18:08:22 <pingou> should be all there
18:08:23 <nirik> I think we got it all pretty much?
18:08:33 <pingou> including the bugs we missed :)
18:08:38 <nirik> #info got Mirrormanager 2 pretty much all implemented
18:08:50 <nirik> #info got some test coverage added to Mirrormanager 2
18:09:09 <nirik> #info got Mirrormanager2 mostly added to ansible and deployed in staging
18:09:19 <pingou> https://admin.stg.fedoraproject.org/mirrormanager2/
18:09:26 <nirik> There is still some testing work and finishing things up in stg, but it's largely all there. ;)
18:09:42 <pingou> the mirrorlist needs a little tweaking :)
18:09:51 <nirik> did we decide on a timeline to deploy?
18:09:56 <Khaytsus> Sorry, my server is flipping out again, I"m reading back ;)
18:09:56 <pingou> we did not
18:09:56 <nirik> in prod I mean
18:10:28 <nirik> well, it would be good to get this rolled out before any freezes or milestones...
18:10:35 <nirik> perhaps mid january?
18:10:46 <pingou> once we get mirrorlist up and running we'll be able to test mm2 nicely
18:10:51 <pingou> but mid-january sounds good
18:11:15 <nirik> we could also do a phased approach... do mirrorlists first, then the rest...
18:11:18 <nirik> or vice versa.
18:11:23 <nirik> as long as the data is compatible
18:11:34 <pingou> should be :)
18:12:01 <nirik> #info targeting mid january for production deployment. Might do some kind of phased approach with mirrorlist vs the rest.
18:12:12 <nirik> anything else to note on MM2?
18:12:46 <pingou> looks good to me :)
18:12:48 <nirik> we also then spent serveral days working on ansible migrations.
18:13:01 <nirik> #info ansible migrations took up the rest of the FAD time.
18:13:18 <nirik> threebean: where were we on proxy playbooks? still needs more converting?
18:13:34 <nirik> #info puiterwijk got nameservers in ansible and all redeployed with it.
18:14:13 <nirik> #info various other playbooks written, just need to finish things up.
18:14:21 <nirik> #info down to 57 machines in puppet.
18:14:56 <nirik> anything else to note about the ansible side of the FAD?
18:15:38 <nirik> #topic Applications status / discussion
18:15:49 <puiterwijk> nirik: well, just one more thing about the FAD not about ansible
18:15:51 <nirik> anything in the applications side this week? I see there was a new pkgdb deployed
18:15:56 <nirik> puiterwijk: oh?
18:16:00 <puiterwijk> oh, can also go here
18:16:21 <puiterwijk> got single login into badges upstream, and a spec for single logout. more to come in the near future
18:16:33 <nirik> oh yeah.
18:16:48 <puiterwijk> (that was also during the FAD, though not officially part :) )
18:17:00 <nirik> #info puiterwijk got single login in badges, and spec for single logout (more to come)
18:17:01 <lmacken> I worked on polishing up the atomic tree composer code and integrated it into bodhi during the first 1.5 days of the FAD
18:17:18 <nirik> lmacken: oh yeah, we were going to look at landing that sometime, right?
18:17:20 <lmacken> got bodhi mashes running in stg for the first time, and with atomic tree support too
18:17:37 <lmacken> nirik: yeah, I'd be happy to push that out today or tomorrow
18:17:47 <threebean> nirik: on proxy playbooks the redirects, websites, and reversepassproxies are done.
18:17:55 <nirik> lmacken: we might run it by dgilmore... but I think it's the way he wanted it.
18:17:56 <tflink> any farther thoughts on when bodhi2 might be deployed?
18:18:17 <lmacken> nirik: i'll have it compose trees to a one-off location and have him +1 the output before we throw it live
18:18:28 <threebean> prolly 75% done... there are some more little bits that need doing, with a commented-out TODO list in playbooks/groups/proxies-reverseproxy.yml
18:18:30 <lmacken> and walters too
18:18:31 <nirik> tflink: not sure, but we should try and come up with a timeline.
18:18:47 <tflink> not much concrete movement in qa apps, mostly discussion and planning. still working on an el7 build for blockerbugs so that can be migrated but keep hitting epel7 package problems
18:19:14 <nirik> lmacken: sounds good.
18:19:33 <nirik> tflink: let me know if I can help... or if you get to the end of the well and we can cut them over. ;)
18:19:38 <pingou> nirik: btw, rebuilt fas03 fine yesterday?
18:20:05 <nirik> #info proxy playbooks are about 75% done. TODO list is in playbooks/groups/proxies-reverseproxy.yml
18:20:08 <tflink> we're likely to try getting beaker working as more than just a dev machine next year, will be starting more conversations about how to deploy and secure that once it gets farther along
18:20:24 <nirik> pingou: I rebuilt fas03 and fas02. Both seem working fine.
18:20:29 <pingou> cool!
18:20:37 <tflink> nirik: yeah, I want to make sure Taskotron is ready for the switchover since we rely on bodhi for notifications right now
18:21:02 <tflink> nirik: it's mostly epel7 builds that were never tested ATM. things that build fine until you try to use them and they explode :-/
18:21:02 <puiterwijk> nirik: well, I've seen some 2FA problems earlier today. but might've been just net timeouts or me overthinking things
18:21:20 <nirik> puiterwijk: hum, ok. It's working fine for me...
18:21:32 <puiterwijk> nirik: not all the time. next time I see it, I'll report back
18:21:37 <puiterwijk> and/or debug
18:22:00 <nirik> lmacken: thoughts on bodhi2 deployment? can we add a bodhi2 instance in stg now and see what else we need?
18:22:15 <lmacken> nirik: setting up a stg instance sounds good to me
18:22:33 <lmacken> nirik: the frontend should be close to ready, and having a stg setup that we can mash in now will help me polish up the final masher bits
18:22:49 <nirik> cool. I can make one this afternoon. ;)
18:22:51 <lmacken> (until we mash in koji)
18:23:00 <lmacken> nirik: awesome, thanks
18:23:32 <nirik> I'd be happy if we could roll it out in jan too... or sooner, but that may not be practical.
18:23:56 <tflink> nirik: you mean a production bodhi2?
18:24:06 <tflink> or the stg instance
18:24:14 <nirik> yeah. I want it out. It's been forever.
18:24:19 <nirik> stg I plan on doing today. ;)
18:24:29 <nirik> perhaps that will give us a feel for when we could go to prod
18:24:42 * tflink starts scrambling to get Taskotron ready for that
18:24:44 <lmacken> yeah, let's shoot for sometime in jan/feb :)
18:25:03 <nirik> tflink: is that impractical from your side? or how much code changes?
18:25:04 <lmacken> there's backwards compat work that still needs to be done in python-fedora, etc.
18:25:35 <nirik> oh true.
18:25:37 <tflink> nirik: the biggest problem is that we need to get fedmsg integration, i assume that API changes will have minimal impact from our end
18:26:01 <nirik> tflink: ok. having a stg instance should help there too... (so you have something to test/see)
18:26:12 <lmacken> threebean and I can help with the fedmsg stuff on all fronts
18:26:21 <nirik> the other big application deployment I want to see is mailman3 soonish. ;)
18:26:30 * pingou looks at abompard
18:26:32 <nirik> abompard: ^ any news on mailman3
18:27:35 <nirik> #info hopefully roll out a bodhi2 instance in stg asap and see what all we need for production
18:27:57 <nirik> cool. Any other applications side news?
18:28:28 <nirik> #topic Sysadmin status / discussion
18:28:33 <mirek-hm> openstack had just problems ergo we copr had problems today and there were lots of failing builds today. /me lost the patience and I'm scratching the idea of installin Juno as the installation still does not work and I want to install Icehouse, which works (err worked last time I tried that).
18:28:50 <nirik> mirek-hm: yeah, seems reasonable.
18:29:06 <nirik> mirek-hm: I asked Smooge to reinstall those. If he can't get to it soon I will.
18:29:50 <nirik> I'm also going to send out a outage announcement for next week... I'm going to plan an outage block monday/tuesday/wed.
18:29:59 <nirik> #info monday outage - updates/reboots
18:30:02 <mirek-hm> nirik: my plan is that once installed, I will run there few VM and will check how stable it is, and eventually start migrating copr
18:30:08 <nirik> #info tuesday outage - database server migrations
18:30:19 <nirik> #info wed outage - more ansible/rhel7 migrations
18:30:28 <nirik> mirek-hm: ok, sounds good.
18:30:43 <threebean> nirik: would you like to see the proxy stuff ready by then (next week)?
18:31:04 <puiterwijk> threebean: I think that might be a bit soon for something that complex
18:31:12 <nirik> threebean: I think it's too soon to roll that to prod... also, we can do that outside outage
18:31:21 * threebean nods
18:31:28 <nirik> ie, take a proxy out, convert it, add it back carefully and watch
18:31:28 <threebean> will get it ready and then let it sit, then.
18:31:48 <nirik> if it's ready looking I can try some conversions over the holidays... ;)
18:31:52 <pingou> nirik: on the other side threebean tested with with ansible --check to see if it changed anything
18:32:17 <nirik> yeah...
18:32:26 <pingou> the only thing I would have is: next week might be a little soon with the holidays around the corner :)
18:32:34 <pingou> but early jan sounds good to me
18:32:51 <nirik> I think the next step is to finish things up and put it in as proxy01.stg...
18:33:07 <nirik> then we can test it more... then if it looks all good there, we can start rolling out to prod proxies.
18:33:26 <nirik> also, at the same time we can try and update colo sites to rhel7 virthosts.
18:33:28 <puiterwijk> nirik: or we can push it to proxy01 and have everyone use that again. might provide good testing :)
18:33:43 <nirik> heh. indeed.
18:33:46 <pingou> ^^
18:35:22 <nirik> Anything else on the sysadmin side?
18:35:31 <nirik> #topic nagios recap
18:35:37 <puiterwijk> .tiny https://admin.fedoraproject.org/nagios/cgi-bin//summary.cgi?report=1&displaytype=3&timeperiod=last7days&smon=11&sday=1&syear=2014&shour=0&smin=0&ssec=0&emon=11&eday=6&eyear=2014&ehour=24&emin=0&esec=0&hostgroup=all&servicegroup=all&host=all&alerttypes=3&statetypes=3&hoststates=7&servicestates=120&limit=25
18:35:37 <zodbot> puiterwijk: http://tinyurl.com/pzzwx2r
18:35:57 <puiterwijk> oh, hah. copied the wrong one I guess
18:35:59 <nirik> we had quite the pile of alerts this week... due to the proxy geoip issue puiterwijk tracked down and fixed (thanks again!)
18:36:40 <puiterwijk> we had lots of traffic as well (thanks to all contributors for Fedora 21!)
18:36:44 <nirik> hopefully things will relax a bit now. Anything to reduce alerts over the holidays would be good.
18:37:28 <nirik> #topic Upcoming Tasks/Items
18:37:28 <nirik> https://apps.fedoraproject.org/calendar/list/infrastructure/
18:37:37 <nirik> anything upcoming that folks would like to note or schedule?
18:37:56 <nirik> aside from mirrormanager2, mailman3 and bodhi2 deployed by jan. ;)
18:38:04 <pingou> \ó/
18:38:11 <adimania> note: do we have an updated list of what is left for ansible?
18:38:20 <puiterwijk> nirik: RH shutdown end of december?
18:38:34 <nirik> #info we will meet next week as normal, but not the following 2 weeks
18:38:58 <nirik> adimania: I have a list. I am going to probibly post to the list... if you want to look at working on some more. ;)
18:39:19 <nirik> a lot of the ones left in puppet already have playbooks, just a matter of downtime and moving them.
18:39:25 <nirik> thus the outage blocks next week
18:39:35 <adimania> nirik, yes. I am willing to. If you can post the list today or tomorrow, then I can try to do something over the weekend :)
18:39:43 <nirik> cool. ;)
18:40:33 <nirik> #action nirik to send list of ansible work we still need done to list.
18:40:39 <nirik> #topic Open Floor
18:40:42 <nirik> anything for open floor?
18:41:26 <puiterwijk> if people want to add Single login to Fedora apps, please get in contact with me?
18:41:43 <puiterwijk> help add*
18:41:43 <nirik> #info talk to puiterwijk if you want single login on your fedora app
18:43:01 <nirik> ok, thanks for coming everyone. :) Lets continue in #fedora-admin, #fedora-noc and #fedora-apps.
18:43:04 <nirik> #endmeeting
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 819 bytes
Desc: OpenPGP digital signature
URL: <http://lists.fedoraproject.org/pipermail/infrastructure/attachments/20141211/7c5c2cd9/attachment-0001.sig>