Summary/Minutes from today's Fedora Infrastructure meeting (2014-06-19)

Thu Jun 19 18:49:28 UTC 2014

============================================
#fedora-meeting: Infrastructure (2014-06-19)
============================================

Meeting started by nirik at 18:12:35 UTC. The full logs are available at
fedora-meeting/2014/fedora-meeting.2014-06-19-18.12.log.html .

Meeting summary
---------------
* aloha  (nirik, 18:12:36)

* New folks introductions and Apprentice tasks.  (nirik, 18:13:27)

* Applications status / discussion  (nirik, 18:16:56)
  * LINK: https://apps.fedoraproject.org/github2fedmsg/   (threebean,
    18:17:07)
  * fedoauth is updated in stg. Please test it against as many sites as
    you can... it's going to production this weekend.  (nirik, 18:17:21)
  * copr upgrade also coming up tomorrow. There will be a short outage.
    (nirik, 18:23:56)

* Sysadmin status / discussion  (nirik, 18:31:25)
  * rhel7 setup almost in place to start moving machines to it. Will
    work on kickstarts next week.  (nirik, 18:36:01)
  * ansible using ssh pipeline/controlpersist on lockbox, should be much
    faste.r  (nirik, 18:36:18)
  * smooge and nirik will be heading out to phx2 datacenter in late
    july... 22nd to 25th  (nirik, 18:36:44)

* nagios recap  (nirik, 18:37:48)

* Upcoming Tasks/Items  (nirik, 18:41:39)
  * LINK: https://apps.fedoraproject.org/calendar/list/infrastructure/
    (nirik, 18:41:39)

* Open Floor  (nirik, 18:42:47)

Meeting ended at 18:46:03 UTC.

Action Items
------------

Action Items, by person
-----------------------
* **UNASSIGNED**
  * (none)

People Present (lines said)
---------------------------
* nirik (63)
* oddshocks (19)
* tflink (14)
* roshi (10)
* threebean (7)
* ootbro (7)
* ursabot (6)
* smooge (5)
* danofsatx-work (3)
* puiterwijk (1)
* randomuser (1)
* abadger1999 (1)
* lmacken (1)
* ausmarton (1)
* janeznemanic (1)
* mpduty (1)
* lorddemon (1)
* mdomsch (0)
* pingou (0)
* relrod (0)
* dgilmore (0)
--
18:12:35 <nirik> #startmeeting Infrastructure (2014-06-19)
18:12:35 <ursabot> Meeting started Thu Jun 19 18:12:35 2014 UTC.  The chair is nirik. Information about MeetBot at http://wiki.debian.org/MeetBot.
18:12:35 <ursabot> Useful Commands: #action #agreed #help #info #idea #link #topic.
18:12:35 <nirik> #meetingname infrastructure
18:12:35 <ursabot> The meeting name has been set to 'infrastructure'
18:12:36 <nirik> #topic aloha
18:12:36 <nirik> #chair smooge relrod nirik abadger1999 lmacken dgilmore mdomsch threebean pingou puiterwijk
18:12:36 <ursabot> Current chairs: abadger1999 dgilmore lmacken mdomsch nirik pingou puiterwijk relrod smooge threebean
18:12:46 <oddshocks> woohoo
18:12:52 <nirik> who all is around for a late starting, net splitting meeting? ;)
18:12:53 * oddshocks present and accounted for
18:13:09 * ausmarton here
18:13:15 * puiterwijk is around but busy
18:13:17 <abadger1999> Greetings
18:13:27 <nirik> #topic New folks introductions and Apprentice tasks.
18:13:28 * threebean is here
18:13:32 * danofsatx-work is nuking sketty
18:13:38 * lmacken 
18:13:44 <danofsatx-work> and there's the beep...
18:13:46 <nirik> any new folks around who want to introduce themselves? or apprentices with questions, comments or items to note?
18:14:10 * roshi is here
18:14:18 <janeznemanic> hi
18:14:20 * randomuser is milling around
18:14:30 <roshi> just lurking trying to get a better understanding of our infra :)
18:14:48 <roshi> learn what I can about how we do things on the infra side
18:15:06 <oddshocks> oh god
18:15:07 <oddshocks> why
18:15:09 <roshi> :)
18:15:20 <oddshocks> 2x netsplit combo
18:15:44 <oddshocks> lost kevin and patrick again
18:15:46 * roshi thought you were meaning "oh god why?" to checking out our infra
18:15:56 <oddshocks> roshi: oh no, it's not that bad, I promise ;)
18:16:21 <oddshocks> we just lost about 30 connections to netsplit for the second time in 5 mins
18:16:22 <roshi> :D
18:16:28 <oddshocks> same ones
18:16:39 <nirik> ursabot: ping
18:16:39 <ursabot> pong
18:16:39 <oddshocks> and they're back
18:16:39 <nirik> yeah...
18:16:42 <nirik> freenode isn't too staable today
18:16:56 <nirik> #topic Applications status / discussion
18:17:00 * tflink shows up late, apologies
18:17:01 <nirik> any applications news?
18:17:07 <threebean> https://apps.fedoraproject.org/github2fedmsg/
18:17:20 <threebean> pushed that out this past week.
18:17:21 <nirik> #info fedoauth is updated in stg. Please test it against as many sites as you can... it's going to production this weekend.
18:17:42 <nirik> threebean: pretty cool. Is there any kind of overview of projects connected to it? or it's all per user?
18:18:03 <threebean> all per user, but we could make an overview dashboard thing if it would be useful
18:18:21 <nirik> just my curiosity, dunno how important that is.
18:18:44 * threebean nods
18:18:58 <oddshocks> Fedimg is making its strides! Working out some libcloud issues with #libcloud, but things are looking good! Got all the fedmsg meta code merged in to the fedmsg meta infra repo... docs started on RTFD... makin' progress. that's all for me
18:19:20 <nirik> oddshocks: cool.
18:19:49 <threebean> oddshocks: your fedmsg.meta stuff is released too and on its way to the updates-testing yum repos
18:19:52 <nirik> oddshocks: oh, I wonder... do we want to look at having it upload to the infrastructure private cloud too? dunno how useful it would be, but we could look into that if it is.
18:20:23 <oddshocks> threebean: awesom!
18:20:34 <oddshocks> nirik: I 100% do
18:20:58 <nirik> should just need an account that can upload I would suspect.
18:22:10 <nirik> any other app news?
18:22:10 <oddshocks> nirik: our goal is definitely automatic uploads to external AND internal cloud providers. I was originally uploading to our internal Fedora FTP server too but I was told that is The Wrong Thing To Do and that I should just have nightly builds uploaded to the internal fedora FTP server, but I am 100% down for internal cloud image uploads
18:22:35 <oddshocks> nirik: but any info anyone can get me on getting the images on our internal cloud server is super-helpful
18:22:38 <oddshocks> and can be done with ease
18:22:42 <oddshocks> :)
18:23:04 <nirik> oddshocks: sure, we can talk out of meeting on it sometime... if we do nightlies we should also expire/delete older ones (ie, I don't think we want to keep them forever)
18:23:17 <oddshocks> Definitely. Sounds good.
18:23:56 <nirik> #info copr upgrade also coming up tomorrow. There will be a short outage.
18:25:03 * tflink doesn't know if this counts as application status (or if it's too far out) but we're hoping to get taskotron farther deployed this week and next
18:25:10 <nirik> cool.
18:25:57 <nirik> tflink: oh, there's an issue with resultsdb-stg01... I tried to put it in the stg group, but it can't talk to stg fas. ;( I have a RHIT ticket to fix that...
18:26:08 <nirik> we do want the *stg* qa stuff to be in infra stg right?
18:26:20 <nirik> (for fas, and talking to proxy01.stg and koji01.stg, etc?)
18:27:30 <tflink> nirik: resultsdb doesn't have auth right now, so fas isn't a huge issue
18:27:49 <nirik> well, for local accounts/sudo
18:27:57 <tflink> yeah, I wasn't thinking about that
18:28:13 <tflink> I'm fine eihter way, stg means that my yubikey doesn't work but I can deal with that
18:28:13 <nirik> I can get it sorted out. I would expect we want them to interact with stg
18:28:22 <nirik> ok
18:28:39 <tflink> unless the 2fa issue with stg fas was worked out
18:28:41 <nirik> hopefully the proxy stuff works for you now.
18:28:49 <tflink> er 2fa with yubikey and stg fas
18:29:02 <nirik> I'm not sure which issue? wanting one yubikey to work with the same slot for both?
18:29:15 <tflink> for a while, yubikey couldn't be used with fas stg for 2fa
18:29:28 <tflink> I was told to use google authenticator instead
18:29:40 <nirik> huh, I haven't tried, but I can...
18:30:07 <tflink> I hooked the load testing setup up to resultsdb-stg for a test
18:30:21 <tflink> so it'll get tested out more shortly, even if the names don't quite match :)
18:30:38 <tflink> ie, taskotron-dev.fp.o working with taskotron.stg.fp.o/resultsdb
18:30:42 <nirik> ok.
18:31:25 <nirik> #topic Sysadmin status / discussion
18:31:33 <nirik> ok, lets see on the sysadmin side...
18:31:58 <nirik> I have all our ducks in a row finally for rhel7 setup... so we will be setting up kickstarts and moving to rhel7 instances soon where we can.
18:32:32 <nirik> We will have to see where we are application stack wise before moving any of the application instances over.
18:32:44 <nirik> and we also talked about python3 sometime.
18:32:45 * oddshocks nods
18:33:04 <lorddemon> \me supee
18:33:34 <nirik> but a lot of simpler stuff can move to 7 sooner... like download servers, virthosts, the mailman instances (so we can deploy hyperkitty with rhel7 instead of f19)
18:33:55 * mpduty is here
18:34:01 <ootbro> nirik: does it look like things are back on track?
18:34:04 <ootbro> I'm still hanging on
18:34:04 <ootbro> roshi:  I'm starting work on getting that documentation created
18:34:11 <ootbro> \ still watching over the meeting attempting to start
18:34:28 <nirik> ootbro: we started and have had a few minutes with no net split. ;)
18:34:45 <roshi> which documentation ootbro?
18:34:51 * roshi doesn't recognize your nick
18:35:07 <ootbro> nirik: thanks.. I've been watching the lines scroll past here
18:35:16 <nirik> I've gotten ansible on lockbox using controlpersist and ssh pipelining, so playbook runs should be much faster. This is using a openssh scratch build with that support.
18:35:25 <ootbro> roshi: documentation on how the Fedora infrastructure is laid out
18:35:43 <roshi> ah, sweet
18:36:01 <roshi> sorry, I ask a lot of people for documentation on things and can't keep it all straight in my head :)
18:36:01 <nirik> #info rhel7 setup almost in place to start moving machines to it. Will work on kickstarts next week.
18:36:18 <nirik> #info ansible using ssh pipeline/controlpersist on lockbox, should be much faste.r
18:36:24 <ootbro> not a problem.....  I picked up the project a few weeks ago, with the announcement on here
18:36:44 <nirik> #info smooge and nirik will be heading out to phx2 datacenter in late july... 22nd to 25th
18:37:48 <nirik> #topic nagios recap
18:38:19 <nirik> ?tiny https://admin.fedoraproject.org/nagios/cgi-bin//summary.cgi?report=1&displaytype=3&timeperiod=last7days&smon=5&sday=1&syear=2014&shour=0&smin=0&ssec=0&emon=5&eday=15&eyear=2014&ehour=24&emin=0&esec=0&hostgroup=all&servicegroup=all&host=all&alerttypes=3&statetypes=2&hoststates=3&servicestates=56&limit=25
18:38:20 <ursabot> nirik: http://tinyurl.com/q8j48o9
18:38:32 <nirik> our bug alerter is virthost-comm02 this week.
18:38:46 <nirik> it's got some kind of disk issue going on thats making it slow.
18:38:59 <nirik> we are waiting for it to finish raid check and then likely will try a reboot.
18:39:36 <tflink> nirik: when were you thinking the reboot would be?
18:39:47 <nirik> tflink: not sure yet... this raid check is taking forever. ;(
18:40:04 <nirik> [==================>..]  check = 91.9% (447933184/487298048) finish=653.5min speed=1003K/sec
18:40:04 <smooge> I think we will be rebooting it before the raid check is done
18:40:12 <smooge> oh wait
18:40:18 <nirik> it's getting close
18:40:18 <smooge> thats different from when I looked
18:40:28 <nirik> tflink: so, perhaps tomorrow?
18:40:36 <smooge> that is 1 MB/sec on a bunch of 3 GB/sec disks
18:40:45 <smooge> of course they are busy
18:40:56 <tflink> ok, I think I have some stuff on there that's not set up to rebound on reboot so please let me know when you reboot it
18:40:56 <nirik> yeah, it's definitely not right. ;)
18:41:20 <nirik> tflink: can do... we also need to coordinate with arm folks as the arm koji instance is on that virthost too
18:41:39 <nirik> #topic Upcoming Tasks/Items
18:41:39 <nirik> https://apps.fedoraproject.org/calendar/list/infrastructure/
18:41:56 <nirik> anyhing upcoming anyone would like to note or schedule?
18:42:47 <nirik> #topic Open Floor
18:42:53 <nirik> anyone have items for open floor?
18:44:16 <nirik> ok, if nothing else will end the meeting in a minute, hopefully before the next net.split. ;)
18:44:24 <threebean> :)
18:44:47 <danofsatx-work> my email issues are fixed, and I'll be RHCE certified next Friday ;)
18:44:59 <nirik> danofsatx-work: congrats
18:46:00 <nirik> ok, thanks for coming everyone. Do continue in #fedora-admin, #fedora-apps and #fedora-noc. ;)
18:46:03 <nirik> #endmeeting
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 836 bytes
Desc: not available
URL: <http://lists.fedoraproject.org/pipermail/infrastructure/attachments/20140619/d1f0449d/attachment-0001.sig>