Summary/Minutes from today's Fedora Infrastructure meeting (2014-05-01)

[Kevin Fenzi]

============================================
#fedora-meeting: Infrastructure (2014-05-01)
============================================


Meeting started by nirik at 18:00:03 UTC. The full logs are available at
http://meetbot.fedoraproject.org/fedora-meeting/2014-05-01/infrastructure.2014-05-01-18.00.log.html
.



Meeting summary
---------------
* greetings starfighters  (nirik, 18:00:03)

* New folks introductions and Apprentice tasks  (nirik, 18:02:13)
  * LINK: http://fedoraproject.com/easyfix   (webpigeon, 18:07:12)

* Applications status / discussion  (nirik, 18:08:02)
  * some work on a flask re-write of mirrormanager ongoing  (nirik,
    18:14:14)
  * ui work on pkgdb2 ongoing  (nirik, 18:14:24)
  * hyperkitty came up in the news a few times this week in slashdot and
    lwn, pointing to our stg instance.  (nirik, 18:14:48)
  * LINK: https://fedorahosted.org/fedora-infrastructure/ticket/4044
    (threebean, 18:17:15)
  * LINK: http://threebean.org/blog/fedmsg-collectd-ng/   (threebean,
    18:17:19)

* Sysadmin status / discussion  (nirik, 18:18:47)
  * storage move had soe issues, but hopefully we have worked them out
    now.  (nirik, 18:21:14)
  * new bvirthosts are on-line  (nirik, 18:21:20)
  * LINK: https://admin.fedoraproject.org/nagios/ is our main nagios
    (nirik, 18:33:10)

* Upcoming Tasks/Items  (nirik, 18:35:09)
  * LINK: https://apps.fedoraproject.org/calendar/list/infrastructure/
    (nirik, 18:35:09)
  * LINK: https://fedoraproject.org/wiki/FAD_Bodhi2_Taskotron_2014
    (threebean, 18:35:54)
  * nirik will be out saturday to next thursday.  (nirik, 18:36:06)

* Open Floor  (nirik, 18:40:09)

Meeting ended at 18:56:22 UTC.




Action Items
------------





Action Items, by person
-----------------------
* **UNASSIGNED**
  * (none)




People Present (lines said)
---------------------------
* nirik (108)
* threebean (23)
* henderbj (15)
* pingou (15)
* bwood09 (13)
* smooge (13)
* mattdm (9)
* zodbot_ (7)
* ootbro (6)
* danrimal (5)
* Daredel (4)
* danofsatx-work (3)
* nj0y (3)
* relrod (2)
* webpigeon (2)
* janeznemanic (1)
* mpduty (1)
* lmacken (1)
* ghostalker (1)
* danofsatx|kvirc (1)
* oddshocks (1)
* dgilmore (1)
* mdomsch (0)
* puiterwijk (0)
* abadger1999 (0)
--
18:00:03 <nirik> #startmeeting Infrastructure (2014-05-01)
18:00:03 <zodbot_> Meeting started Thu May  1 18:00:03 2014 UTC.  The chair is nirik. Information about MeetBot at http://wiki.debian.org/MeetBot.
18:00:03 <zodbot_> Useful Commands: #action #agreed #halp #info #idea #link #topic.
18:00:03 <nirik> #meetingname infrastructure
18:00:03 <nirik> #topic greetings starfighters
18:00:03 <nirik> #chair smooge relrod nirik abadger1999 lmacken dgilmore mdomsch threebean pingou puiterwijk
18:00:03 <zodbot_> The meeting name has been set to 'infrastructure'
18:00:03 <zodbot_> Current chairs: abadger1999 dgilmore lmacken mdomsch nirik pingou puiterwijk relrod smooge threebean
18:00:10 * relrod waves
18:00:26 * pingou 
18:00:31 * webpigeon waves
18:00:48 <janeznemanic> hi
18:01:04 * lmacken 
18:01:33 <danofsatx-work> I'm here, but if Konversation wigs out on me again
18:01:47 <danofsatx|kvirc> I'll be here instead
18:02:00 <nirik> :)
18:02:06 <nirik> ok, lets go ahead and get started....
18:02:08 * threebean is here
18:02:13 <nirik> #topic New folks introductions and Apprentice tasks
18:02:27 <danrimal> hi, i am new here
18:02:34 <nirik> any new folks want to do a quick one line introduction of themselves? or apprentices with questions or comments?
18:02:46 <danrimal> yes, sure
18:02:48 <danrimal> I am sysadmin, my job is high load and high availability application as web, mail and databases services as well as bgp and ospf networking
18:03:00 <ootbro> I'll also jump in as one of the newbies
18:03:02 <danrimal> and i am interested in sysadmin things, if available
18:03:31 <ootbro> I've been using Linux for years and now want to contribute here.  I think the testing FIG is my best starting point, with an eye toward sysadmin-main (eventually)
18:03:38 <nirik> danrimal: welcome! sure thing... see me in #fedora-admin after the meeting and I can get you setup in our apprentice program...
18:03:52 <nirik> ootbro: welcome again. ;)
18:03:59 <ootbro> thanks
18:04:22 <nj0y> As I introduced myself in the mailing, i'm a sysadmin/engineer from switzerland and very interested getting in charge here. And i search some fig to join. I think also fig testing or fig web is a good place for me.
18:05:00 <nirik> nj0y: welcome also. ;)
18:05:16 * ghostalker is here
18:05:17 <nj0y> thanks, glad i'm here.
18:05:30 <nirik> always good to have new folks around... do chime in with questions or comments anytime...
18:05:31 <mpduty> .fasinfo mohanprakash
18:05:32 <zodbot_> mpduty: User: mohanprakash, Name: Mohan Prakash, email: mpduty at gmail.com, Creation: 2013-12-27, IRC Nick: mpduty, Timezone: Asia/Kolkata, Locale: en, GPG key ID: 0xAF620142, Status: active
18:05:36 <zodbot_> mpduty: Unapproved Groups: l10n-editor l10n-commits marketing
18:05:39 <zodbot_> mpduty: Approved Groups: fi-apprentice cvsl10n cla_done cla_fpca
18:06:26 <nirik> I can assist anyone after the meeting over in #fedora-admin who wants to join our apprentice group or would like to be pointed at easyfix tickets, etc. :)
18:06:31 <nirik> Welcome again everyone!
18:06:52 <ootbro> many thanks....   I could use the help in getting started
18:07:01 <nj0y> me too.
18:07:08 <danrimal> ok, thanks
18:07:12 <webpigeon> http://fedoraproject.com/easyfix
18:07:23 * mattdm is lurking
18:07:35 * bwood09 is here
18:07:46 <nirik> see also http://fedoraproject.org/wiki/Infrastructure/GettingStarted and https://fedoraproject.org/wiki/Infrastructure_Apprentice
18:08:02 <nirik> #topic Applications status / discussion
18:08:14 <nirik> any application side news from the previous week or upcoming?
18:08:27 * pingou has done a good chunk of work on mirrormanager2 this week
18:08:42 <pingou> and over the last two days I am on the re-design of some the page of pkgdb2
18:08:48 * Daredel is late
18:08:48 <pingou> including http://209.132.184.188/package/R-DBI/
18:08:53 <nirik> pingou: this is the flask re-write of it? or is it tg2? or ?
18:08:57 <pingou> nirik: flask
18:09:10 <pingou> but I only worked on the UI
18:09:14 <nirik> have you contacted mdomsch any on it? (I know he's not been around)
18:09:24 <threebean> pingou: that looks *much* nicer.
18:09:43 <relrod> yeah, that looks nice :)
18:09:56 * nirik is waiting for load. ;)
18:10:08 <nirik> I bet a bunch of us clicked it at the same time.
18:10:35 <danofsatx-work> i waited a bit, came up fine for me ;)
18:10:48 <nirik> or... it could be my pesky wireless. ;(
18:10:53 <danofsatx-work> which is amazing, considering what I've been fighting with locally......
18:11:33 <nirik> pingou: how is 'package administrator' determined?
18:11:41 <pingou> threebean: designed by mizmo, I can't beat that :)
18:11:47 * oddshocks here late, in lecture as usual
18:11:53 <pingou> nirik: Contacts are the POC, Admins are the users with approveacls
18:12:56 <nirik> pingou: ok, so anyone with any approveacls?
18:13:20 <pingou> yes
18:13:36 <pingou> nirik: or pending approveacls (then there is a (?) icon next to them)
18:13:51 <nirik> ok, cool.
18:14:14 <nirik> #info some work on a flask re-write of mirrormanager ongoing
18:14:24 <nirik> #info ui work on pkgdb2 ongoing
18:14:48 <nirik> #info hyperkitty came up in the news a few times this week in slashdot and lwn, pointing to our stg instance.
18:15:24 <threebean> are we any closer to cutting another list over?
18:15:41 <threebean> erm, by that I mean changing some existing lists from mailman2 to mailman3
18:15:54 <nirik> I sent abompard some issues and he was going to fix them up... then we were going to see where we were.
18:16:05 <nirik> hopefully soon tho.
18:16:09 <threebean> cool.  newly queued stuff..
18:16:11 * threebean nods
18:16:12 <nirik> I'd be happy to move the infra list.
18:16:20 <threebean> yeah, agreed
18:16:36 <nirik> there may also be some fixes from this recent press on it...
18:17:10 <threebean> unrelated, janeznemanic and I have been working on some fedmsg monitoring stuff and made some progress this week
18:17:15 <threebean> https://fedorahosted.org/fedora-infrastructure/ticket/4044
18:17:17 <nirik> excellent.
18:17:19 <threebean> http://threebean.org/blog/fedmsg-collectd-ng/
18:17:34 <threebean> collectd is in place and fun.  nagios checks coming soon.
18:17:56 * bwood09 starts reading the entirety of threebean.org
18:18:17 <nirik> any other application type news? or shall we move on to sysadmin?
18:18:47 <nirik> #topic Sysadmin status / discussion
18:19:02 <nirik> smooge got some of our new build virthosts up and running yesterday.
18:19:21 <smooge> yay
18:19:22 <nirik> Tuesday night we moved our backend storage from one netapp to another less loaded one...
18:19:28 <smooge> hahah
18:19:29 <nirik> but we have had some issues since then. ;(
18:19:34 <smooge> boo
18:19:58 <nirik> It's looking a lot like those issues are related to some virthosts having an emulated realtek network card instead of virtio.
18:20:07 <bwood09> nirik, will those new virthosts need to be added to nagios and the such?
18:20:09 <nirik> something in the move caused them to start dropping packets like mad
18:20:16 <nirik> bwood09: they will indeed. ;)
18:20:42 <nirik> I can file a ticket on them after the meeting.
18:20:45 <nirik> or smooge can
18:20:47 <bwood09> I'm going to go through today and tomorrow and take care of the nagios stuff, so if you drop a ticket for them in easyfix-- yeah
18:20:49 <bwood09> lol
18:20:51 <nirik> or really anyone can. ;)
18:21:14 <nirik> #info storage move had soe issues, but hopefully we have worked them out now.
18:21:20 <nirik> #info new bvirthosts are on-line
18:21:57 * smooge opens an easyfix ticket that someone can open an easyfix ticket to add monitoring for several hosts
18:22:11 <henderbj> Hello all... i am late... already read previous messages
18:22:19 <bwood09> Also, not sure if this is the place to do this, but I want to get on with the sysadmin-hosted group
18:22:26 * pingou gtg
18:22:33 <nirik> welcome henderbj
18:22:36 <smooge> bye pingou
18:22:38 <nirik> bye pingou
18:23:46 <nirik> bwood09: what sorts of things do you want to work on there? any tickets in specific? or just adding new projects and such?
18:24:27 <nirik> we had some plans in there we could look at again and see if you might want to work on them...
18:24:30 <bwood09> I'm going to look at the tickets today and see if there's anything I want to tackle. Recently, most of my experience has been git, svn, hg, and bzr so I figure I'd be a good fit
18:24:52 <nirik> sure thing. Let me (or any other hosted sponsor know) and we can see about helping you along.
18:24:58 <bwood09> Alrighty
18:25:10 <nirik> on nagios... we had a lot more alerts this last week I fear...
18:25:21 <nirik> 273 I see since last thursday.
18:25:45 <threebean> oo
18:25:47 <bwood09> What's the norm for those?
18:25:47 <nirik> the vast majority of which I think were related in one way or another to the storage move.
18:25:58 <threebean> this is a fun new routine.  :p
18:26:01 <dgilmore> damn storage
18:26:32 * nirik looks back at the previous weeks
18:27:18 <nirik> 77 the week before
18:27:34 <bwood09> oh wow
18:28:18 <nirik> I'd like to reduce them as much as we can... I fear it will be impossible to make them 0 without making them not alert when theres problems users will notice.
18:28:35 <henderbj> well, that's normal... a lot of alarms when someone touches anything!
18:29:13 <nirik> well, most of the 'normal' ones are network related. We have a very wide network... so if our monitoring host can't reach some datacenter, it alerts.
18:30:07 <nirik> some of the ones this last week were also from a datacenter where we started to see packet loss... they were being hit by a DDOS.
18:30:12 <smooge> or where we aren't losing pings but they are taking close to a second to travel around the world
18:30:29 <nirik> anyhow, if anyone wants to dig thru nagios logs and propose changes that would be lovely. ;)
18:30:53 <henderbj> i will be testing nagios on my own testing machine
18:31:04 <nirik> we may be able to tune the network related ones down some, but not too far.
18:31:21 <henderbj> When get into it, i will pick something about nagios to help
18:31:42 <ootbro> question.....   is there a way in nagios to not try a set of hosts if a "core" host is unreachable due to a network outage?
18:31:44 <nirik> henderbj: sounds great. Feel free to ask in #fedora-noc or #fedora-admin if you have any questions about our setup
18:31:54 <nirik> ootbro: yeah, it has dependencies...
18:32:01 <henderbj> Tnx, nirik, sure
18:32:09 <nirik> I think they should be in pretty good shape now, I revamped them all a while back
18:32:32 <nirik> so if say virthost01 is down, it will only alert about that, not the vm's running on it also
18:32:42 <nirik> or a router is down, etc.
18:33:10 <nirik> https://admin.fedoraproject.org/nagios/ is our main nagios
18:33:22 <nirik> and https://admin.fedoraproject.org/nagios-external/ is a smaller one we have at a secondary datacenter
18:33:43 <nirik> anyone should be able to login with their fedora account login/pass
18:34:29 <nirik> ok, any other sysadmin related stuff?
18:35:09 <nirik> #topic Upcoming Tasks/Items
18:35:09 <nirik> https://apps.fedoraproject.org/calendar/list/infrastructure/
18:35:13 <threebean> good stuff
18:35:20 <nirik> anything upcoming anyone would like to schedule or note?
18:35:34 * pingou has none
18:35:44 <threebean> heh, kinda like a broken record... but we have the bodhi2 FAD upcoming in June
18:35:47 <nirik> I'd like to note that I will be GONE from saturday until thursday (back late wed night)
18:35:48 <smooge> just more hardware to install
18:35:54 <threebean> https://fedoraproject.org/wiki/FAD_Bodhi2_Taskotron_2014
18:36:02 <threebean> nothing new to note.. just reminding that its happening.
18:36:06 <nirik> #info nirik will be out saturday to next thursday.
18:36:11 <pingou> oh, during the meeting I pushed the change the 'Manage ACL' page: see http://209.132.184.188/package/guake/acl/commit/ (replace guake by a package you own)
18:36:22 <nirik> if you need me for anything before then, please find me today/tomorrow. ;)
18:36:43 <threebean> nirik: if you have specific things you need taken care of while you're gone, feel free to tell us either here or offline.
18:36:43 <smooge> during that time threebean will technically be in charge but in an undisclosed bunker. I will be available as Alexander Haig
18:37:16 * threebean promotes smooge
18:37:27 <nirik> can do. ;) I will have cell saturday and wed, but won't even have that the rest of the time. Hurray wilderness! :)
18:37:29 <pingou> I'm out from Sunday to Saturday next week
18:37:45 <bwood09> I'm probably going to be out for the same ^
18:37:54 <bwood09> Supposed to be going to Georgia
18:38:00 <pingou> I'll likely check on emails once in a while, but I'll try to stay away from irc :)
18:38:06 <nirik> popular vacation week. ;)
18:38:35 <smooge> nirik, threebean with you and pingou gone.. should we go to warm slush for changes?
18:39:12 <nirik> well, I'd say to be carefull sure... dunno if we need anything formal
18:39:16 <smooge> eg changes need at least a IRC +1 from someone else who can review it before commit/push
18:39:25 <nirik> since I won't have phone, I don't care... can't bother me. ;)
18:39:45 <threebean> you'll come back and we'll have a chef setup in place
18:39:53 <nirik> :)
18:40:02 * smooge goes to find his contacts in the Smoke Jumpers to see if they can fix that
18:40:06 <nirik> anyhow...
18:40:09 <nirik> #topic Open Floor
18:40:20 <nirik> anyone have anything for open floor? questions? comments?
18:40:41 <mattdm> nirik yeah I have one
18:40:43 <ootbro> I was able to get into nagios with my regular FP id
18:40:51 <mattdm> just filed https://fedorahosted.org/fedora-infrastructure/ticket/4350
18:40:51 <henderbj> i have one... a moment please
18:41:13 <henderbj> can apprentice members ssh to lockbox01?
18:41:17 <mattdm> colin walters requests a slightly-less ad hoc place to do ostree experimetnation fedora
18:41:39 <Daredel> hi, i get late for the New folks introductions and Apprentice tasks, i'm new and really exited about contributing to the community
18:41:46 <nirik> mattdm: hum, ok, I already promised walters one of our old virthosts once we move a new one in... is that for this same thing or something different?
18:42:04 <mattdm> nirik I *think* this is the same thing? maybe he is just getting antsy? :)
18:42:15 <nirik> henderbj: absolutely. See the ssh access link off the apprentice page. ;)
18:42:31 <nirik> Daredel: welcome! are you interested in sysadmin or application devel or both?
18:42:32 * mattdm did not know about that. or forgot if i did
18:43:03 <nirik> mattdm: ok. We have been backloged by heartbleed, then virthosts getting shipped the wrong place, then storage hell, etc. We are getting there tho.
18:43:11 <Daredel> i think both, but most of all devel
18:43:21 <mattdm> nirik ok I will relay that.
18:43:34 <nirik> smooge: did we decide what 2 old virthosts we were going to save? one for ostree the other for cloud lockbox?
18:44:01 <nirik> Daredel: great. See me after the meeting in #fedora-admin and I can help set you up with the apprentice group... #fedora-apps can help with application devel stuff. :)
18:44:13 <henderbj> I read it before.. but from bastion01 i get: Permission denied (publickey).
18:44:14 <Daredel> ok thanks :D
18:44:42 <bwood09> henderbj, how are you authenticating? And did you upload your public key to FAS?
18:44:54 <nirik> henderbj: can assist you after the meeting in #fedora-admin, but you should be doing 'ssh lockbox01.phx2.fedoraproject.org' from your home machine, it should use bastion01 as a proxy...
18:45:02 <smooge> nirik, I have not yet. I keep doing so and then forgetting which 2 I saved and start over
18:45:19 <nirik> smooge: yeah, we should see if we can hurry on one for ostree stuff.
18:45:43 <nirik> mattdm: we will try and hurry it along.
18:45:51 <mattdm> nirik thanks. :)
18:45:58 <mattdm> nirik is the previous ticket https://fedorahosted.org/fedora-infrastructure/ticket/4200 ?
18:46:09 <henderbj> I created the ~/.ssh/config file, then ssh to bastion01 , and from there, i did: ssh lockbox01.phx2.fedoraproject.org, and get as response: Permission denied (publickey).
18:46:16 <nirik> mattdm: could be yeah
18:47:04 <nirik> henderbj: you can't do it that way.. ;) bastion doesn't (and shouldn't) have your config and keys on it... you should run the 'ssh lockbox01.phx2.fedoraproject.org' from your home machine. The config takes care of the proxying part.
18:47:17 <henderbj> ok... i will trying to connect after the meeting
18:47:50 <nirik> we will get it working. :)
18:47:51 <smooge> mattdm, we are having to do a lot of yak shaving to get these boxes available. it may be mid may
18:49:07 <nirik> anyhow, we will get there as soon as we can.
18:49:23 <nirik> smooge: lets both go over them and come up with a pair...
18:50:26 <nirik> ok, anything else? or shall we call it a meeting?
18:51:05 <henderbj> well, about easyfix tickets
18:51:36 <nirik> sure, shoot...
18:51:37 <henderbj> are those easyfix tickets from 2011-2012 really need any work done?
18:51:50 <nirik> if they are still open, yes.
18:52:17 <nirik> they may have been things that weren't urgent enough for someone else to do...
18:52:41 <threebean> henderbj: if you have one or two in particular in mind, drop a link to them in channel
18:52:56 <nirik> if they don't need anything anymore, we can close them. ;)
18:53:06 <threebean> otherwise, I can only guess...
18:53:49 <henderbj> i reviewed this one: https://fedorahosted.org/fedora-infrastructure/ticket/3617
18:54:50 <threebean> yeah, I'm pretty sure that one still needs work
18:54:50 <henderbj> After my "quick" review, i didn0t find anything to do... i left it because it was too old ;)
18:54:51 <nirik> yeah, probibly needs the current output added, but I can do that if you want to work on it. ;)
18:56:00 <nirik> ok, lets all move over to #fedora-admin, #fedora-noc and #fedora-apps...
18:56:04 <henderbj> Ok... if any question i will post it on the ticket to get going to close it
18:56:15 <nirik> thanks for coming everyone. And welcome again to all the new folks. ;)
18:56:19 <nirik> henderbj: sounds great.
18:56:22 <nirik> #endmeeting
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 836 bytes
Desc: not available
URL: <http://lists.fedoraproject.org/pipermail/infrastructure/attachments/20140501/573bab7e/attachment-0001.sig>