The infrastructure team will be having it's weekly meeting tomorrow, 2014-05-01 at 18:00 UTC in #fedora-meeting on the freenode network.
Suggested topics:
#topic New folks introductions and Apprentice tasks.
If any new folks want to give a quick one line bio or any apprentices would like to ask general questions, they can do so in this part of the meeting. Don't be shy!
#topic Applications status / discussion
Check in on status of our applications: pkgdb, fas, bodhi, koji, community, voting, tagger, packager, dpsearch, etc. If there's new releases, bugs we need to work around or things to note.
#topic Sysadmin status / discussion
Here we talk about sysadmin related happenings from the previous week, or things that are upcoming.
#topic nagios/alerts recap
Here we go over the last weeks alerts and see if we can find ways to make it so they don't happen again.
#topic Upcoming Tasks/Items
https://apps.fedoraproject.org/calendar/list/infrastructure/
#topic Open Floor
Submit your agenda items, as tickets in the trac instance and send a note replying to this thread.
More info here:
https://fedoraproject.org/wiki/Infrastructure/Meetings#Meetings
Thanks
kevin
I am not new to the list, I have been liking for some time but not too active. My interests are mainly in monitoring, I have looked through the tickets marked for easy completion but nothing really concerning monitoring. I am fairly good with python and would like to learn more about monitoring or deployment and automated testing so if anyone needs of knows of any specific monitoring type projects I would be happy to assist where I can and I will continue to keep an eye on the tickets for anything I feel I can complete reasonably effectively. NOTICE:
This message, and any attachments, contain(s) information that may be confidential or protected by privilege from disclosure and is intended only for the individual or entity named above. No one else may disclose, copy, distribute or use the contents of this message for any purpose. Its unauthorized use, dissemination or duplication is strictly prohibited and may be unlawful. If you receive this message in error or you otherwise are not an authorized recipient, please immediately delete the message and any attachments and notify the sender.
============================================ #fedora-meeting: Infrastructure (2014-05-01) ============================================
Meeting started by nirik at 18:00:03 UTC. The full logs are available at http://meetbot.fedoraproject.org/fedora-meeting/2014-05-01/infrastructure.20... .
Meeting summary --------------- * greetings starfighters (nirik, 18:00:03)
* New folks introductions and Apprentice tasks (nirik, 18:02:13) * LINK: http://fedoraproject.com/easyfix (webpigeon, 18:07:12)
* Applications status / discussion (nirik, 18:08:02) * some work on a flask re-write of mirrormanager ongoing (nirik, 18:14:14) * ui work on pkgdb2 ongoing (nirik, 18:14:24) * hyperkitty came up in the news a few times this week in slashdot and lwn, pointing to our stg instance. (nirik, 18:14:48) * LINK: https://fedorahosted.org/fedora-infrastructure/ticket/4044 (threebean, 18:17:15) * LINK: http://threebean.org/blog/fedmsg-collectd-ng/ (threebean, 18:17:19)
* Sysadmin status / discussion (nirik, 18:18:47) * storage move had soe issues, but hopefully we have worked them out now. (nirik, 18:21:14) * new bvirthosts are on-line (nirik, 18:21:20) * LINK: https://admin.fedoraproject.org/nagios/ is our main nagios (nirik, 18:33:10)
* Upcoming Tasks/Items (nirik, 18:35:09) * LINK: https://apps.fedoraproject.org/calendar/list/infrastructure/ (nirik, 18:35:09) * LINK: https://fedoraproject.org/wiki/FAD_Bodhi2_Taskotron_2014 (threebean, 18:35:54) * nirik will be out saturday to next thursday. (nirik, 18:36:06)
* Open Floor (nirik, 18:40:09)
Meeting ended at 18:56:22 UTC.
Action Items ------------
Action Items, by person ----------------------- * **UNASSIGNED** * (none)
People Present (lines said) --------------------------- * nirik (108) * threebean (23) * henderbj (15) * pingou (15) * bwood09 (13) * smooge (13) * mattdm (9) * zodbot_ (7) * ootbro (6) * danrimal (5) * Daredel (4) * danofsatx-work (3) * nj0y (3) * relrod (2) * webpigeon (2) * janeznemanic (1) * mpduty (1) * lmacken (1) * ghostalker (1) * danofsatx|kvirc (1) * oddshocks (1) * dgilmore (1) * mdomsch (0) * puiterwijk (0) * abadger1999 (0) -- 18:00:03 <nirik> #startmeeting Infrastructure (2014-05-01) 18:00:03 <zodbot_> Meeting started Thu May 1 18:00:03 2014 UTC. The chair is nirik. Information about MeetBot at http://wiki.debian.org/MeetBot. 18:00:03 <zodbot_> Useful Commands: #action #agreed #halp #info #idea #link #topic. 18:00:03 <nirik> #meetingname infrastructure 18:00:03 <nirik> #topic greetings starfighters 18:00:03 <nirik> #chair smooge relrod nirik abadger1999 lmacken dgilmore mdomsch threebean pingou puiterwijk 18:00:03 <zodbot_> The meeting name has been set to 'infrastructure' 18:00:03 <zodbot_> Current chairs: abadger1999 dgilmore lmacken mdomsch nirik pingou puiterwijk relrod smooge threebean 18:00:10 * relrod waves 18:00:26 * pingou 18:00:31 * webpigeon waves 18:00:48 <janeznemanic> hi 18:01:04 * lmacken 18:01:33 <danofsatx-work> I'm here, but if Konversation wigs out on me again 18:01:47 <danofsatx|kvirc> I'll be here instead 18:02:00 <nirik> :) 18:02:06 <nirik> ok, lets go ahead and get started.... 18:02:08 * threebean is here 18:02:13 <nirik> #topic New folks introductions and Apprentice tasks 18:02:27 <danrimal> hi, i am new here 18:02:34 <nirik> any new folks want to do a quick one line introduction of themselves? or apprentices with questions or comments? 18:02:46 <danrimal> yes, sure 18:02:48 <danrimal> I am sysadmin, my job is high load and high availability application as web, mail and databases services as well as bgp and ospf networking 18:03:00 <ootbro> I'll also jump in as one of the newbies 18:03:02 <danrimal> and i am interested in sysadmin things, if available 18:03:31 <ootbro> I've been using Linux for years and now want to contribute here. I think the testing FIG is my best starting point, with an eye toward sysadmin-main (eventually) 18:03:38 <nirik> danrimal: welcome! sure thing... see me in #fedora-admin after the meeting and I can get you setup in our apprentice program... 18:03:52 <nirik> ootbro: welcome again. ;) 18:03:59 <ootbro> thanks 18:04:22 <nj0y> As I introduced myself in the mailing, i'm a sysadmin/engineer from switzerland and very interested getting in charge here. And i search some fig to join. I think also fig testing or fig web is a good place for me. 18:05:00 <nirik> nj0y: welcome also. ;) 18:05:16 * ghostalker is here 18:05:17 <nj0y> thanks, glad i'm here. 18:05:30 <nirik> always good to have new folks around... do chime in with questions or comments anytime... 18:05:31 <mpduty> .fasinfo mohanprakash 18:05:32 <zodbot_> mpduty: User: mohanprakash, Name: Mohan Prakash, email: mpduty@gmail.com, Creation: 2013-12-27, IRC Nick: mpduty, Timezone: Asia/Kolkata, Locale: en, GPG key ID: 0xAF620142, Status: active 18:05:36 <zodbot_> mpduty: Unapproved Groups: l10n-editor l10n-commits marketing 18:05:39 <zodbot_> mpduty: Approved Groups: fi-apprentice cvsl10n cla_done cla_fpca 18:06:26 <nirik> I can assist anyone after the meeting over in #fedora-admin who wants to join our apprentice group or would like to be pointed at easyfix tickets, etc. :) 18:06:31 <nirik> Welcome again everyone! 18:06:52 <ootbro> many thanks.... I could use the help in getting started 18:07:01 <nj0y> me too. 18:07:08 <danrimal> ok, thanks 18:07:12 <webpigeon> http://fedoraproject.com/easyfix 18:07:23 * mattdm is lurking 18:07:35 * bwood09 is here 18:07:46 <nirik> see also http://fedoraproject.org/wiki/Infrastructure/GettingStarted and https://fedoraproject.org/wiki/Infrastructure_Apprentice 18:08:02 <nirik> #topic Applications status / discussion 18:08:14 <nirik> any application side news from the previous week or upcoming? 18:08:27 * pingou has done a good chunk of work on mirrormanager2 this week 18:08:42 <pingou> and over the last two days I am on the re-design of some the page of pkgdb2 18:08:48 * Daredel is late 18:08:48 <pingou> including http://209.132.184.188/package/R-DBI/ 18:08:53 <nirik> pingou: this is the flask re-write of it? or is it tg2? or ? 18:08:57 <pingou> nirik: flask 18:09:10 <pingou> but I only worked on the UI 18:09:14 <nirik> have you contacted mdomsch any on it? (I know he's not been around) 18:09:24 <threebean> pingou: that looks *much* nicer. 18:09:43 <relrod> yeah, that looks nice :) 18:09:56 * nirik is waiting for load. ;) 18:10:08 <nirik> I bet a bunch of us clicked it at the same time. 18:10:35 <danofsatx-work> i waited a bit, came up fine for me ;) 18:10:48 <nirik> or... it could be my pesky wireless. ;( 18:10:53 <danofsatx-work> which is amazing, considering what I've been fighting with locally...... 18:11:33 <nirik> pingou: how is 'package administrator' determined? 18:11:41 <pingou> threebean: designed by mizmo, I can't beat that :) 18:11:47 * oddshocks here late, in lecture as usual 18:11:53 <pingou> nirik: Contacts are the POC, Admins are the users with approveacls 18:12:56 <nirik> pingou: ok, so anyone with any approveacls? 18:13:20 <pingou> yes 18:13:36 <pingou> nirik: or pending approveacls (then there is a (?) icon next to them) 18:13:51 <nirik> ok, cool. 18:14:14 <nirik> #info some work on a flask re-write of mirrormanager ongoing 18:14:24 <nirik> #info ui work on pkgdb2 ongoing 18:14:48 <nirik> #info hyperkitty came up in the news a few times this week in slashdot and lwn, pointing to our stg instance. 18:15:24 <threebean> are we any closer to cutting another list over? 18:15:41 <threebean> erm, by that I mean changing some existing lists from mailman2 to mailman3 18:15:54 <nirik> I sent abompard some issues and he was going to fix them up... then we were going to see where we were. 18:16:05 <nirik> hopefully soon tho. 18:16:09 <threebean> cool. newly queued stuff.. 18:16:11 * threebean nods 18:16:12 <nirik> I'd be happy to move the infra list. 18:16:20 <threebean> yeah, agreed 18:16:36 <nirik> there may also be some fixes from this recent press on it... 18:17:10 <threebean> unrelated, janeznemanic and I have been working on some fedmsg monitoring stuff and made some progress this week 18:17:15 <threebean> https://fedorahosted.org/fedora-infrastructure/ticket/4044 18:17:17 <nirik> excellent. 18:17:19 <threebean> http://threebean.org/blog/fedmsg-collectd-ng/ 18:17:34 <threebean> collectd is in place and fun. nagios checks coming soon. 18:17:56 * bwood09 starts reading the entirety of threebean.org 18:18:17 <nirik> any other application type news? or shall we move on to sysadmin? 18:18:47 <nirik> #topic Sysadmin status / discussion 18:19:02 <nirik> smooge got some of our new build virthosts up and running yesterday. 18:19:21 <smooge> yay 18:19:22 <nirik> Tuesday night we moved our backend storage from one netapp to another less loaded one... 18:19:28 <smooge> hahah 18:19:29 <nirik> but we have had some issues since then. ;( 18:19:34 <smooge> boo 18:19:58 <nirik> It's looking a lot like those issues are related to some virthosts having an emulated realtek network card instead of virtio. 18:20:07 <bwood09> nirik, will those new virthosts need to be added to nagios and the such? 18:20:09 <nirik> something in the move caused them to start dropping packets like mad 18:20:16 <nirik> bwood09: they will indeed. ;) 18:20:42 <nirik> I can file a ticket on them after the meeting. 18:20:45 <nirik> or smooge can 18:20:47 <bwood09> I'm going to go through today and tomorrow and take care of the nagios stuff, so if you drop a ticket for them in easyfix-- yeah 18:20:49 <bwood09> lol 18:20:51 <nirik> or really anyone can. ;) 18:21:14 <nirik> #info storage move had soe issues, but hopefully we have worked them out now. 18:21:20 <nirik> #info new bvirthosts are on-line 18:21:57 * smooge opens an easyfix ticket that someone can open an easyfix ticket to add monitoring for several hosts 18:22:11 <henderbj> Hello all... i am late... already read previous messages 18:22:19 <bwood09> Also, not sure if this is the place to do this, but I want to get on with the sysadmin-hosted group 18:22:26 * pingou gtg 18:22:33 <nirik> welcome henderbj 18:22:36 <smooge> bye pingou 18:22:38 <nirik> bye pingou 18:23:46 <nirik> bwood09: what sorts of things do you want to work on there? any tickets in specific? or just adding new projects and such? 18:24:27 <nirik> we had some plans in there we could look at again and see if you might want to work on them... 18:24:30 <bwood09> I'm going to look at the tickets today and see if there's anything I want to tackle. Recently, most of my experience has been git, svn, hg, and bzr so I figure I'd be a good fit 18:24:52 <nirik> sure thing. Let me (or any other hosted sponsor know) and we can see about helping you along. 18:24:58 <bwood09> Alrighty 18:25:10 <nirik> on nagios... we had a lot more alerts this last week I fear... 18:25:21 <nirik> 273 I see since last thursday. 18:25:45 <threebean> oo 18:25:47 <bwood09> What's the norm for those? 18:25:47 <nirik> the vast majority of which I think were related in one way or another to the storage move. 18:25:58 <threebean> this is a fun new routine. :p 18:26:01 <dgilmore> damn storage 18:26:32 * nirik looks back at the previous weeks 18:27:18 <nirik> 77 the week before 18:27:34 <bwood09> oh wow 18:28:18 <nirik> I'd like to reduce them as much as we can... I fear it will be impossible to make them 0 without making them not alert when theres problems users will notice. 18:28:35 <henderbj> well, that's normal... a lot of alarms when someone touches anything! 18:29:13 <nirik> well, most of the 'normal' ones are network related. We have a very wide network... so if our monitoring host can't reach some datacenter, it alerts. 18:30:07 <nirik> some of the ones this last week were also from a datacenter where we started to see packet loss... they were being hit by a DDOS. 18:30:12 <smooge> or where we aren't losing pings but they are taking close to a second to travel around the world 18:30:29 <nirik> anyhow, if anyone wants to dig thru nagios logs and propose changes that would be lovely. ;) 18:30:53 <henderbj> i will be testing nagios on my own testing machine 18:31:04 <nirik> we may be able to tune the network related ones down some, but not too far. 18:31:21 <henderbj> When get into it, i will pick something about nagios to help 18:31:42 <ootbro> question..... is there a way in nagios to not try a set of hosts if a "core" host is unreachable due to a network outage? 18:31:44 <nirik> henderbj: sounds great. Feel free to ask in #fedora-noc or #fedora-admin if you have any questions about our setup 18:31:54 <nirik> ootbro: yeah, it has dependencies... 18:32:01 <henderbj> Tnx, nirik, sure 18:32:09 <nirik> I think they should be in pretty good shape now, I revamped them all a while back 18:32:32 <nirik> so if say virthost01 is down, it will only alert about that, not the vm's running on it also 18:32:42 <nirik> or a router is down, etc. 18:33:10 <nirik> https://admin.fedoraproject.org/nagios/ is our main nagios 18:33:22 <nirik> and https://admin.fedoraproject.org/nagios-external/ is a smaller one we have at a secondary datacenter 18:33:43 <nirik> anyone should be able to login with their fedora account login/pass 18:34:29 <nirik> ok, any other sysadmin related stuff? 18:35:09 <nirik> #topic Upcoming Tasks/Items 18:35:09 <nirik> https://apps.fedoraproject.org/calendar/list/infrastructure/ 18:35:13 <threebean> good stuff 18:35:20 <nirik> anything upcoming anyone would like to schedule or note? 18:35:34 * pingou has none 18:35:44 <threebean> heh, kinda like a broken record... but we have the bodhi2 FAD upcoming in June 18:35:47 <nirik> I'd like to note that I will be GONE from saturday until thursday (back late wed night) 18:35:48 <smooge> just more hardware to install 18:35:54 <threebean> https://fedoraproject.org/wiki/FAD_Bodhi2_Taskotron_2014 18:36:02 <threebean> nothing new to note.. just reminding that its happening. 18:36:06 <nirik> #info nirik will be out saturday to next thursday. 18:36:11 <pingou> oh, during the meeting I pushed the change the 'Manage ACL' page: see http://209.132.184.188/package/guake/acl/commit/ (replace guake by a package you own) 18:36:22 <nirik> if you need me for anything before then, please find me today/tomorrow. ;) 18:36:43 <threebean> nirik: if you have specific things you need taken care of while you're gone, feel free to tell us either here or offline. 18:36:43 <smooge> during that time threebean will technically be in charge but in an undisclosed bunker. I will be available as Alexander Haig 18:37:16 * threebean promotes smooge 18:37:27 <nirik> can do. ;) I will have cell saturday and wed, but won't even have that the rest of the time. Hurray wilderness! :) 18:37:29 <pingou> I'm out from Sunday to Saturday next week 18:37:45 <bwood09> I'm probably going to be out for the same ^ 18:37:54 <bwood09> Supposed to be going to Georgia 18:38:00 <pingou> I'll likely check on emails once in a while, but I'll try to stay away from irc :) 18:38:06 <nirik> popular vacation week. ;) 18:38:35 <smooge> nirik, threebean with you and pingou gone.. should we go to warm slush for changes? 18:39:12 <nirik> well, I'd say to be carefull sure... dunno if we need anything formal 18:39:16 <smooge> eg changes need at least a IRC +1 from someone else who can review it before commit/push 18:39:25 <nirik> since I won't have phone, I don't care... can't bother me. ;) 18:39:45 <threebean> you'll come back and we'll have a chef setup in place 18:39:53 <nirik> :) 18:40:02 * smooge goes to find his contacts in the Smoke Jumpers to see if they can fix that 18:40:06 <nirik> anyhow... 18:40:09 <nirik> #topic Open Floor 18:40:20 <nirik> anyone have anything for open floor? questions? comments? 18:40:41 <mattdm> nirik yeah I have one 18:40:43 <ootbro> I was able to get into nagios with my regular FP id 18:40:51 <mattdm> just filed https://fedorahosted.org/fedora-infrastructure/ticket/4350 18:40:51 <henderbj> i have one... a moment please 18:41:13 <henderbj> can apprentice members ssh to lockbox01? 18:41:17 <mattdm> colin walters requests a slightly-less ad hoc place to do ostree experimetnation fedora 18:41:39 <Daredel> hi, i get late for the New folks introductions and Apprentice tasks, i'm new and really exited about contributing to the community 18:41:46 <nirik> mattdm: hum, ok, I already promised walters one of our old virthosts once we move a new one in... is that for this same thing or something different? 18:42:04 <mattdm> nirik I *think* this is the same thing? maybe he is just getting antsy? :) 18:42:15 <nirik> henderbj: absolutely. See the ssh access link off the apprentice page. ;) 18:42:31 <nirik> Daredel: welcome! are you interested in sysadmin or application devel or both? 18:42:32 * mattdm did not know about that. or forgot if i did 18:43:03 <nirik> mattdm: ok. We have been backloged by heartbleed, then virthosts getting shipped the wrong place, then storage hell, etc. We are getting there tho. 18:43:11 <Daredel> i think both, but most of all devel 18:43:21 <mattdm> nirik ok I will relay that. 18:43:34 <nirik> smooge: did we decide what 2 old virthosts we were going to save? one for ostree the other for cloud lockbox? 18:44:01 <nirik> Daredel: great. See me after the meeting in #fedora-admin and I can help set you up with the apprentice group... #fedora-apps can help with application devel stuff. :) 18:44:13 <henderbj> I read it before.. but from bastion01 i get: Permission denied (publickey). 18:44:14 <Daredel> ok thanks :D 18:44:42 <bwood09> henderbj, how are you authenticating? And did you upload your public key to FAS? 18:44:54 <nirik> henderbj: can assist you after the meeting in #fedora-admin, but you should be doing 'ssh lockbox01.phx2.fedoraproject.org' from your home machine, it should use bastion01 as a proxy... 18:45:02 <smooge> nirik, I have not yet. I keep doing so and then forgetting which 2 I saved and start over 18:45:19 <nirik> smooge: yeah, we should see if we can hurry on one for ostree stuff. 18:45:43 <nirik> mattdm: we will try and hurry it along. 18:45:51 <mattdm> nirik thanks. :) 18:45:58 <mattdm> nirik is the previous ticket https://fedorahosted.org/fedora-infrastructure/ticket/4200 ? 18:46:09 <henderbj> I created the ~/.ssh/config file, then ssh to bastion01 , and from there, i did: ssh lockbox01.phx2.fedoraproject.org, and get as response: Permission denied (publickey). 18:46:16 <nirik> mattdm: could be yeah 18:47:04 <nirik> henderbj: you can't do it that way.. ;) bastion doesn't (and shouldn't) have your config and keys on it... you should run the 'ssh lockbox01.phx2.fedoraproject.org' from your home machine. The config takes care of the proxying part. 18:47:17 <henderbj> ok... i will trying to connect after the meeting 18:47:50 <nirik> we will get it working. :) 18:47:51 <smooge> mattdm, we are having to do a lot of yak shaving to get these boxes available. it may be mid may 18:49:07 <nirik> anyhow, we will get there as soon as we can. 18:49:23 <nirik> smooge: lets both go over them and come up with a pair... 18:50:26 <nirik> ok, anything else? or shall we call it a meeting? 18:51:05 <henderbj> well, about easyfix tickets 18:51:36 <nirik> sure, shoot... 18:51:37 <henderbj> are those easyfix tickets from 2011-2012 really need any work done? 18:51:50 <nirik> if they are still open, yes. 18:52:17 <nirik> they may have been things that weren't urgent enough for someone else to do... 18:52:41 <threebean> henderbj: if you have one or two in particular in mind, drop a link to them in channel 18:52:56 <nirik> if they don't need anything anymore, we can close them. ;) 18:53:06 <threebean> otherwise, I can only guess... 18:53:49 <henderbj> i reviewed this one: https://fedorahosted.org/fedora-infrastructure/ticket/3617 18:54:50 <threebean> yeah, I'm pretty sure that one still needs work 18:54:50 <henderbj> After my "quick" review, i didn0t find anything to do... i left it because it was too old ;) 18:54:51 <nirik> yeah, probibly needs the current output added, but I can do that if you want to work on it. ;) 18:56:00 <nirik> ok, lets all move over to #fedora-admin, #fedora-noc and #fedora-apps... 18:56:04 <henderbj> Ok... if any question i will post it on the ticket to get going to close it 18:56:15 <nirik> thanks for coming everyone. And welcome again to all the new folks. ;) 18:56:19 <nirik> henderbj: sounds great. 18:56:22 <nirik> #endmeeting
infrastructure@lists.fedoraproject.org
-
Jones, Matt -
Kevin Fenzi