============================================ #fedora-meeting: Infrastructure (2014-08-14) ============================================
Meeting started by nirik at 17:59:55 UTC. The full logs are available at http://meetbot.fedoraproject.org/fedora-meeting/2014-08-14/infrastructure.20... .
Meeting summary --------------- * aloha (nirik, 17:59:55) * LINK: https://fedoraproject.org/wiki/User:Adionditsak (adionditsak, 18:01:03) * LINK: https://www.linkedin.com/pub/anders-aarvik/26/133/47b (adionditsak, 18:01:05)
* New folks introductions and Apprentice tasks (nirik, 18:02:51)
* Applications status / discussion (nirik, 18:10:09) * copr update to seperate queues per arch, hopefully arm support before too long. (nirik, 18:11:41) * more optimizing on FMN ongoing (nirik, 18:11:54) * some progress on new private openstack cloud. Hopefully more next week. (nirik, 18:18:19) * oddshocks and nirik to get fedimg uploading to old infrastructure private cloud. (nirik, 18:21:00) * LINK: http://jenkins.cloud.fedoraproject.org/job/fedora-rube/ (threebean, 18:21:20) * more rube tests needed (nirik, 18:22:28)
* Sysadmin status / discussion (nirik, 18:23:05) * got fasClient fedmsg enabled and running from lockbox01. (nirik, 18:23:25) * new retrace machines being provisioned now. (nirik, 18:24:38) * qa09/virthost-comm03 next to be provisioned. (nirik, 18:24:52)
* nagios/alerts recap (nirik, 18:26:06) * LINK: http://tinyurl.com/ms5u5qm (nirik, 18:28:05)
* Many many thanks to Toshio (nirik, 18:33:00)
* Upcoming Tasks/Items (nirik, 18:35:14) * LINK: https://apps.fedoraproject.org/calendar/list/infrastructure/ (nirik, 18:35:15)
* Open Floor (nirik, 18:38:59)
Meeting ended at 18:45:06 UTC.
Action Items ------------
Action Items, by person ----------------------- * **UNASSIGNED** * (none)
People Present (lines said) --------------------------- * nirik (88) * threebean (16) * adionditsak (15) * lanica (9) * mapyth (7) * oddshocks (7) * mirek-hm (7) * lmacken (6) * zodbot (5) * danielbruno (5) * bochecha (4) * ootbro (3) * adimania_ (3) * abadger1999 (2) * nmzaheer (2) * lbazan (1) * relrod (1) * smooge (1) * mdomsch (0) * puiterwijk (0) * pingou (0) * dgilmore (0) -- 17:59:55 <nirik> #startmeeting Infrastructure (2014-08-14) 17:59:55 <zodbot> Meeting started Thu Aug 14 17:59:55 2014 UTC. The chair is nirik. Information about MeetBot at http://wiki.debian.org/MeetBot. 17:59:55 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic. 17:59:55 <nirik> #meetingname infrastructure 17:59:55 <nirik> #topic aloha 17:59:55 <nirik> #chair smooge relrod nirik abadger1999 lmacken dgilmore mdomsch threebean pingou puiterwijk 17:59:55 <zodbot> The meeting name has been set to 'infrastructure' 17:59:55 <zodbot> Current chairs: abadger1999 dgilmore lmacken mdomsch nirik pingou puiterwijk relrod smooge threebean 18:00:03 <smooge> aloha 18:00:07 <nirik> :) 18:00:08 <abadger1999> Greetings 18:00:09 <bochecha> hi everyone 18:00:20 * lmacken 18:00:28 <adionditsak> Greetings - i will copy paste a small introduction in 1 minute :-) 18:00:59 <adionditsak> I am a very new (1 week old) contributor/ambassador of Fedora, and i look forward to get started for real. I just joined the FI apprentice group, 1 hour ago, and i am excited to dig into stuff. My name is Anders Aarvik, i am from Denmark, and i am a young IT guy with 10 years of Linux usage, lately professionally as a sysadmin (developer earlier). Tell me if you would like to know something else about me. L 18:01:00 <adionditsak> ook forward to talk with you in the future. 18:01:00 <adionditsak> Sincerely, 18:01:01 <adionditsak> Anders Aarvik (adionditsak) 18:01:03 <adionditsak> https://fedoraproject.org/wiki/User:Adionditsak 18:01:05 <adionditsak> https://www.linkedin.com/pub/anders-aarvik/26/133/47b 18:01:20 * lanica is here for the infrastructure meeting. 18:01:46 * adimania_ is here 18:01:53 * mirek-hm is here 18:02:03 * danielbruno here 18:02:07 <ootbro> here 18:02:09 <oddshocks> here 18:02:10 * nmzaheer is here 18:02:12 * relrod here 18:02:28 <nirik> adionditsak: welcome. ;) 18:02:51 <nirik> #topic New folks introductions and Apprentice tasks 18:03:14 <ootbro> From the apprentice front .... the second diagram is now ready for review at http://www.jt-sw.com/docs/index.nsf/docs/fpo-proxy 18:03:24 <nirik> adionditsak: were you more interested in sysadmin tasks or development 18:03:31 <adionditsak> Thanks :-) Inserted a small intro further up 18:03:48 <adionditsak> Mostly system administration, but maybe a bit of development in the future aswell. 18:03:57 <danielbruno> Hello guys, I contribute to Fedora for 5 ~ 6 years and I intend to contirbute on infra group now 18:04:16 <nirik> ootbro: looks accurate to me. ;) 18:04:33 <nirik> adionditsak: cool. We have #fedora-apps for our application developers too. 18:04:35 <ootbro> thanks. this version has the extra notes that weren't in the early preview I e-mailed you last week. 18:04:38 <nirik> danielbruno: welcome. :) 18:04:49 <danielbruno> nirik, thank you :) 18:05:08 <threebean> (here) 18:05:23 <adionditsak> Great :-) I will hang out in both channels then. What is the fedora-ops for? Saw it under administration aswell. 18:05:54 <bochecha> adionditsak, operators of IRC channels 18:05:58 <nirik> adionditsak: thats unrelated to infrastructure. thats for irc channel operators. 18:06:10 <adionditsak> Ah. 18:06:12 <danielbruno> more info about me: https://fedoraproject.org/wiki/User:Dbruno and http://danielbruno.eti.br 18:06:16 <nirik> adionditsak: we also have #fedora-noc for sysadmin type things. It also gets alerts and outage info, etc. 18:06:37 <adionditsak> Great - i will make a note on this 18:07:02 <nirik> danielbruno: you were interested in sysadmin tasks? or development? or both? :) 18:07:18 <nirik> danielbruno: I can add you to our apprentice group to look around after the meeting... see me in #fedora-admin. 18:07:39 <danielbruno> nirik, sysadmin tasks 18:08:07 <nirik> ok, great. 18:08:41 <nirik> welcome again to all the new folks. Do hang out in our channels and ask questions anytime... 18:08:58 <adionditsak> Thank you :-) 18:09:07 <nirik> any other new folks? 18:09:51 <nmzaheer> Hi guys..i am just observing for now 18:10:06 <nirik> nmzaheer: ok. :) Feel free. 18:10:09 <nirik> #topic Applications status / discussion 18:10:26 <nirik> Any application development news/plans/discussion this week? 18:10:47 <mirek-hm> I upgraded Copr to new version. Main feature is that we know have queues separate for each architecture, so Copr code is completly ready for new arches. And adding those ARM is just matter of creating playbook for spinning up and terminating (or reprovision) a machine. 18:10:53 <nirik> lmacken: what all do we need to do to get bodhi2 in staging? shouldn't be too far should it? 18:10:58 <mirek-hm> s/know/now/ 18:11:02 <threebean> I'm working on some further optimizations to fmn, https://apps.fedoraproject.org/notifications 18:11:04 <nirik> mirek-hm: awesome! 18:11:12 <threebean> it's getting faster but still not yet where I'd like it to be. 18:11:16 <lmacken> nirik: we'll need to port the ansible playbooks, since right now bodhi.dev is setup by hand in a virtualenv with paster 18:11:34 <lmacken> bodhi.dev is also dead in the water right now with an expired koji cert, and I haven't figured out how to fix it 18:11:41 <nirik> #info copr update to seperate queues per arch, hopefully arm support before too long. 18:11:54 <nirik> #info more optimizing on FMN ongoing 18:12:21 <nirik> lmacken: ok. Happy to help with that. I think we can redo bodhi01.stg with bodhi2 as soon as it's ready for it. 18:12:58 <lmacken> nirik: cool, sounds good. we're almost ready to dive into that port, I just want to polish up the masher service and systemdify it 18:13:07 <nirik> fair enough. 18:13:18 <lmacken> or I guess we can just piggy-back on the moksha-hub service instead 18:13:19 <nirik> mirek-hm: did you get any further with cloud networking? 18:14:11 <mirek-hm> nirik: nope. I had to catch up my normal work after Flock (rebuilding SCLs for el7) 18:14:17 <mirek-hm> hopefuly next week 18:14:26 <nirik> ok, no worries. 18:14:30 <mapyth> Hello Everybody! 18:14:30 <mapyth> I would like to update on the status of bugspad. As decided, I was focussing on making the currently implemented 18:14:30 <mapyth> features more robust, and as bug free as possible. In the mean time, my mentor has started doing a parallel work 18:14:31 <mapyth> in python. This would allow to get a performance metric, for Python vs Golang. :) 18:14:43 <nirik> happily it seems like the old cloud has been reasonably stable for copr recently. 18:14:51 <mirek-hm> yes 18:15:23 <threebean> mapyth: cool :) 18:15:23 <nirik> mapyth: cool. I still need to arrange a demo/talk with your mentor. Hopefully tomorrow or early next week. 18:15:36 <threebean> I forget if we've asked already.. but is there a demo instance of bugspad somewhere that we could play with? 18:15:52 <nirik> threebean: I think there is in the cloud... mapyth ? 18:16:05 <mapyth> nirik, threebean Please please check out, http://209.132.184.128/ (my instance) 18:16:28 <mirek-hm> nirik: last day on Flock I discovered https://github.com/openstack-ansible/openstack-ansible-modules , which should easy writing our playbook for our OS instance. asamalik tested it yesterday and it works fine. jruzicka promised that he may try to package it for Fedora 18:16:41 <mapyth> nirik: Currently only basic features are done, and I am testing it for robustness. 18:17:18 <nirik> mirek-hm: oh nice. I think I found that a while back and it wasn't active, but looks like it has been recently now. 18:17:26 <nirik> mapyth: great 18:17:40 <mapyth> nirik: :) 18:18:19 <nirik> #info some progress on new private openstack cloud. Hopefully more next week. 18:18:26 <nirik> any other application type news? 18:18:39 <threebean> mapyth: neat :) 18:18:53 * nirik was hoping abompard would be here and we could talk hyperkitty plans, but will catch him when he's around. 18:19:07 <oddshocks> nirik: Is our internal openstack stuff ready to accept image uploads from Fedimg? 18:19:29 <nirik> oddshocks: we could set that up on the old one... the new one is still in a state of flux. 18:19:37 <oddshocks> cool. 18:19:38 <nirik> oddshocks: what all info do you need? a login/pass? 18:19:52 <oddshocks> nirik: yeah, whatever creds. i can check specifically right now 18:20:16 <nirik> ok. I can get you those... ideally it would upload, but purge old ones or else the list will get very large. 18:20:41 <oddshocks> nirik: yep just looks like auth user and auth pass for openstack clouds 18:20:52 <oddshocks> nirik: right. cool. 18:21:00 <nirik> #info oddshocks and nirik to get fedimg uploading to old infrastructure private cloud. 18:21:04 <oddshocks> I don't know how openstack does image registration but I'll figure it out 18:21:10 <threebean> oh, I almost forgot. we nearly have rube running in jenkins. 18:21:17 <nirik> excellent. ;) 18:21:18 <threebean> or, it's running.. but still failing on a test or two 18:21:20 <threebean> http://jenkins.cloud.fedoraproject.org/job/fedora-rube/ 18:21:34 <nirik> what triggers runs? just timed? 18:21:46 <threebean> so, for people who don't know, rube is an automated test suite for our staging webapps. it opens up firefox with selenium and tries to click through everything. 18:22:04 <threebean> getting people to contribute more tests would be really helpful -> https://github.com/fedora-infra/rube/ 18:22:11 <threebean> nirik: yeah, just timed at the moment. 18:22:28 <nirik> #info more rube tests needed 18:23:05 <nirik> #topic Sysadmin status / discussion 18:23:12 <nirik> ok, on the sysadmin side... 18:23:13 <lmacken> threebean: can you hook me up with admin privs for it? 18:23:19 * threebean nods 18:23:25 <nirik> #info got fasClient fedmsg enabled and running from lockbox01. 18:23:43 <threebean> big win there :) 18:23:50 <nirik> Hopefully this will stop all those fasClient timeouts from when things were running from cron. 18:24:03 <nirik> one nitpick... fedmsg shows me as running the playbook. :) 18:24:19 <nirik> or whatever user restarted the hub on lockbox01 last. 18:24:38 <nirik> #info new retrace machines being provisioned now. 18:24:52 <nirik> #info qa09/virthost-comm03 next to be provisioned. 18:25:34 <nirik> I'm not sure I want to start too much on the sysadmin side this week, as we might go into freeze next tuesday. 18:25:54 <nirik> but the usual porting of stuff from puppet to ansible and rhel6 to rhel7 is still ongoing. 18:26:06 <nirik> #topic nagios/alerts recap 18:26:16 * adimania_ needs to finish a couple of tickets before freeze 18:27:01 <nirik> I also closed out some older tickets... 18:28:05 <nirik> http://tinyurl.com/ms5u5qm 18:28:34 <nirik> hum, that might not be the right report. 18:29:41 <nirik> .tiny https://admin.fedoraproject.org/nagios/cgi-bin//summary.cgi?report=1&dis... 18:29:46 <zodbot> nirik: http://tinyurl.com/ng2ouvc 18:30:05 <nirik> so, we have gotten a lot of packet loss on vpns, etc... 18:30:11 <nirik> hopefully thats cleared up 18:30:27 <nirik> busgateway01 is still producing a lot of alerts. 18:31:06 <nirik> threebean: can we bump the time on that busgateway monitoring some more? 18:31:36 * threebean nods 18:32:05 <threebean> not sure why its just nauncier that keeps bugging out there.. :/ 18:33:00 <nirik> #topic Many many thanks to Toshio 18:33:25 <nirik> As some of you know, abadger1999 is stepping back from Fedora (hopefully only for a little while) 18:33:39 <nirik> Just thought I would mention it here and say many thanks to him for all his work. ;) 18:33:42 <abadger1999> Thanks nirik :-) 18:34:34 <nirik> If theres anything fedora related you want from him, catch him soon. :) 18:35:14 <nirik> #topic Upcoming Tasks/Items 18:35:15 <nirik> https://apps.fedoraproject.org/calendar/list/infrastructure/ 18:35:27 <nirik> Anything upcoming anyone would like to note or schedule? 18:35:43 <nirik> we may be going into freeze next tuesday. We will see if we have a test compose by then. 18:36:15 * lbazan here 18:36:45 <bochecha> nirik, does staging get frozen as well? 18:36:51 <nirik> also, there is another mass rebuild starting later today... so even though we aren't in freeze, but carefull about any changes that could impact builders 18:37:05 <nirik> bochecha: nope. Staging never freezes. ;) 18:37:43 <bochecha> alright, just wondering if we'd be able to move forward with puppet->ansible migrations during the freeze :) 18:37:47 <nirik> there's a script in ansible repo that lists whats frozen and not 18:37:55 <nirik> yep. we should be able to no problem. 18:38:59 <nirik> #topic Open Floor 18:39:07 <nirik> ok, any items for open floor? questions, comments? 18:39:10 <adimania_> if I can recall, we might not be able to run playbooks on frozen stuff. 18:39:26 <lanica> I've been working on trying to recreate the openid issue as most recently seen on noc01 ... Ticket #4479 https://fedorahosted.org/fedora-infrastructure/ticket/4479 18:39:51 <lanica> I have yet to recreate it in my lab. 18:39:52 <nirik> adimania_: right. When a host is frozen, any changes to it need to be sent to the list first and get at least 2 +1's from releng or sysadmin-main before being applied. 18:40:02 <nirik> lanica: yeah, it's a weird one. ;( 18:40:03 <adionditsak> Do we usually use Puppet or Ansible? I master Puppet, but i am interested in Ansible. Have seen you talk about Ansible, but just logged into a server with puppet agent. 18:40:16 <lanica> I'm not sure how long it takes to trigger or if perhaps it is an issue that involves the front proxy servers 18:40:18 <nirik> adionditsak: both right now. We are moving from puppet to ansible. 18:40:28 <nirik> some hosts are still in puppet 18:40:34 <lanica> If anyone has more data or ideas...please add it to the ticket! 18:40:56 <adionditsak> Ok cool :-) 18:41:22 <lanica> I also do not have access to noc01 to try to collect more information, which adds a degree of difficulty. If there are other servers that I can be given access to that have had the issue in the past...that would be helpful. 18:41:51 <nirik> lanica: we don't actually have any other hosts using mod_auth_openid. ;( This was the first one. 18:42:10 <lanica> Ok, I somehow missed that 18:42:24 <nirik> lanica: you should be able to login to log01 tho, and look under /var/log/hosts/noc01*/ and see if there's any more info in logs. ;( 18:43:04 <nirik> doubt there will be, but possible I guess. ;) 18:43:10 <lanica> Ok, thanks. puiterwijk did collect the basic error messages that seemed most relevant and sent them to me and I added it to the ticket. I'll dig through there some more in case there are some hints. 18:43:29 <nirik> thanks for looking into it 18:43:35 <lanica> NP 18:44:15 <nirik> alright. Anything else? or shall we call it a meeting? 18:45:03 <nirik> Thanks for coming everyone. Do follow up over in #fedora-admin, #fedora-apps and #fedora-noc. ;) 18:45:06 <nirik> #endmeeting
meetingminutes@lists.fedoraproject.org
-
Kevin Fenzi