Meeting Log - 2010-03-25

Thu Mar 25 20:33:58 UTC 2010

20:00 < mmcgrath> #startmeeting Infrastructure
20:00 < zodbot> Meeting started Thu Mar 25 20:00:19 2010 UTC.  The chair is mmcgrath. Information about MeetBot at http://wiki.debian.org/MeetBot.
20:00 < mmcgrath> Who's here?
20:00  * skvidal is here
20:00 < zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
20:00 -!- zodbot changed the topic of #fedora-meeting to:  (Meeting topic: Infrastructure)
20:00  * lmacken 
20:00  * a-k is here
20:00 -!- djf_jeff [~jeff at 69.70.231.230] has joined #fedora-meeting
20:01 -!- rwmjones [~rjones at 94-30-104-162.xdsl.murphx.net] has quit Ping timeout: 248 seconds
20:01 -!- inode0 [~inode0 at fedora/inode0] has joined #fedora-meeting
20:02 < mmcgrath> #topic Fedora 13 beta
20:02 -!- zodbot changed the topic of #fedora-meeting to: Fedora 13 beta (Meeting topic: Infrastructure)
20:02 < mmcgrath> hah
20:02 < mmcgrath> Lets get started
20:02 < mmcgrath> https://fedorahosted.org/fedora-infrastructure/report/9
20:03 < mmcgrath> .ticket 2058
20:03 < zodbot> mmcgrath: #2058 (Verify Mirror Space) - Fedora Infrastructure - Trac - https://fedorahosted.org/fedora-infrastructure/ticket/2058
20:03 < mmcgrath> I'll get this one
20:03 < mmcgrath> .ticket 2059
20:03 < zodbot> mmcgrath: #2059 (Release Day ticket) - Fedora Infrastructure - Trac - https://fedorahosted.org/fedora-infrastructure/ticket/2059
20:03 < mmcgrath> This one's just a tracker ticket.  I'll take it too
20:03 < mmcgrath> .ticket 2060
20:03 < zodbot> mmcgrath: #2060 (Verify releng permissions) - Fedora Infrastructure - Trac - https://fedorahosted.org/fedora-infrastructure/ticket/2060
20:03 < mmcgrath> smooge: want to get that one again?
20:04 < mmcgrath> we'll come back to that.
20:04 < mmcgrath> .tiny 2061
20:04 < zodbot> mmcgrath: Error: '2061' is not a valid url.
20:05 < mmcgrath> .ticket 2061
20:05 < mmcgrath> sorry :/
20:05 < zodbot> mmcgrath: #2061 (MM redirects) - Fedora Infrastructure - Trac - https://fedorahosted.org/fedora-infrastructure/ticket/2061
20:05 < mmcgrath> mdomsch usually gets this one (I believe it's automated now and just requires verification)
20:05 < mmcgrath> .ticket 2062
20:05 < zodbot> mmcgrath: #2062 (Infrastructure Change Freeze) - Fedora Infrastructure - Trac - https://fedorahosted.org/fedora-infrastructure/ticket/2062
20:05 < mmcgrath> I'll get that, we are frozen.
20:05 < smooge> morning
20:05 < smooge> sorry got stuck on phone
20:05 < mmcgrath> smooge: hey, want to get 2060?
20:05 < smooge> yes
20:05 < mmcgrath> k
20:05 < mmcgrath> and the last ticket
20:06 < mmcgrath> .ticket 2063 doesn't need to be done until after the launch
20:06 < smooge> could I get 2058 .. I find them related
20:06 < zodbot> mmcgrath: Error: "2063 doesn't need to be done until after the launch" is not a valid integer.
20:06 < smooge> hehheeh
20:06 -!- djf_jeff [~jeff at 69.70.231.230] has quit Quit: I quit
20:06 < mmcgrath> zodbot: you are testing me!
20:06 < mmcgrath> ok, anyone have any questions or comments related to the release?
20:06 < mmcgrath> Oxf13: you around?
20:07 < Oxf13> I am
20:07 < mmcgrath> what are our odds of slipping at the moment?
20:07 < Oxf13> I'd put it at 50% chance
20:07 -!- biertie [~bert at fedora/biertie] has joined #fedora-meeting
20:07 < Oxf13> There is one blocker we're worried about, but we have a patch in hand, it just needs testing, then I can make the RC
20:07 < skvidal> mmcgrath: so - when it's a 50% chance of rain - you carry an umbrella :)
20:07 < mmcgrath> cool.
20:07 < Oxf13> we have a compressed amount of time to test the RC
20:08 < Oxf13> and not really any time to fix anything that's wrong with the RC and validate a second RC before the go / no go time
20:08 < mmcgrath> Oxf13: I'll work with you later today or tomorrow to verify mirror space, we expecting this to be the same size(ish) as the alpha?
20:08 < Oxf13> yes
20:08 < mmcgrath> k, sounds good.
20:08 < mmcgrath> If no one has anything else, I'll move on?
20:08  * smooge remembers the days of having 8 or 9 RC's
20:09 < smooge> nothing else
20:09 < mmcgrath> #topic func updates
20:09 -!- zodbot changed the topic of #fedora-meeting to: func updates (Meeting topic: Infrastructure)
20:09 < mmcgrath> So after some coding and some testing, the func updates before the freeze went pretty well I thought.
20:09 < skvidal> does anyone else want to work on that project?
20:09 < mmcgrath> still a few kinks to work out but it was much easier then our current method and required much less attention.
20:09 < mmcgrath> skvidal: as in you're done with it or want some help?
20:10 < skvidal> want some help
20:10 < skvidal> I got asked to work on something else this week
20:10 < mmcgrath> skvidal: I have some cycles during the freeze.  though I can't promise I won't make things worse :)
20:10 < skvidal> and that's been taking my focus
20:11 < skvidal> so it's not dropped
20:11 < mmcgrath> skvidal: well I'm sure I'll be pinging you soon(ish)
20:11 < skvidal> but I won't be able to spend as much time on it until I get the mock vm stuff out
20:11 < mmcgrath> <nod>
20:11 < mmcgrath> anyone else have questions or comments on that?
20:11 < skvidal> the func+yum thing is lightweight
20:11 < skvidal> and entry-level easy to work on
20:11 < skvidal> http://fedorapeople.org/gitweb/skvidal/func-yum.git
20:12 < skvidal> lots of easy wins
20:12 < mmcgrath> skvidal: thanks
20:12 < mmcgrath> Ok, next topic
20:12 < mmcgrath> #topic Collectd
20:12 -!- zodbot changed the topic of #fedora-meeting to: Collectd (Meeting topic: Infrastructure)
20:12 < mmcgrath> So we've been using collectd for a bit now.
20:12 < mmcgrath> what do people think?
20:12 < abadger1999> skvidal: Thanks for getting func working well again.
20:12 < skvidal> abadger1999: so much to do to make things 'well'
20:12 < smooge> skvidal, I would like to help
20:12 < skvidal> but it is unbroken
20:13 < abadger1999> :-)
20:13 < smooge> my python is broken, but I really want to help
20:13 < smooge> sorry meant help on func
20:13 < smooge> collectd I have found useful
20:13 < abadger1999> mmcgrath: It's helped us fix something already.  It's generally good.
20:13 < smooge> looking at app04 I can see where it is heavily running into some issues
20:13 < mmcgrath> smooge: me too, we've already found several problems just by having it in place.
20:14 < skvidal> what did collectd help you fix?
20:14 -!- rwmjones [~rjones at 94-30-104-162.xdsl.murphx.net] has joined #fedora-meeting
20:14 < mmcgrath> skvidal: it helpped us find the outage blips as being realted to db2.
20:14 < skvidal> ah
20:14 < skvidal> cool
20:14 -!- pcalarco_afk [~pcalarco at fedora/pcalarco] has quit Quit: Ex-Chat
20:15 < mmcgrath> other tools could have found it, but just the way we have it setup right now (every 10s) allowed us to see that load spike in such a short window was related to disk, and even more then that disk writes.
20:15 < mmcgrath> which got us looking.
20:15 < mmcgrath> and while it's not totally fixed I do think we're in better shape.  I think we just need to adjust our backup system a bit.
20:15 < mmcgrath> but that'll be a post-freeze thing.
20:15 < mmcgrath> so here's the only got'cha with collectd.
20:15 < skvidal> its a massive suck?
20:15 < mmcgrath> https://admin.fedoraproject.org/collectd/bin/index.cgi?hostname=log01&plugin=disk&timespan=604800&action=show_selection&ok_button=OK
20:15 < mmcgrath> heh
20:16 < mmcgrath> it's the disk IO required to do rrd files.
20:16 < mmcgrath> there's lots of tricks we can (and do) use to fix that.
20:16 < mmcgrath> but as we grow, it's something to watch.
20:16 < mmcgrath> you'll notice that on the 19th I figured out that automatically polling every tcp port in use and recording that info was too expensive for us :)
20:16 < mmcgrath> duh
20:16 < mmcgrath> but yeah, something to watch.
20:17 < mmcgrath> there's also non-rrdtool collection methods we can use if we really need to that would also be useful
20:17 < mmcgrath> anywho, anyone have any questions on that?
20:17 < mmcgrath> I've been slowly adding more useful stuff
20:17 < mmcgrath> like -
20:17 < mmcgrath> .tiny https://admin.fedoraproject.org/collectd/bin/index.cgi?hostname=db1&plugin=mysql&timespan=86400&action=show_selection&ok_button=OK
20:17 < zodbot> mmcgrath: http://tinyurl.com/ye6nrvl
20:18 < mmcgrath> anywho, no more questions there so that's good.
20:18 < mmcgrath> #topic Monitoring
20:18 -!- zodbot changed the topic of #fedora-meeting to: Monitoring (Meeting topic: Infrastructure)
20:18 < smooge> mmcgrath, we did rrdtools in a ram drive
20:18 < mmcgrath> smooge: yeah that's basically what some of the tunables in the rrdtool plugin do.
20:18 < mmcgrath> Ok, so we're basically back to nagios and collectd.
20:19 < smooge> yeah.. we set it up like /var/spool/mqueue/t and had a 2 GB partition for it ..
20:19 < mmcgrath> hydh has been working on some stuff but I know he's been busy
20:19 < smooge> I want to say our community help on nagios has been cool
20:19 < smooge> and great
20:19 < mmcgrath> I'd like to get some basic event handlers finalized as well as proper deps in place.
20:19 < mmcgrath> smooge: indeed
20:20 < mmcgrath> anyone have any questions about what we're up to in nagios and where we're headed?
20:20 < mmcgrath> alrighty, well with that
20:20 < mmcgrath> #topic Open Floor
20:20 -!- zodbot changed the topic of #fedora-meeting to: Open Floor (Meeting topic: Infrastructure)
20:20 < mmcgrath> anyone have anything they'd like to discuss?
20:20 < mmcgrath> a-k: anything new on search engines?
20:20 -!- XulLunch is now known as XulWork
20:21 < a-k> I'm still looking at mnoGoSearch with PostgreSQL
20:21 < a-k> I haven't had a chance to try crawling with it yet, but if it goes well I'll put it in pub test next week
20:21 < a-k> BTW is there a preference for MySQL vs PostgreSQL? I know/think we have them both around....
20:21 < dgilmore> a-k: no preference
20:21 < a-k> I'm okay with that.  That's about it for now.
20:22 < mmcgrath> anyone have anything else they'd like to discuss/
20:22 < gholms> Random question?
20:22 < gholms> Did you folks notice the collectd server-side daemon putting a lot of load on the machine?  I ask because that's what I experienced at $dayjob.
20:23 < mmcgrath> gholms: yeah, we fixed it with the suggestions here...
20:23  * mmcgrath gets link
20:23 < mmcgrath> http://collectd.org/wiki/index.php/Inside_the_RRDtool_plugin
20:24 < gholms> Ooh, that looks useful.  Thanks.
20:24 < mmcgrath> gholms: yup yup
20:24 < smooge> 1) going to work on log reviews over freeze. Now that we have over 50% free logs I was going to see what I could get out of it daily.
20:24 < smooge> I think someone else was working on this earlier so will hook up with them and see where it goes
20:24 < mmcgrath> smooge: cool
20:24 < mmcgrath> yeah someone was but I don't know the status
20:25 < smooge> then I am pretty much building my home/slicehost network to 'clone' F-I so I can test stuff here a bit better.
20:25 < smooge> my goal will be to see how far I can take epylog before it screams in terror at our data
20:25 < mmcgrath> smooge: sounds good, let us know if you need anything
20:25 < mmcgrath> well me
20:25 < mmcgrath> :)
20:26 -!- fab_ [~bellet at bellet.info] has quit Ping timeout: 248 seconds
20:26 < mmcgrath> ok, well with that I'll close the meeting in 30
20:26 < mmcgrath> 15
20:26 < mmcgrath> #endmeeting
20:26 -!- zodbot changed the topic of #fedora-meeting to: Channel is used by various Fedora groups and committees for their regular meetings | Note that meetings often get logged | For questions about using Fedora please ask in #fedora | See http://fedoraproject.org/wiki/Meeting_channel for meeting schedule
20:26 < zodbot> Meeting ended Thu Mar 25 20:26:56 2010 UTC.  Information about MeetBot at http://wiki.debian.org/MeetBot .
20:27 < mmcgrath> thanks for coming everyone!
20:27 < zodbot> Minutes:        http://meetbot.fedoraproject.org/fedora-meeting/2010-03-25/fedora-meeting.2010-03-25-20.00.html
20:27 < zodbot> Minutes (text): http://meetbot.fedoraproject.org/fedora-meeting/2010-03-25/fedora-meeting.2010-03-25-20.00.txt
20:27 < zodbot> Log:            http://meetbot.fedoraproject.org/fedora-meeting/2010-03-25/fedora-meeting.2010-03-25-20.00.log.html
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 198 bytes
Desc: not available
Url : http://lists.fedoraproject.org/pipermail/infrastructure/attachments/20100325/0b057700/attachment.bin