3:33 p.m.
20:00 < mmcgrath> #startmeeting Infrastructure
20:00 < zodbot> Meeting started Thu Mar 25 20:00:19 2010 UTC. The chair is
mmcgrath. Information about MeetBot at http://wiki.debian.org/MeetBot.
20:00 < mmcgrath> Who's here?
20:00 * skvidal is here
20:00 < zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
20:00 -!- zodbot changed the topic of #fedora-meeting to: (Meeting topic:
Infrastructure)
20:00 * lmacken
20:00 * a-k is here
20:00 -!- djf_jeff [~jeff(a)69.70.231.230] has joined #fedora-meeting
20:01 -!- rwmjones [~rjones(a)94-30-104-162.xdsl.murphx.net] has quit Ping timeout: 248
seconds
20:01 -!- inode0 [~inode0@fedora/inode0] has joined #fedora-meeting
20:02 < mmcgrath> #topic Fedora 13 beta
20:02 -!- zodbot changed the topic of #fedora-meeting to: Fedora 13 beta (Meeting topic:
Infrastructure)
20:02 < mmcgrath> hah
20:02 < mmcgrath> Lets get started
20:02 < mmcgrath> https://fedorahosted.org/fedora-infrastructure/report/9
20:03 < mmcgrath> .ticket 2058
20:03 < zodbot> mmcgrath: #2058 (Verify Mirror Space) - Fedora Infrastructure - Trac
- https://fedorahosted.org/fedora-infrastructure/ticket/2058
20:03 < mmcgrath> I'll get this one
20:03 < mmcgrath> .ticket 2059
20:03 < zodbot> mmcgrath: #2059 (Release Day ticket) - Fedora Infrastructure - Trac
- https://fedorahosted.org/fedora-infrastructure/ticket/2059
20:03 < mmcgrath> This one's just a tracker ticket. I'll take it too
20:03 < mmcgrath> .ticket 2060
20:03 < zodbot> mmcgrath: #2060 (Verify releng permissions) - Fedora Infrastructure
- Trac - https://fedorahosted.org/fedora-infrastructure/ticket/2060
20:03 < mmcgrath> smooge: want to get that one again?
20:04 < mmcgrath> we'll come back to that.
20:04 < mmcgrath> .tiny 2061
20:04 < zodbot> mmcgrath: Error: '2061' is not a valid url.
20:05 < mmcgrath> .ticket 2061
20:05 < mmcgrath> sorry :/
20:05 < zodbot> mmcgrath: #2061 (MM redirects) - Fedora Infrastructure - Trac -
https://fedorahosted.org/fedora-infrastructure/ticket/2061
20:05 < mmcgrath> mdomsch usually gets this one (I believe it's automated now
and just requires verification)
20:05 < mmcgrath> .ticket 2062
20:05 < zodbot> mmcgrath: #2062 (Infrastructure Change Freeze) - Fedora
Infrastructure - Trac - https://fedorahosted.org/fedora-infrastructure/ticket/2062
20:05 < mmcgrath> I'll get that, we are frozen.
20:05 < smooge> morning
20:05 < smooge> sorry got stuck on phone
20:05 < mmcgrath> smooge: hey, want to get 2060?
20:05 < smooge> yes
20:05 < mmcgrath> k
20:05 < mmcgrath> and the last ticket
20:06 < mmcgrath> .ticket 2063 doesn't need to be done until after the launch
20:06 < smooge> could I get 2058 .. I find them related
20:06 < zodbot> mmcgrath: Error: "2063 doesn't need to be done until after
the launch" is not a valid integer.
20:06 < smooge> hehheeh
20:06 -!- djf_jeff [~jeff(a)69.70.231.230] has quit Quit: I quit
20:06 < mmcgrath> zodbot: you are testing me!
20:06 < mmcgrath> ok, anyone have any questions or comments related to the release?
20:06 < mmcgrath> Oxf13: you around?
20:07 < Oxf13> I am
20:07 < mmcgrath> what are our odds of slipping at the moment?
20:07 < Oxf13> I'd put it at 50% chance
20:07 -!- biertie [~bert@fedora/biertie] has joined #fedora-meeting
20:07 < Oxf13> There is one blocker we're worried about, but we have a patch in
hand, it just needs testing, then I can make the RC
20:07 < skvidal> mmcgrath: so - when it's a 50% chance of rain - you carry an
umbrella :)
20:07 < mmcgrath> cool.
20:07 < Oxf13> we have a compressed amount of time to test the RC
20:08 < Oxf13> and not really any time to fix anything that's wrong with the RC
and validate a second RC before the go / no go time
20:08 < mmcgrath> Oxf13: I'll work with you later today or tomorrow to verify
mirror space, we expecting this to be the same size(ish) as the alpha?
20:08 < Oxf13> yes
20:08 < mmcgrath> k, sounds good.
20:08 < mmcgrath> If no one has anything else, I'll move on?
20:08 * smooge remembers the days of having 8 or 9 RC's
20:09 < smooge> nothing else
20:09 < mmcgrath> #topic func updates
20:09 -!- zodbot changed the topic of #fedora-meeting to: func updates (Meeting topic:
Infrastructure)
20:09 < mmcgrath> So after some coding and some testing, the func updates before the
freeze went pretty well I thought.
20:09 < skvidal> does anyone else want to work on that project?
20:09 < mmcgrath> still a few kinks to work out but it was much easier then our
current method and required much less attention.
20:09 < mmcgrath> skvidal: as in you're done with it or want some help?
20:10 < skvidal> want some help
20:10 < skvidal> I got asked to work on something else this week
20:10 < mmcgrath> skvidal: I have some cycles during the freeze. though I can't
promise I won't make things worse :)
20:10 < skvidal> and that's been taking my focus
20:11 < skvidal> so it's not dropped
20:11 < mmcgrath> skvidal: well I'm sure I'll be pinging you soon(ish)
20:11 < skvidal> but I won't be able to spend as much time on it until I get the
mock vm stuff out
20:11 < mmcgrath> <nod>
20:11 < mmcgrath> anyone else have questions or comments on that?
20:11 < skvidal> the func+yum thing is lightweight
20:11 < skvidal> and entry-level easy to work on
20:11 < skvidal> http://fedorapeople.org/gitweb/skvidal/func-yum.git
20:12 < skvidal> lots of easy wins
20:12 < mmcgrath> skvidal: thanks
20:12 < mmcgrath> Ok, next topic
20:12 < mmcgrath> #topic Collectd
20:12 -!- zodbot changed the topic of #fedora-meeting to: Collectd (Meeting topic:
Infrastructure)
20:12 < mmcgrath> So we've been using collectd for a bit now.
20:12 < mmcgrath> what do people think?
20:12 < abadger1999> skvidal: Thanks for getting func working well again.
20:12 < skvidal> abadger1999: so much to do to make things 'well'
20:12 < smooge> skvidal, I would like to help
20:12 < skvidal> but it is unbroken
20:13 < abadger1999> :-)
20:13 < smooge> my python is broken, but I really want to help
20:13 < smooge> sorry meant help on func
20:13 < smooge> collectd I have found useful
20:13 < abadger1999> mmcgrath: It's helped us fix something already. It's
generally good.
20:13 < smooge> looking at app04 I can see where it is heavily running into some
issues
20:13 < mmcgrath> smooge: me too, we've already found several problems just by
having it in place.
20:14 < skvidal> what did collectd help you fix?
20:14 -!- rwmjones [~rjones(a)94-30-104-162.xdsl.murphx.net] has joined #fedora-meeting
20:14 < mmcgrath> skvidal: it helpped us find the outage blips as being realted to
db2.
20:14 < skvidal> ah
20:14 < skvidal> cool
20:14 -!- pcalarco_afk [~pcalarco@fedora/pcalarco] has quit Quit: Ex-Chat
20:15 < mmcgrath> other tools could have found it, but just the way we have it setup
right now (every 10s) allowed us to see that load spike in such a short window was related
to disk, and even more then that disk writes.
20:15 < mmcgrath> which got us looking.
20:15 < mmcgrath> and while it's not totally fixed I do think we're in
better shape. I think we just need to adjust our backup system a bit.
20:15 < mmcgrath> but that'll be a post-freeze thing.
20:15 < mmcgrath> so here's the only got'cha with collectd.
20:15 < skvidal> its a massive suck?
20:15 < mmcgrath>
https://admin.fedoraproject.org/collectd/bin/index.cgi?hostname=log01&...
20:15 < mmcgrath> heh
20:16 < mmcgrath> it's the disk IO required to do rrd files.
20:16 < mmcgrath> there's lots of tricks we can (and do) use to fix that.
20:16 < mmcgrath> but as we grow, it's something to watch.
20:16 < mmcgrath> you'll notice that on the 19th I figured out that
automatically polling every tcp port in use and recording that info was too expensive for
us :)
20:16 < mmcgrath> duh
20:16 < mmcgrath> but yeah, something to watch.
20:17 < mmcgrath> there's also non-rrdtool collection methods we can use if we
really need to that would also be useful
20:17 < mmcgrath> anywho, anyone have any questions on that?
20:17 < mmcgrath> I've been slowly adding more useful stuff
20:17 < mmcgrath> like -
20:17 < mmcgrath> .tiny
https://admin.fedoraproject.org/collectd/bin/index.cgi?hostname=db1&p...
20:17 < zodbot> mmcgrath: http://tinyurl.com/ye6nrvl
20:18 < mmcgrath> anywho, no more questions there so that's good.
20:18 < mmcgrath> #topic Monitoring
20:18 -!- zodbot changed the topic of #fedora-meeting to: Monitoring (Meeting topic:
Infrastructure)
20:18 < smooge> mmcgrath, we did rrdtools in a ram drive
20:18 < mmcgrath> smooge: yeah that's basically what some of the tunables in the
rrdtool plugin do.
20:18 < mmcgrath> Ok, so we're basically back to nagios and collectd.
20:19 < smooge> yeah.. we set it up like /var/spool/mqueue/t and had a 2 GB
partition for it ..
20:19 < mmcgrath> hydh has been working on some stuff but I know he's been busy
20:19 < smooge> I want to say our community help on nagios has been cool
20:19 < smooge> and great
20:19 < mmcgrath> I'd like to get some basic event handlers finalized as well as
proper deps in place.
20:19 < mmcgrath> smooge: indeed
20:20 < mmcgrath> anyone have any questions about what we're up to in nagios and
where we're headed?
20:20 < mmcgrath> alrighty, well with that
20:20 < mmcgrath> #topic Open Floor
20:20 -!- zodbot changed the topic of #fedora-meeting to: Open Floor (Meeting topic:
Infrastructure)
20:20 < mmcgrath> anyone have anything they'd like to discuss?
20:20 < mmcgrath> a-k: anything new on search engines?
20:20 -!- XulLunch is now known as XulWork
20:21 < a-k> I'm still looking at mnoGoSearch with PostgreSQL
20:21 < a-k> I haven't had a chance to try crawling with it yet, but if it goes
well I'll put it in pub test next week
20:21 < a-k> BTW is there a preference for MySQL vs PostgreSQL? I know/think we have
them both around....
20:21 < dgilmore> a-k: no preference
20:21 < a-k> I'm okay with that. That's about it for now.
20:22 < mmcgrath> anyone have anything else they'd like to discuss/
20:22 < gholms> Random question?
20:22 < gholms> Did you folks notice the collectd server-side daemon putting a lot
of load on the machine? I ask because that's what I experienced at $dayjob.
20:23 < mmcgrath> gholms: yeah, we fixed it with the suggestions here...
20:23 * mmcgrath gets link
20:23 < mmcgrath> http://collectd.org/wiki/index.php/Inside_the_RRDtool_plugin
20:24 < gholms> Ooh, that looks useful. Thanks.
20:24 < mmcgrath> gholms: yup yup
20:24 < smooge> 1) going to work on log reviews over freeze. Now that we have over
50% free logs I was going to see what I could get out of it daily.
20:24 < smooge> I think someone else was working on this earlier so will hook up
with them and see where it goes
20:24 < mmcgrath> smooge: cool
20:24 < mmcgrath> yeah someone was but I don't know the status
20:25 < smooge> then I am pretty much building my home/slicehost network to
'clone' F-I so I can test stuff here a bit better.
20:25 < smooge> my goal will be to see how far I can take epylog before it screams
in terror at our data
20:25 < mmcgrath> smooge: sounds good, let us know if you need anything
20:25 < mmcgrath> well me
20:25 < mmcgrath> :)
20:26 -!- fab_ [~bellet(a)bellet.info] has quit Ping timeout: 248 seconds
20:26 < mmcgrath> ok, well with that I'll close the meeting in 30
20:26 < mmcgrath> 15
20:26 < mmcgrath> #endmeeting
20:26 -!- zodbot changed the topic of #fedora-meeting to: Channel is used by various
Fedora groups and committees for their regular meetings | Note that meetings often get
logged | For questions about using Fedora please ask in #fedora | See
http://fedoraproject.org/wiki/Meeting_channel for meeting schedule
20:26 < zodbot> Meeting ended Thu Mar 25 20:26:56 2010 UTC. Information about
MeetBot at http://wiki.debian.org/MeetBot .
20:27 < mmcgrath> thanks for coming everyone!
20:27 < zodbot> Minutes:
http://meetbot.fedoraproject.org/fedora-meeting/2010-03-25/fedora-meeting...
20:27 < zodbot> Minutes (text):
http://meetbot.fedoraproject.org/fedora-meeting/2010-03-25/fedora-meeting...
20:27 < zodbot> Log:
http://meetbot.fedoraproject.org/fedora-meeting/2010-03-25/fedora-meeting...