Meeting Log - 2010-03-11

[Ricky Zhou]

20:02 < smooge> #startmeeting Fedora Infrastructure 2010-03-11
20:02 < zodbot> Meeting started Thu Mar 11 20:02:43 2010 UTC.  The chair is smooge. Information about MeetBot at http://wiki.debian.org/MeetBot.
20:02 < zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
20:02 -!- zodbot changed the topic of #fedora-meeting to:  (Meeting topic: Fedora Infrastructure 2010-03-11)
20:02 < smooge> thankyou zodbot
20:03 < smooge> mmcgrath is finishing up a meeting so I am just helping along
20:03 < smooge> #topic Roll Call
20:03 -!- zodbot changed the topic of #fedora-meeting to: Roll Call (Meeting topic: Fedora Infrastructure 2010-03-11)
20:03  * ricky 
20:03 < smooge> smooge
20:03  * a-k is here
20:03  * nirik is around. 
20:04 < smooge> mdomsch said he was here a second ago before I started meeting
20:04 -!- rdieter_ [~rdieter at pcp084943pcs.unl.edu] has joined #fedora-meeting
20:04 < smooge> mmcgrath will be here soon.
20:04 -!- biertie [~bert at fedora/biertie] has joined #fedora-meeting
20:04 -!- rdieter [~rdieter at fedora/rdieter] has quit Ping timeout: 265 seconds
20:04  * ayoung is here
20:04 < smooge> ok let me pull up the other standard things we do in the meeting
20:05 < smooge> #topic Meeting Tickets
20:05 -!- zodbot changed the topic of #fedora-meeting to: Meeting Tickets (Meeting topic: Fedora Infrastructure 2010-03-11)
20:06 < smooge> According to trac.. there are no tickets for this meeting :)
20:06 < smooge> am i missing anything people know?
20:07 < smooge> ok moving on
20:07 < smooge> #topic Alpha Release - https://fedorahosted.org/fedora-infrastructure/report/9
20:07 -!- zodbot changed the topic of #fedora-meeting to: Alpha Release - https://fedorahosted.org/fedora-infrastructure/report/9 (Meeting topic: Fedora Infrastructure 2010-03-11)
20:07 < smooge> alpha release was on Tuesday
20:08 < smooge> ok tickets
20:08 < smooge> sorry for my slowness guys.. my greps aren't as fast as I hoped
20:08 < smooge> .ticket 1944
20:08 < zodbot> smooge: #1944 (Fedora 13 Alpha Partial Infrastructure Freeze 16/Feb - 3/March) - Fedora Infrastructure - Trac - https://fedorahosted.org/fedora-infrastructure/ticket/1944
20:08 < a-k> # topic?
20:09 < mmcgrath> yo
20:09  * mmcgrath here :)
20:09 < smooge> hi mmcgrath
20:09 < mmcgrath> herro
20:09 < mmcgrath> smooge: you want to keep going or you want me to take over?
20:09 < smooge> hopefully I am helping here ... but its been real quiet. y
20:09 -!- gholms|work [~gholms at lateralus.spa.umn.edu] has joined #fedora-meeting
20:09 < smooge> you can take over...
20:09 < mmcgrath> alrighty
20:09 < smooge> you get people to say things :)
20:10 < smooge> #chair mmcgrath
20:10 < zodbot> Current chairs: mmcgrath smooge
20:10 < mmcgrath> hehehe
20:10 < mmcgrath> so this alpha release went fine, but with oddities.
20:10 < mmcgrath> I'll go ahead and close
20:10 < mmcgrath> .ticket 1944
20:10 < mmcgrath> and
20:10 < zodbot> mmcgrath: #1944 (Fedora 13 Alpha Partial Infrastructure Freeze 16/Feb - 3/March) - Fedora Infrastructure - Trac - https://fedorahosted.org/fedora-infrastructure/ticket/1944
20:10 < mmcgrath> .ticket 1990
20:10 < zodbot> mmcgrath: #1990 (Release Day Ticket) - Fedora Infrastructure - Trac - https://fedorahosted.org/fedora-infrastructure/ticket/1990
20:10 < mmcgrath> lets talk about .ticket 1992
20:10 < mmcgrath> .ticket 1992
20:10 < zodbot> mmcgrath: #1992 (Lessons Learned) - Fedora Infrastructure - Trac - https://fedorahosted.org/fedora-infrastructure/ticket/1992
20:11 < mmcgrath> So the first problem we actually ran into was that bapp1 had puppet disabled.
20:11 < mmcgrath> ricky: you around by chance?
20:11 < ricky> Yup
20:11 < mmcgrath> so having puppet disabled on bapp1 did what exactly?
20:12 < ricky> syncStatic is run by the apache user on bapp01 now, so that prevented it from getting the change to update the website
20:12 < mmcgrath> got'cha, so new syncStatic didn't make it on to the server so new website didn't make it to the proxy servers.
20:12 < mmcgrath> this is a monitoring thing, so when we get to monitoring which servers have puppet disabled, that one will go away
20:12 < mmcgrath> The other thign that happened was with our i2 netapp
20:13 < mmcgrath> Smooge and I just got off a meeting with the storage team about this.
20:13 < mmcgrath> basically it took several hours to transfer 16G worth of blocks.
20:13 < mmcgrath> the temporary fix was to put our sync traffic at a higher QoS then other traffic
20:13 < mmcgrath> long term though there was actually something wrong with the link... again.
20:13 < mmcgrath> they know and they're working on it
20:14 < mmcgrath> but it's certainly something we're going to want to track ourselves.
20:14 < mmcgrath> But, really the alpha went well even with those things... for non i-2 users anyway.
20:14 < mmcgrath> we had 85%+ good mirror rate
20:15 < mmcgrath> And the last thing I'd say is boot.fedoraproject.org
20:15 < smooge> and once the I-2 got up I think they had little issues too
20:15 < mmcgrath> Oxf13: ping
20:15 < mmcgrath> smooge: indeed
20:15 < mmcgrath> well, right now I just did boot.fedoraproject.org
20:16 < mmcgrath> but it dawns on me that's sort of a releng task.
20:16 < mmcgrath> one that I'm happy to do
20:16 < mmcgrath> but I wanted to check with Oxf13 as to where he things the SOP should sit, in Infrastructure or RELENG.
20:16 -!- kedars [~kedars at socks.wantstofly.org] has quit Ping timeout: 276 seconds
20:16 < mmcgrath> it's a minor distinction in this case, but making one group responsible will ensure that it always gets done :)
20:16 < mmcgrath> So anyway
20:17 < mmcgrath> anyone have any other questions about the alpha release?
20:17 < mmcgrath> alrighty
20:17 < mmcgrath> well
20:17 < mmcgrath> #topic Next freeze
20:17 -!- zodbot changed the topic of #fedora-meeting to: Next freeze (Meeting topic: Fedora Infrastructure 2010-03-11)
20:17 < mmcgrath> The next freeze is coming upretty quick
20:17 < mmcgrath> #link http://fedoraproject.org/wiki/Schedule
20:18 < mmcgrath> Infrastructure will be freezing on the 23rd.
20:18 < mmcgrath> that's less then 2 weeks.
20:18 < mmcgrath> so keep that in mind as you're deploying new things.
20:18 < smooge> so we should go slushy
20:18 < smooge> ?
20:18 < mmcgrath> slushy?
20:19 < ricky> Ice, slushy, freeze?  :-)
20:19 < mdomsch> I have some MM fixes to get out, but may not make it before the next freeze :-(
20:19 -!- rdieter_ is now known as rdieter
20:19 < mmcgrath> mdomsch: anything I can do to help?
20:19  * mdomsch needs to test/fix one part, but it's been a month, so I forget which part it was ;-(
20:20 < smooge> sorry I meant, if you are going to be testing stuff for over 2 weeks.. not the time to do it.
20:20 -!- kedars [~kedars at socks.wantstofly.org] has joined #fedora-meeting
20:20 < mmcgrath> yeah
20:20 < mdomsch> mmcgrath, nothing huge; I'll get to it, or not...
20:20 < mmcgrath> I'll try to send a couple of reminders as the time gets closer.
20:20 < mmcgrath> the freeze always sneak up on us :)
20:21 < mmcgrath> anyone have anything else on that?
20:21 < smooge> we have updates to do this/next week
20:21 < mmcgrath> skvidal: ping
20:21 < skvidal> pong
20:21 < mmcgrath> smooge: lets talk about that
20:21 < mmcgrath> #topic Monthly Update
20:21 < smooge> okie dokie
20:21 -!- zodbot changed the topic of #fedora-meeting to: Monthly Update (Meeting topic: Fedora Infrastructure 2010-03-11)
20:21 < mmcgrath> skvidal: ok, so the list I've been working from with you is very nearing completion.
20:21 -!- rwmjones [~rjones at 94-30-104-162.xdsl.murphx.net] has quit Ping timeout: 256 seconds
20:22 < mmcgrath> skvidal: think we'll be in any place to update soon?
20:22 < skvidal> yah - I think likely
20:22 < skvidal> though not sure TOMORROW is gonna happen
20:22 < smooge> thats fine tuesday sound ok?
20:23 < skvidal> sounds possible - depends what gets set on fire over the weekend
20:23 < mmcgrath> tuseday of next week sounds good, the freeze starts one week after that.
20:23 < smooge> yeah which was why I didn't want monday :)
20:23 < skvidal> unless there are A LOT of yum bugs that show up in f13a over the weekend
20:23 < skvidal> then I should have the time
20:23 < smooge> wghat are the changes you are working on?
20:24 < mmcgrath> smooge: I've been removing old hosts from puppet
20:24 < mmcgrath> skvidal's been working on having func use those hosts and coming up with a solid update script
20:24 < mmcgrath> or func program
20:24 < skvidal> smooge: you talking to me or mmcgrath?
20:24 < mmcgrath> I'm not sure what they're called.
20:24 < skvidal> okay
20:24 < skvidal> so here's all it ids
20:24 < skvidal> err is
20:25 < skvidal> 1. make it so our func certs aren't constantly screwed up
20:25 < smooge> ah
20:25 < skvidal> 2. make it so our func minions match our puppet minions
20:25 < skvidal> 3. have a script to let us do searches/installs/updates/etc from puppet1 w/o having to schlep all over the place
20:26 < skvidal> the script I worked on a couple of weeks ago
20:26 < skvidal> and got something mostly functional - but with room for improvement
20:26 < mmcgrath> smooge: do you know if this update will require a reboot?
20:26 < skvidal> the first 2 is what I spent the week working on to solve the problem that our func minions were completely wrong
20:26 < skvidal> and mangled badly from the phx2 move + rename stuff
20:27 < smooge> well it depends on if a kernel update gets dropped over the weekend :). At the moment I don't think so
20:27 < mmcgrath> yeah so I've been most of my last afternoon and this morning renaming hosts.  I think I've got 3 hosts left and a little additional cleanup to do
20:28 -!- bpepple|lt [~bpepple|l at 64.134.160.158] has joined #fedora-meeting
20:28 < mmcgrath> OK
20:28 < smooge> no the worst will be an openssh restart
20:28 < mmcgrath> so anyone have anything else on this?
20:28 < mmcgrath> <nod>
20:28 < skvidal> umm
20:28 < skvidal> I have a couple more things
20:28 < mmcgrath> skvidal: have at it
20:29 < skvidal> so I'll see if I can get  a new func pkg out for all the hosts and a mechanism to update their minion.conf files to point to the puppet certificates
20:29 < skvidal> that's going to be the REALLY fun part :)
20:29 < mmcgrath> If you need any puppeting help let me know
20:30 < mmcgrath> you talking about a new func epel package or in the infra repo/
20:30 < skvidal> is minion.conf on the boxes puppet controlled now?
20:30 < skvidal> infra repo
20:30 < skvidal> for now
20:30 < skvidal> it'll eventually make it over to epel
20:30 < skvidal> but this is new cod
20:30 < ricky> I think so
20:30 < skvidal> e
20:30 < smooge> and if it isn't I will be happy to schlep what needs to be done
20:31 < skvidal> anyway - that's really all
20:31 < skvidal> the func-yum overlord script is fairly simple and could be added to by anyone as we go along
20:31 < skvidal> that's what's going to do the update/list updates/etc work
20:31 < gholms|work> I just wanted to mention that that is the greatest script name ever.
20:32 < mmcgrath> skvidal: and just becuase I'm overly paranoid... the time out bug?  all fixed in the new version?
20:32 < skvidal> mmcgrath: the timeout bug was fixed long long ago
20:32 < skvidal> we've been running an old version of func for quite sometime
20:33 < dgilmore> sorry im late
20:33 < mmcgrath> k
20:33 < mmcgrath> dgilmore: no worries
20:33 < skvidal> dgilmore: BETRAYER!
20:33 < smooge> the only thing I wanted to ask was what would be prefered for making func 'groups/classes'
20:33  * skvidal giggles
20:33 < smooge> skvidal, hey hey hey.. thats me
20:33 < mmcgrath> Ok, anyone have anything else on this?  If not we'll move on.
20:33 < skvidal> dgilmore: sorry, I just thought it was hilarious!
20:34 < dgilmore> skvidal: i had to get food for dinner and screwed up times
20:34 < smooge> nothing form me
20:34 < mmcgrath> ok
20:34 < mmcgrath> #topic Search Engine
20:34 -!- zodbot changed the topic of #fedora-meeting to: Search Engine (Meeting topic: Fedora Infrastructure 2010-03-11)
20:34 < mmcgrath> a-k: ping, want to take this?
20:34 < a-k> Sure
20:34 < a-k> I looking again at one of the candidates I eliminated from consideration earlier
20:34 < a-k> It advertises the most complete support for Unicode char sets
20:35 < a-k> I had eliminated it because I couldn't get it to work with SQLite
20:35 < a-k> ... and its user forum had no answered questions about how to get SQLite to work
20:35 < mmcgrath> which one was this?
20:35 < a-k> mnoGoSearch
20:35 < Oxf13> pong
20:35 < a-k> So I'm not there yet, but is there a db server I can use in pub test or do I need to install my own?
20:36 < a-k> Either MySQL and PostgreSQL should work
20:36 < mmcgrath> a-k: we'll have one for when we move to staging and production, but on the pt servers, just yum and install one
20:36 < a-k> OK.  I'm still working on it locally, but I'll move to pt when I'm ready
20:36 < mmcgrath> cool
20:36 < a-k> I think that's it for now
20:37 < mmcgrath> a-k: thanks
20:37 < mmcgrath> Anyone have any questions for a-k about that?
20:37 < dgilmore> a-k: do we really care for sqlite support?
20:37 < a-k> I thought SQLite would be easier/preferable to MySQL or PostgreSQL
20:38 < mmcgrath> easier for a demo but probably not for production
20:38 < a-k> None of the other candidates had needed an external db
20:38 < mmcgrath> interesting
20:38 < mmcgrath> they all had their own local store then?
20:38 < a-k> The other ones use their own local db
20:38 < mmcgrath> yeah
20:38 < mmcgrath> alright, anyone have anything else?
20:39 < mmcgrath> alrighty
20:39 < mmcgrath> #topic Monitoring
20:39 -!- zodbot changed the topic of #fedora-meeting to: Monitoring (Meeting topic: Fedora Infrastructure 2010-03-11)
20:39 < mmcgrath> SO we talked about this on infrastructure for a bit.
20:39 -!- sonargal [~Test at fedora/SonarGal] has quit Quit: used jmIrc
20:39 < mmcgrath> it leaves us in an ackward position
20:39 < mmcgrath> do we just dump zabbix now?
20:39 < smooge> mmcgrath, did you want to get Oxf13 while he was here.
20:39 < mmcgrath> go back to nagios for a bit?
20:39 < dgilmore> mmcgrath: yes
20:39 < mmcgrath> smooge: oh right
20:40 < mmcgrath> Oxf13: did you want me to put the boot.fedoraproject.org as a releng SOP or an infrastructure SOP?  it seems more releng, I'm happy to do it for as long as I'm part of Fedora but it should be documented :)
20:40 < Oxf13> that's a good question.
20:40 < Oxf13> it does sound relengy
20:41 < mmcgrath> it's pretty easy to maintain, I just want to make sure it actually gets done every release.
20:41 < mmcgrath> Oxf13: you're call.  I'll write it up this week sometime, just let me know :)
20:41  * dgilmore thinks is a releng thing 
20:42  * mdomsch says releng ;-)
20:42 < mmcgrath> alrighty, well unless Oxf13 says otherwise I'll put it as a marketing sop :-P
20:42 < mmcgrath> naw, I'll just put it in releng for now and if we change our minds later we can move it.
20:42 < mmcgrath> ok, back to monitoring.
20:42 < mdomsch> mmcgrath, unless you write a script to run when releng pushes a tree, to update bko automatically.  then it's an infra ticket that gets opened and closed automagically
20:42 < ricky> I think zabbix has been taking away effort on improving our nagios monitoring, so I'm up for dumping
20:43 < Oxf13> releng works for us, and if you come with content that's even better (:
20:43 < mmcgrath> ricky: so the problem then is trending.
20:43 < mmcgrath> our cacti install
20:43 < mmcgrath> doesn't even seem to exist anymore :)
20:44 < ricky> Apart from request times for our websites, what do you want to be able to monitor that cacti can't?
20:44 < smooge> can I work on tht with someone?
20:44 < ricky> Well, easily can't
20:44 < mmcgrath> ricky: well, the problem is that we have to enter data in two locations
20:44 < mmcgrath> and that has inherit problems.
20:44 < mmcgrath> and doing custom trending in cacti can be tricky at times.
20:45 < ricky> What kind of problems?  You still get your notification when something goes down and the general idea of movements, right?
20:45 < mmcgrath> anyway, lets take a look at our current zabbix install
20:45 < mmcgrath> err nagios install
20:45 < mmcgrath> and see how it goes.
20:45 < mmcgrath> ricky: ehh, I use trending a lot, to see when things started.
20:45 < mmcgrath> the alert is when a threshold started, but it's usually only a small part of the story.
20:45 < ricky> I've also not been crazy about running the zabbix agent public facing either.
20:46 < mmcgrath> for example when MM had the bloated pickle issues.
20:46 < mmcgrath> we didn't get the alert until days after MM was upgraded.
20:46 < mmcgrath> without trending we wouldn't have noticed when the problems started
20:46 < ricky> Yeah, but cacti would still have given you the big picture you needed, right?
20:46 < dgilmore> ricky: i rember you disabled catci
20:46 < mmcgrath> if it were in there.
20:46 < mmcgrath> trying to keep cacti and nagios in sync is goign to be a pretty big pain.
20:47 < mmcgrath> especially when we start wanting to monitor artibrary bits of info with cacti
20:47 < smooge> ricky, that is my major problem with it also
20:47 < mmcgrath> it gets complex pretty quick
20:47 < smooge> I think we should have one agent per server.. and thats func
20:47 < ricky> dgilmore: Probably, I don't running public facing stuff we're not using :-)
20:47 -!- rwmjones [~rjones at 94-30-104-162.xdsl.murphx.net] has joined #fedora-meeting
20:47 < dgilmore> ricky: from memory there was a security bug in cacti
20:48 < mmcgrath> ricky: FWIW, I still use zabbix, it's sitting on my desktop now and beeps at me :)
20:48 < dgilmore> and rather than fix it you disabled it
20:48 < dgilmore> but i could be remebering wrong
20:48 < mmcgrath> that could be
20:48 < ricky> Sounds like me alright
20:48 < mmcgrath> well no rush to make a decision today.  nagios alerts are still going out and paged alerts are going out
20:49 < mmcgrath> and at the moment, zabbix does have trending of some of the important things we need
20:49 < mmcgrath> like /mnt/koji usage
20:49 < ricky> The question is whether we should be spending time on improving nagios monitoring
20:49 < ricky> Like adding checks and keeping it in sync with hosts.  That stuff has mostly stagnated since we looked at zabbix
20:49 < mmcgrath> yeah
20:49 < mmcgrath> I'm not sure how out of sync they are.
20:49 < mmcgrath> and zabbix has a lot of stuff nagios doesn't monitor.
20:49 < ricky> Is that a yeah to "we should be spending time on it" ?  :-)
20:49 < mmcgrath> but that nagios doesn't really need to
20:50 < mmcgrath> just yeah, that stuff has stagnated
20:50 < mmcgrath> I'll take a look at things and try to get a better estimate of the work that needs to be done.
20:50 < dgilmore> ricky: i know we have hosts not in nagios
20:51 < mmcgrath> yeah
20:51 < mmcgrath> but most of our exernal service are still properly montiored by it
20:52 < mmcgrath> properly(ish)
20:52 < mmcgrath> ok, so more work to do, I'll get on that.
20:52 < mmcgrath> anyone have anything else?  If not I'll open the floor
20:52 < smooge> all I want in a monitoring solution is something where adding a host to be monitorer can be done via puppet +files
20:53 < mmcgrath> yeah
20:53 < mmcgrath> ok
20:53 < ricky> Same here - the question seems to be what we want for a trending solution :-/
20:53 < smooge> I hate clicking on things or depending on having to click on things. But open floor
20:53 < mmcgrath> #topic open floor
20:53 -!- zodbot changed the topic of #fedora-meeting to: open floor (Meeting topic: Fedora Infrastructure 2010-03-11)
20:53 < mmcgrath> ricky: I would love it if something like sar had the ability for arbitrary input :)
20:54  * gholms|work raises hand
20:54 < mmcgrath> gholms|work: yo
20:54 < gholms|work> Do you folks still plan on moving to Zenoss once 3.0 is released?
20:54 < mmcgrath> gholms|work: I don't think we ever planned on moving to Zenoss
20:54 < mmcgrath> so not to still, and no to zenoss 3.0 :)
20:54 < gholms|work> Heh, ok.
20:54  * gholms|work wonders where that idea came from...
20:54 < mmcgrath> I thought zenoss wasn't totally free?
20:55 < mmcgrath> I know it's not in Fedora yet, and that's a requisite
20:55 < gholms|work> It has a FOSS version.  The big hangup was that it relied on bundled Python 2.4.
20:55 < mmcgrath> Ok, anyone have anything else to discuss?
20:56 < ricky> Just want to get this some visibility
20:56 < ricky> Jason Walsh has been working on poseidon, an OpenID provider writtin in pylons to replace the broken stuff in FAS
20:56 < ricky> So I'll be trying to get it setup with FAS auth on a publictest soon
20:57 < mmcgrath> ricky: excellent
20:57 < mmcgrath> happy to hear that one's been making progress.
20:57 < ricky> Which will hopefully be a good workout for the auth middleware in python-fedora :-)
20:57 < mmcgrath> ricky: are we the only users / potential users?
20:57 < ricky> Sorry, not sure what you mean
20:58 < mmcgrath> does anyone else use poseidon?
20:58 < ricky> Oh, no - it was started and written for this
20:58 < mmcgrath> k
20:58 < dgilmore> ricky: do you think we could have a way to use openid, present the cla and loow wiki edits etc
20:58 < dgilmore> and things like bodhi we could use openid sans  cla
20:59 < dgilmore> for feedback
20:59 < ricky> Hm, that could get messy, as openids not linked to an existing FAS account could create all sorts of corner cases
20:59 < dgilmore> or am i all sorts of crazy
20:59 < mmcgrath> dgilmore: I think this is more of a provider
20:59 -!- mbonnet is now known as mbonnet_
20:59 < mmcgrath> to accept openid from places would require a different sort of work
20:59 < dgilmore> mmcgrath: ok
20:59 < ricky> This is more of a "packagers can login to upstream bugzillas, etc. with their OpenID" sort of thing.
21:00 < ricky> (If those exist - or even just commenting on blogs)
21:00 < dgilmore> cool
21:00  * abadger1999 shows up in time for open mic^Wfloor
21:00 < dgilmore> just throwing my crazy ideas out there
21:00 < mmcgrath> yup yup
21:00 < mmcgrath> time to close the meeting, if no one has any objections we'll close in 30
21:01  * dgilmore wants to hear abadger1999 
21:01 < ricky> ... sing :-)
21:01 < abadger1999> dgilmore: Interpretive dance :-)
21:01 < gholms|work> Over IRC?!?
21:01 < dgilmore> abadger1999: seen it already
21:01 < mmcgrath> we can talk about this next time or in #fedora-admin if we want, lets not hold everyone here :)
21:01 < dgilmore> mmcgrath: close it up
21:01 < mmcgrath> #endmeeting
21:01 -!- zodbot changed the topic of #fedora-meeting to: Channel is used by various Fedora groups and committees for their regular meetings | Note that meetings often get logged | For questions about using Fedora please ask in #fedora | See http://fedoraproject.org/wiki/Meeting_channel for meeting schedule
21:01 < zodbot> Meeting ended Thu Mar 11 21:01:31 2010 UTC.  Information about MeetBot at http://wiki.debian.org/MeetBot .
21:01 < zodbot> Minutes:        http://meetbot.fedoraproject.org/fedora-meeting/2010-03-11/fedora-meeting.2010-03-11-20.02.html
21:01 < mmcgrath> thanks for coming everyone@
21:01 < zodbot> Minutes (text): http://meetbot.fedoraproject.org/fedora-meeting/2010-03-11/fedora-meeting.2010-03-11-20.02.txt
21:01 < zodbot> Log:            http://meetbot.fedoraproject.org/fedora-meeting/2010-03-11/fedora-meeting.2010-03-11-20.02.log.html
21:01 < mmcgrath> !
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 198 bytes
Desc: not available
Url : http://lists.fedoraproject.org/pipermail/infrastructure/attachments/20100311/05fcba5d/attachment.bin