Meeting Log - 2007-11-01

Fri Nov 2 02:21:18 UTC 2007

15:55  * jima runs and hides
15:58 -!- fchiulli [i=824c4010 at gateway/web/cgi-irc/ircatwork.com/x-a6b81cd860eb6651] has joined #fedora-meeting
16:00 < jima> AIEEE NOOO
16:00 -!- mmcgrath changed the topic of #fedora-meeting to: Infrastructure -- Who's here?
16:00 < mmcgrath> Ping
16:00 < mmcgrath> who's around?
16:01 < markhsa> Mark H
16:01 < paulobanon_> me
16:01 < markhsa> me
16:01 < f13> I'm here
16:01 < fchiulli> fchiulli is listening
16:01  * jeremy is around-ish
16:01  * quaid is lurking
16:01 < bklemm> bklemm is listening in too
16:01  * jima imagines fchiulli and bklemm are new to irc :)
16:02  * jima 
16:02 < mmcgrath> spevack: abadger1999 poelcat mbacovsk ricky ivazquez warren skvidal lmacken dgilmore: ping
16:02 < fchiulli> Give that man a cigar.
16:02 < mmcgrath> I know some of them won't be in today.
16:02 < skvidal> pong
16:02 < abadger1999> pong
16:02 < warren> pong
16:02 < poelcat> mmcgrath: pong
16:02 < jima> dgilmore was busy as heck earlier, not sure about now.
16:03 -!- giarc_w [i=hidden-u at gnat.asiscan.com] has joined #fedora-meeting
16:03 < lmacken> pong
16:03 < mmcgrath> Welp, lets get started then.
16:03 -!- mmcgrath changed the topic of #fedora-meeting to: Infrastructure -- Release is go!
16:03 < mmcgrath> .tiny https://hosted.fedoraproject.org/projects/fedora-infrastructure/query?status=new&status=assigned&status=reopened&milestone=Fedora+8
16:03 < mmcgrathbot> mmcgrath: http://tinyurl.com/292qq8
16:04 < mmcgrath> So yeah, AFAIK we will release Fedora 8 on the 8th next week.
16:04 < mmcgrath> Is anyone confused about what they should or shouldn't be doing at this point?
16:04 < jima> should: looking for bugs
16:04 < spoleeba> mmcgrath, duck and cover?
16:04 < warren> meh
16:04 < jima> shouldn't: changing infrastructure without asking :)
16:05 < skvidal> jima: wait? what?
16:05 < skvidal> should NOT
16:05 < jima> spoleeba: i thought we did that every day ;)
16:05 < skvidal> oh shit
16:05 < abadger1999> :-)
16:05  * mmcgrath smacks skvidal 
16:05 -!- nim-nim [n=nim-nim at fedora/nim-nim] has quit "Leaving."
16:05 < skvidal> mmcgrath: hey, I wasn't the one who brought proxy4 online yesterday! :)
16:05 < mmcgrath> Just a reminder, ANOTHER one :)  After tonight, no changes to infrastructure without contacting the list first.
16:05 < jima> skvidal: ...?
16:05 < skvidal> jima: I'm making fun of mmcgrath
16:06 < mmcgrath> jesse also had a wish to extend this period for next release.  I'm generally in favor of that though I'd like to see hwo this change freeze goes.
16:06  * dgilmore is sitting on the cold floor of the colo 
16:06 < mmcgrath> There are a few critical things I'd like to talk about.
16:06 -!- mmcgrath changed the topic of #fedora-meeting to: Infrastructure - mmcgrath (schedule)
16:06 < jima> hmm, should i be getting approval to work on ticket #220? :)
16:06 < mmcgrath> jima: lets get to that in a bit
16:06 < jima> 'k
16:06 -!- nim-nim [n=nim-nim at fedora/nim-nim] has joined #fedora-meeting
16:07 < mmcgrath> So, if I've done my job right, I won't need to be around for this release.  Having said that.  I'm going to try to be.
16:07 < paulobanon_> no changes no fun!
16:07 < mmcgrath> skvidal is going to be the point person during the release, even if I am around.
16:07 < skvidal> mmcgrath: do not be around
16:07 < skvidal> seriously - don't do it
16:07 < mmcgrath> :)
16:07 < jima> skvidal: see, you shouldn't have made fun of him -- he just got payback.
16:07 < skvidal> we'll deal w/o you :)
16:07 < mmcgrath> So here's my schedule.
16:07 < skvidal> jima: I [was] volunteered
16:08 < mmcgrath> I'm off tomorrow and will largely be unavailable until Monday.
16:08 < jima> skvidal: exactly my point. ;)
16:08 < jima> mmcgrath: have fun! :)
16:08 < mmcgrath> I'll be taking half days Monday - Thursday (release day) and on the 9th I'll be gone.
16:08 < mmcgrath> just gone.  Like I wasn't ever here.
16:08 < jima> damn ninja.
16:08 < mmcgrath> and I'll be back for good on the 13th.
16:08 < jeremy> who's mmcgrath?
16:08 < skvidal> mmcgrath: wow, like keyser sose
16:08 < mmcgrath> Hopefully with a tan.  Though its likely I'll burst into flames the first time I come in contact with the sun.
16:09 < skvidal> KEYSER SOSE KEYSER SOSE!
16:09 < mmcgrath> :)
16:09 < jima> i think anything bursts into flame when it comes even *close* to the sun.
16:09 < mmcgrath> So mark it down and remember that we talked about it in the meeting.
16:09 < mmcgrath> Next topic!
16:09 < paulobanon_> wedding topic ?!
16:09 -!- mmcgrath changed the topic of #fedora-meeting to: Infrastructure -- Current nown oddities.
16:09 -!- mmcgrath changed the topic of #fedora-meeting to: Infrastructure -- Current known oddities.
16:09 < mmcgrath> damn
16:09 < mmcgrath> Ok.
16:09 < jima> known oddity: "nown" is missing a 'k'
16:09 < skvidal> so, next week after we remove all of mmcgrath's accounts what will we be doing
16:09 < skvidal> ?
16:09 < skvidal> oh
16:09 < skvidal> damn
16:10 < skvidal> wrong channel
16:10 < jima> skvidal: that was for #fedora-evil, huh?
16:10 < mmcgrath> heh
16:10 < skvidal> it was
16:10 < mmcgrath> https://hosted.fedoraproject.org/projects/fedora-infrastructure/ticket/222
16:10 < mmcgrath> .ticket 222
16:10 < mmcgrathbot> mmcgrath: #222 (sysctl on the proxy servers) - Fedora Infrastructure - Trac
16:10 < mmcgrath> I should have had this done a week ago but testing it and being busy with other stuff put it off.
16:10 < mmcgrath> Long story short.
16:10 -!- stahnma_ [n=stahnma at c-76-18-178-254.hsd1.tn.comcast.net] has joined #fedora-meeting
16:10 < yingbull> I should probably say hi - I was around earlier in the fall, but got a bad case of the $DAYJOB, and had fedora account issues that slowed me down getting involved (and without anything tasked to me yet, I just sorta faded off).  Back around now, tho.   So... hi.  ;-)
16:10 < mmcgrath> The sysctl configuration optimizations have been done on all the proxy servers.
16:10 < mmcgrath> yingbull: welcome!
16:11 < yingbull> made it to this meeting at least, hopefully its a good start.  :-) Sorry to interrupt.
16:11 < mmcgrath> So here's the scoop, we've got optimized proxy servers.  I've done all the testing I can to ensure that we are getting peak performance from them.
16:11  * jima waves to yingbull
16:11 < jima> mmcgrath: seems like a clever idea, yeah.
16:11 < jima> is that all the proxies or just 1-2?
16:11 < mmcgrath> Having said that, I don't feel they're tested enough to just deploy.  So to mitigate this risk my *optimized* config is in /root/sysctl.conf and the normal config thats been deployed for many months is in /etc/sysctl.conf
16:11 < mmcgrath> jima: all of them.
16:12 < mmcgrath> So if anyone sees the proxy servers getting overwhelmed, and feel it might be tuned wrong, just run "sysctl -p"
16:12 < mmcgrath> if you want to go back to the 'optimized configs' run "sysctl -p /root/sysctl.conf"
16:12 < mmcgrath> profit.
16:12  * jima almosts asks "where's a copy of the optimized one in puppet?" then remembers mmcgrath provided us with a copy.
16:12 < mmcgrath> Thats an odditiy I thought I'd mention.
16:12 -!- clarkbw [i=clarkbw at nat/redhat/x-ae3597e3eeab15df] has joined #fedora-meeting
16:12 < mmcgrath> since it doesn't make any obvious sense.
16:13 -!- londo [n=georgiou at 82.133.49.59] has quit Read error: 104 (Connection reset by peer)
16:13 -!- clarkbw [i=clarkbw at nat/redhat/x-ae3597e3eeab15df] has quit Read error: 104 (Connection reset by peer)
16:13 < mmcgrath> Anyone have any questions on that?  Did anyone see anything obviously wrong with the sysctl.conf I sent to the list?
16:13 < lmacken> looked sane to me
16:13 < jeremy> seem fine to me
16:13 < mmcgrath> ok, moving on.
16:14 < mmcgrath> Oddity number 2.
16:14 -!- rdieter is now known as rdieter_away
16:14 < mmcgrath> The wiki notifications today.
16:14 < mmcgrath> We've been seeing higher than normal wiki notifications as of today.
16:14 < mmcgrath> I don't think the wiki is any worse off today then it was yesterday.  I think its because we've been monitoring the wiki with caching enabled and I recently made that cache more aggressive than normal.
16:15 < mmcgrath> The notifications we're getting are from the only non-cached page.
16:15 < mmcgrath> Which means A) the wiki is unresponsive at times.
16:15 < mmcgrath> but B) I've not witnessed this myself, only seen the nagios alerts.
16:16 < mmcgrath> I'm watching this closely and trying to tune the wiki a bit but so far nothing has made the load any better.
16:16 < mmcgrath> Long story short, the caching I added sees between a 50% and 75% hit rate.
16:16 < jima> that's good.
16:16 < mmcgrath> that means only 50% - 25% of the requests actually hit the app server.
16:16 < mmcgrath> this is something we did not have for the F7 release.
16:16 -!- clarkbw [i=clarkbw at nat/redhat/x-29205887cfe9d537] has joined #fedora-meeting
16:16 < mmcgrath> The bulk of our work will be done by the proxy servers this time around.
16:17 < jima> but god help the actual wiki for the hits that do get through. :|
16:17 < mmcgrath> of which we have 4, the 2 new ones have more ram and are x86_64 boxes.  So I think we're in good shape.
16:17 < mmcgrath> The wiki, as always, concerns me.
16:17 -!- k0k [n=k0k at fedora/k0k] has quit Read error: 110 (Connection timed out)
16:17 < mmcgrath> It just does.
16:17  * jima concurs
16:17 < mmcgrath> I've moved app3 off of xen1 and on to xen2 to dedicate more resources to xen1 (which currently contains the wiki and one proxy server)
16:17 < jeremy> we should probably just be aggressive about making "important" pages static and for the rest, if the wiki goes down, the wiki goes down
16:17 < poelcat> skvidal: is the problems I had yesterday app1 and with updating wiki pages is fixed?
16:17 < mmcgrath> unfortunately thats the best setup I've found so far.
16:18 < skvidal> poelcat: for certain values of fixed, yes
16:18 < mmcgrath> Any questions about the wiki right now?
16:19 < jima> mmcgrath: yes, but my questions keep timing out.
16:19 < mmcgrath> I've also turned off the translate app for app1 for now.
16:19  * mmcgrath smacks jima.
16:19 < mmcgrath> especially over the next many days.  If you have ANY problems with the wiki, please email me the time, what you were trying to do and your IP address.
16:20 < mmcgrath> Ok, moving on.
16:20 < mmcgrath> mdomsch is not around but all that is left on that end is to disable the primary mirrors initially (download.fedora.redhat.com) from the mirror list and public list.
16:20  * jima scratches head
16:21 -!- mmcgrath changed the topic of #fedora-meeting to: Infrastructure -- Specific tickets open.
16:21 < jima> disable them from mirrorlist? isn't that tweakable via mirrormanager?
16:21 < mmcgrath> .ticket 93
16:21 < mmcgrathbot> mmcgrath: #93 (Fedora 8) - Fedora Infrastructure - Trac
16:21 < mmcgrath> We'll close that when everything is done
16:21 < mmcgrath> .ticket 97
16:21 < mmcgrathbot> mmcgrath: #97 (Fedora 8 Website) - Fedora Infrastructure - Trac
16:21 < mmcgrath> Ricky and I have been working on this together, expect to see the F8 website by the end of the day but with the F7 graphics and such.
16:21 < mmcgrath> We'll have a special build for F8 for the mirrors by the end of the day on Monday.
16:21 < mmcgrath> .ticket 99
16:21 < mmcgrathbot> mmcgrath: #99 (Mirror Coordination) - Fedora Infrastructure - Trac
16:22 < mmcgrath> Currently assigned to mdomsch either he or jesse keating will likely notify the mirrors when the bits are ready.
16:22 < mmcgrath> .ticket 100
16:22 < mmcgrathbot> mmcgrath: #100 (Web Content on the mirrors) - Fedora Infrastructure - Trac
16:22 < mmcgrath> already discussed, this will be done with ticket #97
16:22 < mmcgrath> .ticket 101
16:22 < mmcgrathbot> mmcgrath: #101 (Fedora Infrastructure Change Freeze) - Fedora Infrastructure - Trac
16:22 < mmcgrath> I'll close this ticket AFTER tonight.
16:22 < mmcgrath> ticket .137
16:22 < mmcgrath> .ticket 137
16:22 < mmcgrath> :)
16:22 < mmcgrathbot> mmcgrath: #137 (Sponsorship prominence) - Fedora Infrastructure - Trac
16:23 < mmcgrath> This is actually in good shape and live.  Its linked at the bottom of the page http://fedoraproject.org/sponsors
16:23 < mmcgrath> .ticket 210
16:23 < mmcgrathbot> mmcgrath: #210 (Disable F8 in Mirrormanager) - Fedora Infrastructure - Trac
16:23 < notting> g
16:24 < mmcgrath> This is also currently assigned to mdomsch (who's not here) but I'll follow up with him to make sure this is done correctly when the bits go
16:24 < mmcgrath> .ticket 211
16:24 < mmcgrathbot> mmcgrath: #211 (Enable Bodhi Updates) - Fedora Infrastructure - Trac
16:24 < mmcgrath> lmacken: ?
16:24 < mmcgrath> This is assigned to you, still all good?
16:24 < lmacken> shouldn't be a problem
16:24 < mmcgrath> any ETA?
16:25 < lmacken> it's just a bitflip.. whenever we want to unleash
16:25 -!- stahnma_ is now known as stahnma
16:25 < mmcgrath> k, I'll leave that up to you and releng then :)
16:25 < mmcgrath> .ticket 54
16:25 < mmcgrathbot> mmcgrath: #54 (Postfix Server) - Fedora Infrastructure - Trac
16:25 < mmcgrath> This, unfortunately, will probably be the only ticket we didn't completely finish.  Mostly an -ENOTIME and a -EALREADYHAVESOMETHING
16:26 < mmcgrath> bastion's been fine, this was a bigger priority when the mailman list stuff was being discussed and we had OTRS.
16:26 < mmcgrath> and the last ticket
16:26 < mmcgrath> .ticket 136
16:26 < mmcgrathbot> mmcgrath: #136 (Translated Website) - Fedora Infrastructure - Trac
16:26 < mmcgrath> We've 'frozen' the website and the translation team is currently at work on it
16:26 < jima> yay for translators!
16:26  * mmcgrath notes
16:26 < mmcgrath> http://translate.fedoraproject.org/module/fedora-web
16:27 < mmcgrath> I think this is actually a big deal.
16:27 < mmcgrath> but the final 'freeze' will be on monday night.
16:27 < mmcgrath> So really, we're in good shape.
16:27 < mmcgrath> Hopefully we won't have to juggle much at release time.
16:27 < mmcgrath> Anyone have any questions or concerns regarding me, our infrastructure, stuff not staying up or getting overloaded?
16:28 < mmcgrath> skvidal: you've got to have questions about stuff right?
16:29 < skvidal> mostly i'm reading
16:29 -!- EvilBob [n=bob at fedora/pdpc.sustaining.BobJensen] has quit Remote closed the connection
16:29  * mmcgrath also notes http://fedoraproject.org/wiki/Infrastructure/SOP/Release
16:29 < mmcgrath> still exists and is valid.
16:29 < skvidal> but right now my primary thoughts are making sure we have all the steps down for dynamically making static pages when we need them
16:29 < mmcgrath> skvidal: we should make an SOP for that.
16:30 < mmcgrath> skvidal: OH, thats actually a good point.
16:30 < skvidal> but we can't get to the SOP :)
16:30 < skvidal> so it's better just to have a note
16:30 < skvidal> so i'm thinking
16:30 < mmcgrath> *if* the wiki gets overloaded and you can't get to it, log into RH VPN and go to app2.fedora.phx.redhat.com/wiki/
16:30 < mmcgrath> since we can't balance the wiki its always on one or the other, just go to the other :)
16:30 < skvidal> 1. save the page in firefox
16:30 < mmcgrath> skvidal: do it with wget.
16:30 < skvidal> 2. upload the files to the static path on the proxies
16:31 < skvidal> wget, in the past, didn't grab css for me
16:31 < mmcgrath> hmm, k,  we can test that after the meeting.
16:31 < skvidal> okie doke
16:31 < mmcgrath> I'd also encourage everyone to become familiar with our caching setup.  especially the headers.
16:32 < mmcgrath> .headers http://fedoraproject.org/wiki/
16:32 < mmcgrathbot> mmcgrath: apptime: D=77335, proxytime: D=167438, expires: Thu, 01 Nov 2007 21:32:01 GMT, vary: Cookie,User-Agent,Accept-Language, connection: close, server: Apache/2.2.3 (Red Hat), appserver: (null), proxyserver: proxy3.fedoraproject.org, date: Thu, 01 Nov 2007 20:32:01 GMT, content-type: text/html; charset=utf-8
16:32  * mmcgrath notes proxy and app time
16:32 < mmcgrath> .headers http://fedoraproject.org/wiki/
16:32 < mmcgrathbot> mmcgrath: apptime: D=77335, content-length: 13644, proxytime: D=87, expires: Thu, 01 Nov 2007 21:32:01 GMT, vary: Cookie,User-Agent,Accept-Language, connection: close, server: Apache/2.2.3 (Red Hat), appserver: (null), proxyserver: (null), date: Thu, 01 Nov 2007 20:32:15 GMT, content-type: text/html; charset=utf-8, age: 15
16:32  * mmcgrath notes "age" this time.
16:32 < mmcgrath> stuff like that is important for understanding why USER A) gets cached copy and why user B) doesn't.
16:33 < mmcgrath> If you want to flush all the cache, just "rm -rf /srv/web/cache/*" on the proxy box.
16:33 -!- londo [n=georgiou at 82.133.49.59] has joined #fedora-meeting
16:33 < mmcgrath> blah blah.
16:33 < mmcgrath> Ok, if no one has any questions about anything else I'll open the floor?
16:33 < skvidal> mmcgrath: one more question
16:33 < skvidal> on the proxy boxes
16:33 < skvidal> we have /srv/web/static-web
16:34 < skvidal> is that path old/broken?
16:34 < mmcgrath> That is the old path.
16:34  * mmcgrath notes proxy4
16:34 < mmcgrath> it doesn't have it.
16:34 < skvidal> right - I was more thinking of cleaning that up
16:34 < mmcgrath> <nod>
16:34 < mmcgrath> I thought I'd moved that to /tmp/
16:35 < mmcgrath> perhaps I moved it back during the conversion.
16:35 < mmcgrath> but yeah, that should now be "/srv/web/fedoraproject.org/" or "/srv/web/start.fedoraproject.org/"
16:35 < skvidal> looks like you copied it to temp :)
16:35 < skvidal> but it never got nuked
16:35  * mmcgrath also notes that AFAIK, search.fedoraproejct.org won't be ready.
16:36 < mmcgrath> but thats not really our concern, just pointing it out.
16:36 < yingbull> my bit:  I'll want something to do soon.   I need to fix my fedora account, but will want something to work on to get started.   Else I'm likely to let $DAYJOB eat my $NIGHTLIFE, and without anything depending/waiting on me, fade away again. :-)   Anyone got something that I can thing them about once my fedora account is working?
16:36 < mmcgrath> yingbull: how's your mysql skills?
16:36 -!- clarkbw [i=clarkbw at nat/redhat/x-29205887cfe9d537] has quit Read error: 110 (Connection timed out)
16:37 < yingbull> mmcgrath: not bad, haven't had a huge need for it lately, but I did work with it before.
16:37 < yingbull> Used it for a big web-app at the last place I worked.
16:37 < mmcgrath> yingbull: ping me after the meeting in #fedora-admin, we'll find something definitive for you.
16:37 < yingbull> Will do.   Save me from $DAYJOB, please! :-)
16:37 < mmcgrath> heh
16:37 < mmcgrath> ok
16:37 -!- mmcgrath changed the topic of #fedora-meeting to: Infrastructure -- Open Floor
16:38 < mmcgrath> Does anyone have anything specific they'd like to discuss?
16:38 < mmcgrath> I'd encourage everyone to start thinking of F9 goals.
16:38  * jima muses
16:39 < mmcgrath> Anyone have anything else?  If not we'll close the meeting.
16:39 < jima> mmcgrath: ticket 220?
16:40 < mmcgrath> .ticket 220
16:40 < mmcgrathbot> mmcgrath: #220 (nagios monitoring for puppet1) - Fedora Infrastructure - Trac
16:40 < mmcgrath> jima: ahh, yeah I just wanted to make sure it was beign monitored.
16:40 < jima> it's not. next question? :)
16:41 < mmcgrath> well, can we make it monitored?  ICMP, disk, puppetmaster, http.
16:41 < mmcgrath> standard stuff.
16:41 < jima> is it running nrpe?
16:41 < mmcgrath> it should be.
16:41 < jima> yep, it is.
16:42 < jima> should i make those changes to nagios? no conflicts with the infrastructure freeze?
16:42 < mmcgrath> jima: I'd put that as a low priority but if you can't get it done tonight, I'd wait until after the release.
16:42 < jima> 'k.
16:42 < nim-nim> ticket 215 anyone?
16:42 < mmcgrath> .ticket 215
16:42 < mmcgrathbot> mmcgrath: #215 (New mailing list request: fedora-fonts-bugs-list) - Fedora Infrastructure - Trac
16:42 < mmcgrath> nim-nim: that one's on me.  I'll put that request in now.
16:43 < nim-nim> mmcgrath: thanks!
16:43  * mmcgrath has been a little busy :)
16:43  * nim-nim understands
16:43 < mmcgrath> I'd also like to get the rest of the hosted repos created that are in the queue by tonight.
16:43 < nim-nim> still, nim-nim needs to get his SIG in shape too
16:44 < nim-nim> many thanks!
16:44 -!- JSchmitt [n=s4504kr at fedora/JSchmitt] has quit "Konversation terminated!"
16:44 < mmcgrath> Ok, anyone have anything else?
16:45 < mmcgrath> If not I'll close the meeting in 30.
16:45 < mmcgrath> 15
16:45 < mmcgrath> 5
16:45 < mmcgrath> Good luck to us all next week :)
16:46 -!- mmcgrath changed the topic of #fedora-meeting to: Infrasturcture -- Meeting Closed.
16:46 -!- fchiulli [i=824c4010 at gateway/web/cgi-irc/ircatwork.com/x-a6b81cd860eb6651] has left #fedora-meeting []
16:46 < mmcgrath> Thanks for coming everyone.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: not available
Url : http://lists.fedoraproject.org/pipermail/infrastructure/attachments/20071101/9101e1b7/attachment.bin