Meeting Log - 2010-04-22

Ricky Zhou ricky at fedoraproject.org
Thu Apr 22 20:40:35 UTC 2010


20:00 < mmcgrath> #startemeeting Infrastructure
20:00 < mmcgrath> #startmeeting Infrastructure
20:00 < zodbot> Meeting started Thu Apr 22 20:00:28 2010 UTC.  The chair is mmcgrath. Information about MeetBot at http://wiki.debian.org/MeetBot.
20:00 < zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
20:00 -!- a-k [~akistler at 2002:638e:131f:3:20d:56ff:fe10:bb8d] has joined #fedora-meeting
20:00 -!- zodbot changed the topic of #fedora-meeting to:  (Meeting topic: Infrastructure)
20:00 < mmcgrath> zodbot: do as I mean not as I say
20:01 < gholms|work> Hehe
20:01  * nirik is hanging around in the cheap seats. 
20:01 < mmcgrath> #topic who's here?
20:01 -!- zodbot changed the topic of #fedora-meeting to: who's here? (Meeting topic: Infrastructure)
20:01  * ricky 
20:01  * a-k is
20:01 < smooge> here
20:01  * Infern4us 
20:01 < smooge> needs coffee
20:02  * mdomsch  
20:02 < mmcgrath> Ok, lets get started
20:02 < mmcgrath> #topic Final Release
20:02 -!- zodbot changed the topic of #fedora-meeting to: Final Release (Meeting topic: Infrastructure)
20:02 < mmcgrath> The final F13 release is on the way here pretty quick.
20:02 < mmcgrath> Our final freeze goes into place on the 4th IIRC.
20:02 -!- ldimaggi_ [~ldimaggi at nat/redhat/x-aeysqvshujydvowu] has quit Quit: Leaving
20:03 < mmcgrath> Anyone have any questions or concerns about that?
20:03 < mmcgrath> any major projects to get deployed before then?
20:03 < mmcgrath> I have only 2 major change
20:03 < mmcgrath> s
20:04 -!- mjg59 [~mjg59 at cavan.codon.org.uk] has quit Remote host closed the connection
20:04 < mmcgrath> Alrighty, well we can move on.
20:04 < ricky> What are the changes?
20:04 < smooge> hmm isn't that the same day as U-10.04?
20:04 < mmcgrath> ricky: going into those right now
20:04 < mmcgrath> #topic Insight
20:04 -!- zodbot changed the topic of #fedora-meeting to: Insight (Meeting topic: Infrastructure)
20:04 < mmcgrath> stickster: ping
20:05 < smooge> mmcgrath, I have no projects for that time. I was going to deploy rsyslog next week
20:05 < mmcgrath> I'm wondering if there's anything we can get in place today now so there's less to do later.
20:05 < stickster> mmcgrath: pong
20:05 < mdomsch> not much time before the freeze then
20:05 < mmcgrath> stickster: hey, so is there any insight bits that can be done now?
20:05 < mmcgrath> anything that, even though the whole project isn't ready, parts could be deployed now?
20:06 < stickster> mmcgrath: There are still both styling and technical bits that have critical or blocker bugs attached
20:06 < mmcgrath> stickster: I'm thinking even if the base stuff is in place and not advertised it'd help increase the chances of success.
20:07 < mmcgrath> what are the nature of the changes that are still to be made?  packaging?  upstream stuff?
20:07  * hydh is here too
20:07 -!- yawns [~yawns at 170.170.59.133] has joined #fedora-meeting
20:07 -!- mjg59 [~mjg59 at cavan.codon.org.uk] has joined #fedora-meeting
20:07 < stickster> mmcgrath: There are problems with the authentication that still need to be solved, then upstreamed to the fedora-zikula module and released
20:07 < stickster> The styling bugs are not as pernicious but will take some time to resolve
20:07 < mmcgrath> so in your estimation, we still on track for deployment later in the month?
20:08 < stickster> mmcgrath: http://lists.fedoraproject.org/pipermail/logistics/2010-April/000510.html
20:08 < mmcgrath> stickster: also, how much of this code is stuff we'll have to maintain?
20:08 < stickster> No, we agreed to push off to post-GA
20:08 < mmcgrath> ah, k.
20:08 < stickster> There's not much code we have to maintain
20:08 < mmcgrath> I missed that, sorry.
20:08 < stickster> AuthFAS module is about it.
20:08 < mmcgrath> excellent.
20:08 < stickster> And that's fairly understandable
20:08 < mmcgrath> stickster: ok, thanks for the latest.  Anything else?
20:09 < stickster> It's the other issues we still have to solve that weren't ready for our go/no-go that caused us to wave off.
20:09 < stickster> logistics@ list is where discussion is taking place about what we're going to do next.
20:09 < stickster> eof
20:09 < mmcgrath> stickster: thanks
20:09 < mmcgrath> ok, next topic
20:09 < mmcgrath> #topic netapp migration
20:09 -!- zodbot changed the topic of #fedora-meeting to: netapp migration (Meeting topic: Infrastructure)
20:10 < mmcgrath> This is something I wanted to have done before the beta but failed to do so
20:10 < smooge> ok what is this?
20:10 < mmcgrath> basically I need to move alt and whatever is left on the secondary1 drives, to the netapp.
20:10 < mmcgrath> smooge: so they'll show up on download.fedora.redhat.com
20:11 < mmcgrath> any questions or concerns about that?
20:11 < mdomsch> nah, they're small
20:11 < mmcgrath> For me the big one is trying to figure out exactly how to let everyone continue to upload their content.
20:11 < smooge> not really
20:12 < mmcgrath> AFAIK it'll all be the same way.
20:12 < mmcgrath> Ok, moving on :)
20:12 < smooge> oh.. there is that
20:12 < mdomsch> log into a server that has it mounted r/w
20:12 < mdomsch> right now that's secondary1 for alt
20:12 < smooge> who is allowed to do this?
20:12 < mmcgrath> mdomsch: well I'm thinking they'd still be allowed to do that
20:12 < mmcgrath> but then I'm not sure what to do with secondary1's actual drives :)
20:12 < mdomsch> altvideo group can for /pub/alt/video/
20:12 < mmcgrath> maybe just have them sync from the netapp and continue to expose.
20:13 < mmcgrath> smooge: there's an SOP
20:13  * mmcgrath gets it
20:13 < mdomsch> yeah
20:13 < mmcgrath> smooge: http://fedoraproject.org/wiki/Content_Hosting_Infrastructure_SOP
20:13 < mmcgrath> giving users direct access to the netapp concerns me a bit
20:13 < mmcgrath> but really it's a completely different share then the /pub/fedora and /pub/epel stuff
20:13 < mmcgrath> and the only thing they could do is fill the disk up which A) we monitor and B) is easy to fix
20:13 < tremble> Which netapp modules are yu using?
20:14 < smooge> ok so it will need to be a seperate partition/log-volume on the netapp
20:14 < mmcgrath> tremble: I always forget.
20:14 < mmcgrath> smooge: <nod> it already is.
20:14 < mmcgrath> smooge: oh wait, not a seperate 'partition' in that way
20:14 < mmcgrath> since we don't really know what future expansion will be
20:15 -!- tatica_dua [~tatica at 190.79.205.119] has joined #fedora-meeting
20:15 < mmcgrath> this will allow either side of the house to grow without us having to guess.
20:15 < mdomsch> so alt.fp.o becomes a new VM too?
20:15 < skvidal> oh crap, the meeting
20:15 < tremble> FWIW $POE uses a netapp.
20:15 < skvidal> sorry about being late
20:15 < smooge> ah well I was wondering about setting up a netapp quota and not having to worry about filling
20:15 < mmcgrath> mdomsch: I haven't figured that part out yet, I might just see if download.fedora.redhat.com will start accepting alt.fedoraproject.org
20:15 < mmcgrath> smooge: that could work too.
20:16 < smooge> it allows for us to also seperate out differing snapshot schedules and such
20:16 < mmcgrath> <nod>
20:16 < mmcgrath> so anyone have any other questions or comments on that/
20:16 < mmcgrath> ?
20:17 < smooge> no we can talk offline
20:17 < mmcgrath> k
20:17 < mmcgrath> next topic!
20:18 < tremble> At $POE we've found that having 1 or 2 aaggregates and multiple thin provisioned volumes works well as long as you monitor the aggregates
20:18 < mmcgrath> #topic collectd
20:18 -!- zodbot changed the topic of #fedora-meeting to: collectd (Meeting topic: Infrastructure)
20:18 < mmcgrath> So I've added some more collectd modules
20:18 < mmcgrath> Of particular interest are these 3
20:18 < mmcgrath> ping test:
20:18 < mmcgrath> .tiny https://admin.fedoraproject.org/collectd/bin/index.cgi?hostname=log01&plugin=ping&timespan=3600&action=show_selection&ok_button=OK
20:18 < zodbot> mmcgrath: http://tinyurl.com/zddwih
20:19 < mmcgrath> postgres connections:
20:19 < mmcgrath> .tiny https://admin.fedoraproject.org/collectd/bin/index.cgi?hostname=db02&plugin=pg_conns&timespan=3600&action=show_selection&ok_button=OK
20:19 < zodbot> mmcgrath: http://tinyurl.com/zddv5d
20:19 < mmcgrath> mdomsch: you might be interested in what happened to mirrormanager there in the last hour
20:19  * gholms|work hopes that doesn't cause extra URLs to show up in the minutes
20:19 < mmcgrath> .tiny https://admin.fedoraproject.org/collectd/bin/index.cgi?hostname=proxy3&plugin=haproxy&timespan=3600&action=show_selection&ok_button=OK
20:19 < zodbot> mmcgrath: http://tinyurl.com/zddtjd
20:20 < mmcgrath> and that's the last one, haproxy by site
20:20 < mdomsch> looking
20:20 < ricky> Is that response time I see?  Veeery nice
20:20 < mmcgrath> ricky: which one?  the haproxy one?
20:21 < mdomsch> ricky, what unites?
20:21 < mmcgrath> nope that's actually...
20:21 < mdomsch> units?
20:21 < mmcgrath> stot: requests/s
20:21 < mmcgrath> econ: errors/s
20:21 < mmcgrath> eresp: err responses/s
20:21 < mdomsch> so, every 10 minutes on the dot, we spike in mirrorlist requests
20:21 < ricky> Ah, OK
20:21 < mmcgrath> econ: is error connections /s
20:21 < mmcgrath> ricky: there's LOTS we can get out of haproxy if you want to add something
20:21 < mdomsch> for about a minute then it drops back down
20:21 < mmcgrath> response time is on my list.
20:21 < mmcgrath> mdomsch: what did you think about MM db connections there?
20:22 < smooge> hmm the tiny urls dont seem to work
20:22 < mmcgrath> smooge: pooh, interesting
20:22 < mmcgrath> use the longer ones then :)
20:23 < mdomsch> mmcgrath, blow that out over a larger time scale...
20:23 < mmcgrath> yeah it's pretty common
20:23  * mdomsch bets that's the crawler with 80 threads
20:23 < mmcgrath> mdomsch: that could very well be.
20:23 < mdomsch> tailing off at the end of the run
20:23 < mmcgrath> yeah
20:23 < mdomsch> it tries to keep 80 threads running at once, starting a new one as one completes
20:24 < mdomsch> so it'll flatline around 80, then tail off, the jump back to 80 for a while
20:24 < smooge> beb back in a sec
20:24 < mmcgrath> <nod>
20:24 < mmcgrath> but yeah, we now have more visibility into our applications then ever before.
20:24 < mmcgrath> we've learned a great deal about our environments just in the last couple of weeks from collectd.
20:24 < mdomsch> yep, that's what it's doing.  Nice graphs. :-)
20:24 < mmcgrath> in particular it's the 10s resolution.
20:24 < mmcgrath> it is just so much detail that we were missing before.
20:24 < smooge> ok dog thrown outside
20:25 < mmcgrath> anyone have any questions / requests?
20:25 < smooge> no thanks for this
20:25 < smooge> oh one question
20:26 < smooge> what does the ping test against?
20:26 -!- mchua_afk is now known as mchua
20:26 < mmcgrath> We can have it run from everywhere but right now I've got it running on log1 (which is the central ping server)
20:26 < mmcgrath> maybe I should have used noc1.
20:26 < mmcgrath> anywho.
20:26 < mmcgrath> it then pings out to the hosts from there
20:26 < mmcgrath> just an ICMP ping
20:26 < mmcgrath> then tracks latency, std dev, and drop rate.
20:26 -!- liknus [~liknus at athedsl-4499885.home.otenet.gr] has quit Ping timeout: 246 seconds
20:27 < mmcgrath> How do you add more hosts?
20:27 < mmcgrath> I'm glad you asked mmcgrath  :)
20:27 < mmcgrath>     collectd::ping { 'ping':
20:27 < mmcgrath>         hosts => ['tummy1.fedoraproject.org', 'telia1.fedoraproject.org', 'serverbeach4.fedoraproject.org', 'serverbeach1.fedoraproject.org', 'osuosl1.fedoraproject.org']
20:27 < ricky> Heheh
20:27 < mmcgrath>     }
20:27 < smooge> ah cool
20:27 < mmcgrath> add that to the node or server group you want
20:27 < mmcgrath> and collectd will do the rest.
20:27 < smooge> I think we want noc01/noc02 as the ping testers.
20:27 < mdomsch> mmcgrath, how does haproxy determine if mirror-lists is down ?
20:27 < smooge> but log01 works too
20:28 < mmcgrath> mdomsch:  it hits /mirrorlist every 5 seconds, 3 failures in a row takes that node out.
20:28 -!- sharkcz [~dan at plz1-v-4-17.static.adsl.vol.cz] has quit Quit: Ukončuji
20:28 < ricky> It should go by timeouts we have set or http status codes
20:28 < mmcgrath> smooge: actually that is a good transition into the next topic I wanted to bring up (also monitoring oriented)
20:28 < mdomsch> ah.  then I bet that's the hourly cache refresh non-responsiveness doing it
20:28 < mmcgrath> anyone have any questions or comments on this?
20:28 < mdomsch> that's kind of a short timeout...
20:29 -!- Oxf13 [~jkeating at fedora/ender] has quit Ping timeout: 246 seconds
20:29 < tremble> Suppose it depends wht you consider acceptable down time.
20:29 < ricky> Oh, that actually makes sense.  I wonder how long one cache refresh takes an app server out for
20:30 < mmcgrath> mdomsch: does the refresh staggar at all?
20:30 < mmcgrath> mdomsch: actually that doesn't match up
20:30 < mmcgrath> it's an hourly refresh.
20:30 < mmcgrath> but we see stuff going down more often then that
20:30 < mmcgrath> mdomsch: take a look at proxy3 -
20:30 < mmcgrath> grep mirror /var/log/messages
20:30 < mmcgrath> anywho, we can discuss that more in a bit.
20:30 < mdomsch> ok, we don't have to solve it here
20:31 < mmcgrath> <nod>
20:31 -!- XulLunch is now known as XulWork
20:31 < mmcgrath> #topic Nagios
20:31 -!- zodbot changed the topic of #fedora-meeting to: Nagios (Meeting topic: Infrastructure)
20:31 < mmcgrath> so
20:31 < mmcgrath> right now we have noc1 and noc2.
20:31 < mmcgrath> if we move to nagios3, it becomes easier to merge the two.
20:31 -!- tibbs [~tibbs at fedora/tibbs] has quit Remote host closed the connection
20:31 < mmcgrath> but I'm still wary about having monitoring only in PHX2
20:32 < smooge> how does it merge them?
20:32  * ricky wouldn't mind moving noc2 out of germany though :-/  As nice as it is to get a perspective from there, it often gives alerts on network issues we can't do anything about
20:32 -!- Oxf13 [~jkeating at c-98-237-133-70.hsd1.wa.comcast.net] has joined #fedora-meeting
20:32 -!- Oxf13 [~jkeating at c-98-237-133-70.hsd1.wa.comcast.net] has quit Changing host
20:32 -!- Oxf13 [~jkeating at fedora/ender] has joined #fedora-meeting
20:32 < mmcgrath> smooge: well, nagios3 has a better ability to realize multiple IPs for a given host.
20:32 < mmcgrath> ricky: agreed.
20:32 < smooge> ah but we would still have hairpin problems
20:32 < mmcgrath> smooge: so we can do the whole 'internal' and 'external' test without problems.
20:32 < mmcgrath> yeah
20:33 < mmcgrath> yeah that's it then, there's a blocker there.
20:33 < mmcgrath> because external to PHX2 we can't monitor everything in phx2.
20:33 < mmcgrath> inside phx2 we can't monitor everything in phx2 :)
20:33 < mmcgrath> so we'll probably have to keep that dynamic at least somewhere.
20:33 < mmcgrath> lets think on it a bit
20:33 < smooge> I would make noc03 in ibiblio and go from there
20:34 < mmcgrath> I'm hoping to work with pvangundy on that, he's a volunteer that's been gone for a while.
20:34 < mmcgrath> He's back but has been busy
20:34  * mmcgrath hopes he gets less busy
20:34 < mmcgrath> anywho, anything else on that for the meeting?
20:34 < mmcgrath> alrighty
20:34 < mmcgrath> #topic search engine
20:34 -!- zodbot changed the topic of #fedora-meeting to: search engine (Meeting topic: Infrastructure)
20:34 < mmcgrath> a-k: whats the latest?
20:35 < a-k> I've got DataparkSearch on publictest3
20:35 < mmcgrath> url?
20:35 < a-k> #link http://publictest3.fedoraproject.org/cgi-bin/dpsearch
20:35 < a-k> DataparkSearch forked from mnoGoSearch in 2003
20:35 < a-k> Mostly so far it seems like a broken version of mnoGoSearch
20:35 < a-k> I've indexed only a tiny number of documents from the wiki
20:36 < mmcgrath> :( that's no fun
20:36 < a-k> I'll poke it a little more to see how bad it is
20:36 < a-k> More docs, etc
20:36 < ricky> Search is hard :-/
20:36 < hydh> hehe
20:36 < mmcgrath> a-k: thanks, anything else for now?
20:36 < a-k> I don't think so
20:36 < mmcgrath> alrighty
20:37 < mmcgrath> Well with that I'll open the floor
20:37 < mmcgrath> #topic Open Floor
20:37 -!- zodbot changed the topic of #fedora-meeting to: Open Floor (Meeting topic: Infrastructure)
20:37 < mmcgrath> anyone have anything they'd like to discuss?
20:37 < mmcgrath> if not we'll close in 30
20:39 < mmcgrath> sweet, silence is golden
20:39 < mmcgrath> ok
20:39 < mmcgrath> #endmeeting
20:39 -!- zodbot changed the topic of #fedora-meeting to: Channel is used by various Fedora groups and committees for their regular meetings | Note that meetings often get logged | For questions about using Fedora please ask in #fedora | See http://fedoraproject.org/wiki/Meeting_channel for meeting schedule
20:39 < zodbot> Meeting ended Thu Apr 22 20:39:12 2010 UTC.  Information about MeetBot at http://wiki.debian.org/MeetBot .
20:39 < mmcgrath> thanks for coming everyone!
20:39 < zodbot> Minutes:        http://meetbot.fedoraproject.org/fedora-meeting/2010-04-22/fedora-meeting.2010-04-22-20.00.html
20:39 < zodbot> Minutes (text): http://meetbot.fedoraproject.org/fedora-meeting/2010-04-22/fedora-meeting.2010-04-22-20.00.txt
20:39 < zodbot> Log:            http://meetbot.fedoraproject.org/fedora-meeting/2010-04-22/fedora-meeting.2010-04-22-20.00.log.html
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 198 bytes
Desc: not available
Url : http://lists.fedoraproject.org/pipermail/infrastructure/attachments/20100422/0d94b9db/attachment-0001.bin 


More information about the infrastructure mailing list