#fedora-meeting: Infrastructure (2011-04-14)
Meeting started by nirik at 19:00:17 UTC. The full logs are available at
* Robot Roll Call (nirik, 19:00:17)
* New Folks Intros (nirik, 19:04:20)
* Fedora 15 Beta tickets. (nirik, 19:10:18)
* Upcoming items and outages (nirik, 19:17:47)
* Meeting tickets (nirik, 19:27:56)
* on cvs01, I am waiting to hear back from the last 2 projects on
there. They may move to fedorahosted, or fedorapeople. (nirik,
* Other tickets (nirik, 19:36:46)
* Open Floor (nirik, 19:57:57)
Meeting ended at 20:00:52 UTC.
Action Items, by person
People Present (lines said)
* nirik (112)
* skvidal (28)
* smooge (27)
* CodeBlock (22)
* ricky (17)
* sijis (15)
* zodbot (10)
* ranjibd (6)
* rbergeron (6)
* jsmith (5)
* goozbach (4)
* crashmag (4)
* jpattonwx (2)
* codeblock (0)
19:00:17 <nirik> #startmeeting Infrastructure (2011-04-14)
19:00:17 <zodbot> Meeting started Thu Apr 14 19:00:17 2011 UTC. The chair is nirik.
Information about MeetBot at http://wiki.debian.org/MeetBot
19:00:17 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
19:00:17 <nirik> #meetingname infrastructure
19:00:17 <zodbot> The meeting name has been set to 'infrastructure'
19:00:17 <nirik> #topic Robot Roll Call
19:00:17 <nirik> #chair goozbach smooge skvidal codeblock ricky nirik
19:00:17 <zodbot> Current chairs: codeblock goozbach nirik ricky skvidal smooge
19:00:19 <zodbot> rbergeron: Error: Can't start another meeting, one is in
19:00:21 <rbergeron> ohh
19:00:29 * CodeBlock here
19:00:29 <nirik> humf
19:00:30 <nirik> oops
19:00:32 * rbergeron always does this to you guys
19:00:43 <rbergeron> if anyone pops in, send them to f-m-1
19:00:43 <goozbach> wait, what?
19:00:45 <rbergeron> that's one, not L
19:00:46 <smooge> what
19:00:46 <rbergeron> :)
19:00:58 <goozbach> hah!
19:01:00 <goozbach> ok ok
19:01:02 * goozbach here
19:01:04 <smooge> remember L
19:01:08 <rbergeron> :)
19:01:09 * jpattonwx here, lurking
19:01:49 * jsmith lurks
19:02:40 * sijis is around
19:03:00 * crashmag says hi
19:03:20 <nirik> sorry about that...
19:03:27 <nirik> somehow I didn't realize they were the same time. ;(
19:03:36 <smooge> well we have it first
19:03:47 <smooge> who needs a release anyway
19:04:11 <nirik> ok, lets gets started I guess.
19:04:20 <nirik> #topic New Folks Intros
19:04:28 <nirik> any new folks lurking here this week?
19:04:36 <crashmag> yap :)
19:04:37 <nirik> want to say hi and such?
19:04:45 * jsmith hasn't said hi in a long time :-p
19:05:04 <nirik> crashmag: welcome.
19:05:05 <smooge> I will be able to help people get into infrastructure after this
release I believe
19:05:33 <CodeBlock> I can sponsor some people after the semester ends, too
19:05:43 * skvidal is here
19:05:46 <skvidal> sorry I was late
19:05:52 <CodeBlock> skvidal: no. unexcusable.
19:05:53 <jpattonwx> I think I will wait until after semester here too before
worrying about sponsorship
19:05:55 <nirik> crashmag: any thoughts on areas you are interested in? do lurk in
#fedora-admin and/or #fedora-noc and ask questions and chime in when you can.
19:06:07 <skvidal> CodeBlock: I know - I shall go flagellate myself
19:06:08 <CodeBlock> skvidal: just like my bad grammer. inexcusable*
19:06:45 * nirik points new folks to
lots of good info there.
19:07:07 <crashmag> nirik, well I work for an ITSM company. I realise all of this is
based upon trust, and that takes time. So I was hoping that theres a way at working on
"lower level" tickets to begin with.
19:08:02 <nirik> crashmag: sure. ;) Do look at our trac instance and
19:08:16 <nirik> there are some tasks that don't need much access at all...
19:08:25 <crashmag> started using linux around 02. my geek factor can be measured by
my 3 servers at home. was hoping to help out and learn by doing so :)
19:08:56 <nirik> excellent.
19:09:27 <nirik> often lurking in the above channels can get you a feel for things
going on, and people are usually happy to explain, etc.
19:09:55 <smooge> oooh itsm I just closed some tickets on that
19:10:08 <nirik> ok, shall we touch on Beta tickets?
19:10:18 <nirik> #topic Fedora 15 Beta tickets.
19:10:58 <nirik> so beta will be released tuesday.
19:11:05 <nirik> we have several tickets to make sure that goes well.
19:11:15 <nirik> .ticket 2704
19:11:16 <zodbot> nirik: #2704 ([Fedora 15 Beta] New Website) - Fedora
Infrastructure - Trac - https://fedorahosted.org/fedora-infrastructure/ticket/2704
19:11:45 * nirik sees sijis is answering for websites in the other meeting right now. ;)
19:11:46 * CodeBlock looks at sijis
19:12:39 <nirik> we can come back to this one.
19:12:45 <sijis> ok.
19:12:45 <nirik> .ticket 2707
19:12:46 <zodbot> nirik: #2707 ([Fedora 15 Beta] Permissions on mirrors) - Fedora
Infrastructure - Trac - https://fedorahosted.org/fedora-infrastructure/ticket/2707
19:12:59 <nirik> we need content first, then we can check this.
19:13:16 <nirik> .ticket 2708
19:13:17 <zodbot> nirik: #2708 ([Fedora 15 Beta] Mirror manager repository
redirects) - Fedora Infrastructure - Trac -
19:13:27 <nirik> will need to check with mdomsch on that one.
19:13:56 <nirik> sijis: any changes in size or anything for the Beta website? any
other notable changes?
19:14:41 <sijis> nope. the size will increase a little but not significant
19:14:48 <sijis> about 5-10mb (if that)
19:14:55 <sijis> its because of the countdown banner
19:15:02 <nirik> ok, cool.
19:15:31 <nirik> .ticket 2706
19:15:32 <zodbot> nirik: #2706 ([Fedora 15 Beta] Release Day) - Fedora
Infrastructure - Trac - https://fedorahosted.org/fedora-infrastructure/ticket/2706
19:15:43 <nirik> .ticket 2710
19:15:45 <zodbot> nirik: #2710 ([Fedora 15 Beta] Infra change freeze) - Fedora
Infrastructure - Trac - https://fedorahosted.org/fedora-infrastructure/ticket/2710
19:15:45 <smooge> thats yours as boss
19:16:03 <nirik> Thats for release day issues, and a reminder that we are in freeze.
19:16:06 <nirik> smooge: yep. ;)
19:16:12 <CodeBlock> :P
19:16:23 <nirik> I think thats all the beta tickets. Anyone have anything else Beta
19:16:26 <smooge> I closed the disk space one. we are way under 1TB at the moment
19:17:21 <nirik> excellent.
19:17:47 <nirik> #topic Upcoming items and outages
19:17:52 <smooge> I will be starting a sync of F13 over to archives soon and a sync
of secondary1 over to the netapp also
19:18:12 <nirik> I have a list in the post on the items I know of that are coming up
after this freeze.
19:18:27 <nirik> smooge: ok. 13 has a bit more life, but not too much.
19:18:48 <nirik> whats left on secondary1? staging stuff, some video content from
old fudcons, etc?
19:18:49 <CodeBlock> smooge: not sure what that entails, but if you need some help
and feel like teaching me, by all means, would love to learn :)
19:19:35 <smooge> yeah I sync stuff over just to make it easier for when it is EOL
19:19:44 <nirik> The end of next week we will be having an outage to move db03,
nfs01, and pkgs01 (so it will be an outage for building/packagers).
19:20:10 <smooge> and fas01
19:20:43 <nirik> oh, I didn't have that one on my list. That was a rhel6 guest
on a xen5 host we wanted to move?
19:21:02 <smooge> correct
19:21:14 <smooge> it will cut a bunch of issues down
19:21:27 <smooge> as it silently reboots a lot and sometimes not so silent
19:21:37 <smooge> it just needs to be built onto virthost03
19:21:38 <nirik> ok. will add to the list. We will figure out exact timing on them
and I will send out an outage notice.
19:22:05 <smooge> fas01 contains various certs and such so when it is going down it
will degrade other items
19:22:39 <nirik> ok. We will need to determine that. Ideally we can sync it to a new
one in advance so the outage window is small.
19:23:03 <nirik> db03 will take the longest, as it will be a dump/reload on new
19:23:24 <CodeBlock> indeed
19:23:32 <smooge> pkgs01 may be the simplest
19:23:47 <nirik> yeah, it should be just shutdown in one place and bring up in
19:24:18 <ranjibd> db02 can be done like that too right?
19:24:25 <ranjibd> sorry db03
19:24:43 <nirik> well, no, as we are switching it from rhel5 older db version to
19:24:57 <nirik> it's now a rhel5 box with a custom postgresql on it.
19:25:07 <nirik> it will go to rhel6 with the normal version.
19:25:43 <nirik> eventually we will move the other db's over to rhel6 too... but
not next week
19:26:22 <nirik> after next week we have puppet upgrade and fpca landing, then after
final we are going to upgrade nagios/noc01. ;)
19:26:35 <CodeBlock> yep
19:26:38 <nirik> anyone have any other upcoming items they would like to discuss
19:27:02 * CodeBlock still wants to start playing with moving hosted to EL6, but that is
looking like a summer project
19:27:32 <nirik> yeah, that would be great to get done...
19:27:33 <nirik> someday.
19:27:51 <nirik> ok, meeting tickets...
19:27:56 <nirik> #topic Meeting tickets
19:28:42 <nirik> a grab bag of things. ;)
19:29:05 <nirik> #info on cvs01, I am waiting to hear back from the last 2 projects
on there. They may move to fedorahosted, or fedorapeople.
19:29:43 <nirik> Anyone see a meeting ticket they would like to comment on?
19:31:11 <CodeBlock> what ever happened with moving blogs off of infra?
19:31:31 <CodeBlock> (i.e. ticket 2591)
19:32:22 <nirik> it's not really moving forward. ;) I think ricky was going to
look into it more when he had time...
19:32:38 <nirik> it needs someone talking to wordpress.com
or the like... and then
talking to the blog users.
19:32:41 <smooge> yeah. it needs a key person to push it forward.
19:32:45 <smooge> sorry slow type
19:32:46 <sijis> yeah, i'm not sure what's left on that.
19:33:06 <smooge> I would say if its not in the ticket.. then everything is left on
19:33:17 * jsmith has some friends at wordpress.com
, but they're busy cleaning up from
the recent security incident
19:33:19 <sijis> is it contact wp and see if they will host that stuff for us and
contacting users and seeing if they need help moving (or are we just moving stuff for
19:33:23 <jsmith> I'd be happy to make some introductions
19:33:33 <sijis> oh yeah. i read that this morning
19:33:34 <nirik> jsmith: That would be cool.
19:34:05 <nirik> yeah, contact them and see what they can do for us... then once we
know that, contact all our blog users and tell them about it...
19:34:26 <nirik> I think it will depend on what they want to do if we can help move
them or if users move their own.
19:34:37 <nirik> and we will need to set a sunset date...
19:34:56 <sijis> i believe they can export all their content with a single click.
19:35:07 <nirik> yeah, should be pretty easy.
19:35:37 <nirik> we also want to explain to them why we are doing this and such...
so no one is blindsided or confused.
19:35:56 <sijis> good point. its noted
19:36:27 <nirik> so, if anyone would like to drive that forward, please feel free.
19:36:46 <nirik> #topic Other tickets
19:36:56 <nirik> Any other tickets folks would like to bring up or note?
19:37:32 <ranjibd> nagios upgrade? skvidal was telling last time it didn't work
due to some issues
19:37:47 <skvidal> ranjibd: nirik and CodeBlock have the knowledge there
19:37:57 <nirik> yeah, ran into issues with our irc bot, and backed out the upgrade.
19:38:08 <nirik> CodeBlock is planning on doing it again after f15 final.
19:38:17 <nirik> Hopefully it will go smoothly this time. ;)
19:38:39 <CodeBlock> indeed
19:38:50 <ranjibd> puppet upgrade ? can i help any way?
19:38:53 <sijis> is that why we have zodbot and fedbot?
19:39:45 <smooge> possibly.. it was looking at what we could break out
19:39:48 <nirik> sijis: nope. ;) fedbot is one I run locally, mostly for #fedora.
It's got some non packaged plugins and such, also it's locked down to not respond
to lots of things, which we wouldn't want zodbot to do.
19:40:48 <nirik> ranjibd: it should be pretty easy. Basically just upgrade the
master, upgrade a 5 and 6 client and test... it should go smoothly.
19:41:13 <nirik> I don't think splitting zodbot is going to help any.
19:41:22 <ranjibd> nirik: all the clients have same version ?
19:41:43 <skvidal> splitting zodbot won't help the migration - but it will make
the bot in general make a lot more since where it should live
19:42:03 <skvidal> having our meetingbot on our nagios server makes no sense to me
19:42:22 <nirik> skvidal: yeah... but where should the meeting part go?
19:42:35 <skvidal> nirik: away
19:42:47 <skvidal> nirik: away from our nagios server
19:42:50 <skvidal> it makes me nervous
19:42:59 <nirik> ranjibd: yeah.
19:43:08 <ricky> For what it's worth, there's no reason that zodbot has to
be on the same machine as noc01 to get nagios notifications
19:43:24 <ricky> So moving zodbot to value or something is easy
19:43:27 <nirik> skvidal: ok, we can figure something out.
19:43:37 <nirik> yeah, value might make some sense.
19:43:59 <nirik> we do need to serve the meetbot html stuff.
19:44:24 * nirik makes a note.
19:44:40 <nirik> splitting it won't help the lag on restart unless we split it
out by channels...
19:45:02 <nirik> we may be able to prune the channels it's joining...
19:45:43 <ricky> Sorry, I missed the beginning due to internet issues - what does
splitting zodbot mean?
19:46:00 <sijis> it would be two bot.. zod and bot :)
19:46:19 <sijis> ok. i need a break. (lame joke)
19:46:26 <nirik> there was talk about splitting it into fasbot, nocbot, meetbot
19:46:33 <smooge> smoogebot
19:46:35 <nirik> sijis: :)
19:46:51 <skvidal> I suggested splitting it logically
19:46:55 <skvidal> into its functional components
19:47:03 <skvidal> and it's authenticated-components
19:47:09 <skvidal> nocbot has more info thant meetboot
19:47:21 <skvidal> fasbot has different access than meetbot or nocbot
19:47:39 <skvidal> conflating all 3 seemed like a recipe for problems
19:47:42 <sijis> isn't that just more stuff to oversee than 1 bot?
19:47:57 <skvidal> also - from a services and meta-services standpoint
19:48:18 <skvidal> it does not make sense to have a user-facing service (zodbot)
living on a meta-service server (noc01)
19:48:34 <skvidal> just like it wouldn't make sense to run a webserver for the
wiki on our dhcp server
19:48:41 * nirik nods.
19:48:51 <skvidal> that way if noc01 goes down
19:48:58 <skvidal> we don't have to worry about it impacting user-facing
19:49:00 <nirik> I think moving it is a good idea. Splitting it I am more meh on.
19:49:38 <CodeBlock> I like the idea of splitting it out, but that's me.
19:49:57 <ricky> Same - supybot is pretty simple/stable, and it only really touches
data that's public anyway
19:50:06 <ricky> So while I wouldn't mind splitting it out, I don't see it
as a huge priority or a huge gain
19:50:20 <skvidal> is everyone on board with moving it OFF of noc01?
19:50:24 <CodeBlock> yes
19:50:25 <nirik> yes.
19:50:27 <smooge> yes
19:50:36 <skvidal> great
19:50:37 <smooge> let me rephrase, YES
19:50:38 <sijis> i do agree that it doens't make sense that its on noc1. it
should be in value1/2 as ricky mentioned
19:50:45 <skvidal> value## seems like a winner
19:50:52 <ricky> Yup, only change would be firewall changes to accept supybot-notify
+ pointing nagios at the new IP
19:51:02 <ricky> Ideally it should not be on a VPN host so that we get notifications
if VPN goes down
19:51:04 <nirik> so, how about we move it sooner rather than later... and if someone
wants to drive splitting it down the road, they can
19:51:30 <nirik> value already has a webserver for http://meetbot.fedoraproject.org/
19:51:32 <nirik> ?
19:51:59 <ricky> We currently treat value like an app server, so we could proxy that
through the proxies like we normally do if we want
19:52:13 <ricky> Or it'd be fine to poke a hole for it too.
19:52:34 <nirik> ok.
19:53:01 <nirik> does someone want to take the task of moving it? I guess I can...
19:53:05 <CodeBlock> heh, less to worry about for the noc01/noc03-tmp switchover
19:53:11 <CodeBlock> nirik: I can if you don't want
19:53:32 <nirik> CodeBlock: sure, feel free. :) It will require doing it when no
meetings are around...
19:53:41 <CodeBlock> when are we looking to do it? And what kind of an outage notif
needs sent out?
19:53:51 <smooge> bah.. I think we time it for rbergeron's most important
meeting of the week
19:54:02 <ricky> Probably an outage notification to devel-announce + infrastructure
19:54:10 <nirik> well, we could add it to the pile next week, but thats getting lots
19:54:16 <ricky> And as a college student, I'm sure you'll be around at 3 AM
some days, right? :-D
19:54:20 <nirik> it shouldn't be frozen, so I suppose we could do it most
19:54:29 <nirik> oh, that reminds me...
19:54:38 <ricky> (half-kidding - on one hand, changes at 3 AM aren't that great
- on the other hand, zodbot's dead simple)
19:55:40 <nirik> What would folks think of the following addition to outage
notifications: Once you write up one, have one other person review and approve before
sending. It's so easy to mess up timezone stuff, or the like.
19:56:02 <smooge> that would be good
19:56:03 <ricky> Good idea :-)
19:56:09 <CodeBlock> ricky: Yeah 3AM is perfectly fine for me ;D
19:56:10 <skvidal> I have a counterproposal
19:56:11 <smooge> it would make most of my emails though less funny
19:56:13 <ricky> Doesn't have to be formal or anything, just an "OK"
19:56:20 <skvidal> how about we make all timezones Eastern Standard
19:56:20 <ricky> Right?
19:56:29 <skvidal> so we don't have to mess around with silly things like
timezones at all
19:56:30 <skvidal> :)
19:56:32 <nirik> yeah, no red tape, just another pair of eyes.
19:56:33 * skvidal kids
19:56:39 <nirik> skvidal: fine with me. ;)
19:56:44 <CodeBlock> ;D
19:57:02 <ranjibd> +1 for UCT
19:57:06 <nirik> anyone object to the above? will add to sop...
19:57:21 <jsmith> +1 for second set of eyes
19:57:52 <nirik> cool.
19:57:57 <nirik> #topic Open Floor
19:58:01 <nirik> Anything for open floor?
19:58:39 <CodeBlock> 3 weeks or so left in my semester then I'll have a lot more
time to be around :)
19:58:39 <ricky> Wow, perfect timing... this has got to be a first!
19:58:50 * CodeBlock assumes same for ricky
19:59:15 <ricky> Yup, will have a lot more time this year
19:59:36 <nirik> cool. ;)
19:59:59 <nirik> ok, if nothing else, will close out the meeting in a minute here.
20:00:49 <nirik> Thanks for coming everyone!
20:00:52 <nirik> #endmeeting