5:36 p.m.
20:00 < skvidal> meeting?
20:00 * skvidal awaits
20:00 * dgilmore is here
20:00 < dgilmore> mmcgrath:
20:00 < dgilmore> mmcgrath:
20:00 < dgilmore> mmcgrath:
20:00 < mmcgrath> #startmeeting
20:00 < zodbot> Meeting started Thu Dec 10 20:00:40 2009 UTC. The chair is
mmcgrath. Information about MeetBot at http://wiki.debian.org/MeetBot.
20:00 < zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
20:00 < mmcgrath> #topic Who's here?
20:00 * mmcgrath is
20:00 -!- zodbot changed the topic of #fedora-meeting to: Who's here?
20:00 * dgilmore
20:00 < mmcgrath> smooge: ping
20:00 -!- a-k [n=akistler@2002:638e:1ba0:3:20d:56ff:fe10:bb8d] has joined #fedora-meeting
20:01 < smooge> here
20:01 * a-k is
20:01 < smooge> putting clothes in dryer
20:01 < mmcgrath> So the outage is this weekend, I figured we should all go over the
timelines and such
20:01 < mmcgrath> jwb: This might interest you
20:01 < mmcgrath> Oxf13: ^^
20:02 * skvidal is here
20:02 * jwb is semi here
20:02 < mmcgrath> So here's the basics
20:02 < mmcgrath> #action - Tonight we'll migrate db1, db2 and the vpn to PHX2
20:02 * nirik is sitting in the back in the cheap seats.
20:02 < a-k> mmcgrath: #topic?
20:02 < mmcgrath> #action - Sometime before Sat. Morning mail will start to go
through bastion3 in PHX2
20:02 < mmcgrath> #topic Timeline
20:02 -!- zodbot changed the topic of #fedora-meeting to: Timeline
20:02 < smooge> #topic movement to PHX
20:02 < mmcgrath> #action - Tonight we'll migrate db1, db2 and the vpn to PHX2
20:03 < mmcgrath> #action - Sometime before Sat. Morning mail will start to go
through bastion3 in PHX2
20:03 * mmcgrath doesn't get this bot stuff sometimes
20:03 < mmcgrath> The mail part is still in an unknown state to me
20:03 < mmcgrath> I've got a ticket open, I've pinged people about it's
priority, I'm waiting to hear back.
20:03 < mmcgrath> #action - Smooge is working on some resolvers for us, those should
be in place ASAP but isn't a blocker for the move.
20:03 < mmcgrath> it is a blocker to start turning things on though.
20:04 < dgilmore> mmcgrath: do we have the new bastion host up?
20:04 < mmcgrath> dgilmore: yes, bastion3.fedoraproject.org is up and running
20:04 < mmcgrath> Ok, this takes us to Friday morning.
20:04 < smooge> #action - Smooge will email out named scheme for PHX2 resolve
domain
20:04 < mmcgrath> #action jwb/ someone in releng will tell me as soon as they are
done writing to /mnt
20:05 < mmcgrath> #action at that time I'll mount the public mirrors as read
only.
20:05 < jwb> mmcgrath, will likely be two sets of things there
20:05 < jwb> updates, and rawhide
20:05 < mmcgrath> jwb: totally fine, just let me know when it's done.
20:05 < jwb> k
20:05 < mmcgrath> Ok, this is where the coordination stuff happens.
20:05 < mmcgrath> I'll be flying to PHX at noon on Friday.
20:05 < mmcgrath> smooge: when do you leave?
20:06 < smooge> I leave at ~0900 tomorrow morning
20:06 < mmcgrath> that's localtime?
20:06 < dgilmore> smooge: MST
20:06 -!- Sonar_Guy [i=Who@fedora/sonarguy] has quit Read error: 104 (Connection reset by
peer)
20:06 < mmcgrath> k
20:07 < mmcgrath> We both have access to PHX1 and PHX2 from Friday to Wed I believe
20:07 * SmootherFrOgZ is around
20:07 < smooge> sorry 0800 tomorrow mornign ariving at ~0900 MST
20:07 < mmcgrath> k
20:07 < mmcgrath> So I'll be heading to PHX1 in the afternoon to give everything
a final look over
20:07 < mmcgrath> this is still Friday.
20:08 -!- Sonar_Guy [i=Who@fedora/sonarguy] has joined #fedora-meeting
20:08 < smooge> mmcgrath, when do you get there
20:08 < mmcgrath> I'll probably shut down stuff like db1 since it's not
being used at that point.
20:08 < smooge> nm.. see now
20:08 < mmcgrath> I get in at 3:30
20:08 < smooge> 03:30 or 15:30
20:08 < mmcgrath> 15
20:08 < skvidal> what's the plan for mail at this point?
20:09 * mdomsch is here finally
20:09 < mmcgrath> skvidal: I said that already :)
20:09 < skvidal> oh
20:09 < skvidal> it had just scrolled off
20:09 < skvidal> so I missed it on the divider :(
20:09 < skvidal> sorry
20:09 < mmcgrath> skvidal: I've got a ticket open and have pinged people about
it's priority
20:09 < mmcgrath> no worries.
20:09 < mmcgrath> I'm hoping for that to get done tonight, if it's not done
by then I'm going to need to hand that off to someone to keep bugging people until it
gets done because I won't be around on Friday
20:09 < mmcgrath> skvidal: but the good news is we aren't the blockers on that
AFAIK.
20:10 < mmcgrath> bastion3 is setup and listening and ready to forward mail.
20:10 < mmcgrath> Ok, so here's what is left
20:10 < mmcgrath> #action Friday night they'll be configuring the new netapp
head unit in PHX2
20:10 < mmcgrath> Then sleep
20:10 < mmcgrath> #action Smooge and Mike will meet at PHX1 at 05:15 am
20:11 < mmcgrath> #action Disable nagios
20:11 < mmcgrath> #action make a final backup of sigul and a few other bits (this
will probably be done the night before, just forgot to meniton it)
20:11 < mmcgrath> #action power everything down
20:11 < mmcgrath> #action movers will arrive at 05:45 to move
20:11 < jwb> action pray
20:11 < jwb> :)
20:12 < mmcgrath> The movers will actually be helping us unhook cables and such.
20:12 < mmcgrath> oh
20:12 < dgilmore> mmcgrath: buildsys we should disable friday night
20:12 < mmcgrath> #action I'll be taking a sigul drive, and smooge will have the
other.
20:12 < jwb> ah, good
20:12 < mmcgrath> dgilmore: we can do it that morning if we want or the night
before
20:12 < smooge> mmcgrath, make sure its got both as being bootable.
20:12 < mmcgrath> The movers themselves will actually be loading the entire racks
onto the truck.
20:12 < smooge> sorry thats meant more as I need to ...
20:13 < smooge> mmcgrath, they will be?
20:13 < mmcgrath> yep, I have no idea how that'll work but I bet it's going
to be wicked awesome.
20:13 < smooge> we aren't getting new racks in PHX or they will move them from
one to another there?
20:13 < dgilmore> mmcgrath: lets do it at night make sure all in progress builds
complete
20:13 < mmcgrath> dgilmore: sure
20:13 < mmcgrath> #action meet the movers at PHX2 where they'll unload the
racks
20:14 < mmcgrath> #action we'll then be moving servers from the old racks (which
I guess are now sitting near the new racks) and re-racking everything.
20:14 * mmcgrath will bring bandaids
20:14 < mmcgrath> now this is where part of our story has changed a bit
20:14 < mmcgrath> we were under the impression contractors would be on site to
re-wire everything
20:14 < mmcgrath> that's not true
20:14 < pjones> bandaids? consider stopping by autozone and getting some work
gloves.
20:15 < mmcgrath> Jonathan (the guy normally onsite at PHX2) is going to be
pre-wiring some of the racks.
20:15 < mmcgrath> depending on how far he gets we may have a lot of cabling to do
ourselves or none.
20:15 < smooge> pjones, I have had a rack go through teflon armored gloves.. they
need blood to work
20:15 < mmcgrath> :)
20:15 < mdomsch> as electronics need smoke
20:15 < pjones> surely you mean kevlar?
20:15 < mmcgrath> so lets see where that leaves us...
20:16 < mmcgrath> #action start powering services back on
20:16 < mmcgrath> let me get the priority list...
20:16 < pjones> (and in any case; knives go through kevlar more easily than they go
through canvas. the high velocity of something like a bullet is where kevlar gets its
strength from...)
20:16 < mmcgrath> #action get the app servers online first
20:16 < mmcgrath> #action then the buildsystem
20:17 < mmcgrath> I only put them in that order because the app servers should be
very straight forward.
20:17 < dgilmore> do we have dhcp?
20:17 < mmcgrath> dgilmore: yeah, and we run it
20:17 -!- sijis_afk [n=sijis(a)70.131.83.26] has joined #fedora-meeting
20:17 < dgilmore> should we reconfigure all the builders first for the new networks
20:17 * mmcgrath is skipping the network config stuff for a later part of the meeting
20:17 < dgilmore> ok
20:17 < mmcgrath> because some of it is quite a bit different
20:17 < mmcgrath> but we'll get to that.
20:17 < mmcgrath> Now it's while we're re-racking this stuff that we will
need help from people.
20:18 < mmcgrath> I've already worked on a checklist via CSI
20:18 < mmcgrath> basically I'm looking for people to re-certify these hosts
20:18 < mmcgrath> what does that mean?
20:18 < smooge> pjones, I meant kevlar.. but teflon would expect why I couldn't
hold anything
20:18 < mmcgrath> verify remote management, power and cyclades works, as well as
network.
20:18 < mmcgrath> this is stuff anyone in sysadmin-main can do.
20:19 < mmcgrath> Ok, so any questions about that?
20:19 < smooge> mmcgrath, pointer to checklist
20:19 < mmcgrath> smooge: I'm going to be adding that info to the ticket in a
moment -
20:19 < mmcgrath> https://fedorahosted.org/fedora-infrastructure/ticket/1845
20:19 < smooge> ok cool
20:19 < mmcgrath> Ok, so as far as timeline goes and the physical work of things are
there any questions?
20:20 < mmcgrath> Sometime after the move we'll be moving the db hosts back to
their normal machines.
20:20 < mmcgrath> Ok, so the next topic is about what the PHX2 world will look
like.
20:20 < mmcgrath> #topic PHX2 - what does it look like?
20:20 -!- zodbot changed the topic of #fedora-meeting to: PHX2 - what does it look like?
20:21 < mmcgrath> So in PHX2 we're going to have 3 networks and might add a 4th
one later.
20:21 < mmcgrath> The 3 networks are as follows
20:21 -!- rdieter [n=rdieter(a)ip174-71-102-133.om.om.cox.net] has joined #fedora-meeting
20:21 < mmcgrath> 1) public network - 10.5.126
20:21 < mmcgrath> 2) build network - 10.5.125
20:21 < mmcgrath> 2) storage network - 10.5.127
20:21 < mmcgrath> the majority of our services will end up on the buidl network
20:21 < mmcgrath> that should be 3) storage :)
20:22 < mmcgrath> and the majority of our services will end up on the public
network, not build.
20:22 * mmcgrath needs more sleep.
20:22 < mmcgrath> the build network will be mostly for releng
20:22 < mmcgrath> and the storage network will be for nfs traffic and possibly
backups.
20:22 < mmcgrath> I'm going to be doing tests to find out where backups best
work and have the least impact on other things.
20:23 < mmcgrath> This also means that several of our hosts will now have multiple
IP addresses.
20:23 -!- notting [n=notting@redhat/notting] has joined #fedora-meeting
20:23 < mmcgrath> for example relengX will be on the build and storage network.
20:23 < mmcgrath> we won't be routing them.
20:23 -!- sandorlev [n=sandorle(a)87.97.110.134.pool.invitel.hu] has joined #fedora-meeting
20:23 < mmcgrath> Anyone have any questions on this?
20:23 < sandorlev> hello
20:24 < mmcgrath> Ok. so that's really it.
20:24 < mmcgrath> I'm working on a spreadsheet right now that I'd appreciate
some second looks at.
20:24 < mdomsch> I presume we'll only have DNS A records for the inbound service
IPs on those dual-homed boxes
20:25 < mmcgrath> mdomsch: I believe so.
20:25 < mdomsch> e.g. build1's IP on network 3 won't have an A record
20:25 < mmcgrath> we'll be running our own DNS servers in PHX2, smooge is
working on that now.
20:25 < mmcgrath> mdomsch: correct but it will probably have a reverse.
20:25 < mmcgrath> smooge: what do you think?
20:25 < smooge> i am going to have two boxes eventually but at the moment just one..
ns001.phx2.fedoraproject.org
20:26 < mdomsch> mmcgrath, makes sense
20:26 < mdomsch> reverse is good to keep track of allocated addresses at least
20:27 < smooge> the hosts will just be on the phx2 subdomain which will be a
'hidden' domain (not on public dns since it will contain 192 and 10.x ips)
20:27 < mmcgrath> smooge: also on the meeting yesterday I think it was requested
that we RH to still be able to do lookups and reverse lookups so we might be stuck doing
fedora.phx2.redhat.com
20:27 < mmcgrath> unless there's an easy way to just mask it for them?
20:27 * mmcgrath hasn't thought about that much.
20:27 < smooge> I am going to keep it simple in version one because we don't
ahve a lot of time.. but dual homed would be later
20:28 < smooge> mmcgrath, I think in either case they will be able to do zone
transfers from our servers
20:28 < mmcgrath> smooge: wise idea
20:28 < mmcgrath> Ok, so any questions about this?
20:28 < smooge> so we could call it fedoraproject.int and it should be good
20:29 < mmcgrath> at present there is no firewall preventing access to or from any
of the 3 networks I've listed.
20:29 < mmcgrath> we'll be doing that later, just ran out of time.
20:29 < nb> i would suggest not using .int because .int is a real TLD
20:29 < smooge> nb it was a joke sorry
20:29 -!- nucleo [n=nucleo@fedora/nucleo] has quit Remote closed the connection
20:29 < nb> smooge, oh ok :)
20:29 < mmcgrath> Ok, so anyone else have any questions about this?
20:29 < mmcgrath> I really have no estimate for how long this will take.
20:30 < mmcgrath> I've never moved this many machines before and I have no idea
how much is already done and how many people we'll have helping
20:30 < smooge> mmcgrath, you are leaving on Tuesday correct?
20:30 < smooge> smooge will be leaving on Wednesday so we wont lose coverage
afterwords
20:30 < smooge> and we have problems like "Oh crap that disk drive didn't
make it."
20:30 < mmcgrath> smooge: I'm not sure actually I need to look at my trip info.
20:31 < smooge> in case of real emergency I will just get back in the car and drive
back til we are stable.
20:31 < mmcgrath> so that's really all I have to discuss
20:31 < mmcgrath> anyone else have any questions or comments?
20:31 < mmcgrath> if not I'll get back to my IP list and let smooge get back to
DNS stuff.
20:32 -!- nucleo [n=nucleo@fedora/nucleo] has joined #fedora-meeting
20:32 < nb> anything non-main people can do to help?
20:32 < mmcgrath> #topic open floor
20:32 -!- zodbot changed the topic of #fedora-meeting to: open floor
20:32 < mmcgrath> nb: I'm hoping the non-main people will help with logistics
and troubleshooting
20:32 < nb> ok
20:32 < mmcgrath> when the services start coming back online and interacting with
eachother I suspect there will be lots of little bugs
20:32 < mmcgrath> like say, app2's network is behaving poorly
20:33 < mmcgrath> nb: but also keeping watch in #fedora-admin and #fedora
20:33 * stickster wants to say a big "thank you" from all the people in Fedora
that won't know to whom they should say it... for working so hard to keep us in
business during the move.
20:34 < mmcgrath> nb: but yeah, when stuff comes back online smooge and I will
probably say stuff like "the wiki is back"
20:34 < mmcgrath> we'll need verification of that :)
20:34 -!- sandorlev [n=sandorle(a)87.97.110.134.pool.invitel.hu] has quit "Have to
go"
20:34 < nb> ok
20:34 < sijis_afk> i know i've missed most of the meeting.. but its this
weekend, right?
20:34 < mmcgrath> sijis_afk: correct.
20:34 < mmcgrath> stickster: :)
20:34 < mmcgrath> any other questions?
20:34 < sijis_afk> i should be around and help if possible
20:34 -!- sijis_afk is now known as sijis
20:35 < mmcgrath> when I'm done with this IP list I could use some extra eyes
20:35 * nb will be around saturday and sunday afternoon/evening
20:35 < mmcgrath> Anyone that'll be around in the next hour or so just look, try
to find duplicates or where I've done something stupid.
20:35 < nb> mmcgrath, ok
20:35 < mmcgrath> If no one has anything else, I'll close the meeting in 30
20:36 < mmcgrath> alllrighty
20:36 < mmcgrath> #endmeeting
20:36 -!- zodbot changed the topic of #fedora-meeting to: Channel is used by various
Fedora groups and committees for their regular meetings | Note that meetings often get
logged | For questions about using Fedora please ask in #fedora | See
http://fedoraproject.org/wiki/Communicate/FedoraMeetingChannel for meeting schedule
20:36 < zodbot> Meeting ended Thu Dec 10 20:36:46 2009 UTC. Information about
MeetBot at http://wiki.debian.org/MeetBot .
20:36 < zodbot> Minutes:
http://meetbot.fedoraproject.org/fedora-meeting/2009-12-10/fedora-meeting...
20:36 < zodbot> Minutes (text):
http://meetbot.fedoraproject.org/fedora-meeting/2009-12-10/fedora-meeting...
20:36 < zodbot> Log:
http://meetbot.fedoraproject.org/fedora-meeting/2009-12-10/fedora-meeting...