serverbeach hardware refresh plans / hosted VFAD

[Kevin Fenzi]

I talked about this at todays meeting, but I thought I would note it
here for further input and also see if there was interest in doing a
Virtual Fedora Activity Day next week to finish out hosted plans. 

background: 

We have 5 servers at serverbeach (Thanks for donating them SB Folks!)

They were setup long ago and are getting a bit old in the tooth, so we
are getting 5 new machines to migrate our stuff to. This will allow us
to be on hardware thats newer/better supported and also move to new OS
and re-arrange things. We must migrate to those new machines by
2011-12-11. (around a month from now). 

current: 

serverbeach01 -> a mirror

serverbeach02 -> collab01 (all our lists.fedoraproject.org lists).
                           ns01 

serverbeach03 -> collab02 (a warm spare of collab01, it rsyncs from
there every hour). 

serverbeach04 -> hosted01 (all fedorahosted.org)

serverbeach05 -> hosted02 (a warm spare, but currently down and not
working). 

Plan: 

serverbeach06 -> collab03
                            ns01
serverbeach07 -> collab04

(move these to rhel6 for both host/guest. Also, look at setting up drbd
for /srv on them, so we can easily switch over from 03 to 04 in case of
failure. If we do do drbd, I'd like to schedule at least one failover
to make sure it really is all working as expected. ns01 can go on
either. ) We will also need to add drbd rpms to our repo. 

That leaves us with hosted to deal with, and serverbeach08/09/10 to do
it in (currently). We have discussed lots of plans with hosted, I'd
like to propose meeting up on IRC next Wed to do a virtual Fedora
Activity Day around planning that. Or we could just try and work out
details here on list. 

Things we have discussed: 

* Moving lists.fedorahosted.org to collab*
* Moving projects to a CNAME or aliased setup so we can then move them
  to whatever backend host we like. (what exact changes?)
* Making whatever changes we need now so we could move to spinning out
  some projects into "the cloud" instances. What are those specific
  changes?

Issues: 

* Backups of hosted01 take ~6 hours, which is not very good. ;( (thats
  incrementals, full backups are much slower... days). 
* Hosted01 is pretty heavily loaded. Some of which may just disappear
  when we move to rhel6 (I know some of it will). 
* Ideally we could bring up projects on a new instance in the event of
  a failure of a node. 

I'd like to have a SOP with steps for migrating hosted instances soon.
So, we can say, 'here's what we are going to change and why'. Then we
can get setup and start moving projects that opt-in, then have a flag
day before our migration window to just move everyone else. 

Thoughts?

kevin
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 198 bytes
Desc: not available
Url : http://lists.fedoraproject.org/pipermail/infrastructure/attachments/20111110/fb706721/attachment.bin