serverbeach hardware refresh plans / hosted VFAD

[seth vidal]

On Thu, 10 Nov 2011 14:45:16 -0700
Kevin Fenzi <kevin at scrye.com> wrote:

> I talked about this at todays meeting, but I thought I would note it
> here for further input and also see if there was interest in doing a
> Virtual Fedora Activity Day next week to finish out hosted plans. 
> 
> background: 
> 
> We have 5 servers at serverbeach (Thanks for donating them SB Folks!)
> 
> They were setup long ago and are getting a bit old in the tooth, so we
> are getting 5 new machines to migrate our stuff to. This will allow us
> to be on hardware thats newer/better supported and also move to new OS
> and re-arrange things. We must migrate to those new machines by
> 2011-12-11. (around a month from now). 
> 
> current: 
> 
> serverbeach01 -> a mirror
> 
> serverbeach02 -> collab01 (all our lists.fedoraproject.org lists).
>                            ns01 
> 
> serverbeach03 -> collab02 (a warm spare of collab01, it rsyncs from
> there every hour). 
> 
> serverbeach04 -> hosted01 (all fedorahosted.org)
> 
> serverbeach05 -> hosted02 (a warm spare, but currently down and not
> working). 
> 
> Plan: 
> 
> serverbeach06 -> collab03
>                             ns01
> serverbeach07 -> collab04
> 
> (move these to rhel6 for both host/guest. Also, look at setting up
> drbd for /srv on them, so we can easily switch over from 03 to 04 in
> case of failure. If we do do drbd, I'd like to schedule at least one
> failover to make sure it really is all working as expected. ns01 can
> go on either. ) We will also need to add drbd rpms to our repo. 
> 
> That leaves us with hosted to deal with, and serverbeach08/09/10 to do
> it in (currently). We have discussed lots of plans with hosted, I'd
> like to propose meeting up on IRC next Wed to do a virtual Fedora
> Activity Day around planning that. Or we could just try and work out
> details here on list. 
> 
> Things we have discussed: 
> 
> * Moving lists.fedorahosted.org to collab*
> * Moving projects to a CNAME or aliased setup so we can then move them
>   to whatever backend host we like. (what exact changes?)
> * Making whatever changes we need now so we could move to spinning out
>   some projects into "the cloud" instances. What are those specific
>   changes?
> 
> Issues: 
> 
> * Backups of hosted01 take ~6 hours, which is not very good. ;( (thats
>   incrementals, full backups are much slower... days). 
> * Hosted01 is pretty heavily loaded. Some of which may just disappear
>   when we move to rhel6 (I know some of it will). 
> * Ideally we could bring up projects on a new instance in the event of
>   a failure of a node. 
> 
> I'd like to have a SOP with steps for migrating hosted instances soon.

1. I think we need some volunteer projects. Do we know of any
enterprising folks who would like to be suckers.... err test subjects?


2. For collab2 and hosted2 - would it be possible to run both of
those secondary machines on one of the new sb boxes? Disk-space wise we
should be able to make it work - and it from a failover standpoint it
feels unlikely we would need to failover both hosted1 and collab1 at
the same time.

3. Then do backups from collab2 and hosted2 - until we have our stuff
together for hosted-tng.


> So, we can say, 'here's what we are going to change and why'. Then we
> can get setup and start moving projects that opt-in, then have a flag
> day before our migration window to just move everyone else. 

I suspect we'll need to waddle them all through a little at a time.
Remember getting rid of cvs.fedoraproject.org? like that :(

-sv
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 190 bytes
Desc: not available
Url : http://lists.fedoraproject.org/pipermail/infrastructure/attachments/20111111/5f86ba1d/attachment.bin