serverbeach hardware refresh plans / hosted VFAD

[Jon Stanley]

As someone who has intimate knowledge of some special setups on
hosted, I'd be happy to participate. Just let me know when and where
to show up :)

On Thu, Nov 10, 2011 at 4:45 PM, Kevin Fenzi <kevin at scrye.com> wrote:
> I talked about this at todays meeting, but I thought I would note it
> here for further input and also see if there was interest in doing a
> Virtual Fedora Activity Day next week to finish out hosted plans.
>
> background:
>
> We have 5 servers at serverbeach (Thanks for donating them SB Folks!)
>
> They were setup long ago and are getting a bit old in the tooth, so we
> are getting 5 new machines to migrate our stuff to. This will allow us
> to be on hardware thats newer/better supported and also move to new OS
> and re-arrange things. We must migrate to those new machines by
> 2011-12-11. (around a month from now).
>
> current:
>
> serverbeach01 -> a mirror
>
> serverbeach02 -> collab01 (all our lists.fedoraproject.org lists).
>                           ns01
>
> serverbeach03 -> collab02 (a warm spare of collab01, it rsyncs from
> there every hour).
>
> serverbeach04 -> hosted01 (all fedorahosted.org)
>
> serverbeach05 -> hosted02 (a warm spare, but currently down and not
> working).
>
> Plan:
>
> serverbeach06 -> collab03
>                            ns01
> serverbeach07 -> collab04
>
> (move these to rhel6 for both host/guest. Also, look at setting up drbd
> for /srv on them, so we can easily switch over from 03 to 04 in case of
> failure. If we do do drbd, I'd like to schedule at least one failover
> to make sure it really is all working as expected. ns01 can go on
> either. ) We will also need to add drbd rpms to our repo.
>
> That leaves us with hosted to deal with, and serverbeach08/09/10 to do
> it in (currently). We have discussed lots of plans with hosted, I'd
> like to propose meeting up on IRC next Wed to do a virtual Fedora
> Activity Day around planning that. Or we could just try and work out
> details here on list.
>
> Things we have discussed:
>
> * Moving lists.fedorahosted.org to collab*
> * Moving projects to a CNAME or aliased setup so we can then move them
>  to whatever backend host we like. (what exact changes?)
> * Making whatever changes we need now so we could move to spinning out
>  some projects into "the cloud" instances. What are those specific
>  changes?
>
> Issues:
>
> * Backups of hosted01 take ~6 hours, which is not very good. ;( (thats
>  incrementals, full backups are much slower... days).
> * Hosted01 is pretty heavily loaded. Some of which may just disappear
>  when we move to rhel6 (I know some of it will).
> * Ideally we could bring up projects on a new instance in the event of
>  a failure of a node.
>
> I'd like to have a SOP with steps for migrating hosted instances soon.
> So, we can say, 'here's what we are going to change and why'. Then we
> can get setup and start moving projects that opt-in, then have a flag
> day before our migration window to just move everyone else.
>
> Thoughts?
>
> kevin
>
> _______________________________________________
> infrastructure mailing list
> infrastructure at lists.fedoraproject.org
> https://admin.fedoraproject.org/mailman/listinfo/infrastructure
>