serverbeach hardware refresh plans / hosted VFAD

[Kevin Fenzi]

I talked about this at todays meeting, but I thought I would note it
here for further input and also see if there was interest in doing a
Virtual Fedora Activity Day next week to finish out hosted plans. 

background: 

We have 5 servers at serverbeach (Thanks for donating them SB Folks!)

They were setup long ago and are getting a bit old in the tooth, so we
are getting 5 new machines to migrate our stuff to. This will allow us
to be on hardware thats newer/better supported and also move to new OS
and re-arrange things. We must migrate to those new machines by
2011-12-11. (around a month from now). 

current: 

serverbeach01 -> a mirror

serverbeach02 -> collab01 (all our lists.fedoraproject.org lists).
                           ns01 

serverbeach03 -> collab02 (a warm spare of collab01, it rsyncs from
there every hour). 

serverbeach04 -> hosted01 (all fedorahosted.org)

serverbeach05 -> hosted02 (a warm spare, but currently down and not
working). 

Plan: 

serverbeach06 -> collab03
                            ns01
serverbeach07 -> collab04

(move these to rhel6 for both host/guest. Also, look at setting up drbd
for /srv on them, so we can easily switch over from 03 to 04 in case of
failure. If we do do drbd, I'd like to schedule at least one failover
to make sure it really is all working as expected. ns01 can go on
either. ) We will also need to add drbd rpms to our repo. 

That leaves us with hosted to deal with, and serverbeach08/09/10 to do
it in (currently). We have discussed lots of plans with hosted, I'd
like to propose meeting up on IRC next Wed to do a virtual Fedora
Activity Day around planning that. Or we could just try and work out
details here on list. 

Things we have discussed: 

* Moving lists.fedorahosted.org to collab*
* Moving projects to a CNAME or aliased setup so we can then move them
  to whatever backend host we like. (what exact changes?)
* Making whatever changes we need now so we could move to spinning out
  some projects into "the cloud" instances. What are those specific
  changes?

Issues: 

* Backups of hosted01 take ~6 hours, which is not very good. ;( (thats
  incrementals, full backups are much slower... days). 
* Hosted01 is pretty heavily loaded. Some of which may just disappear
  when we move to rhel6 (I know some of it will). 
* Ideally we could bring up projects on a new instance in the event of
  a failure of a node. 

I'd like to have a SOP with steps for migrating hosted instances soon.
So, we can say, 'here's what we are going to change and why'. Then we
can get setup and start moving projects that opt-in, then have a flag
day before our migration window to just move everyone else. 

Thoughts?

kevin

[Jon Stanley]

As someone who has intimate knowledge of some special setups on
hosted, I'd be happy to participate. Just let me know when and where
to show up :)

On Thu, Nov 10, 2011 at 4:45 PM, Kevin Fenzi <kevin(a)scrye.com&gt; wrote:
>  I talked about this at todays meeting, but I thought I would note it
>  here for further input and also see if there was interest in doing a
>  Virtual Fedora Activity Day next week to finish out hosted plans.
>
>  background:
>
>  We have 5 servers at serverbeach (Thanks for donating them SB Folks!)
>
>  They were setup long ago and are getting a bit old in the tooth, so we
>  are getting 5 new machines to migrate our stuff to. This will allow us
>  to be on hardware thats newer/better supported and also move to new OS
>  and re-arrange things. We must migrate to those new machines by
>  2011-12-11. (around a month from now).
>
>  current:
>
>  serverbeach01 -> a mirror
>
>  serverbeach02 -> collab01 (all our lists.fedoraproject.org lists).
>                            ns01
>
>  serverbeach03 -> collab02 (a warm spare of collab01, it rsyncs from
>  there every hour).
>
>  serverbeach04 -> hosted01 (all fedorahosted.org)
>
>  serverbeach05 -> hosted02 (a warm spare, but currently down and not
>  working).
>
>  Plan:
>
>  serverbeach06 -> collab03
>                             ns01
>  serverbeach07 -> collab04
>
>  (move these to rhel6 for both host/guest. Also, look at setting up drbd
>  for /srv on them, so we can easily switch over from 03 to 04 in case of
>  failure. If we do do drbd, I'd like to schedule at least one failover
>  to make sure it really is all working as expected. ns01 can go on
>  either. ) We will also need to add drbd rpms to our repo.
>
>  That leaves us with hosted to deal with, and serverbeach08/09/10 to do
>  it in (currently). We have discussed lots of plans with hosted, I'd
>  like to propose meeting up on IRC next Wed to do a virtual Fedora
>  Activity Day around planning that. Or we could just try and work out
>  details here on list.
>
>  Things we have discussed:
>
>  * Moving lists.fedorahosted.org to collab*
>  * Moving projects to a CNAME or aliased setup so we can then move them
>   to whatever backend host we like. (what exact changes?)
>  * Making whatever changes we need now so we could move to spinning out
>   some projects into "the cloud" instances. What are those specific
>   changes?
>
>  Issues:
>
>  * Backups of hosted01 take ~6 hours, which is not very good. ;( (thats
>   incrementals, full backups are much slower... days).
>  * Hosted01 is pretty heavily loaded. Some of which may just disappear
>   when we move to rhel6 (I know some of it will).
>  * Ideally we could bring up projects on a new instance in the event of
>   a failure of a node.
>
>  I'd like to have a SOP with steps for migrating hosted instances soon.
>  So, we can say, 'here's what we are going to change and why'. Then we
>  can get setup and start moving projects that opt-in, then have a flag
>  day before our migration window to just move everyone else.
>
>  Thoughts?
>
>  kevin
>
>  _______________________________________________
>  infrastructure mailing list
>  infrastructure(a)lists.fedoraproject.org
>  https://admin.fedoraproject.org/mailman/listinfo/infrastructure
>

[Seth Vidal]

On Thu, 10 Nov 2011 14:45:16 -0700
Kevin Fenzi <kevin(a)scrye.com&gt; wrote:

>  I talked about this at todays meeting, but I thought I would note it
>  here for further input and also see if there was interest in doing a
>  Virtual Fedora Activity Day next week to finish out hosted plans. 
>  
>  background: 
>  
>  We have 5 servers at serverbeach (Thanks for donating them SB Folks!)
>  
>  They were setup long ago and are getting a bit old in the tooth, so we
>  are getting 5 new machines to migrate our stuff to. This will allow us
>  to be on hardware thats newer/better supported and also move to new OS
>  and re-arrange things. We must migrate to those new machines by
>  2011-12-11. (around a month from now). 
>  
>  current: 
>  
>  serverbeach01 -> a mirror
>  
>  serverbeach02 -> collab01 (all our lists.fedoraproject.org lists).
>                             ns01 
>  
>  serverbeach03 -> collab02 (a warm spare of collab01, it rsyncs from
>  there every hour). 
>  
>  serverbeach04 -> hosted01 (all fedorahosted.org)
>  
>  serverbeach05 -> hosted02 (a warm spare, but currently down and not
>  working). 
>  
>  Plan: 
>  
>  serverbeach06 -> collab03
>                              ns01
>  serverbeach07 -> collab04
>  
>  (move these to rhel6 for both host/guest. Also, look at setting up
>  drbd for /srv on them, so we can easily switch over from 03 to 04 in
>  case of failure. If we do do drbd, I'd like to schedule at least one
>  failover to make sure it really is all working as expected. ns01 can
>  go on either. ) We will also need to add drbd rpms to our repo. 
>  
>  That leaves us with hosted to deal with, and serverbeach08/09/10 to do
>  it in (currently). We have discussed lots of plans with hosted, I'd
>  like to propose meeting up on IRC next Wed to do a virtual Fedora
>  Activity Day around planning that. Or we could just try and work out
>  details here on list. 
>  
>  Things we have discussed: 
>  
>  * Moving lists.fedorahosted.org to collab*
>  * Moving projects to a CNAME or aliased setup so we can then move them
>    to whatever backend host we like. (what exact changes?)
>  * Making whatever changes we need now so we could move to spinning out
>    some projects into "the cloud" instances. What are those specific
>    changes?
>  
>  Issues: 
>  
>  * Backups of hosted01 take ~6 hours, which is not very good. ;( (thats
>    incrementals, full backups are much slower... days). 
>  * Hosted01 is pretty heavily loaded. Some of which may just disappear
>    when we move to rhel6 (I know some of it will). 
>  * Ideally we could bring up projects on a new instance in the event of
>    a failure of a node. 
>  
>  I'd like to have a SOP with steps for migrating hosted instances soon. 
1. I think we need some volunteer projects. Do we know of any
enterprising folks who would like to be suckers.... err test subjects?


2. For collab2 and hosted2 - would it be possible to run both of
those secondary machines on one of the new sb boxes? Disk-space wise we
should be able to make it work - and it from a failover standpoint it
feels unlikely we would need to failover both hosted1 and collab1 at
the same time.

3. Then do backups from collab2 and hosted2 - until we have our stuff
together for hosted-tng.


>  So, we can say, 'here's what we are going to change and
> why'. Then we
>  can get setup and start moving projects that opt-in, then have a flag
>  day before our migration window to just move everyone else.  
I suspect we'll need to waddle them all through a little at a time.
Remember getting rid of cvs.fedoraproject.org? like that :(

-sv

[Kevin Fenzi]

On Fri, 11 Nov 2011 09:55:58 -0500
seth vidal <skvidal(a)fedoraproject.org&gt; wrote:

>  1. I think we need some volunteer projects. Do we know of any
>  enterprising folks who would like to be suckers.... err test subjects? 
I'm sure we can find a few projects that wouldn't mind... 

>  2. For collab2 and hosted2 - would it be possible to run both of
>  those secondary machines on one of the new sb boxes? Disk-space wise
>  we should be able to make it work - and it from a failover standpoint
>  it feels unlikely we would need to failover both hosted1 and collab1
>  at the same time. 
That possible, but it might hit I/O on that machine a bit hard. 
Depending on how we sync to those secondaries. 

>  3. Then do backups from collab2 and hosted2 - until we have our
> stuff
>  together for hosted-tng. 
Yeah, if we could get 2 to work that would work too probibly. 
More I/O tho. 

>  > So, we can say, 'here's what we are going to change and
> why'. Then
>  > we can get setup and start moving projects that opt-in, then have a
>  > flag day before our migration window to just move everyone else. 
>  
>  I suspect we'll need to waddle them all through a little at a time.
>  Remember getting rid of cvs.fedoraproject.org? like that :( 
Yep. Until flag day... 

kevin

[Pierre-Yves Chibon]

On Fri, 2011-11-11 at 16:23 -0700, Kevin Fenzi wrote:
>  > 1. I think we need some volunteer projects. Do we know of any
>  > enterprising folks who would like to be suckers.... err test
>  subjects?
>  
>  I'm sure we can find a few projects that wouldn't mind...  
Volunteering for the R2spec project (though it is small and no so
lively).

Pierre

[Pierre-Yves Chibon]

On Mon, 2011-11-14 at 22:38 +0100, Pierre-Yves Chibon wrote:
>  On Fri, 2011-11-11 at 16:23 -0700, Kevin Fenzi wrote:
>  > > 1. I think we need some volunteer projects. Do we know of any
>  > > enterprising folks who would like to be suckers.... err test
>  > subjects?
>  > 
>  > I'm sure we can find a few projects that wouldn't mind... 
>  
>  Volunteering for the R2spec project (though it is small and no so
>  lively).  
And after confirmation from Stanislav, we're ok to test on FedoraReview
(both project don't have mailing list so we won't be testing this).

Pierre

[Jared K. Smith]

On Fri, Nov 11, 2011 at 9:55 AM, seth vidal <skvidal(a)fedoraproject.org&gt; wrote:
>  1. I think we need some volunteer projects. Do we know of any
>  enterprising folks who would like to be suckers.... err test subjects? 
I recently got added as an admin for the "pym" project on Fedora
Hosted, and I'd be happy to let you use pym as a guinea pig project.

--
Jared Smith
Fedora Project Leader