Hosted plans

[Mike McGrath]

On Mon, 8 Aug 2011, Kevin Fenzi wrote:

> Greetings.
>
> I thought I would throw out some plans for hosted moving forward and
> see if we could hash out a plan and some timetable for implementing
> things.
>
> == Current status/setup/problems ==
>
> Hosted is 2 machines. One primary (hosted01) and one spare (hosted02).
> There is an hourly rsync of data from 01->02. 02 is completely standby,
> it takes none of the load or requests, it's simply there in case
> hosted01 dies.
>
> Hosted01 has the following items on it:
>
> * apache/httpd
> 	gitweb
> 	other scm web
> 	trac
> 	mailman archive http access
> 	source downloads (tar.gz, etc)
> 	loggerhead
> 	reviewboard
>
> * rsync
> * scm checkouts
> * scm checkins
> * lists.fedorahosted.org mailman
> * spamassassin for lists.
>
> Issues with current setup:
>
> * hosted01 is under heavy load much of the time. Needs to be more
>   responsive.
> * hosted01 is rhel5 and very old trac. rhel6 has a much newer trac.
> * If hosted01 dies we could loose up to 1 hour of data with switching
>   to hosted02.
> * hosted01/02 are at serverbeach. If they are off line, hosted is off
>   line.
>
> === Short term plan ===
>
> * Create a hosted-lists01/02 (or whatever we want to call them)
> 	sync all lists and list data to it.
> 	make sure dns is happy on it.
> 	Move all lists.fedorahosted.org over to these new machines.
>
> This will move some of the load off the existing hosted01/02 setup,
> and should be a pretty easy thing to do. We will just setup them like
> collab01/02 and hosted01/02 are with a live machine and hot spare
> (unless we can use some shared fs somehow?)
>
> Questions:
>
> - Should these be at serverbeach? Or one there and one somewhere else?
> (There is approx 23GB of mailman archives on hosted01 currently).
>
> * Create a hosted03/04
> 	Setup rhel6 versions of hosted01/02
> 	Setup in similar way, except with no lists (see above).
>
> Ask hosted project owners to 'opt-in' to moving to the new machines.
> Get a small pool of things at first, then move more as any issues are
> fixed. Once everything looks stable, announce a cut over date, and just
> move all the rest over then.
>
> Questions:
> - Should these be at serverbeach? Or one there and one somewhere else?
> (There is approx 90GB of data on hosted01 currently).
>
> * Look at some caching. Could we add memcached on hosted03/04 easily?
> What about varnish?
>

My understanding of the load issues has always been that the trac git
plugin is very inefficient.  It'd probably be worth it to see if the
load problems magically vanish by updating trac and the git plugin before
committing to any archtectural changes related to performance.

	-Mike