Hosted plans

[Dennis Gilmore]

On Monday, August 08, 2011 11:54:30 AM Kevin Fenzi wrote:
> Greetings.
> 
> I thought I would throw out some plans for hosted moving forward and
> see if we could hash out a plan and some timetable for implementing
> things.
> 
> == Current status/setup/problems ==
> 
> Hosted is 2 machines. One primary (hosted01) and one spare (hosted02).
> There is an hourly rsync of data from 01->02. 02 is completely standby,
> it takes none of the load or requests, it's simply there in case
> hosted01 dies.
> 
> Hosted01 has the following items on it:
> 
> * apache/httpd
> 	gitweb
> 	other scm web
> 	trac
> 	mailman archive http access
> 	source downloads (tar.gz, etc)
> 	loggerhead
> 	reviewboard
> 
> * rsync
> * scm checkouts
> * scm checkins
> * lists.fedorahosted.org mailman
> * spamassassin for lists.
> 
> Issues with current setup:
> 
> * hosted01 is under heavy load much of the time. Needs to be more
>   responsive.
> * hosted01 is rhel5 and very old trac. rhel6 has a much newer trac.
> * If hosted01 dies we could loose up to 1 hour of data with switching
>   to hosted02.
> * hosted01/02 are at serverbeach. If they are off line, hosted is off
>   line.
> 
> === Short term plan ===
> 
> * Create a hosted-lists01/02 (or whatever we want to call them)
> 	sync all lists and list data to it.
> 	make sure dns is happy on it.
> 	Move all lists.fedorahosted.org over to these new machines.
> 
> This will move some of the load off the existing hosted01/02 setup,
> and should be a pretty easy thing to do. We will just setup them like
> collab01/02 and hosted01/02 are with a live machine and hot spare
> (unless we can use some shared fs somehow?)
> 
> Questions:
> 
> - Should these be at serverbeach? Or one there and one somewhere else?
> (There is approx 23GB of mailman archives on hosted01 currently).
> 
> * Create a hosted03/04
> 	Setup rhel6 versions of hosted01/02
> 	Setup in similar way, except with no lists (see above).
> 
> Ask hosted project owners to 'opt-in' to moving to the new machines.
> Get a small pool of things at first, then move more as any issues are
> fixed. Once everything looks stable, announce a cut over date, and just
> move all the rest over then.
> 
> Questions:
> - Should these be at serverbeach? Or one there and one somewhere else?
> (There is approx 90GB of data on hosted01 currently).
> 
> * Look at some caching. Could we add memcached on hosted03/04 easily?
> What about varnish?
> 
> Questions:
> - What of the things we have would use caching well?
> 
> === Long term plan ===
> 
> We kicked around some plans here in several of our meetings, but did a
> poor job of recording that anywhere. ;)
> 
> Some things we talked about:
> 
> * A caching frontend of some kind. Could cache web requests and take
>   load off the main machines.
> * Some way to distribute the data, so if serverbeach were off line we
>   could still switch to and use another machine.
> * Adding more/different services.
> 	IRC commit bot
> 	redmine (still not in fedora, but people are working on it)
> 	your idea here.
> 
> So, what do folks think? Should we try and come up with more concrete
> longer term plans before acting on the short term ones?
> Are there other paths forward that make sense here?
> 
> kevin

another thing ive wanted to test and look into is moving to using something 
like glusterfs as the backend storage so that we can scale beyond more than 
one box without users needing to know what box there proect is stored on.

Dennis
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 198 bytes
Desc: This is a digitally signed message part.
Url : http://lists.fedoraproject.org/pipermail/infrastructure/attachments/20110808/81eac2b8/attachment.bin