Greetings.
I thought I would throw out some plans for hosted moving forward and
see if we could hash out a plan and some timetable for implementing
things.
== Current status/setup/problems ==
Hosted is 2 machines. One primary (hosted01) and one spare (hosted02).
There is an hourly rsync of data from 01->02. 02 is completely standby,
it takes none of the load or requests, it's simply there in case
hosted01 dies.
Hosted01 has the following items on it:
* apache/httpd
gitweb
other scm web
trac
mailman archive http access
source downloads (tar.gz, etc)
loggerhead
reviewboard
* rsync
* scm checkouts
* scm checkins
*
lists.fedorahosted.org mailman
* spamassassin for lists.
Issues with current setup:
* hosted01 is under heavy load much of the time. Needs to be more
responsive.
* hosted01 is rhel5 and very old trac. rhel6 has a much newer trac.
* If hosted01 dies we could loose up to 1 hour of data with switching
to hosted02.
* hosted01/02 are at serverbeach. If they are off line, hosted is off
line.
=== Short term plan ===
* Create a hosted-lists01/02 (or whatever we want to call them)
sync all lists and list data to it.
make sure dns is happy on it.
Move all
lists.fedorahosted.org over to these new machines.
This will move some of the load off the existing hosted01/02 setup,
and should be a pretty easy thing to do. We will just setup them like
collab01/02 and hosted01/02 are with a live machine and hot spare
(unless we can use some shared fs somehow?)
Questions:
- Should these be at serverbeach? Or one there and one somewhere else?
(There is approx 23GB of mailman archives on hosted01 currently).
* Create a hosted03/04
Setup rhel6 versions of hosted01/02
Setup in similar way, except with no lists (see above).
Ask hosted project owners to 'opt-in' to moving to the new machines.
Get a small pool of things at first, then move more as any issues are
fixed. Once everything looks stable, announce a cut over date, and just
move all the rest over then.
Questions:
- Should these be at serverbeach? Or one there and one somewhere else?
(There is approx 90GB of data on hosted01 currently).
* Look at some caching. Could we add memcached on hosted03/04 easily?
What about varnish?
Questions:
- What of the things we have would use caching well?
=== Long term plan ===
We kicked around some plans here in several of our meetings, but did a
poor job of recording that anywhere. ;)
Some things we talked about:
* A caching frontend of some kind. Could cache web requests and take
load off the main machines.
* Some way to distribute the data, so if serverbeach were off line we
could still switch to and use another machine.
* Adding more/different services.
IRC commit bot
redmine (still not in fedora, but people are working on it)
your idea here.
So, what do folks think? Should we try and come up with more concrete
longer term plans before acting on the short term ones?
Are there other paths forward that make sense here?
kevin
another thing ive wanted to test and look into is moving to using something
like glusterfs as the backend storage so that we can scale beyond more than
one box without users needing to know what box there proect is stored on.
Dennis