Some architectural changes

[Mike McGrath]

So we've finally completed some consolidation we've been working on for
the past couple of months.  We've also added a new class of server (bappX
servers).  What does this all mean?

Well, app1+ are now all identical, for the most part.  This allows us to
think of our app servers as truly being part of a farm.  The exception to
this right now is those apps requiring our nfs mount in PHX.  I'm still
debating what to do about this but so far haven't really come up with a
good answer.

We'll be distributing load with haproxy.  Just because an app is on a
server doesn't mean that it will get hit.  This allows us to distribute
load intelligently and hopefully allow us to get local proxies preferring
local app servers so geoIP dns can be deployed.

The bapp servers (there's only 1 right now) will be our job control
servers.  Right now we kind of just have cron jobs spread all over the
place and in some instances this has caused problems with production
traffic.  For example, while damned lies is checking out and updating all
of its repos to get the latest translation stats for
http://translate.fedoraproject.org/ app1 and app2 would typically see high
load and if you happened to hit one of them while trying to get to pkgdb,
you might experience slow response time.

Now though we'll be running this job on bapp1 and pushing or pulling the
relevant content via rsync.  We use this model pretty successfully for
other applications like mirrormanager and smolt, but now we have a
dedicated server for it.  Additionally we can add the bapp servers to the
haproxy farms as a backup server, no traffic goes to it unless its needed.

	-Mike