Some architectural changes

[Mike McGrath]

So we've finally completed some consolidation we've been working on for
the past couple of months.  We've also added a new class of server (bappX
servers).  What does this all mean?

Well, app1+ are now all identical, for the most part.  This allows us to
think of our app servers as truly being part of a farm.  The exception to
this right now is those apps requiring our nfs mount in PHX.  I'm still
debating what to do about this but so far haven't really come up with a
good answer.

We'll be distributing load with haproxy.  Just because an app is on a
server doesn't mean that it will get hit.  This allows us to distribute
load intelligently and hopefully allow us to get local proxies preferring
local app servers so geoIP dns can be deployed.

The bapp servers (there's only 1 right now) will be our job control
servers.  Right now we kind of just have cron jobs spread all over the
place and in some instances this has caused problems with production
traffic.  For example, while damned lies is checking out and updating all
of its repos to get the latest translation stats for
http://translate.fedoraproject.org/ app1 and app2 would typically see high
load and if you happened to hit one of them while trying to get to pkgdb,
you might experience slow response time.

Now though we'll be running this job on bapp1 and pushing or pulling the
relevant content via rsync.  We use this model pretty successfully for
other applications like mirrormanager and smolt, but now we have a
dedicated server for it.  Additionally we can add the bapp servers to the
haproxy farms as a backup server, no traffic goes to it unless its needed.

	-Mike

[Stephen John Smoogen]

On Tue, Oct 7, 2008 at 4:41 PM, Mike McGrath <mmcgrath(a)redhat.com&gt; wrote:
>  So we've finally completed some consolidation we've been
> working on for
>  the past couple of months.  We've also added a new class of server (bappX
>  servers).  What does this all mean?
>  
tres tres cool


-- 
Stephen J Smoogen. -- BSD/GNU/Linux
How far that little candle throws his beams! So shines a good deed
in a naughty world. = Shakespeare. "The Merchant of Venice"

[Anand Capur]

>  > So we've finally completed some consolidation we've been working on for
>  > the past couple of months.  We've also added a new class of server (bappX
>  > servers).  What does this all mean?
>  >
>
>  tres tres cool
>
>
>  --
>  Stephen J Smoogen. -- BSD/GNU/Linux
>  How far that little candle throws his beams! So shines a good deed
>  in a naughty world. = Shakespeare. "The Merchant of Venice"
>  
Yeah, +++1. Sounds VERY cool :D. Just shows how cool we are!
-Anand

[Bret McMillan]

Mike McGrath wrote:

>  Well, app1+ are now all identical, for the most part.  This allows us
> to
>  think of our app servers as truly being part of a farm.  The exception to
>  this right now is those apps requiring our nfs mount in PHX.  I'm still
>  debating what to do about this but so far haven't really come up with a
>  good answer. 
2 choices come to mind:

1) accept that they're a different category, and partition them accordingly

2) could maybe tinker around w/ some things like MogileFS or HDFS or 
something even crazier :)

>  We'll be distributing load with haproxy.  Just because an app is
> on a
>  server doesn't mean that it will get hit.  This allows us to distribute
>  load intelligently and hopefully allow us to get local proxies preferring
>  local app servers so geoIP dns can be deployed. 
Do all apps get deployed to all servers in this category?

>  The bapp servers (there's only 1 right now) will be our job
> control
>  servers.  Right now we kind of just have cron jobs spread all over the
>  place and in some instances this has caused problems with production
>  traffic. 
Cool.  How many bapp servers are we running these days, and how do we 
choose what gets run where?

--Bret

[Mike McGrath]

On Wed, 8 Oct 2008, Bret McMillan wrote:

>  Mike McGrath wrote:
>
>  > Well, app1+ are now all identical, for the most part.  This allows us to
>  > think of our app servers as truly being part of a farm.  The exception to
>  > this right now is those apps requiring our nfs mount in PHX.  I'm still
>  > debating what to do about this but so far haven't really come up with a
>  > good answer.
>
>  2 choices come to mind:
>
>  1) accept that they're a different category, and partition them accordingly
>
>  2) could maybe tinker around w/ some things like MogileFS or HDFS or something
>  even crazier :)
>
>  > We'll be distributing load with haproxy.  Just because an app is on a
>  > server doesn't mean that it will get hit.  This allows us to distribute
>  > load intelligently and hopefully allow us to get local proxies preferring
>  > local app servers so geoIP dns can be deployed.
>
>  Do all apps get deployed to all servers in this category?
>  
Not all apps but most web apps.  In particular this is our 'new fedora
release day' stuff which includes the wiki, docs, get.fp.o, and the mirror
list.  Koji wouldn't be on these hosts nor would asterisk.

>  > The bapp servers (there's only 1 right now) will be our job
> control
>  > servers.  Right now we kind of just have cron jobs spread all over the
>  > place and in some instances this has caused problems with production
>  > traffic.
>
>  Cool.  How many bapp servers are we running these days, and how do we choose
>  what gets run where?
>  
Just one and when bapp2 gets made it'll be more of an 'active' and
'passive' setup.  So all of them will go on one until such a time comes
that we're seeing performance issues then we'll re-evaluate.

	-Mike