Smolt deployment

[Toshio Kuratomi]

On Sun, 2007-03-04 at 20:23 -0500, Luke Macken wrote:
> On Sat, Mar 03, 2007 at 05:31:07PM -0500, seth vidal wrote:
> > How performant is the tg server? In the past the python webserver was
> > not exactly a barn burner when it came to performance. It worked, but it
> > didn't hold up well under heavy load. Having apache in front helps but
> > just like with zope, if the app is slow, the app is slow.
> > 
> > any load testing done, yet?
> 
> AFAIK, no load testing has been done with our TurboGears apps, but I'm
> definitely in favor of doing it before F7.  I recall kim0 and paulobanon
> did some load tests in the past, but I haven't seen the results yet.
> Does anyone recommend any load generation tools?
> 
> A presentation was given at this years PyCon called "Scaling Python for
> High-Load Web Sites"[0], I definitely recommend checking it out.
> 
Really cool.  My reading of the talk is if our loads match up with their
sample application then we're probably okay with just a single cherrypy
instance behind apache for nearly everything.  Load balancing could get
us the rest of the way for all of our "internal" apps (meaning: Apps
meant for contributors to the project rather than the Fedora Userbase.)
Of course, in your proposal, once we have one thing behind the load
balancers we should be able to put everything behind the load balancers
without too much effort.

The wiki/plone, bugzilla and other end-user facing applications need
more than that.  Unfortunately, we aren't in charge of coding those so
we don't have as many choices in terms of getting it to scale at the
moment.  With moin moin, for instance, my impression is that moin
wouldn't be able to lock files if we had two instances running so we're
unable to use load balancing as an optimization.

> I recommend that we load balance dynamic page requests from our proxy
> servers to our application servers, and let the proxies serve out cached
> static content.  We definitely want to hide hide CherryPy behind apache,
> because having HTTP/1.1 and SSL support is nice, among many other
> benefits.  Whether or not we use mod_{python,proxy,rewrite} to connect to
> CherryPy is up for discussion.  mod_python is the fastest option, and the
> only downfall really is that it is harder to configure, and that you have to
> restart Apache every time you cange your CherryPy code.  I give a +1 for
> mod_python, at least until WSGI support in CherryPy solidifies.
> 
It appears that TG + mod_python is very slow ATM::
  http://tinyurl.com/3xyznr

> Since each application server will have its own connection pool with the
> db servers, increasing our scalability will simply consist of adding
> another Xen guest behind our load balancer.
> 
Why do we even need to add Xen guests?  From the pycon talk it looked
like just adding additional cherrypy servers would increase our ability
to serve more pages.  

We'd want to run benchmarks to see but I'd suspect that having one guest
with five cherrypy instances that we load balance between will give us
more bang for the resources used than five guests on the same Xen host
running one cherrypy server apiece.

Additional guests could enhance reliability, though.  If our load
balancer detects whether a guest has stopped responding and serves
requests to the other guests that are running the cherrypy servers, we
could take a guest down for maintenance and then return it to the pool
without interrupting service.  Having them on separate Xen hosts would
mean we could lose a physical machine and still survive (at half
capacity).

> So from here we might want to look into creating a standard guest image optimized
> for our TurboGears Xen guests.  publictest2 was running FC6 (it still might be,
> but as far as I can tell it seems to be down), and I'm not sure what our
> other TG systems are running, but I think we should be consistent.  I tend to
> lean towards RHEL{5,4}, which will help us get TurboGears & friends whipped into
> shape for EPEL 
> 
RHEL4 would be python2.3.  RHEL5 is python2.4 like FC6.  F7 will be
python2.5....

python2.4 has decorators which TG makes heavy use of so I think we want
to have at least that version.  It'll feel constraining to run 2.5 for
local development on our home machines with Fedora7+ but having to
develop for python 2.4 because that's what comes with RHEL5 (Unified
try:except:finally and ternary operators being the features I'll miss
the most) but I suspect that's a tradeoff that we'll want to make so we
aren't upgrading every six months.

-Toshio

> What do you all think?
> 
> luke
> 
> [0]: http://www.polimetrix.com/pycon/slides/

-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: This is a digitally signed message part
Url : http://lists.fedoraproject.org/pipermail/infrastructure/attachments/20070305/bb92f433/attachment.bin