Web torture results

[Ahmed Kamal]

Hi,
Paulo and me (kim0) have been working on testing a caching setup for the
wiki. A test migration to Moin 1.5 is complete, squid is now configured as a
reverse caching proxy. We've done some stress tests on the current setup.
Attached are some extracted results that (I think) are of interest. Mainly
the number of requests served per second, and the average time for serving a
request. Also, paulo pointed that caching differs per file type, the tests
have been done on three different file types (html, png image, and css)

Test Setup:
=========
1- All connections were initiated from proxy1
2- Proxy2 had squid caching turned on
3- Testing for html/png/css done, sweeping the number of concurrent
connections
4- Turn off squid caching on proxy2
5- Testing for html/png/css done, sweeping the number of concurrent
connections again

Interesting notes:
============
1- Serving PNG is 10X faster than html
2- Serving CSS is 10X faster than PNG!
3- Serving html is really the bottleneck. Unfortunately, Moin developers
acknowledged current version (1.5) is not cache friendly. Work for making
1.6 cache friendly is undergoing
4- Using squid currently only seems to double our PNG serving rate, nothing
else
5- The application server hits swapping (about 0.5GB) at full load (~300
concurrent connections), for some reason the requests/second served is still
high!! (Is our cache disks that fast)
6- The test did not stress the server bad enough to run out of swap space,
not sure if this is needed though!

I can send the full results date if anyone is interested.

Best Regards

[Paulo Santos]

Hi all,

The PNG link i gave kim0 was just an example of an image. Currently we are
caching all general type of images (gif, jpg, png).
As Ahmed pointed, current version of Moin isn't still html friendly,
hopefully this will be addressed at Moin 1.6.x (our current version is Moin
1.3.4 and our testing was based on Moin 1.5.6).
Moin 1.5.6, still bring alot of improvements, so in my opinion, even though
that Moin html is not cache friendly we should still be thinking in  the
migration. Also as a workaround, we can still create a static website that
with the help of mod_headers, it could be forced to be cached.
Last but not least, let's not forget that this was a local test. A more real
stress test, is still going to be prepared and should include a server
requesting to the future architecture (including load balancer and both
proxies), this will improve even more our ability to deliver the desired
content.

We would like some coments on this tests (past and future), so that we can
improve what needs to be improved. As a final note, i would like all of you
to start testing the future platform[1]. Just play with Moin, and give some
feedback on the WikiMigration page[2].

[1] http://webtest.fedora.redhat.com/wiki
[2] http://fedoraproject.org/wiki/Infrastructure/WikiMigration

Once again thanks all for the help and feedback...


Paulo

On 12/23/06, Ahmed Kamal <email.ahmedkamal(a)googlemail.com&gt; wrote:
>  Hi,
>  Paulo and me (kim0) have been working on testing a caching setup for the
>  wiki. A test migration to Moin 1.5 is complete, squid is now configured as
>  a reverse caching proxy. We've done some stress tests on the current setup.
>  Attached are some extracted results that (I think) are of interest. Mainly
>  the number of requests served per second, and the average time for serving a
>  request. Also, paulo pointed that caching differs per file type, the tests
>  have been done on three different file types (html, png image, and css)
>
>  Test Setup:
>  =========
>  1- All connections were initiated from proxy1
>  2- Proxy2 had squid caching turned on
>  3- Testing for html/png/css done, sweeping the number of concurrent
>  connections
>  4- Turn off squid caching on proxy2
>  5- Testing for html/png/css done, sweeping the number of concurrent
>  connections again
>
>  Interesting notes:
>  ============
>  1- Serving PNG is 10X faster than html
>  2- Serving CSS is 10X faster than PNG!
>  3- Serving html is really the bottleneck. Unfortunately, Moin developers
>  acknowledged current version ( 1.5) is not cache friendly. Work for making
>  1.6 cache friendly is undergoing
>  4- Using squid currently only seems to double our PNG serving rate,
>  nothing else
>  5- The application server hits swapping (about 0.5GB) at full load (~300
>  concurrent connections), for some reason the requests/second served is still
>  high!! (Is our cache disks that fast)
>  6- The test did not stress the server bad enough to run out of swap space,
>  not sure if this is needed though!
>
>  I can send the full results date if anyone is interested.
>
>  Best Regards
>
>  _______________________________________________
>  Fedora-infrastructure-list mailing list
>  Fedora-infrastructure-list(a)redhat.com
>  https://www.redhat.com/mailman/listinfo/fedora-infrastructure-list
>
>
>
>

[Mike McGrath]

On 12/23/06, Paulo Santos <paulo.banon(a)googlemail.com&gt; wrote:
>  > 3- Serving html is really the bottleneck. Unfortunately, Moin
> developers
>  acknowledged current version ( 1.5) is not cache friendly. Work for making 
It sounds like we really won't get much of a benefit till 1.6.  Our
biggest issue last time appeard to be the dynamic generation of
content, converting them to static content helped cpu load on fpserv
quite a bit.

How did the appserver handle the stress test?  How about the proxy
server?  Not just in code generation and such but actual cpuload, that
type of thing?

             -Mike

[Ahmed Kamal]

Under full load (~200 concurrent connections), this was the status

Proxy2 machine
===========
1- RAM 100% used, machine not swapping though (Linux ram utilization is a
bit complex though)
2- Squid was saturating the CPU (80~90%)
3- Lots and lots of httpd processess, each taking 0% cpu, and minimal ram
~0.7%
4- load average: 1.06, 0.94, 0.75. top is showing squid cpu utilization >
85%, while total cpu utilization = 7%!! This seems to be a bug?! So, I cant
trust the load numbers either

App1 server
=========
1- RAM 100% used, 0.5GB swapping
2- CPU saturated by large number of httpd processes
3- load average: 150.35, 143.94, 104.04. Cacti showing load peaking to 400,
while top doesnt go that high!
https://admin.fedoraproject.org/cacti/graph.php?action=zoom&local_gra...

Best Regards


On 12/24/06, Mike McGrath <mmcgrath(a)fedoraproject.org&gt; wrote:
>  On 12/23/06, Paulo Santos <paulo.banon(a)googlemail.com&gt; wrote:
>  > > 3- Serving html is really the bottleneck. Unfortunately, Moin
>  developers
>  > acknowledged current version ( 1.5) is not cache friendly. Work for
>  making
>
>  It sounds like we really won't get much of a benefit till 1.6.  Our
>  biggest issue last time appeard to be the dynamic generation of
>  content, converting them to static content helped cpu load on fpserv
>  quite a bit.
>
>  How did the appserver handle the stress test?  How about the proxy
>  server?  Not just in code generation and such but actual cpuload, that
>  type of thing?
>
>               -Mike
>

[Ask Bjørn Hansen]

On Dec 24, 2006, at 23:55, Ahmed Kamal wrote:

>  1- RAM 100% used, 0.5GB swapping 
You should configure the web server for fewer concurrent  
connections.  Having it swap is most certainly bad for performance  
(unless it's swapping unrelated and inactive programs).


  - ask

-- 
http://develooper.com/ - http://askask.com/

[Ask Bjørn Hansen]

Instead of Squid it might be worthwhile looking at Varnish for the  
reverse-proxy cache.

	http://phk.freebsd.dk/pubs/varnish_tech.pdf

	http://varnish.linpro.no/


  - ask

[Elliot Lee]

Hey Ahmed,

It's neat to see these results! A few comments:

I don't see anywhere where variables such as the file size, request  
size, HTTP pipelining, etc. are taken into account. Optimally, the  
test would have cases that exercise points along all these  
dimensions. E.g. a load test of serving an empty file would help us  
understand what the absolute best case performance is within the  
constraints the current systems.

The other thing that'd be useful to have is a well-defined capacity  
goal that helps everyone understand where things ought to be. For a  
straw-man example, let's say 20 home page loads/second, including all  
associated image/CSS/JS files, by logged-in users. I don't know what  
the actual needs are, but a look at the web logs during the FC6  
release should help.

 From what's mentioned below, I also didn't quite understand whether  
squid was sitting in front of apache serving straight HTML/PNG/CSS,  
or whether it was moinmoin that contained all the content. Hopefully  
that's just something I missed in an earlier message.

My current favorite treatise on optimizing overall web performance -  
http://www.die.net/musings/page_load_time/

Hope this helps,
-- Elliot

On Dec 23, 2006, at 3:19 PM, Ahmed Kamal wrote:

>  Hi,
>  Paulo and me (kim0) have been working on testing a caching setup  
>  for the wiki. A test migration to Moin 1.5 is complete, squid is  
>  now configured as a reverse caching proxy. We've done some stress  
>  tests on the current setup. Attached are some extracted results  
>  that (I think) are of interest. Mainly the number of requests  
>  served per second, and the average time for serving a request.  
>  Also, paulo pointed that caching differs per file type, the tests  
>  have been done on three different file types (html, png image, and  
>  css)
>
>  Test Setup:
>  =========
>  1- All connections were initiated from proxy1
>  2- Proxy2 had squid caching turned on
>  3- Testing for html/png/css done, sweeping the number of concurrent  
>  connections
>  4- Turn off squid caching on proxy2
>  5- Testing for html/png/css done, sweeping the number of concurrent  
>  connections again
>
>  Interesting notes:
>  ============
>  1- Serving PNG is 10X faster than html
>  2- Serving CSS is 10X faster than PNG!
>  3- Serving html is really the bottleneck. Unfortunately, Moin  
>  developers acknowledged current version ( 1.5) is not cache  
>  friendly. Work for making 1.6 cache friendly is undergoing
>  4- Using squid currently only seems to double our PNG serving rate,  
>  nothing else
>  5- The application server hits swapping (about 0.5GB) at full load  
>  (~300 concurrent connections), for some reason the requests/second  
>  served is still high!! (Is our cache disks that fast)
>  6- The test did not stress the server bad enough to run out of swap  
>  space, not sure if this is needed though!
>
>  I can send the full results date if anyone is interested.
>
>  Best Regards
>  <results.ods>
>  _______________________________________________
>  Fedora-infrastructure-list mailing list
>  Fedora-infrastructure-list(a)redhat.com
>  https://www.redhat.com/mailman/listinfo/fedora-infrastructure-list