mirrorlist boxes

[seth vidal]

To mdomsch in particular but in general to the list so folks know what
we're doing:

Matt,
 I grabbed your induce-stress script (which rocks btw) and I tested out
how mirrorlist-osuosl runs under a load of 200 wgets with 2 cpus. 
While the %cpu scaled up - the response time stayed pretty reasonable. It
did okay - much more performant than before - but still in the 3-4s
range in terms of responses. 

So we bumped all systems to 4 cpus, 4gb of ram, x86_64. None of the
systems are on overcommitted vhosts. They seem to be handling a much
heavier load now. We can increase specs in a variety of ways, add
servers and even switch back to i686 relatively quickly, if you see
problems. 

I've disabled the app servers from the mirrorlist responses in haproxy.
I only commented them out - so putting them back is simpler. 

I also modified mm_sync_out so it does each host and waits 60s for the
mirrorlist_server process to come back before going onto the next
mirrorlist box.

I've left bapp02 syncing the mirrorlist files to the app servers for
now until we are positive we can decom that service from them.

i've been watching it for a while the numbers are hovering around:

112ms for osuosl
140ms for ibiblio
10ms for phx2

Thanks,
-sv