I'm seeking two +1s to apply some work I did and tested in staging to
the production proxies.
The gist is that we currently proxy persistent tcp connections for the
fedmsg firehose through our proxies to busgateway01 (which these days
numbers around 700 connections but could scale up unexpectedly). We
suspect that this has been causing our vpn issues since all fedmsg
traffic is sent to the proxies for every connection (700 times the
traffic normally on the bus).
The following patch 'turns on' a scheme where each proxy will run its
own fedmsg-gateway slave. Haproxy connects remote requests to these
slaves, and the slaves in turn connect just once to the master
fedmsg-gateway on busgateway01. Traffic should then only be sent once
to each proxy (over the vpn in most cases).
I tested it pretty thoroughly in stg (removed the daemon and its
config and re-ran puppet a few times to make sure things were put in
place in the correct order).
Patch attached.
-Ralph