Freeze Break Request - Introduce fedmsg-gateway slaves on the proxies

[Ralph Bean]

I'm seeking two +1s to apply some work I did and tested in staging to
the production proxies.

The gist is that we currently proxy persistent tcp connections for the
fedmsg firehose through our proxies to busgateway01 (which these days
numbers around 700 connections but could scale up unexpectedly).  We
suspect that this has been causing our vpn issues since all fedmsg
traffic is sent to the proxies for every connection (700 times the
traffic normally on the bus).

The following patch 'turns on' a scheme where each proxy will run its
own fedmsg-gateway slave.  Haproxy connects remote requests to these
slaves, and the slaves in turn connect just once to the master
fedmsg-gateway on busgateway01.  Traffic should then only be sent once
to each proxy (over the vpn in most cases).

I tested it pretty thoroughly in stg (removed the daemon and its
config and re-ran puppet a few times to make sure things were put in
place in the correct order).

Patch attached.

-Ralph
-------------- next part --------------
From e88462f501ff79bf1f346f717b453d7a58d72517 Mon Sep 17 00:00:00 2001
From: Ralph Bean <rbean at redhat.com>
Date: Thu, 4 Sep 2014 20:02:01 +0000
Subject: [PATCH] Apply fedmsg-gateway-slave work to prod proxies.

---
 manifests/services/proxy.pp       |    4 +---
 modules/haproxy/files/haproxy.cfg |    7 +++----
 2 files changed, 4 insertions(+), 7 deletions(-)

diff --git a/manifests/services/proxy.pp b/manifests/services/proxy.pp
index 4f2c8d0..d73b957 100644
--- a/manifests/services/proxy.pp
+++ b/manifests/services/proxy.pp
@@ -20,9 +20,7 @@ class proxy {
         varnish::varnish{ 'proxy.vcl': }
     }
 
-    if $environment == "staging" {
-        include haproxy::fedmsg-gateway-slave
-    }
+    include haproxy::fedmsg-gateway-slave
 
 
     $fpo_ips = [
diff --git a/modules/haproxy/files/haproxy.cfg b/modules/haproxy/files/haproxy.cfg
index 76f05d3..981d366 100644
--- a/modules/haproxy/files/haproxy.cfg
+++ b/modules/haproxy/files/haproxy.cfg
@@ -202,9 +202,8 @@ listen fedmsg-websockets 0.0.0.0:9938
 
 # This, unlike the websockets entry just above, is listening directly to the
 # outside world with no stunnel inbetween.
-# Simply redirect tcp connections to busgateway01 on the same port.  There, an
-# instance of fedmsg-gateway will be publishing every message that comes across
-# the bus internally.  This is for consumption by desktop applications.
+# Simply redirect tcp connections to a local fedmsg-gateway slave.  It should be
+# forwarding messages from the master gateway on busgateway01.
 listen fedmsg-raw-zmq-outbound 0.0.0.0:9940
     mode tcp
     balance roundrobin
@@ -212,7 +211,7 @@ listen fedmsg-raw-zmq-outbound 0.0.0.0:9940
     timeout queue 5000
     timeout server 86400000
     timeout connect 86400000
-    server  busgateway01 busgateway01:9940 weight 1 maxconn 16384
+    server  localhost localhost:9942 weight 1 maxconn 16384
 
 # While the above fedmsg-raw-zmq-outbound forwards incoming connections to an
 # instance of the "fedmsg-gateway" daemon (which pushes internal messages out),
-- 
1.7.2.1

-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 473 bytes
Desc: not available
URL: <http://lists.fedoraproject.org/pipermail/infrastructure/attachments/20140904/22f516a9/attachment.sig>