Freeze Break Request - Introduce fedmsg-gateway slaves on the proxies

[Ralph Bean]

I'm seeking two +1s to apply some work I did and tested in staging to
the production proxies.

The gist is that we currently proxy persistent tcp connections for the
fedmsg firehose through our proxies to busgateway01 (which these days
numbers around 700 connections but could scale up unexpectedly).  We
suspect that this has been causing our vpn issues since all fedmsg
traffic is sent to the proxies for every connection (700 times the
traffic normally on the bus).

The following patch 'turns on' a scheme where each proxy will run its
own fedmsg-gateway slave.  Haproxy connects remote requests to these
slaves, and the slaves in turn connect just once to the master
fedmsg-gateway on busgateway01.  Traffic should then only be sent once
to each proxy (over the vpn in most cases).

I tested it pretty thoroughly in stg (removed the daemon and its
config and re-ran puppet a few times to make sure things were put in
place in the correct order).

Patch attached.

-Ralph

[Ricky Elrod]

+1

-re

On 09/04/2014 04:08 PM, Ralph Bean wrote:
>  I'm seeking two +1s to apply some work I did and tested in
> staging to
>  the production proxies.
>  
>  The gist is that we currently proxy persistent tcp connections for the
>  fedmsg firehose through our proxies to busgateway01 (which these days
>  numbers around 700 connections but could scale up unexpectedly).  We
>  suspect that this has been causing our vpn issues since all fedmsg
>  traffic is sent to the proxies for every connection (700 times the
>  traffic normally on the bus).
>  
>  The following patch 'turns on' a scheme where each proxy will run its
>  own fedmsg-gateway slave.  Haproxy connects remote requests to these
>  slaves, and the slaves in turn connect just once to the master
>  fedmsg-gateway on busgateway01.  Traffic should then only be sent once
>  to each proxy (over the vpn in most cases).
>  
>  I tested it pretty thoroughly in stg (removed the daemon and its
>  config and re-ran puppet a few times to make sure things were put in
>  place in the correct order).
>  
>  Patch attached.
>  
>  -Ralph
>  
>  
>  
>  _______________________________________________
>  infrastructure mailing list
>  infrastructure(a)lists.fedoraproject.org
>  https://admin.fedoraproject.org/mailman/listinfo/infrastructure
>

[Kevin Fenzi]

+1 here. 

:) 

kevin

[Luke Macken]

Looks good to me, +1.

luke

On Thu, Sep 04, 2014 at 01:08:16PM -0700, Ralph Bean wrote:
>  I'm seeking two +1s to apply some work I did and tested in
> staging to
>  the production proxies.
>  
>  The gist is that we currently proxy persistent tcp connections for the
>  fedmsg firehose through our proxies to busgateway01 (which these days
>  numbers around 700 connections but could scale up unexpectedly).  We
>  suspect that this has been causing our vpn issues since all fedmsg
>  traffic is sent to the proxies for every connection (700 times the
>  traffic normally on the bus).
>  
>  The following patch 'turns on' a scheme where each proxy will run its
>  own fedmsg-gateway slave.  Haproxy connects remote requests to these
>  slaves, and the slaves in turn connect just once to the master
>  fedmsg-gateway on busgateway01.  Traffic should then only be sent once
>  to each proxy (over the vpn in most cases).
>  
>  I tested it pretty thoroughly in stg (removed the daemon and its
>  config and re-ran puppet a few times to make sure things were put in
>  place in the correct order).
>  
>  Patch attached.
>  
>  -Ralph 
>  From e88462f501ff79bf1f346f717b453d7a58d72517 Mon Sep 17 00:00:00
> 2001
>  From: Ralph Bean <rbean(a)redhat.com&gt;
>  Date: Thu, 4 Sep 2014 20:02:01 +0000
>  Subject: [PATCH] Apply fedmsg-gateway-slave work to prod proxies.
>  
>  ---
>   manifests/services/proxy.pp       |    4 +---
>   modules/haproxy/files/haproxy.cfg |    7 +++----
>   2 files changed, 4 insertions(+), 7 deletions(-)
>  
>  diff --git a/manifests/services/proxy.pp b/manifests/services/proxy.pp
>  index 4f2c8d0..d73b957 100644
>  --- a/manifests/services/proxy.pp
>  +++ b/manifests/services/proxy.pp
>  @@ -20,9 +20,7 @@ class proxy {
>           varnish::varnish{ 'proxy.vcl': }
>       }
>   
>  -    if $environment == "staging" {
>  -        include haproxy::fedmsg-gateway-slave
>  -    }
>  +    include haproxy::fedmsg-gateway-slave
>   
>   
>       $fpo_ips = [
>  diff --git a/modules/haproxy/files/haproxy.cfg b/modules/haproxy/files/haproxy.cfg
>  index 76f05d3..981d366 100644
>  --- a/modules/haproxy/files/haproxy.cfg
>  +++ b/modules/haproxy/files/haproxy.cfg
>  @@ -202,9 +202,8 @@ listen fedmsg-websockets 0.0.0.0:9938
>   
>   # This, unlike the websockets entry just above, is listening directly to the
>   # outside world with no stunnel inbetween.
>  -# Simply redirect tcp connections to busgateway01 on the same port.  There, an
>  -# instance of fedmsg-gateway will be publishing every message that comes across
>  -# the bus internally.  This is for consumption by desktop applications.
>  +# Simply redirect tcp connections to a local fedmsg-gateway slave.  It should be
>  +# forwarding messages from the master gateway on busgateway01.
>   listen fedmsg-raw-zmq-outbound 0.0.0.0:9940
>       mode tcp
>       balance roundrobin
>  @@ -212,7 +211,7 @@ listen fedmsg-raw-zmq-outbound 0.0.0.0:9940
>       timeout queue 5000
>       timeout server 86400000
>       timeout connect 86400000
>  -    server  busgateway01 busgateway01:9940 weight 1 maxconn 16384
>  +    server  localhost localhost:9942 weight 1 maxconn 16384
>   
>   # While the above fedmsg-raw-zmq-outbound forwards incoming connections to an
>   # instance of the "fedmsg-gateway" daemon (which pushes internal messages
> out),
>  -- 
>  1.7.2.1
>   



>  _______________________________________________
>  infrastructure mailing list
>  infrastructure(a)lists.fedoraproject.org
>  https://admin.fedoraproject.org/mailman/listinfo/infrastructure