[freeze break request] Switch phx2 app servers back to proxy01.

Ricky Elrod codeblock at elrod.me
Fri Dec 6 18:06:06 UTC 2013


After talking in #fedora-noc, we would like to make phx2 app servers
talk to other app servers via proxy01, rather than (potentially) going
out to a server across the country for the benefit of load balancing.

The issue this solves is that because apps hosted on admin.fp.o talk to
each other using the same admin.fp.o roundrobin that users use, when an
application hits a random proxy and that proxy has gone down for some
reason, the application would block its thread waiting for a response.
Eventually this would happen on enough applications that we would get a
flood of alerts, and ultimately cause downtime.

By using (only) proxy01, it means re-adding a single point of failure,
but it seems likely that the case of "proxy01 is down so everything else
is down" would mean that phx2 was having a network issue anyway, meaning
we'd be in the same position (i.e., app servers unreachable). It seems
unlikely that proxy01 will just die at random, and if it does then that
is a whole new issue that we should address. So yes, the single point of
failure is bad, but it seems marginally better than what we have been
seeing lately.

I would like +1's to push this to puppet, which effectively reverts
1bac8c9a and 23ceebd5.

[codeblock at lockbox01 puppet]$ git show
commit f0445cffd64d3db980a4b689517fd4b95f6e7686
Author: Ricky Elrod <codeblock at lockbox01.phx2.fedoraproject.org>
Date:   Fri Dec 6 17:55:38 2013 +0000

    Make phx2 boxen use proxy01 for admin.fp.o again

diff --git a/manifests/services/phx.pp b/manifests/services/phx.pp
index 72c24be..317bfc2 100644
--- a/manifests/services/phx.pp
+++ b/manifests/services/phx.pp
@@ -11,11 +11,10 @@ class phx {
   }
   case $environment {
     'production' : {
-#        host { 'admin.fedoraproject.org':
-          #ip => '10.5.126.52',
-          #ip => '66.35.62.166',
-#          ensure => absent,
-#        }
+        host { 'admin.fedoraproject.org':
+          ip => '10.5.126.52',
+          ensure => present,
+        }
         host { 'cvs.fedoraproject.org':
             ip => '10.5.125.151',
             host_aliases => ['cvs']

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 490 bytes
Desc: OpenPGP digital signature
URL: <http://lists.fedoraproject.org/pipermail/infrastructure/attachments/20131206/5c02b67d/attachment.sig>


More information about the infrastructure mailing list