Freeze break request: Switch back to bastion02 for now

Kevin Fenzi kevin at scrye.com
Fri Sep 16 18:58:10 UTC 2011


We've continued to have issues with bastion03 and this bug:
https://bugzilla.redhat.com/show_bug.cgi?id=725332

It's been requiring a reboot every day or two, resulting in 5-10min of
downtime and about 90 pages. ;( 

I hate changes on fridays and more so during a freeze, but I think we
need to switch back to bastion02 for now to avoid this issue until we
can get a fix. 

So, I would like to: 

- commit the following patch. 
- puppet update nameservers to get the new info. 
- puppet update bastion02/03 to get openvpn running on 02 and stopped
  on 03
- Make sure everything reconnects. 

Unfortunately this will result in a small outage, but no worse than the
bastion03 ones have been. If we don't want to do it now, I can wait
until the next time bastion03 freaks out and just change it then, since
it should be all prepped below: 

diff --git a/manifests/nodes/bastion02.phx2.fedoraproject.org.pp b/manifests/nodes/bastion02.phx2.fe
index 4018ec9..1a0ee7c 100644
--- a/manifests/nodes/bastion02.phx2.fedoraproject.org.pp
+++ b/manifests/nodes/bastion02.phx2.fedoraproject.org.pp
@@ -1,6 +1,5 @@
 node bastion02{
-    # Moving openvpn over to bastion03
-    $enable_openvpn = false
+    $enable_openvpn = true
     include phx
     $syncFasAliases = true
     include gateway
diff --git a/manifests/nodes/bastion03.phx2.fedoraproject.org.pp b/manifests/nodes/bastion03.phx2.fe
index 8c5fca9..b7b0f32 100644
--- a/manifests/nodes/bastion03.phx2.fedoraproject.org.pp
+++ b/manifests/nodes/bastion03.phx2.fedoraproject.org.pp
@@ -3,7 +3,7 @@ node bastion03{
     # comment out the line below when bastion02 is down or going to be down.
     # Under normal situations, only one bastion host should be running openvpn
     # or we'll end up with a split-brain problem in the network
-    #$enable_openvpn = false
+    $enable_openvpn = false
     include phx
     $syncFasAliases = true
     include gateway
diff --git a/modules/bind/files/master/fedoraproject.org b/modules/bind/files/master/fedoraproject.o
index 5b72f2d..f3dc836 100644
--- a/modules/bind/files/master/fedoraproject.org
+++ b/modules/bind/files/master/fedoraproject.org
@@ -1,6 +1,6 @@
 $TTL 3600
 @                IN        SOA     ns01.fedoraproject.org.        hostmaster.fedoraproject.org. (
-                        2011091301 ; Serial
+                        2011091601 ; Serial
                         1H ; refresh
                         10M ; retry
                         4W ; expire
@@ -85,7 +85,7 @@ autoqa-stg      IN  A       209.132.181.9
 ; need rhit to fix. 
 bastion01       IN  A       209.132.181.2
 bastion02       IN  A       209.132.181.3
-bastion         IN  A       209.132.181.2
+bastion         IN  A       209.132.181.3
 bastion-comm01  IN  A       209.132.181.13
 backup02        IN  A       152.19.134.140
 blogs           IN  CNAME   wildcard
diff --git a/modules/bind/files/master/phx2.fedoraproject.org b/modules/bind/files/master/phx2.fedor
index b8caea3..7c9eed7 100644
--- a/modules/bind/files/master/phx2.fedoraproject.org
+++ b/modules/bind/files/master/phx2.fedoraproject.org
@@ -28,7 +28,7 @@ bapp01          IN      A       10.5.126.38
 bapp1           IN      CNAME   bapp01
 bapp02          IN      A       10.5.126.39
 bapp2           IN      CNAME   bapp02
-bastion         IN      CNAME   bastion03
+bastion         IN      CNAME   bastion02
 ;bastion01       IN      A       10.5.126.13
 ;bastion1        IN      CNAME   bastion01
 bastion02       IN      A       10.5.126.11
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 198 bytes
Desc: not available
Url : http://lists.fedoraproject.org/pipermail/infrastructure/attachments/20110916/24a9cf92/attachment.bin 


More information about the infrastructure mailing list