Search domains in our environment (Proposal)

[Mike McGrath]

Note: I'm treading into an area which I've always deemed bad practice so 
poke, prod and question where required.

Right now we are using /etc/hosts relatively heavily in our 
environment.  This is to help us clean up our apache configs and further 
blur the line of our servers and where they live.  The suggestion in the 
past has been to host our own DNS server in PHX that provides a common 
view to fedoraproject.org but inside PHX.  (You'll not that you cannot 
get to fedoraproject.org from inside PHX).  Now that we have a 
vpn.fedoraproject.org domain, this allows us to do some dns trickery 
that we could not do before.

So, for example, on bastion you can see this in action.  The current 
search location set to:

search fedora.phx.redhat.com vpn.fedoraproject.org fedoraproject.org

So on bastion you can ping app1 which will use 10.8.34.59.  However if I 
ping proxy3 (which is not in phx) I'll get address 192.168.1.7.  and if 
I ping torrent (which is not in phx and not on the vpn) I'll get address 
152.3.220.165.

In theory, this will allow us to do interesting things in our german 
colo (they have the server now BTW, we are just waiting on IP info, it 
just got there yesterday).  The trick here is having each group of 
servers have a preference for the local address.  There's no reason for 
proxy1 to contact app1 over the vpn as they're on the same LAN.  And 
there could, in theory, be instances where we'd want the serverbeach 
servers to have preference for other serverbeach servers.  In cases of 
geographically separated servers this actually does add a tiny amount of 
redundancy.  In that if a link goes down or dns goes down but the box 
does have connectivity to the internet still somehow, it might be able 
to get to the vpn instead of its direct connection.  Again, tiny but 
there especially true when we get our redundant VPN server installed.

So what does this mean?
* You'll be able to get to any vpn host in our environment without 
having to know where it is.

* We'll have to change any reference to fqdn's where our servers are 
contacting other servers.  This will allow us to move servers around, 
even to other data centers, without having to change the configs.

* The proxy servers are in a slightly special situation right now.  
We're using hosts entries on the proxy servers mostly because our DNS 
server in PHX flaked out on us once.  We can re-examine this setup even 
still, to be consistent I'd like to switch to using non-fqdn access to 
our application servers.

* We will have to be diligent in making sure all of our hosts have 
unique names as we've basically made the domain names negligent.

* This will allow us to have a preference for vpn, remote or local 
traffic on a per machine basis should the need arise.  (so for example, 
We get part of a DR site up and PHX goes down.  We could very easily 
login to proxy3, change the search from vpn being first to local as both 
app5 and proxy3 are in tummy.com and we can be more efficient that way)

Comments?  +1's?  -1's?  I'm basically going for ease of use among the 
admins and since most people "ssh puppet1" instead of "ssh 
puppet1.fedora.phx.redhat.com" I think in our diverse environment it 
will be worth it and is easier then hosting a separate DNS server in 
each of our locations.

    -Mike