Note: I'm treading into an area which I've always deemed bad practice so poke, prod and question where required.
Right now we are using /etc/hosts relatively heavily in our environment. This is to help us clean up our apache configs and further blur the line of our servers and where they live. The suggestion in the past has been to host our own DNS server in PHX that provides a common view to fedoraproject.org but inside PHX. (You'll not that you cannot get to fedoraproject.org from inside PHX). Now that we have a vpn.fedoraproject.org domain, this allows us to do some dns trickery that we could not do before.
So, for example, on bastion you can see this in action. The current search location set to:
search fedora.phx.redhat.com vpn.fedoraproject.org fedoraproject.org
So on bastion you can ping app1 which will use 10.8.34.59. However if I ping proxy3 (which is not in phx) I'll get address 192.168.1.7. and if I ping torrent (which is not in phx and not on the vpn) I'll get address 152.3.220.165.
In theory, this will allow us to do interesting things in our german colo (they have the server now BTW, we are just waiting on IP info, it just got there yesterday). The trick here is having each group of servers have a preference for the local address. There's no reason for proxy1 to contact app1 over the vpn as they're on the same LAN. And there could, in theory, be instances where we'd want the serverbeach servers to have preference for other serverbeach servers. In cases of geographically separated servers this actually does add a tiny amount of redundancy. In that if a link goes down or dns goes down but the box does have connectivity to the internet still somehow, it might be able to get to the vpn instead of its direct connection. Again, tiny but there especially true when we get our redundant VPN server installed.
So what does this mean? * You'll be able to get to any vpn host in our environment without having to know where it is.
* We'll have to change any reference to fqdn's where our servers are contacting other servers. This will allow us to move servers around, even to other data centers, without having to change the configs.
* The proxy servers are in a slightly special situation right now. We're using hosts entries on the proxy servers mostly because our DNS server in PHX flaked out on us once. We can re-examine this setup even still, to be consistent I'd like to switch to using non-fqdn access to our application servers.
* We will have to be diligent in making sure all of our hosts have unique names as we've basically made the domain names negligent.
* This will allow us to have a preference for vpn, remote or local traffic on a per machine basis should the need arise. (so for example, We get part of a DR site up and PHX goes down. We could very easily login to proxy3, change the search from vpn being first to local as both app5 and proxy3 are in tummy.com and we can be more efficient that way)
Comments? +1's? -1's? I'm basically going for ease of use among the admins and since most people "ssh puppet1" instead of "ssh puppet1.fedora.phx.redhat.com" I think in our diverse environment it will be worth it and is easier then hosting a separate DNS server in each of our locations.
-Mike
Mike McGrath wrote:
Comments? +1's? -1's? I'm basically going for ease of use among the admins and since most people "ssh puppet1" instead of "ssh puppet1.fedora.phx.redhat.com" I think in our diverse environment it will be worth it and is easier then hosting a separate DNS server in each of our locations.
I forgot to mention one other concern. A MitM attack or DNS poisoning. This possibility does exist, but exists in our environment as is anyway. This is something we should look at mitigating but other than running a DNS server at every site, I'm not totally sure how to fix it. I consider all of our donations as partnerships. After all, they have local access to the box. At the same time though it is something we should count as a risk and mitigate as much as possible.
-Mike
On Dec 19, 2007 4:06 PM, Mike McGrath mmcgrath@redhat.com wrote:
Mike McGrath wrote:
Comments? +1's? -1's? I'm basically going for ease of use among the admins and since most people "ssh puppet1" instead of "ssh puppet1.fedora.phx.redhat.com" I think in our diverse environment it will be worth it and is easier then hosting a separate DNS server in each of our locations.
I forgot to mention one other concern. A MitM attack or DNS poisoning. This possibility does exist, but exists in our environment as is anyway. This is something we should look at mitigating but other than running a DNS server at every site, I'm not totally sure how to fix it. I consider all of our donations as partnerships. After all, they have local access to the box. At the same time though it is something we should count as a risk and mitigate as much as possible.
As far as I can tell the only way to lower the risk of DNS poisoning is local DNS servers. Having them getting DNS files from a central host via a signed methodology would be not much different than /etc/hosts except you can use other tricks and failovers
Stephen John Smoogen wrote:
On Dec 19, 2007 4:06 PM, Mike McGrath mmcgrath@redhat.com wrote:
Mike McGrath wrote:
Comments? +1's? -1's? I'm basically going for ease of use among the admins and since most people "ssh puppet1" instead of "ssh puppet1.fedora.phx.redhat.com" I think in our diverse environment it will be worth it and is easier then hosting a separate DNS server in each of our locations.
I forgot to mention one other concern. A MitM attack or DNS poisoning. This possibility does exist, but exists in our environment as is anyway. This is something we should look at mitigating but other than running a DNS server at every site, I'm not totally sure how to fix it. I consider all of our donations as partnerships. After all, they have local access to the box. At the same time though it is something we should count as a risk and mitigate as much as possible.
As far as I can tell the only way to lower the risk of DNS poisoning is local DNS servers. Having them getting DNS files from a central host via a signed methodology would be not much different than /etc/hosts except you can use other tricks and failovers
We could also implement stricter IP tables rules regarding creating external TCP connections.
-Mike
On Dec 19, 2007 4:15 PM, Mike McGrath mmcgrath@redhat.com wrote:
Stephen John Smoogen wrote:
On Dec 19, 2007 4:06 PM, Mike McGrath mmcgrath@redhat.com wrote:
Mike McGrath wrote:
Comments? +1's? -1's? I'm basically going for ease of use among the admins and since most people "ssh puppet1" instead of "ssh puppet1.fedora.phx.redhat.com" I think in our diverse environment it will be worth it and is easier then hosting a separate DNS server in each of our locations.
I forgot to mention one other concern. A MitM attack or DNS poisoning. This possibility does exist, but exists in our environment as is anyway. This is something we should look at mitigating but other than running a DNS server at every site, I'm not totally sure how to fix it. I consider all of our donations as partnerships. After all, they have local access to the box. At the same time though it is something we should count as a risk and mitigate as much as possible.
As far as I can tell the only way to lower the risk of DNS poisoning is local DNS servers. Having them getting DNS files from a central host via a signed methodology would be not much different than /etc/hosts except you can use other tricks and failovers
We could also implement stricter IP tables rules regarding creating external TCP connections.
Yes that would help on MitM attacks but not much on the DNS side. Since we are looking for redundancy, could we draw a picture of what it should look like in the end? Need it to see what we have and how we are improving things in the future and what other ideas might be useful.
Hope this makes sense.. on painkillers.
On Wed, 2007-12-19 at 16:33 -0700, Stephen John Smoogen wrote:
On Dec 19, 2007 4:15 PM, Mike McGrath mmcgrath@redhat.com wrote:
Stephen John Smoogen wrote:
On Dec 19, 2007 4:06 PM, Mike McGrath mmcgrath@redhat.com wrote:
Mike McGrath wrote:
Comments? +1's? -1's? I'm basically going for ease of use among the admins and since most people "ssh puppet1" instead of "ssh puppet1.fedora.phx.redhat.com" I think in our diverse environment it will be worth it and is easier then hosting a separate DNS server in each of our locations.
I forgot to mention one other concern. A MitM attack or DNS poisoning. This possibility does exist, but exists in our environment as is anyway. This is something we should look at mitigating but other than running a DNS server at every site, I'm not totally sure how to fix it. I consider all of our donations as partnerships. After all, they have local access to the box. At the same time though it is something we should count as a risk and mitigate as much as possible.
As far as I can tell the only way to lower the risk of DNS poisoning is local DNS servers. Having them getting DNS files from a central host via a signed methodology would be not much different than /etc/hosts except you can use other tricks and failovers
We could also implement stricter IP tables rules regarding creating external TCP connections.
Yes that would help on MitM attacks but not much on the DNS side. Since we are looking for redundancy, could we draw a picture of what it should look like in the end? Need it to see what we have and how we are improving things in the future and what other ideas might be useful.
The reason for all of this is the firewall in place at the PHX colo. If that wasn't there we wouldn't need any of the games at all. We could just have foo.fedoraproject.org be resolveable from anywhere and foo.vpn.fedoraproject.org just mean 'go over the vpn to get to it'.
seth 'big fan of simple networking' vidal -sv
The reason for all of this is the firewall in place at the PHX colo. If that wasn't there we wouldn't need any of the games at all. We could just have foo.fedoraproject.org be resolveable from anywhere and foo.vpn.fedoraproject.org just mean 'go over the vpn to get to it'.
seth 'big fan of simple networking' vidal -sv
+1, but do we still need the firewall for other things? -Anand
On Wed, 2007-12-19 at 18:54 -0500, Anand Capur wrote:
The reason for all of this is the firewall in place at the PHX colo. If that wasn't there we wouldn't need any of the games at all. We could just have foo.fedoraproject.org be resolveable from anywhere and foo.vpn.fedoraproject.org just mean 'go over the vpn to get to it'. seth 'big fan of simple networking' vidal -sv+1, but do we still need the firewall for other things?
So the firewall is something that came with the space. It's red hat's firewall and I don't think we have any choice for the hosts inside phx.
In general, I'm a much bigger fan of hosts-based firewalling and clamping down on exposure paths that way than an edge firewall for a network. In this case it would also make our setup a good bit simpler if we didn't have the edge firewall at all.
-sv
seth vidal wrote:
On Wed, 2007-12-19 at 18:54 -0500, Anand Capur wrote:
The reason for all of this is the firewall in place at the PHX colo. If that wasn't there we wouldn't need any of the games at all. We could just have foo.fedoraproject.org be resolveable from anywhere and foo.vpn.fedoraproject.org just mean 'go over the vpn to get to it'. seth 'big fan of simple networking' vidal -sv+1, but do we still need the firewall for other things?
So the firewall is something that came with the space. It's red hat's firewall and I don't think we have any choice for the hosts inside phx.
In general, I'm a much bigger fan of hosts-based firewalling and clamping down on exposure paths that way than an edge firewall for a network. In this case it would also make our setup a good bit simpler if we didn't have the edge firewall at all.
Just so my stance on this is also public. In general I also agree that it is good to remove the PHX firewall from the mix. The biggest being IP space. (think about the builders and such). There's also a firewall there that we could re-implement ourselves. While long term I do want to re-think our interactions with PHX but I can't say for sure exactly what that will be. If, for example, we got funding to host all non-buildsystem stuff in our new German colo, many of these problems might go away.
I'd very much like to research the alternatives but for now I think the search domain method would suit us well.
-Mike
On Wed, 2007-12-19 at 19:24 -0600, Mike McGrath wrote:
seth vidal wrote:
On Wed, 2007-12-19 at 18:54 -0500, Anand Capur wrote:
The reason for all of this is the firewall in place at the PHX colo. If that wasn't there we wouldn't need any of the games at all. We could just have foo.fedoraproject.org be resolveable from anywhere and foo.vpn.fedoraproject.org just mean 'go over the vpn to get to it'. seth 'big fan of simple networking' vidal -sv+1, but do we still need the firewall for other things?
So the firewall is something that came with the space. It's red hat's firewall and I don't think we have any choice for the hosts inside phx.
In general, I'm a much bigger fan of hosts-based firewalling and clamping down on exposure paths that way than an edge firewall for a network. In this case it would also make our setup a good bit simpler if we didn't have the edge firewall at all.
Just so my stance on this is also public. In general I also agree that it is good to remove the PHX firewall from the mix. The biggest being IP space. (think about the builders and such). There's also a firewall there that we could re-implement ourselves. While long term I do want to re-think our interactions with PHX but I can't say for sure exactly what that will be. If, for example, we got funding to host all non-buildsystem stuff in our new German colo, many of these problems might go away.
I'd very much like to research the alternatives but for now I think the search domain method would suit us well.
option 2:
all hosts we maintain are written in /etc/hosts or hosts.db or something comparable specific to the site.
that would keep mitm down to a minimum, too, but it means keeping that file current.
-sv
seth vidal wrote:
On Wed, 2007-12-19 at 19:24 -0600, Mike McGrath wrote:
seth vidal wrote:
On Wed, 2007-12-19 at 18:54 -0500, Anand Capur wrote:
The reason for all of this is the firewall in place at the PHX colo. If that wasn't there we wouldn't need any of the games at all. We could just have foo.fedoraproject.org be resolveable from anywhere and foo.vpn.fedoraproject.org just mean 'go over the vpn to get to it'. seth 'big fan of simple networking' vidal -sv+1, but do we still need the firewall for other things?
So the firewall is something that came with the space. It's red hat's firewall and I don't think we have any choice for the hosts inside phx.
In general, I'm a much bigger fan of hosts-based firewalling and clamping down on exposure paths that way than an edge firewall for a network. In this case it would also make our setup a good bit simpler if we didn't have the edge firewall at all.
Just so my stance on this is also public. In general I also agree that it is good to remove the PHX firewall from the mix. The biggest being IP space. (think about the builders and such). There's also a firewall there that we could re-implement ourselves. While long term I do want to re-think our interactions with PHX but I can't say for sure exactly what that will be. If, for example, we got funding to host all non-buildsystem stuff in our new German colo, many of these problems might go away.
I'd very much like to research the alternatives but for now I think the search domain method would suit us well.
option 2:
all hosts we maintain are written in /etc/hosts or hosts.db or something comparable specific to the site.
that would keep mitm down to a minimum, too, but it means keeping that file current.
Does search in resolv.conf work with multiple /etc/hosts entries. If so we could do that though, like DNS, we'd need to maintain multiple hostnames / ip's. If that doesn't work then we'd have to maintain multiple /etc/hosts files.
-Mike
On 12/19/07, Mike McGrath mmcgrath@redhat.com wrote:
I forgot to mention one other concern. A MitM attack or DNS poisoning. This possibility does exist, but exists in our environment as is anyway. This is something we should look at mitigating but other than running a DNS server at every site, I'm not totally sure how to fix it. I consider all of our donations as partnerships. After all, they have local access to the box. At the same time though it is something we should count as a risk and mitigate as much as possible.
I believe that DNSSEC is supposed to be the solution to the MitM/DNS poisoning problem. It's been a while since I messed with it, but with DNSSEC your DNS entries get signed with a public key and then properly configured systems will check the signatures on all lookups involving fedora*.org. Having this as a part of the standard setup in Fedora's BIND package would be awesomely cool because then every Fedora machine would be protected against someone spoofing their DNS and possibly causing problems.
I've been meaning to set this up for my personal domain so I could work on the details over the holiday break...
Jeff
Jeffrey Ollie wrote:
On 12/19/07, Mike McGrath mmcgrath@redhat.com wrote:
I forgot to mention one other concern. A MitM attack or DNS poisoning. This possibility does exist, but exists in our environment as is anyway. This is something we should look at mitigating but other than running a DNS server at every site, I'm not totally sure how to fix it. I consider all of our donations as partnerships. After all, they have local access to the box. At the same time though it is something we should count as a risk and mitigate as much as possible.
I believe that DNSSEC is supposed to be the solution to the MitM/DNS poisoning problem. It's been a while since I messed with it, but with DNSSEC your DNS entries get signed with a public key and then properly configured systems will check the signatures on all lookups involving fedora*.org. Having this as a part of the standard setup in Fedora's BIND package would be awesomely cool because then every Fedora machine would be protected against someone spoofing their DNS and possibly causing problems.
I've been meaning to set this up for my personal domain so I could work on the details over the holiday break...
If you find a solution that might work for us while you're setting it up let us know, its certainly an avenue worth looking at.
-Mike
Jeffrey Ollie wrote:
On 12/19/07, Mike McGrath mmcgrath@redhat.com wrote:
I forgot to mention one other concern. A MitM attack or DNS poisoning. This possibility does exist, but exists in our environment as is anyway. This is something we should look at mitigating but other than running a DNS server at every site, I'm not totally sure how to fix it. I consider all of our donations as partnerships. After all, they have local access to the box. At the same time though it is something we should count as a risk and mitigate as much as possible.
I believe that DNSSEC is supposed to be the solution to the MitM/DNS poisoning problem. It's been a while since I messed with it, but with DNSSEC your DNS entries get signed with a public key and then properly configured systems will check the signatures on all lookups involving fedora*.org. Having this as a part of the standard setup in Fedora's BIND package would be awesomely cool because then every Fedora machine would be protected against someone spoofing their DNS and possibly causing problems.
I've been meaning to set this up for my personal domain so I could work on the details over the holiday break...
Also it appears that Paul Wounters is giving a session at FUDCon called "Integrating DNSSEC -- Proposal and demonstration of DNSSEC aware software.
-Mike
On 12/20/07, Mike McGrath mmcgrath@redhat.com wrote:
Also it appears that Paul Wounters is giving a session at FUDCon called "Integrating DNSSEC -- Proposal and demonstration of DNSSEC aware software.
Heh, now if someone would just buy me a plane ticket and pay for my hotel I could afford to show up :). What's the chances of some of these sessions being webcasted live, or at least recorded?
Jeff
Jeffrey Ollie wrote:
On 12/20/07, Mike McGrath mmcgrath@redhat.com wrote:
Also it appears that Paul Wounters is giving a session at FUDCon called "Integrating DNSSEC -- Proposal and demonstration of DNSSEC aware software.
Heh, now if someone would just buy me a plane ticket and pay for my hotel I could afford to show up :). What's the chances of some of these sessions being webcasted live, or at least recorded?
Some of them actually will be, at least they were last year. I'll hit up the FAB and see if we have any official resources to commit.
-Mike
infrastructure@lists.fedoraproject.org
-
Anand Capur -
Jeffrey Ollie -
Mike McGrath -
Seth Vidal -
Stephen John Smoogen