Weird routing problem

[Turbo Fredriksson]

I have two physical hosts (Correo and Alexander), running two XEN
instances on one of them (Ferrari and Amarillo on Correo) and one
on the other (Graham on Alexander)...

Picture at http://bayour.com/misc/VoIP.jpg.


On the firewall/gateway (192.168.1.1) I route 192.168.3.0/24 to Correo
(192.168.1.7) and 192.168.4.0/24 to Alexander (192.168.1.6). This so
that I can access the XEN hosts from the internal network. Very basic...

And all my VoIP phones is on it's (about to be on a) separate network
with the firewall/gateway as default gateway.


On Alexander:
=============
  * /etc/xen/graham.cfg
    kernel          = '/boot/vmlinuz-2.6.18-5-xen-amd64'
    ramdisk         = '/boot/initrd.img-2.6.18-5-xen-amd64'
    memory          = '2500'
    root            = '/dev/sda1 ro'
    disk            = [ 'file:/home/xen/domains/graham/disk.img,sda1,w',
'file:/home/xen/domains/graham/swap.img,sda2,w' ]
    name            = 'graham'
    vif             = [ 'ip=192.168.4.11' ]
    on_poweroff     = 'destroy'
    on_reboot       = 'restart'
    on_crash        = 'restart'

  * /etc/xen/xend-config.sxp
    (xend-http-server yes)
    (xend-unix-server yes)
    (xend-tcp-xmlrpc-server no)
    (xend-unix-xmlrpc-server yes)
    (xend-relocation-server yes)
    (xend-unix-path /var/lib/xend/xend-socket)
    (xend-port            8000)
    (xend-relocation-port 8002)
    (xend-address 'alexander')
    (xend-relocation-address 'alexander')
    (console-limit 1024)
    (network-script network-route)
    (vif-script     vif-route)
    (dom0-min-mem 196)
    (dom0-cpus 2)
    (enable-dump yes)
    (vnc-listen '0.0.0.0')

  * ifconfig (trimmed - only 'lo' if removed)
    eth0      Link encap:Ethernet  HWaddr 00:1C:23:C4:28:92  
              inet addr:192.168.1.6  Bcast:192.168.1.255  Mask:255.255.255.0
              inet6 addr: fe80::21c:23ff:fec4:2892/64 Scope:Link
              UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
    
    eth0:0    Link encap:Ethernet  HWaddr 00:1C:23:C4:28:92  
              inet addr:192.168.4.1  Bcast:192.168.4.255  Mask:255.255.255.0
              UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
    
    vif5.0    Link encap:Ethernet  HWaddr FE:FF:FF:FF:FF:FF  
              inet addr:192.168.1.6  Bcast:192.168.1.255  Mask:255.255.255.255
              inet6 addr: fe80::fcff:ffff:feff:ffff/64 Scope:Link
              UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
  
  * route -n
    Kernel IP routing table
    Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
    192.168.4.11    0.0.0.0         255.255.255.255 UH    0      0        0 vif5.0
    192.168.4.0     0.0.0.0         255.255.255.0   U     0      0        0 eth0
    192.168.1.0     0.0.0.0         255.255.255.0   U     0      0        0 eth0
    0.0.0.0         192.168.1.1     0.0.0.0         UG    0      0        0 eth0

  * iptables -L -n
    Chain INPUT (policy ACCEPT)
    target     prot opt source               destination         
    
    Chain FORWARD (policy ACCEPT)
    target     prot opt source               destination         
    ACCEPT     0    --  192.168.4.11         0.0.0.0/0           PHYSDEV match
--physdev-in vif5.0 
    ACCEPT     udp  --  0.0.0.0/0            0.0.0.0/0           PHYSDEV match
--physdev-in vif5.0 udp spt:68 dpt:67 
    
    Chain OUTPUT (policy ACCEPT)
    target     prot opt source               destination         

  * iptables -L -n -t nat
    Chain PREROUTING (policy ACCEPT)
    target     prot opt source               destination         
    
    Chain POSTROUTING (policy ACCEPT)
    target     prot opt source               destination         
    
    Chain OUTPUT (policy ACCEPT)
    target     prot opt source               destination

On Graham:
==========
  * ifconfig (trimmed - only 'lo' if removed)
    eth0      Link encap:Ethernet  HWaddr 00:16:3E:00:AB:28  
              inet addr:192.168.4.11  Bcast:192.168.4.255  Mask:255.255.255.0
              UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1

  * route -n
    Kernel IP routing table
    Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
    192.168.4.0     0.0.0.0         255.255.255.0   U     0      0        0 eth0
    0.0.0.0         192.168.4.1     0.0.0.0         UG    0      0        0 eth0

  * iptables -L -n
    FATAL: Could not load /lib/modules/2.6.18-5-xen-amd64/modules.dep: No such file or
directory
    iptables v1.3.6: can't initialize iptables table `filter': iptables who? (do
you need to insmod?)
    Perhaps iptables or your kernel needs to be upgraded.

Correo with the XEN hosts Ferrari and Amarillo basically look identical (only different
networks).

As seen, I do NOT use NAT here. I wanted to use true routed network... And it seems to
work.
My primary Asterisk server (the one that do all the routing - the one on Alexander only
deals
with the PSTN trafik) runs on Graham and it can be accessed from the outside - with port
forwarding on the firewall/gateway and it can also contact external Asterisk servers (I
run
one at home to deal with my private VoIP).


The DNS runs on Correo, but it can not be reached (queried) from Graham!

----- s n i p -----
graham# ping -c 5 correo
ping: unknown host correo

graham# ping -c 5 192.168.1.7
PING 192.168.1.7 (192.168.1.7) 56(84) bytes of data.
64 bytes from 192.168.1.7: icmp_seq=1 ttl=62 time=0.270 ms
64 bytes from 192.168.1.7: icmp_seq=2 ttl=62 time=0.260 ms
64 bytes from 192.168.1.7: icmp_seq=3 ttl=62 time=0.264 ms
64 bytes from 192.168.1.7: icmp_seq=4 ttl=62 time=0.273 ms
64 bytes from 192.168.1.7: icmp_seq=5 ttl=62 time=0.257 ms

--- 192.168.1.7 ping statistics ---
5 packets transmitted, 5 received, 0% packet loss, time 4000ms
rtt min/avg/max/mdev = 0.257/0.264/0.273/0.021 ms

graham# traceroute -n 192.168.1.7
traceroute to 192.168.1.7 (192.168.1.7), 30 hops max, 52 byte packets
 1  192.168.1.6  0.285 ms  0.091 ms  0.090 ms
 2  192.168.1.7  0.323 ms  0.262 ms  0.258 ms

graham# telnet 192.168.1.7 53
Trying 192.168.1.7...
Connected to 192.168.1.7.
Escape character is '^]'.
correo
Connection closed by foreign host.

graham# host graham 192.168.1.7
;; reply from unexpected source: 192.168.1.1#53, expected 192.168.1.7#53
;; reply from unexpected source: 192.168.1.1#53, expected 192.168.1.7#53
;; connection timed out; no servers could be reached
----- s n i p -----

Also, scp or ssh FROM Graham to Correo don't work, but the other way
around works fine...


Looking at the answer that 'host' gave me, I now see that the connection
goes via the firewall/gateway which is not directly obvious - Alexander
(which is Graham's default GW) is on the same network as Correo...


PS. I solved this specific DNS problem with a caching DNS server on
    Alexander, but scp/ssh (etc) naturally still don't work because
    of this weird problem... I just can't see it! Maybe a set of
    (many :) extra eyes can... Thanx!

[Dale Bewley]

On Thu, 2008-04-10 at 14:55 +0200, Turbo Fredriksson wrote:
>  I have two physical hosts (Correo and Alexander), running two XEN
>  instances on one of them (Ferrari and Amarillo on Correo) and one
>  on the other (Graham on Alexander)...
>  
>  Picture at http://bayour.com/misc/VoIP.jpg.
>  
>  
>  On the firewall/gateway (192.168.1.1) I route 192.168.3.0/24 to Correo
>  (192.168.1.7) and 192.168.4.0/24 to Alexander (192.168.1.6). This so
>  that I can access the XEN hosts from the internal network. Very basic...
>  
>  And all my VoIP phones is on it's (about to be on a) separate network
>  with the firewall/gateway as default gateway.
>  
>  
>  On Alexander:
>  =============
>    * /etc/xen/graham.cfg
>      kernel          = '/boot/vmlinuz-2.6.18-5-xen-amd64'
>      ramdisk         = '/boot/initrd.img-2.6.18-5-xen-amd64'
>      memory          = '2500'
>      root            = '/dev/sda1 ro'
>      disk            = [ 'file:/home/xen/domains/graham/disk.img,sda1,w',
> 'file:/home/xen/domains/graham/swap.img,sda2,w' ]
>      name            = 'graham'
>      vif             = [ 'ip=192.168.4.11' ]
>      on_poweroff     = 'destroy'
>      on_reboot       = 'restart'
>      on_crash        = 'restart'
>  
>    * /etc/xen/xend-config.sxp
>      (xend-http-server yes)
>      (xend-unix-server yes)
>      (xend-tcp-xmlrpc-server no)
>      (xend-unix-xmlrpc-server yes)
>      (xend-relocation-server yes)
>      (xend-unix-path /var/lib/xend/xend-socket)
>      (xend-port            8000)
>      (xend-relocation-port 8002)
>      (xend-address 'alexander')
>      (xend-relocation-address 'alexander')
>      (console-limit 1024)
>      (network-script network-route)
>      (vif-script     vif-route)
>      (dom0-min-mem 196)
>      (dom0-cpus 2)
>      (enable-dump yes)
>      (vnc-listen '0.0.0.0')
>  
>    * ifconfig (trimmed - only 'lo' if removed)
>      eth0      Link encap:Ethernet  HWaddr 00:1C:23:C4:28:92  
>                inet addr:192.168.1.6  Bcast:192.168.1.255  Mask:255.255.255.0
>                inet6 addr: fe80::21c:23ff:fec4:2892/64 Scope:Link
>                UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
>      
>      eth0:0    Link encap:Ethernet  HWaddr 00:1C:23:C4:28:92  
>                inet addr:192.168.4.1  Bcast:192.168.4.255  Mask:255.255.255.0
>                UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
>      
>      vif5.0    Link encap:Ethernet  HWaddr FE:FF:FF:FF:FF:FF  
>                inet addr:192.168.1.6  Bcast:192.168.1.255  Mask:255.255.255.255
>                inet6 addr: fe80::fcff:ffff:feff:ffff/64 Scope:Link
>                UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
>    
>    * route -n
>      Kernel IP routing table
>      Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
>      192.168.4.11    0.0.0.0         255.255.255.255 UH    0      0        0 vif5.0
>      192.168.4.0     0.0.0.0         255.255.255.0   U     0      0        0 eth0
>      192.168.1.0     0.0.0.0         255.255.255.0   U     0      0        0 eth0
>      0.0.0.0         192.168.1.1     0.0.0.0         UG    0      0        0 eth0
>  
>    * iptables -L -n
>      Chain INPUT (policy ACCEPT)
>      target     prot opt source               destination         
>      
>      Chain FORWARD (policy ACCEPT)
>      target     prot opt source               destination         
>      ACCEPT     0    --  192.168.4.11         0.0.0.0/0           PHYSDEV match
> --physdev-in vif5.0 
>      ACCEPT     udp  --  0.0.0.0/0            0.0.0.0/0           PHYSDEV match
> --physdev-in vif5.0 udp spt:68 dpt:67 
>      
>      Chain OUTPUT (policy ACCEPT)
>      target     prot opt source               destination         
>  
>    * iptables -L -n -t nat
>      Chain PREROUTING (policy ACCEPT)
>      target     prot opt source               destination         
>      
>      Chain POSTROUTING (policy ACCEPT)
>      target     prot opt source               destination         
>      
>      Chain OUTPUT (policy ACCEPT)
>      target     prot opt source               destination
>  
>  On Graham:
>  ==========
>    * ifconfig (trimmed - only 'lo' if removed)
>      eth0      Link encap:Ethernet  HWaddr 00:16:3E:00:AB:28  
>                inet addr:192.168.4.11  Bcast:192.168.4.255  Mask:255.255.255.0
>                UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
>  
>    * route -n
>      Kernel IP routing table
>      Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
>      192.168.4.0     0.0.0.0         255.255.255.0   U     0      0        0 eth0
>      0.0.0.0         192.168.4.1     0.0.0.0         UG    0      0        0 eth0
>  
>    * iptables -L -n
>      FATAL: Could not load /lib/modules/2.6.18-5-xen-amd64/modules.dep: No such file or
> directory
>      iptables v1.3.6: can't initialize iptables table `filter': iptables who? (do
> you need to insmod?)
>      Perhaps iptables or your kernel needs to be upgraded.
>  
>  Correo with the XEN hosts Ferrari and Amarillo basically look identical (only different
>  networks).
>  
>  As seen, I do NOT use NAT here. I wanted to use true routed network... And it seems to
> work.
>  My primary Asterisk server (the one that do all the routing - the one on Alexander only
> deals
>  with the PSTN trafik) runs on Graham and it can be accessed from the outside - with port
>  forwarding on the firewall/gateway and it can also contact external Asterisk servers (I
> run
>  one at home to deal with my private VoIP).
>  
>  
>  The DNS runs on Correo, but it can not be reached (queried) from Graham!
>  
>  ----- s n i p -----
>  graham# ping -c 5 correo
>  ping: unknown host correo
>  
>  graham# ping -c 5 192.168.1.7
>  PING 192.168.1.7 (192.168.1.7) 56(84) bytes of data.
>  64 bytes from 192.168.1.7: icmp_seq=1 ttl=62 time=0.270 ms
>  64 bytes from 192.168.1.7: icmp_seq=2 ttl=62 time=0.260 ms
>  64 bytes from 192.168.1.7: icmp_seq=3 ttl=62 time=0.264 ms
>  64 bytes from 192.168.1.7: icmp_seq=4 ttl=62 time=0.273 ms
>  64 bytes from 192.168.1.7: icmp_seq=5 ttl=62 time=0.257 ms
>  
>  --- 192.168.1.7 ping statistics ---
>  5 packets transmitted, 5 received, 0% packet loss, time 4000ms
>  rtt min/avg/max/mdev = 0.257/0.264/0.273/0.021 ms
>  
>  graham# traceroute -n 192.168.1.7
>  traceroute to 192.168.1.7 (192.168.1.7), 30 hops max, 52 byte packets
>   1  192.168.1.6  0.285 ms  0.091 ms  0.090 ms
>   2  192.168.1.7  0.323 ms  0.262 ms  0.258 ms
>  
>  graham# telnet 192.168.1.7 53
>  Trying 192.168.1.7...
>  Connected to 192.168.1.7.
>  Escape character is '^]'.
>  correo
>  Connection closed by foreign host.
>  
>  graham# host graham 192.168.1.7
>  ;; reply from unexpected source: 192.168.1.1#53, expected 192.168.1.7#53
>  ;; reply from unexpected source: 192.168.1.1#53, expected 192.168.1.7#53
>  ;; connection timed out; no servers could be reached
>  ----- s n i p -----
>  
>  Also, scp or ssh FROM Graham to Correo don't work, but the other way
>  around works fine...
>  
>  
>  Looking at the answer that 'host' gave me, I now see that the connection
>  goes via the firewall/gateway which is not directly obvious - Alexander
>  (which is Graham's default GW) is on the same network as Correo...
>  
>  
>  PS. I solved this specific DNS problem with a caching DNS server on
>      Alexander, but scp/ssh (etc) naturally still don't work because
>      of this weird problem... I just can't see it! Maybe a set of
>      (many :) extra eyes can... Thanx! 
Did you figure it out yet?

I can not quite tell what you're doing. Who is that blue router in your
diagram? Maybe you are using a router icon to indicate a switch?

At any rate if I had to guess it sounds like you are expecting to speak
from eth0:0 IP when you are actually speaking from eth0 IP. This you
could problably confirm with a tcpdump on the target machine while you
probe from the other. If so, you could fix that with policy routing AKA
source routing. i.e. `ip rule help`. Look at your ARP tables too just in
case.

[Turbo Fredriksson]

Quoting Dale Bewley <dlbewley(a)lib.ucdavis.edu&gt;:

>  Did you figure it out yet? 
No I didn't.

>  I can not quite tell what you're doing. Who is that blue router
> in your
>  diagram? Maybe you are using a router icon to indicate a switch? 
I am, sorry. Didn't find any switch icons in dia.

>  At any rate if I had to guess it sounds like you are expecting to
> speak
>  from eth0:0 IP when you are actually speaking from eth0 IP. This you
>  could problably confirm with a tcpdump on the target machine while you
>  probe from the other. If so, you could fix that with policy routing AKA
>  source routing. i.e. `ip rule help`. Look at your ARP tables too just in
>  case. 
I'll have a look. Thanx for the idea...