Routing not working

[Dan Track]

Hi,

I've got a server where I've got two interfaces (vlan 200 and vlan
300). If I get traffic coming into vlan 200 but sourced from a network
that vlan 200 subnet doesn't know about, the server should send the
traffic out of it's default gateway i.e. vlan 300, but this isn't
happening. If I do a tcpdump on both interfaces I can see traffic
coming in on vlan 200 but failing to even be present on the return in
vlan 300 (not even present on vlan 200 - did it for sanity sake).
However, if I put in an explicit route in stating that this unknown
network exists out of vlan 200 then everything works fine.

My routing table is like this:

ip route
166.14.134.144/28 dev vlan200  proto kernel  scope link  src 166.14.134.154
159.156.137.32/28 dev vlan300  proto kernel  scope link  src 159.156.137.42
127.0.0.0/8 dev lo  scope link
default via 159.156.137.33 dev vlan300

vlan 200 and vlan 300 sit on the same bonded interface. Trunking is
set on the switch.

It's almost as though the OS (kernel) is saying that you can't take
input from one int and then send it out another int. Are there some
parameters I need to change?

Thanks for any help.

Dan

[fedora]

echo "1" > /proc/sys/net/ipv4/ip_forward

suomi

On 2011-05-31 09:34, Dan Track wrote:
> Hi,
>
> I've got a server where I've got two interfaces (vlan 200 and vlan
> 300). If I get traffic coming into vlan 200 but sourced from a network
> that vlan 200 subnet doesn't know about, the server should send the
> traffic out of it's default gateway i.e. vlan 300, but this isn't
> happening. If I do a tcpdump on both interfaces I can see traffic
> coming in on vlan 200 but failing to even be present on the return in
> vlan 300 (not even present on vlan 200 - did it for sanity sake).
> However, if I put in an explicit route in stating that this unknown
> network exists out of vlan 200 then everything works fine.
>
> My routing table is like this:
>
> ip route
> 166.14.134.144/28 dev vlan200  proto kernel  scope link  src 166.14.134.154
> 159.156.137.32/28 dev vlan300  proto kernel  scope link  src 159.156.137.42
> 127.0.0.0/8 dev lo  scope link
> default via 159.156.137.33 dev vlan300
>
> vlan 200 and vlan 300 sit on the same bonded interface. Trunking is
> set on the switch.
>
> It's almost as though the OS (kernel) is saying that you can't take
> input from one int and then send it out another int. Are there some
> parameters I need to change?
>
> Thanks for any help.
>
> Dan

[Dan Track]

On Tue, May 31, 2011 at 8:48 AM, fedora  wrote:
> echo "1" > /proc/sys/net/ipv4/ip_forward
>
> suomi

Thanks,

Forgot to mention, that's already done.

cat /proc/sys/net/ipv4/ip_forward
1

Any other thoughts on this?

Thanks
Dan

[Dan Track]

On Tue, May 31, 2011 at 8:53 AM, Dan Track  wrote:
> On Tue, May 31, 2011 at 8:48 AM, fedora  wrote:
> > echo "1" > /proc/sys/net/ipv4/ip_forward
> >
> > suomi
>
> Thanks,
>
> Forgot to mention, that's already done.
>
> cat /proc/sys/net/ipv4/ip_forward
> 1
>
> Any other thoughts on this?
>
> Thanks
> Dan

Hey,

Just for future reference I figured out the problem. You need to run
on rp_filter on the interfaces you wish to include in the routing
process.

Dan

[Tom Horsley]

On Tue, 31 May 2011 14:17:44 +0100
Dan Track wrote:
> Just for future reference I figured out the problem. You need to run
> on rp_filter on the interfaces you wish to include in the routing
> process.

What exactly does that mean, and how do I do it? I've never
heard of it before, but it could easily be the reason
I haven't been able to get my USB wi-fi dongle working
as an access point in f15 while doing all the exact same
things I did in f14 (where it works perfectly).

[Genes MailLists]

On 05/31/2011 09:29 AM, Tom Horsley wrote:
> On Tue, 31 May 2011 14:17:44 +0100
> Dan Track wrote:
> > Just for future reference I figured out the problem. You need to run
> > on rp_filter on the interfaces you wish to include in the routing
> > process.
>
> What exactly does that mean, and how do I do it? I've never
> heard of it before, but it could easily be the reason
> I haven't been able to get my USB wi-fi dongle working
> as an access point in f15 while doing all the exact same
> things I did in f14 (where it works perfectly).

rp_filter (/proc/sys/net/ipv4/conf/*/rp_filter) attempts to avoid src
IP spoofing by checking src IP of packet and ensuring that it goes out
the way it came - to be a little more specific - if the 'best route' to
that src ip is not the same interface the packet came in on, rp_filter
will drop the packet.

Usually its fine (correct) to leave rp_filter on - be thoughtful if
you're doing something funky with routing tables.

Thats my recollection anyway ... you turn it on/off echo 1/0 into the
/proc/sys/xxx

gene/

[Tom Horsley]

On Tue, 31 May 2011 09:41:53 -0400
Genes MailLists wrote:
> Usually its fine (correct) to leave rp_filter on - be thoughtful if
> you're doing something funky with routing tables.

I never know what I'm doing with networking :-). I just
find prescriptions in google and try them, never
managing to get a clue what they mean. This is something
new to try (I'll have to check and see if the settings
are different in f14 and f15 by default). Thanks.

[Tom Horsley]

On Tue, 31 May 2011 09:29:09 -0400
Tom Horsley wrote:
> I haven't been able to get my USB wi-fi dongle working
> as an access point in f15 while doing all the exact same
> things I did in f14 (where it works perfectly).

Well, I finally got a change to try this, and nothing
I do to rp_filter (which seemed to have the same values
on f15 it has on f14 anyway) seems to have an effect.
I just can't get packets to do NAT routing when my
cellphone connects (at least that would explain my
symptoms). The phone connects, it talks to the DHCP
server and gets the expected IP addr in my private
subnet, but it can't talk to anyone else.

At this point I'm tempted to change the name of
the interface back to eth0 from the new em1 name
it gets in f15 just to see if maybe someone has
an eth0 hard coded somewhere :-).

Anyone have any pointers for debugging routing?
Anyone know of something that maybe changed in
this vicinity in f15?

[Genes MailLists]

On 06/01/2011 10:02 AM, Tom Horsley wrote:
> On Tue, 31 May 2011 09:29:09 -0400
> Tom Horsley wrote:
> At this point I'm tempted to change the name of
> the interface back to eth0 from the new em1 name
> it gets in f15 just to see if maybe someone has
> an eth0 hard coded somewhere :-).
>
> Anyone have any pointers for debugging routing?
> Anyone know of something that maybe changed in
> this vicinity in f15?

Mmm ... perhaps you could assemble more information for people to look
at ...

I'd suggest the following to start with - for the system set up and
what you believe should be working. Then please explain exactly what is
not working (packets from where to where) and what you are trying to do
(in terms of the interfaces etc named below) - are you using ipv4 only
or is ipv6 in play too ?

ifconfig
  iwconfig

ip route
  ip addr show

cat /proc/sys/net/ipv4/ip_forward

# you -may- at some point want to log martians but probably not yet
  for i in /proc/sys/net/ipv4/conf/*/log_martians
  do
    echo 1 > $i
  done

Also your relevant iptables rules could be important here too hold off
for the moment till people understand exactly what you're trying to do
with what interface and routing what to where.

gene

[Genes MailLists]

On 06/01/2011 10:02 AM, Tom Horsley wrote:
> On Tue, 31 May 2011 09:29:09 -0400
> Tom Horsley wrote:
> Anyone have any pointers for debugging routing?
> Anyone know of something that maybe changed in
> this vicinity in f15?

One quick question - did you change your iptables rules to use the new
interface names ?

[Tom Horsley]

On Wed, 01 Jun 2011 10:25:27 -0400
Genes MailLists wrote:
> On 06/01/2011 10:02 AM, Tom Horsley wrote:
> > On Tue, 31 May 2011 09:29:09 -0400
> > Tom Horsley wrote:
> > Anyone have any pointers for debugging routing?
> > Anyone know of something that maybe changed in
> > this vicinity in f15?
>
> One quick question - did you change your iptables rules to use the new
> interface names ?

Yep. I found and fixed all the eth0 names in any of the
config files involved.

Most of the info about what I'm doing is described here:

http://home.comcast.net/~tomhorsley/hardware/rtl8192cu/rtl8192cu.html#Succes...

This works great under fedora 14, but my phone can't talk
to anything (other than the DHCP server) on fedora 15 doing
all the same stuff. It gets the wi-fi connection, but thats it.

[Genes MailLists]

On 06/01/2011 10:46 AM, Tom Horsley wrote:
> On Wed, 01 Jun 2011 10:25:27 -0400
>
> Most of the info about what I'm doing is described here:
>
> http://home.comcast.net/~tomhorsley/hardware/rtl8192cu/rtl8192cu.html#Succes...
>
> This works great under fedora 14, but my phone can't talk
> to anything (other than the DHCP server) on fedora 15 doing
> all the same stuff. It gets the wi-fi connection, but thats it.

I'll try and go over what you did - unfortunately, I am totally
unfamiliar with hostapd or why you need it (is it related to your dongle
somehow ?) and how it interacts with other parts of
firewall/routing/networking.

In the meantime can you provide the outputs of what I suggested
previously for the F15 system. Are you seeing any AVC's by the way ?

[Tom Horsley]

On Wed, 01 Jun 2011 11:07:54 -0400
Genes MailLists wrote:
> I'll try and go over what you did - unfortunately, I am totally
> unfamiliar with hostapd or why you need it (is it related to your dongle
> somehow ?) and how it interacts with other parts of
> firewall/routing/networking.

That's what turns it into an access point rather than running the
dongle in normal mode where it is a client of some other access point.

I'll have to wait till I have a chance to boot f15 again to get
the additional info, but I have selinux disabled, so I doubt
I'm getting any AVCs.

[Genes MailLists]

On 06/01/2011 11:24 AM, Tom Horsley wrote:
> On Wed, 01 Jun 2011 11:07:54 -0400
> Genes MailLists wrote:
> > I'll try and go over what you did - unfortunately, I am totally
> > unfamiliar with hostapd or why you need it (is it related to your dongle
> > somehow ?) and how it interacts with other parts of
> > firewall/routing/networking.
>
> That's what turns it into an access point rather than running the
> dongle in normal mode where it is a client of some other access point.
>
> I'll have to wait till I have a chance to boot f15 again to get
> the additional info, but I have selinux disabled, so I doubt
> I'm getting any AVCs.

Ok - and the hostapd is the fedora F15 version installed via yum?

[Tom Horsley]

On Wed, 01 Jun 2011 12:23:04 -0400
Genes MailLists wrote:
> Ok - and the hostapd is the fedora F15 version installed via yum?

Yep, that's the one I normally use. I also tried building from
the latest source on the hostapd home page, but got the same
results.

[Tom Horsley]

On Wed, 01 Jun 2011 11:07:54 -0400
Genes MailLists wrote:
> In the meantime can you provide the outputs of what I suggested
> previously for the F15 system. Are you seeing any AVC's by the way ?

I got the info from f15, but I also found when I rebooted
that the wi-fi on the phone couldn't connect this time because
the dhcpd died for no obvious reason after my script started
it. Also, I'm not using any ipv6 on f14 or f15 (as far as I know).

Anyway, other than substituting em1 for eth0, the output I gathered
on f15 looked identical to the output on f14:

ifconfig says:

em1       Link encap:Ethernet  HWaddr 00:1E:4F:9A:D6:E5  
          inet addr:10.134.30.143  Bcast:10.134.30.255  Mask:255.255.255.0
          inet6 addr: fe80::21e:4fff:fe9a:d6e5/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:2468 errors:0 dropped:0 overruns:0 frame:0
          TX packets:700 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:218060 (212.9 KiB)  TX bytes:76510 (74.7 KiB)
          Interrupt:21 Memory:febe0000-fec00000

lo        Link encap:Local Loopback  
          inet addr:127.0.0.1  Mask:255.0.0.0
          inet6 addr: ::1/128 Scope:Host
          UP LOOPBACK RUNNING  MTU:16436  Metric:1
          RX packets:1081 errors:0 dropped:0 overruns:0 frame:0
          TX packets:1081 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0 
          RX bytes:83988 (82.0 KiB)  TX bytes:83988 (82.0 KiB)

mon.wlan0 Link encap:UNSPEC  HWaddr 94-44-52-45-F3-3A-E0-F1-00-00-00-00-00-00-00-00  
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:78 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:6855 (6.6 KiB)  TX bytes:0 (0.0 b)

wlan0     Link encap:Ethernet  HWaddr 94:44:52:45:F3:3A  
          inet addr:192.168.9.1  Bcast:192.168.9.255  Mask:255.255.255.0
          inet6 addr: fe80::9644:52ff:fe45:f33a/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:55 errors:0 dropped:4 overruns:0 frame:0
          TX packets:51 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:13720 (13.3 KiB)  TX bytes:13844 (13.5 KiB)

Here's iwconfig:

lo        no wireless extensions.

em1       no wireless extensions.

wlan0     IEEE 802.11bg  Mode:Master  Frequency:2.462 GHz  Tx-Power=27 dBm   
          Retry  long limit:7   RTS thr:off   Fragment thr:off
          Power Management:on

mon.wlan0  IEEE 802.11bg  Mode:Monitor  Tx-Power=27 dBm   
          Retry  long limit:7   RTS thr:off   Fragment thr:off
          Power Management:off

Here's ip route:

10.134.30.0/24 dev em1  proto kernel  scope link  src 10.134.30.143 
192.168.9.0/24 dev wlan0  proto kernel  scope link  src 192.168.9.1 
169.254.0.0/16 dev em1  scope link  metric 1002 
default via 10.134.30.196 dev em1

Here's ip addr show:

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue state UNKNOWN 
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: em1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 00:1e:4f:9a:d6:e5 brd ff:ff:ff:ff:ff:ff
    inet 10.134.30.143/24 brd 10.134.30.255 scope global em1
    inet6 fe80::21e:4fff:fe9a:d6e5/64 scope link 
       valid_lft forever preferred_lft forever
3: wlan0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000
    link/ether 94:44:52:45:f3:3a brd ff:ff:ff:ff:ff:ff
    inet 192.168.9.1/24 brd 192.168.9.255 scope global wlan0
    inet6 fe80::9644:52ff:fe45:f33a/64 scope link 
       valid_lft forever preferred_lft forever
4: mon.wlan0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UNKNOWN qlen 1000
    link/ieee802.11/radiotap 94:44:52:45:f3:3a brd ff:ff:ff:ff:ff:ff

Here's /proc/sys/net/ipv4/ip_forward

1

I'm starting to wonder if the ralink driver it is using is
just flaky on f15.

[Genes MailLists]

On 06/01/2011 12:32 PM, Tom Horsley wrote:
> On Wed, 01 Jun 2011 11:07:54 -0400
> ifconfig says:
>
> em1       Link encap:Ethernet  HWaddr 00:1E:4F:9A:D6:E5  
>           inet addr:10.134.30.143  Bcast:10.134.30.255  Mask:255.255.255.0
>           inet6 addr: fe80::21e:4fff:fe9a:d6e5/64 Scope:Link

ipv6 looks on to me ..
> mon.wlan0 Link encap:UNSPEC  HWaddr 94-44-52-45-F3-3A-E0-F1-00-00-00-00-00-00-00-00  
>           UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1

I suppose this has something to do with hostapd ?
> wlan0     Link encap:Ethernet  HWaddr 94:44:52:45:F3:3A  
>           inet addr:192.168.9.1  Bcast:192.168.9.255  Mask:255.255.255.0
>           inet6 addr: fe80::9644:52ff:fe45:f33a/64 Scope:Link

ditto
>
> Here's ip route:
>
> 10.134.30.0/24 dev em1  proto kernel  scope link  src 10.134.30.143 
> 192.168.9.0/24 dev wlan0  proto kernel  scope link  src 192.168.9.1 
> 169.254.0.0/16 dev em1  scope link  metric 1002 
> default via 10.134.30.196 dev em1

I assume the default route is to your internet gateway ..
> Here's ip addr show:
> 2: em1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
>     link/ether 00:1e:4f:9a:d6:e5 brd ff:ff:ff:ff:ff:ff
>     inet 10.134.30.143/24 brd 10.134.30.255 scope global em1
>     inet6 fe80::21e:4fff:fe9a:d6e5/64 scope link 
>        valid_lft forever preferred_lft forever
> 3: wlan0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000
>     link/ether 94:44:52:45:f3:3a brd ff:ff:ff:ff:ff:ff
>     inet 192.168.9.1/24 brd 192.168.9.255 scope global wlan0
>     inet6 fe80::9644:52ff:fe45:f33a/64 scope link 
>        valid_lft forever preferred_lft forever
> 4: mon.wlan0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UNKNOWN qlen 1000
>     link/ieee802.11/radiotap 94:44:52:45:f3:3a brd ff:ff:ff:ff:ff:ff
>
> Here's /proc/sys/net/ipv4/ip_forward
>
> 1
>
> I'm starting to wonder if the ralink driver it is using is
> just flaky on f15.

Possible - I think I'd try turning off IPV6 first .. to make sure its
not interfering in some way .. probably not but might not hurt to test that.

[Tom Horsley]

On Wed, 01 Jun 2011 13:49:30 -0400
Genes MailLists wrote:
> > I'm starting to wonder if the ralink driver it is using is
> > just flaky on f15.
>
> Possible - I think I'd try turning off IPV6 first .. to make sure its
> not interfering in some way .. probably not but might not hurt to test that.

Actually, in the flaky behavior department, I just tried booting f15
one more time, and this time everything started working when I ran
the script. I think the fact that it sometimes works says I'm probably
doing routing correctly, but some other random problem is screwing me
up. Maybe I need some delays, or I should be doing things in a different
order or something. It seems like wlan0 sometimes just stops working
when I poke through the logs. The dongle does get quite warm when it
is running, maybe it is just overheating.

[Genes MailLists]

On 06/01/2011 02:21 PM, Tom Horsley wrote:
> Actually, in the flaky behavior department, I just tried booting f15
> one more time, and this time everything started working when I ran
> the script. I think the fact that it sometimes works says I'm probably
> doing routing correctly, but some other random problem is screwing me
> up. Maybe I need some delays, or I should be doing things in a different
> order or something. It seems like wlan0 sometimes just stops working
> when I poke through the logs. The dongle does get quite warm when it
> is running, maybe it is just overheating.

Ah ... could well be ... maybe its time for a re-dongle :-)

[Tom Horsley]

On Wed, 01 Jun 2011 14:31:21 -0400
Genes MailLists wrote:
> Ah ... could well be ... maybe its time for a re-dongle :-)

My next step is finding the digital thermometer we have around
here somewhere and measuring the temp on f14 compared to f15 :-).

Thanks for all the help.

[Genes MailLists]

On 06/01/2011 02:39 PM, Tom Horsley wrote:
> On Wed, 01 Jun 2011 14:31:21 -0400
> Genes MailLists wrote:
> > Ah ... could well be ... maybe its time for a re-dongle :-)
>
> My next step is finding the digital thermometer we have around
> here somewhere and measuring the temp on f14 compared to f15 :-).
>
> Thanks for all the help.

sure but I didn't actually help ...

[Tom Horsley]

> > Ah ... could well be ... maybe its time for a re-dongle :-)
>
> My next step is finding the digital thermometer we have around
> here somewhere and measuring the temp on f14 compared to f15 :-).

Since the kernel update a day or two ago, this wi-fi dongle has
started working perfectly again in f15 just like it did in f14,
so I guess there was a driver bug of some kind that got fixed.
(No longer gets any hotter in f15 than it did in f14 either).

[JD]

On 06/01/11 10:49, Genes MailLists wrote:
> > ifconfig says:
> >
> > em1       Link encap:Ethernet  HWaddr 00:1E:4F:9A:D6:E5
> >            inet addr:10.134.30.143  Bcast:10.134.30.255  Mask:255.255.255.0
> >            inet6 addr: fe80::21e:4fff:fe9a:d6e5/64 Scope:Link
>  ipv6 looks on to me ..

If you blacklist ipv6 (which I tried) then some installed software,
and some interfaces will belch out error messages, such as:

May 26 21:13:19 localhost kernel: bridge: Unknown symbol 
ipv6_dev_get_saddr (err 0)

You might want to look at

/etc/sysconfig/network-scripts/ifdown-ipv6

[Andre Speelmans]

> If you blacklist ipv6 (which I tried) then some installed software,
> and some interfaces will belch out error messages, such as:
>
> May 26 21:13:19 localhost kernel: bridge: Unknown symbol
> ipv6_dev_get_saddr (err 0)
>
> You might want to look at
>
> /etc/sysconfig/network-scripts/ifdown-ipv6

I always turned it off by adding:
NETWORKING_IPV6=off
to /etc/sysconfig/network.

Not sure if that will still work on Fedora 15, did fine on RHEL 5.

[JB]

Tom Horsley <horsley1953 <at> gmail.com> writes:
> ...
> Here's ip route:
>
> 10.134.30.0/24 dev em1  proto kernel  scope link  src 10.134.30.143 
> 192.168.9.0/24 dev wlan0  proto kernel  scope link  src 192.168.9.1 
> 169.254.0.0/16 dev em1  scope link  metric 1002 
> default via 10.134.30.196 dev em1 
> ...

I would give it a try ...

This line:
> 169.254.0.0/16 dev em1  scope link  metric 1002

is about Zero configuration networking (zeroconf)
http://en.wikipedia.org/wiki/Zero_configuration_networking
which is implemented on F15 with Avahi-daemon.

Disable it permanently, and reboot to get a clean networking/routing table.

JB