9:24 a.m.
I'm trying to migrate some container linux nodes to FCOS.
For my main pod to pod networking I use Calico and that continues to work fine on the new
FCOS nodes.
In addition I have Multus setup which allows me to attach additional network interfaces to
pods by creating network attachment definitions (which define the CNI plugin to use for
that network interface and its config) and then associating those with the few pods that
need them.
A NetworkAttachmentDefinition I have has the following config:
{
"cniVersion": "0.3.1",
"name": "lan",
"type": "bridge",
"bridge": "br-lan",
"ipam": {
"type": "static",
"addresses": [
{ "address": "10.1.0.2/16" }
]
}
}
Which really is as simple as it looks. Basically use the bridge CNI plugin to create a
veth pair so that the additional network interface in the container is added to the br-lan
bridge and configure the container side veth interface with the specified IP address.
If the pod that it annotated with this network attachment definition is moved onto an FCOS
node, its networking on that interface fails to work. From outside, if I tcpdump on the
FCOS node I see traffic traverse eno1 -> bond0 -> br-lan, but then it goes nowhere.
If the pod is on the container linux node that traffic additionally makes its way onto the
veth interface associated with the pod and everything works as expected.
In reverse I see the same thing. If I do something from the pod I see its traffic make it
to the veth interface on the host, but never onto the bridge, but on the CL node it makes
it to the bridge, through the other interfaces and out to the network. It's almost
like the veth interface hasn't been added to the bridge... But it has!
Like I said, this works totally fine on container linux. The networking is configured
exactly the same (other than via NetworkManager rather than systemd-networkd). So my
network interfaces are as follows:
eno1 -> bond0
eno2 -> bond0
bond0 -> br-lan
bond0.100 -> br-something
bond0.200 -> br-somethingelse
When the pod comes up, the bridge CNI plugin via Multus creates the veth interface and
adds the host side interface of the pair into br-lan. This output is from the FCOS node
(the 'master br-lan' part is apparently shows which devices are added to the
br-lan bridge if you haven't got brctl available to see the output that most people
are probably used to seeing):
# ip link | grep br-lan
7: br-lan: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode
DEFAULT group default qlen 1000
11: bond0: <BROADCAST,MULTICAST,MASTER,UP,LOWER_UP> mtu 1500 qdisc noqueue master
br-lan state UP mode DEFAULT group default qlen 1000
31: veth1f6cabd9@if5: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue
master br-lan state UP mode DEFAULT group default
And this from the CL node, which matches:
# ip link | grep br-lan
6: br-lan: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode
DEFAULT group default qlen 1000
10: bond0: <BROADCAST,MULTICAST,MASTER,UP,LOWER_UP> mtu 1500 qdisc noqueue master
br-lan state UP mode DEFAULT group default qlen 1000
218: vetha2c4788e@if5: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue
master br-lan state UP mode DEFAULT group default
ipv4 ip_forward is enabled on both nodes
# cat /proc/sys/net/ipv4/ip_forward
1
rp_filter for all of the above mentioned interfaces is set to 1 on the CL node and 2 on
the FCOS node. As far as I can tell 2 is the same as 1 but more loose about what it allows
so long as it can still route to the source of a packet somehow. So I can't see it
being the problem
Examining the bridge config, the CL node has stp disabled. The FCOS node has stp enabled.
If I investigate further, both bond0 and the veth interface are in the forwarding state.
(3 seems to be the FORWARDING state)
# cat /sys/devices/virtual/net/br-lan/brif/bond0/state
3
# cat /sys/devices/virtual/net/br-lan/brif/veth1f6cabd9/state
3
If I disable stp on the bridge on the FCOS node it makes no difference
I even started comparing everything under /sys/devices/virtual/net/br-lan on both nodes,
but still can't see anything out of the ordinary.
I really can't believe that anything has changed too much in how any of this works
between CL and FCOS, but i've run out of ideas. I've been doing VMs in a similar
way for 10 years or so too, so as far as I can tell the networking itself is sound.
Any thoughts welcomed.