New subject: FCOS / Kubernetes (using docker) / Multus / CNI Bridge plugin - pods unable to access the additional network

4 Sep 2020


      I'm trying to migrate some container linux nodes to FCOS.
For my main pod to pod networking I use Calico and that continues to work fine on the new FCOS nodes.
In addition I have Multus setup which allows me to attach additional network interfaces to pods by creating network attachment definitions (which define the CNI plugin to use for that network interface and its config) and then associating those with the few pods that need them.
A NetworkAttachmentDefinition I have has the following config:
{
 "cniVersion": "0.3.1",
 "name": "lan",
 "type": "bridge",
 "bridge": "br-lan",
 "ipam": {
  "type": "static",
  "addresses": [
   { "address": "10.1.0.2/16" }
  ]
 }
}
Which really is as simple as it looks. Basically use the bridge CNI plugin to create a veth pair so that the additional network interface in the container is added to the br-lan bridge and configure the container side veth interface with the specified IP address.
If the pod that it annotated with this network attachment definition is moved onto an FCOS node, its networking on that interface fails to work. From outside, if I tcpdump on the FCOS node I see traffic traverse eno1 -> bond0 -> br-lan, but then it goes nowhere. If the pod is on the container linux node that traffic additionally makes its way onto the veth interface associated with the pod and everything works as expected.
In reverse I see the same thing. If I do something from the pod I see its traffic make it to the veth interface on the host, but never onto the bridge, but on the CL node it makes it to the bridge, through the other interfaces and out to the network. It's almost like the veth interface hasn't been added to the bridge... But it has!
Like I said, this works totally fine on container linux. The networking is configured exactly the same (other than via NetworkManager rather than systemd-networkd). So my network interfaces are as follows:
eno1 -> bond0
eno2 -> bond0
bond0 -> br-lan
bond0.100 -> br-something
bond0.200 -> br-somethingelse
When the pod comes up, the bridge CNI plugin via Multus creates the veth interface and adds the host side interface of the pair into br-lan. This output is from the FCOS node (the 'master br-lan' part is apparently shows which devices are added to the br-lan bridge if you haven't got brctl available to see the output that most people are probably used to seeing):
# ip link  | grep br-lan
7: br-lan: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default qlen 1000
11: bond0: <BROADCAST,MULTICAST,MASTER,UP,LOWER_UP> mtu 1500 qdisc noqueue master br-lan state UP mode DEFAULT group default qlen 1000
31: veth1f6cabd9@if5: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br-lan state UP mode DEFAULT group default
And this from the CL node, which matches:
# ip link | grep br-lan
6: br-lan: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default qlen 1000
10: bond0: <BROADCAST,MULTICAST,MASTER,UP,LOWER_UP> mtu 1500 qdisc noqueue master br-lan state UP mode DEFAULT group default qlen 1000
218: vetha2c4788e@if5: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br-lan state UP mode DEFAULT group default
ipv4 ip_forward is enabled on both nodes
# cat /proc/sys/net/ipv4/ip_forward
1
rp_filter for all of the above mentioned interfaces is set to 1 on the CL node and 2 on the FCOS node. As far as I can tell 2 is the same as 1 but more loose about what it allows so long as it can still route to the source of a packet somehow. So I can't see it being the problem
Examining the bridge config, the CL node has stp disabled. The FCOS node has stp enabled. If I investigate further, both bond0 and the veth interface are in the forwarding state. (3 seems to be the FORWARDING state)
# cat /sys/devices/virtual/net/br-lan/brif/bond0/state 
3
# cat /sys/devices/virtual/net/br-lan/brif/veth1f6cabd9/state 
3
If I disable stp on the bridge on the FCOS node it makes no difference
I even started comparing everything under /sys/devices/virtual/net/br-lan on both nodes, but still can't see anything out of the ordinary.
I really can't believe that anything has changed too much in how any of this works between CL and FCOS, but i've run out of ideas. I've been doing VMs in a similar way for 10 years or so too, so as far as I can tell the networking itself is sound.
Any thoughts welcomed.