Enabling firewalld is breaking my docker connectivity, need some more eyes on the problem
Hello,
I have docker and firewalld both installed on my Arch Linux machine and have been happy with them so far, although I've recently run into an issue where I've broken what seems to be DNS for internal running containers, and I'm trying to isolate the reason for that. Recently I created a bridge with my primary static ip address and added my physical NIC as a slave on the bridge so that I'd be able to run libvirt VMs that appeared to be machines directly on my local network. That has all worked as expected:
--- # ipb link lo UNKNOWN 00:00:00:00:00:00 <LOOPBACK,UP,LOWER_UP> enp0s25 UP d0:50:99:a5:08:49 <BROADCAST,MULTICAST,UP,LOWER_UP> exbr UP 9a:5f:b2:1e:94:a7 <BROADCAST,MULTICAST,UP,LOWER_UP> virbr0 DOWN 52:54:00:4f:ca:69 <NO-CARRIER,BROADCAST,MULTICAST,UP> virbr0-nic DOWN 52:54:00:4f:ca:69 <BROADCAST,MULTICAST> docker0 DOWN 02:42:dd:21:67:40 <NO-CARRIER,BROADCAST,MULTICAST,UP>
# bridge link show 2: enp0s25: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 master exbr state forwarding priority 32 cost 100 5: virbr0-nic: <BROADCAST,MULTICAST> mtu 1500 master virbr0 state disabled priority 32 cost 100
# ipb a lo UNKNOWN 127.0.0.1/8 ::1/128 enp0s25 UP exbr UP 192.168.2.97/24 fe80::985f:b2ff:fe1e:94a7/64 virbr0 DOWN 192.168.100.1/24 virbr0-nic DOWN docker0 DOWN 172.17.0.1/16 fe80::46e8:567:b3cf:1aa1/64 ---
My problem has suddenly become: My docker containers do not appear to be capable of dns requests unless I *disable* firwalld, something I obviously do not want to do. So, I've taken the approach of turning off the firewalld, restarting docker, and capturing the known good iptables dump, and then tried to compare it to a known bad dump that is produced after I turn on firewalld and restart docker. Interestingly *they are identical* as far as I can tell [1]. Additionally, I tried to TRACE the traffic, and I'm seeing some very confusing info out of the kernel logs:
When things are configured in the "bad" state, the last log I see is:
Sep 08 06:08:49 baldur kernel: TRACE: filter:FORWARD:rule:5 IN=docker0 OUT=exbrPHYSIN=veth7880ce1 MAC=02:42:dd:21:67:40:02:42:ac:11:00:02:08:00 SRC=172.17.0.2 DST=192.168.2.1 LEN=60 TOS=0x00 PREC=0x00 TTL=63 ID=45314 DF PROTO=UDP SPT=45834 DPT=53 LEN=40
In my attached gist, you'll see the full iptables output dump for a "bad" configuration, and according to rule 5 on the filter FOREWARD chain....that's an accept all rule? So I'm very configured why this would be a problem.
In the known working case, here are my traces:
Failed trace output on filter rule 5 Sep 08 06:08:49 baldur kernel: TRACE: filter:DOCKER-ISOLATION-STAGE-1:return:2 IN=docker0 OUT=exbr PHYSIN=veth7880ce1 MAC=02:42:dd:21:67:40:02:42:ac:11:00:02:08:00 SRC=172.17.0.2 DST=192.168.2.1 LEN=60 TOS=0x00 PREC=0x00 TTL=63 ID=45314 DF PROTO=UDP SPT=45834 DPT=53 LEN=40 Sep 08 06:08:49 baldur kernel: TRACE: filter:FORWARD:rule:5 IN=docker0 OUT=exbr PHYSIN=veth7880ce1 MAC=02:42:dd:21:67:40:02:42:ac:11:00:02:08:00 SRC=172.17.0.2 DST=192.168.2.1 LEN=60 TOS=0x00 PREC=0x00 TTL=63 ID=45314 DF PROTO=UDP SPT=45834 DPT=53 LEN=40 Sep 08 06:08:49 baldur kernel: TRACE: mangle:POSTROUTING:policy:1 IN=docker0 OUT=exbr PHYSIN=veth7880ce1 MAC=02:42:dd:21:67:40:02:42:ac:11:00:02:08:00 SRC=172.17.0.2 DST=192.168.2.1 LEN=60 TOS=0x00 PREC=0x00 TTL=63 ID=45314 DF PROTO=UDP SPT=45834 DPT=53 LEN=40
So obviously something about disabling firewalld allows for the packet to pass through the forward chain and hit the mangle POSTROUTING. It's completely unclear to me how to diagnose what might be going on here, any tips appreciated. I have the fully verbose outputs in this gist:
[1] https://gist.github.com/gunnarniels/6826a7ad6f8b800f4c0b68eca98e071e
GN
On Tue, Sep 08, 2020 at 06:35:05AM -0400, Gunnar Niels wrote:
Hello,
I have docker and firewalld both installed on my Arch Linux machine and have been happy with them so far, although I've recently run into an issue where I've broken what seems to be DNS for internal running containers, and I'm trying to isolate the reason for that. Recently I created a bridge with my primary static ip address and added my physical NIC as a slave on the bridge so that I'd be able to run libvirt VMs that appeared to be machines directly on my local network. That has all worked as expected:
# ipb link lo UNKNOWN 00:00:00:00:00:00 <LOOPBACK,UP,LOWER_UP> enp0s25 UP d0:50:99:a5:08:49 <BROADCAST,MULTICAST,UP,LOWER_UP> exbr UP 9a:5f:b2:1e:94:a7 <BROADCAST,MULTICAST,UP,LOWER_UP> virbr0 DOWN 52:54:00:4f:ca:69 <NO-CARRIER,BROADCAST,MULTICAST,UP> virbr0-nic DOWN 52:54:00:4f:ca:69 <BROADCAST,MULTICAST> docker0 DOWN 02:42:dd:21:67:40 <NO-CARRIER,BROADCAST,MULTICAST,UP>
# bridge link show 2: enp0s25: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 master exbr state forwarding priority 32 cost 100 5: virbr0-nic: <BROADCAST,MULTICAST> mtu 1500 master virbr0 state disabled priority 32 cost 100
# ipb a lo UNKNOWN 127.0.0.1/8 ::1/128 enp0s25 UP exbr UP 192.168.2.97/24 fe80::985f:b2ff:fe1e:94a7/64 virbr0 DOWN 192.168.100.1/24 virbr0-nic DOWN docker0 DOWN 172.17.0.1/16 fe80::46e8:567:b3cf:1aa1/64
My problem has suddenly become: My docker containers do not appear to be capable of dns requests unless I *disable* firwalld, something I obviously do not want to do. So, I've taken the approach of turning off the firewalld, restarting docker, and capturing the known good iptables dump, and then tried to compare it to a known bad dump that is produced after I turn on firewalld and restart docker. Interestingly *they are identical* as far as I can tell [1]. Additionally, I tried to TRACE the traffic, and I'm seeing some very confusing info out of the kernel logs:
Looks like you're using a firewalld version that defaults to the nftables backend. Check "nft list ruleset".
When things are configured in the "bad" state, the last log I see is:
Sep 08 06:08:49 baldur kernel: TRACE: filter:FORWARD:rule:5 IN=docker0 OUT=exbrPHYSIN=veth7880ce1 MAC=02:42:dd:21:67:40:02:42:ac:11:00:02:08:00 SRC=172.17.0.2 DST=192.168.2.1 LEN=60 TOS=0x00 PREC=0x00 TTL=63 ID=45314 DF PROTO=UDP SPT=45834 DPT=53 LEN=40
In my attached gist, you'll see the full iptables output dump for a "bad" configuration, and according to rule 5 on the filter FOREWARD chain....that's an accept all rule? So I'm very configured why this would be a problem.
iptables and nftables are _independent_. That means it must pass through both firewalls. If a packets is accepted in iptables it still has to pass through the nftables (firewalld) rules.
In the known working case, here are my traces:
Failed trace output on filter rule 5 Sep 08 06:08:49 baldur kernel: TRACE: filter:DOCKER-ISOLATION-STAGE-1:return:2 IN=docker0 OUT=exbr PHYSIN=veth7880ce1 MAC=02:42:dd:21:67:40:02:42:ac:11:00:02:08:00 SRC=172.17.0.2 DST=192.168.2.1 LEN=60 TOS=0x00 PREC=0x00 TTL=63 ID=45314 DF PROTO=UDP SPT=45834 DPT=53 LEN=40 Sep 08 06:08:49 baldur kernel: TRACE: filter:FORWARD:rule:5 IN=docker0 OUT=exbr PHYSIN=veth7880ce1 MAC=02:42:dd:21:67:40:02:42:ac:11:00:02:08:00 SRC=172.17.0.2 DST=192.168.2.1 LEN=60 TOS=0x00 PREC=0x00 TTL=63 ID=45314 DF PROTO=UDP SPT=45834 DPT=53 LEN=40 Sep 08 06:08:49 baldur kernel: TRACE: mangle:POSTROUTING:policy:1 IN=docker0 OUT=exbr PHYSIN=veth7880ce1 MAC=02:42:dd:21:67:40:02:42:ac:11:00:02:08:00 SRC=172.17.0.2 DST=192.168.2.1 LEN=60 TOS=0x00 PREC=0x00 TTL=63 ID=45314 DF PROTO=UDP SPT=45834 DPT=53 LEN=40
So obviously something about disabling firewalld allows for the packet to pass through the forward chain and hit the mangle POSTROUTING. It's completely unclear to me how to diagnose what might be going on here, any tips appreciated. I have the fully verbose outputs in this gist:
Try adding the docker bridge to the "trusted" firewalld zone.
# firewall-cmd --zone trusted --add-interface docker0
docker (moby) very recently gained integration with firewalld [2].
[2]: https://github.com/moby/libnetwork/pull/2548
[1] https://gist.github.com/gunnarniels/6826a7ad6f8b800f4c0b68eca98e071e
So obviously something about disabling firewalld allows for the packet to pass through the forward chain and hit the mangle POSTROUTING. It's completely unclear to me how to diagnose what might be going on here, any tips appreciated. I have the fully verbose outputs in this gist:
Try adding the docker bridge to the "trusted" firewalld zone.
# firewall-cmd --zone trusted --add-interface docker0docker (moby) very recently gained integration with firewalld [2].
Hey Eric, that seems to have perfectly resolved my problem! I'm confused because it previously worked until I started messing about with my other interfaces, creating a bridge and assigning it master to the physical NIC. I never really touched the docker0 interface, and adding it to trusted suddenly made it start working. My understanding are firewalld zones are INPUT only? Could you elaborate on what traffic was getting blocked prefiously, but by adding the docker0 iface to the trusted zone, I had suddenly whitelisted it?
Thanks again! You're always a huge help.
On Thu, Sep 10, 2020 at 06:06:26AM -0400, Gunnar Niels wrote:
So obviously something about disabling firewalld allows for the packet to pass through the forward chain and hit the mangle POSTROUTING. It's completely unclear to me how to diagnose what might be going on here, any tips appreciated. I have the fully verbose outputs in this gist:
Try adding the docker bridge to the "trusted" firewalld zone.
# firewall-cmd --zone trusted --add-interface docker0docker (moby) very recently gained integration with firewalld [2].
Hey Eric, that seems to have perfectly resolved my problem! I'm confused because it previously worked until I started messing about with my other interfaces, creating a bridge and assigning it master to the physical NIC. I never really touched the docker0 interface, and adding it to trusted suddenly made it start working. My understanding are firewalld zones are INPUT only?
Mostly true. There are exceptions:
- masquerade - forward ports - --set-target also sets the catch-all target for the FORWARD chain. - this is why using the "trusted" zone above works
Could you elaborate on what traffic was getting blocked prefiously, but by adding the docker0 iface to the trusted zone, I had suddenly whitelisted it?
FORWARD traffic. See note above.
firewalld-users@lists.fedorahosted.org
-
Eric Garver -
Gunnar Niels