Hello,
I'm having a bit of an odd issue with firewalld interfering with network routing. Everything here is running CentOS 7; the physical host has kernel version 3.10.0-957.10.1.el7.x86_64. Here is what's going on:
1. I have a physical host, that has several KVM virtual machines. The physical host's eno1 ethernet interface is on my 192.168.2 network. The .2 network is hard wired to other devices and the internet at large via gigabit switch. 2. The physical host also has an interface and network internal to KVM, virbr2, which is 192.168.4 network and used for all of the VM's. 3. I have a VM which acts as a VPN server. It gives out addresses in the 192.168.8 network. 4. For clients in the 192.168.8 network, they can reach servers in the .4 network. Also, servers in the .4 network are able to reach clients with open ports in the .8 network. 5. Clients in the .8 network can NOT reach other devices on the .2 network. Likewise, things on the .2 network can NOT reach anything on the .8 network. The gateway for .8 is properly configured in the physical host as the .4 address of the VPN server. 6. If I turn off firewalld on the physical host, then clients in the .8 network CAN reach things in .2, and vice versa. 6.1. IP v4 forwarding is enabled in both the VPN VM and the physical host. 6.2. Enabling and/or disabling firewalld on the VPN VM does not change any of this behavior. 7. I have tried to put both virbr0 and eth0 in the same network Zone in firewalld- the "trusted" zone. I have also tried to put them in different zones and explicitly configure firewalld. Nothing works.
To make matters more interesting, if I enable logging of dropped packets in firewalld, I get nothing when I attempt to cross the networks... so I can't debug what's going on.
What should my next course of action be?
Thanks!
On Sun, May 19, 2019 at 09:51:03PM -0000, Joshua Kramer wrote:
Hello,
I'm having a bit of an odd issue with firewalld interfering with network routing. Everything here is running CentOS 7; the physical host has kernel version 3.10.0-957.10.1.el7.x86_64. Here is what's going on:
- I have a physical host, that has several KVM virtual machines. The physical host's eno1 ethernet interface is on my 192.168.2 network. The .2 network is hard wired to other devices and the internet at large via gigabit switch.
- The physical host also has an interface and network internal to KVM, virbr2, which is 192.168.4 network and used for all of the VM's.
- I have a VM which acts as a VPN server. It gives out addresses in the 192.168.8 network.
- For clients in the 192.168.8 network, they can reach servers in the .4 network. Also, servers in the .4 network are able to reach clients with open ports in the .8 network.
- Clients in the .8 network can NOT reach other devices on the .2 network. Likewise, things on the .2 network can NOT reach anything on the .8 network. The gateway for .8 is properly configured in the physical host as the .4 address of the VPN server.
- If I turn off firewalld on the physical host, then clients in the .8 network CAN reach things in .2, and vice versa.
6.1. IP v4 forwarding is enabled in both the VPN VM and the physical host. 6.2. Enabling and/or disabling firewalld on the VPN VM does not change any of this behavior. 7. I have tried to put both virbr0 and eth0 in the same network Zone in firewalld- the "trusted" zone. I have also tried to put them in different zones and explicitly configure firewalld. Nothing works.
To make matters more interesting, if I enable logging of dropped packets in firewalld, I get nothing when I attempt to cross the networks... so I can't debug what's going on.
I assume you mean you did this on the host?
# firewalld-cmd --set-log-denied=all
This should log any drops by common catch-all drop/reject rules.
What should my next course of action be?
You can try tcpdump at the various interfaces. That should at minimum let you know where the traffic is getting dropped.
I could never get the --set-log-denied=all to work, so instead I used the "iptables -vnL FORWARD" command. I found that there is a rule that essentially says, "If the destination is 192.168.4.0/24 and the out interface is virbr2, pass it through" as well as "if the source network is 192.168.4.0/24 and the in interface is virbr2 pass it through". However, there were no such rules for the 192.168.8.0/24 network. So I need to duplicate those rules except using 192.168.8.0/24 in place of 192.168.4.0/24.
No big deal, right? I'll just add a Direct rule with the correct parameters. I did that... and the rule is at the BOTTOM of the chain. So the packets never hit that rule because they're dropped farther up the chain. (I did this by using firewall-config GUI, going to Direct Configuration, and entering a Direct rule as "ipv4 / mangle / FORWARD / -1 / -d 192.168.8.0/24 -o virbr2 -j ACCEPT") I thought that setting it to -1 (or even -65535) would put this rule at the top.
How can I get this Direct rule to go to the top?
On Sat, May 25, 2019 at 07:23:56PM -0000, Joshua Kramer wrote:
I could never get the --set-log-denied=all to work, so instead I used the "iptables -vnL FORWARD" command. I found that there is a rule that essentially says, "If the destination is 192.168.4.0/24 and the out interface is virbr2, pass it through" as well as "if the source network is 192.168.4.0/24 and the in interface is virbr2 pass it through".
These rules are added by libvirt. libvirt uses firewalld's direct passthrough interfaces to add them.
However, there were no such rules for the 192.168.8.0/24 network. So I need to duplicate those rules except using 192.168.8.0/24 in place of 192.168.4.0/24.
Probably because libvirt doesn't manage the 192.168.8.0/24 network.
No big deal, right? I'll just add a Direct rule with the correct parameters. I did that... and the rule is at the BOTTOM of the chain. So the packets never hit that rule because they're dropped farther up the chain. (I did this by using firewall-config GUI, going to Direct Configuration, and entering a Direct rule as "ipv4 / mangle / FORWARD / -1 / -d 192.168.8.0/24 -o virbr2 -j ACCEPT")
Did you mean "filter" instead of "mangle"? I don't think mangle makes sense in this scenario.
I thought that setting it to -1 (or even -65535) would put this rule at the top.
How can I get this Direct rule to go to the top?
firewalld creates an empty FORWARD_direct which is jumped to before the rest of firewalld rules. You probably want that.
firewalld creates an empty FORWARD_direct which is jumped to before the rest of firewalld rules. You probably want that.
It doesn't look like this is the case. This is my current FORWARD ruleset:
-P FORWARD ACCEPT -A FORWARD -d 192.168.4.0/24 -o virbr2 -j ACCEPT -A FORWARD -s 192.168.4.0/24 -i virbr2 -j ACCEPT -A FORWARD -i virbr2 -o virbr2 -j ACCEPT -A FORWARD -o virbr2 -j REJECT --reject-with icmp-port-unreachable -A FORWARD -i virbr2 -j REJECT --reject-with icmp-port-unreachable -A FORWARD -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT -A FORWARD -i lo -j ACCEPT -A FORWARD -j FORWARD_direct -A FORWARD -j FORWARD_IN_ZONES_SOURCE -A FORWARD -j FORWARD_IN_ZONES -A FORWARD -j FORWARD_OUT_ZONES_SOURCE -A FORWARD -j FORWARD_OUT_ZONES -A FORWARD -m conntrack --ctstate INVALID -j DROP -A FORWARD -j REJECT --reject-with icmp-host-prohibited
Note that it hits the rule "-A FORWARD -o virbr2 -j REJECT --reject-with icmp-port-unreachable" before it gets to "-A FORWARD -j FORWARD_direct". So the packets will never get to these rules I have in FORWARD_direct:
-N FORWARD_direct -A FORWARD_direct -d 192.168.8.0/24 -o virbr2 -j ACCEPT -A FORWARD_direct -s 192.168.8.0/24 -i virbr2 -j ACCEPT
I wonder if the libvirtd service is adding those REJECT rules that interfere with jumping to FORWARD_direct.
As it turns out, libvirtd DOES in fact create these rules:
https://libvirt.org/firewall.html
...so my next step is to determine the best way to configure libvirtd to add the rules I need.
On Thu, May 30, 2019 at 02:20:56AM -0000, Joshua Kramer wrote:
As it turns out, libvirtd DOES in fact create these rules:
https://libvirt.org/firewall.html
...so my next step is to determine the best way to configure libvirtd to add the rules I need.
You can still use firewalld to add direct rules that occur _before_ the libvirt rules, but you'll have to find a way to trigger the adding of those rules _after_ libvirt adds its rules.
firewalld-users@lists.fedorahosted.org
-
Eric Garver -
Joshua Kramer