IP-routing fails after upgrade F33->F34

[Jouk Jansen]

Hi All,

I'm using one of my Fedora machines as a router between 2 networks. The two
network devices on the machine are called enp0s25 and tun0. On F33 it worked
as expected. However, after an upgrade to F34 It looks like it does not work
anymore.

I tried to give the commands
  firewall-cmd [--permanent]  --direct --add-rule ipv4 filter FORWARD 0 -o enp0s25 -i tun0 -j ACCEPT
  firewall-cmd [--permanent]  --direct --add-rule ipv4 filter FORWARD 0 -i enp0s25 -o tun0 -j ACCEPT
But had no success (not even after restarting firewalld).

"firewall-cmd --list-all" gives the following:
FedoraWorkstation (active)
  target: default
  icmp-block-inversion: no
  interfaces: enp0s25 tun0
  sources:
  services: dhcpv6-client mountd nfs rpc-bind samba-client ssh telnet
  ports: 1025-65535/tcp 1025-65535/udp
  protocols:
  forward: no
  masquerade: yes
  forward-ports:
  source-ports:
  icmp-blocks:
  rich rules:

The strange thing is that "forward" is always "no". (also the masquerade is
always "no" after restarting firewalld, although it is set with --permanent,
but can be set in run-time)

the forwarding variable is defined: 
net.ipv4.conf.all.forwarding = 1

Can someone give me some hints on what I'm missing?

Regards
              Jouk

Pax, vel iniusta, utilior est quam iustissimum bellum.
    (free after Marcus Tullius Cicero (106 b.Chr.-46 b.Chr.)
     Epistularum ad Atticum 7.1.4.3)

Touch not the cat bot a glove
> ------------------------------------------------------------------------------<

Jouk Jansen
    	 
  [e-mail]

Technische Universiteit Delft        tttttttttt  uu     uu  ddddddd
  Kavli Institute of Nanoscience       tttttttttt  uu     uu  dd    dd
  Nationaal centrum voor HREM              tt      uu     uu  dd     dd
  Lorentzweg 1                             tt      uu     uu  dd     dd
  2628 CJ Delft                            tt      uu     uu  dd     dd
  Nederland                                tt      uu     uu  dd    dd
  tel. [phone]                       tt       uuuuuuu   ddddddd
> ------------------------------------------------------------------------------<

[Roberto Ragusa]

On 5/4/21 2:33 PM, Jouk Jansen wrote:
> Hi All,
>
> I'm using one of my Fedora machines as a router between 2 networks. The two
> network devices on the machine are called enp0s25 and tun0. On F33 it worked
> as expected. However, after an upgrade to F34 It looks like it does not work
> anymore.
>
> I tried to give the commands
>    firewall-cmd [--permanent]  --direct --add-rule ipv4 filter FORWARD 0 -o enp0s25 -i tun0 -j ACCEPT
>    firewall-cmd [--permanent]  --direct --add-rule ipv4 filter FORWARD 0 -i enp0s25 -o tun0 -j ACCEPT
> But had no success (not even after restarting firewalld).

Try tcpdump on both interfaces to see what is appearing there.

Then you could have a look at the counters for the related iptables rules to understand if they are being triggered.
(assuming iptables is involved, not sure if firewalld has been switched to nftables backend)

Regards.

[Jouk]

I tried tcpdump only on both devices. I did ping from hrem154.nano.tudelft.nl to 10.9.9.9. this request come in on the enp0s25 device while 10.9.9.9 should go out by the tun0 device. I can see echo requests marked on both devices marked hrem154.nano.tudelft.nl > 10.9.9.9, but nothing in the other direction.
If I ping to 10.9.9.9 on the machine itself I see traffic in 2 directions.

Jouk

[Ed Greshko]

On 04/05/2021 20:33, Jouk Jansen wrote:
> Hi All,
>
> I'm using one of my Fedora machines as a router between 2 networks. The two
> network devices on the machine are called enp0s25 and tun0. On F33 it worked
> as expected. However, after an upgrade to F34 It looks like it does not work
> anymore.
>
> I tried to give the commands
>    firewall-cmd [--permanent]  --direct --add-rule ipv4 filter FORWARD 0 -o enp0s25 -i tun0 -j ACCEPT
>    firewall-cmd [--permanent]  --direct --add-rule ipv4 filter FORWARD 0 -i enp0s25 -o tun0 -j ACCEPT
> But had no success (not even after restarting firewalld).
>
> "firewall-cmd --list-all" gives the following:
> FedoraWorkstation (active)
>    target: default
>    icmp-block-inversion: no
>    interfaces: enp0s25 tun0
>    sources:
>    services: dhcpv6-client mountd nfs rpc-bind samba-client ssh telnet
>    ports: 1025-65535/tcp 1025-65535/udp
>    protocols:
>    forward: no
>    masquerade: yes
>    forward-ports:
>    source-ports:
>    icmp-blocks:
>    rich rules:
>
> The strange thing is that "forward" is always "no". (also the masquerade is
> always "no" after restarting firewalld, although it is set with --permanent,
> but can be set in run-time)
>
> the forwarding variable is defined:
> net.ipv4.conf.all.forwarding = 1
>
> Can someone give me some hints on what I'm missing?

While I don't fully understand your issue, I wonder if the new addition to firewalld may help.

https://firewalld.org/2020/04/intra-zone-forwarding

And, FWIW, firewalld has used nftables since, I think, F32.  You can always check /etc/firewalld.conf
to see what....

FirewallBackend=nftables

is set to.

[Jouk]

Still got a problem when trying to set forwarding on zone FedoraWorkstation, whikle the command on zone home gives success:

[root@foxtrot ~]# firewall-cmd --zone=home --add-forward
success
[root@foxtrot ~]# firewall-cmd --zone=FedoraWorkstation --add-forward
Error: COMMAND_FAILED: 'python-nftables' failed: 
JSON blob:
{"nftables": [{"metainfo": {"json_schema_version": 1}}, {"add": {"rule": {"family": "inet", "table": "firewalld", "chain": "filter_FWDI_FedoraWorkstation_allow", "expr": [{"match": {"left": {"meta": {"key": "oifname"}}, "op": "==", "right": "enp0s25"}}, {"accept": null}]}}}, {"add": {"rule": {"family": "inet", "table": "firewalld", "chain": "filter_FWDI_FedoraWorkstation_allow", "expr": [{"match": {"left": {"meta": {"key": "oifname"}}, "op": "==", "right": "tun0"}}, {"accept": null}]}}}]}

what I want is that the machine act as a router between the outside world (connected to enp0s25) and the local net work 1-0.9.9.x (connected to tun0). so that incomping packages for the 10.9.9.x network on from the outside world reach the machine on the local ndetwork via this machine.

[Ed Greshko]

On 06/05/2021 22:16, Jouk wrote:
> Still got a problem when trying to set forwarding on zone FedoraWorkstation, whikle the command on zone home gives success:
>
> [root@foxtrot ~]# firewall-cmd --zone=home --add-forward
> success
> [root@foxtrot ~]# firewall-cmd --zone=FedoraWorkstation --add-forward
> Error: COMMAND_FAILED: 'python-nftables' failed:
> JSON blob:
> {"nftables": [{"metainfo": {"json_schema_version": 1}}, {"add": {"rule": {"family": "inet", "table": "firewalld", "chain": "filter_FWDI_FedoraWorkstation_allow", "expr": [{"match": {"left": {"meta": {"key": "oifname"}}, "op": "==", "right": "enp0s25"}}, {"accept": null}]}}}, {"add": {"rule": {"family": "inet", "table": "firewalld", "chain": "filter_FWDI_FedoraWorkstation_allow", "expr": [{"match": {"left": {"meta": {"key": "oifname"}}, "op": "==", "right": "tun0"}}, {"accept": null}]}}}]}
>
> what I want is that the machine act as a router between the outside world (connected to enp0s25) and the local net work 1-0.9.9.x (connected to tun0). so that incomping packages for the 10.9.9.x network on from the outside world reach the machine on the local ndetwork via this machine.

In your original post you showed:

FedoraWorkstation (active)
   target: default
   icmp-block-inversion: no
   interfaces: enp0s25 tun0
   sources:

So, both your interfaces are in the FedoraWorkstation zone.  So, it makes no
sense to me to --add-forward to the home zone.  I read it that the
--add-forward is for intra zone forwarding.  With no interfaces in the home
zone there is nothing to forward.

[Joe Zeff]

On 5/6/21 3:38 PM, Ed Greshko wrote:
> I read it that the
> --add-forward is for intra zone forwarding.

Shouldn't that be inter zone (between zones) rather than inter zone 
(inside a zone) here?

[Ed Greshko]

On 07/05/2021 07:41, Joe Zeff wrote:
> On 5/6/21 3:38 PM, Ed Greshko wrote:
> > I read it that the
> > --add-forward is for intra zone forwarding.
>
> Shouldn't that be inter zone (between zones) rather than inter zone (inside a zone) here?

https://firewalld.org/2020/04/intra-zone-forwarding

The original post stated....

FedoraWorkstation (active)
   target: default
   icmp-block-inversion: no
   interfaces: enp0s25 tun0

That would be intra.

[Jouk]

sure you are right. I only added the command for the home zone to show that that one worked, but the same command on the zone  I would like to use , FedoraWorkstation, fails. why?
with the --permanent set is gives success, however after restarting firewalld, the forward seems to be still off.

more or less the same happens with masquerade. I can set it on the running firewall, but when setting it with --permanent, it is lost after restarting firewalld.

[Jouk]

I did a quick check with a fedora 33 machine on which it works. on that machine when I give the firewall-cmd --list-all command the entry "forward: no" is not present at all. So it seems that something changed in respect to forwarding.

[Ed Greshko]

On 07/05/2021 13:57, Jouk wrote:
> I did a quick check with a fedora 33 machine on which it works. on that machine when I give the firewall-cmd --list-all command the entry "forward: no" is not present at all. So it seems that something changed in respect to forwarding.

F34 has --add-forward while f33 does not is due to the upgrade from firewalld-0.8.6-1.fc33 to firewalld-0.9.3-2.fc34.

firewalld-0.9.X introduced the new option.

[Ed Greshko]

On 07/05/2021 13:54, Jouk wrote:
> sure you are right. I only added the command for the home zone to show that that one worked, but the same command on the zone  I would like to use , FedoraWorkstation, fails. why?
> with the --permanent set is gives success, however after restarting firewalld, the forward seems to be still off.
>
> more or less the same happens with masquerade. I can set it on the running firewall, but when setting it with --permanent, it is lost after restarting firewalld.

Unfortunately, I don't think I can model your configuration in a VM.

However, when I add 2 interfaces to a VM I get....

[root@fedora ~]# firewall-cmd --zone=FedoraWorkstation --add-forward
success

Then....

[root@fedora ~]# firewall-cmd --list-all
FedoraWorkstation (active)
   target: default
   icmp-block-inversion: no
   interfaces: enp0s3 enp0s8
   sources:
   services: dhcpv6-client mdns samba-client ssh
   ports: 1025-65535/udp 1025-65535/tcp
   protocols:
   forward: yes
   masquerade: no
   forward-ports:
   source-ports:
   icmp-blocks:
   rich rules:

and....

[root@fedora ~]# firewall-cmd --runtime-to-permanent
success

[root@fedora ~]# systemctl restart firewalld

[root@fedora ~]# firewall-cmd --list-all
FedoraWorkstation (active)
   target: default
   icmp-block-inversion: no
   interfaces: enp0s3 enp0s8
   sources:
   services: dhcpv6-client mdns samba-client ssh
   ports: 1025-65535/udp 1025-65535/tcp
   protocols:
   forward: yes
   masquerade: no
   forward-ports:
   source-ports:
   icmp-blocks:
   rich rules:

You may want to join the [e-mail] list and ask there.
I've gotten good guidance from the folks there.

[Jouk]

OK, but my problem is that I'm not able to get the forward working with the command I gave earlier in this thread.

[Ed Greshko]

On 07/05/2021 16:50, Jouk wrote:
> OK, but my problem is that I'm not able to get the forward working with the command I gave earlier in this thread.

From the web page I see, but don't quite understand, this....

Caveats
When enabled in the default zone, intra zone forwarding can only be applied to the interfaces and sources that have been explicitly added to the current default zone. It can not use a catch-all for all outgoing interfaces as this would allow packets to forward to an interface or source assigned to a different zone.

It *may* be saying that if forwarding is enabled in the default zone (which is fedoraworkstation) it can't be enabled in another
zone.  So, maybe try --remove-forward from home and then adding it to fedoraworkstation?

[Jouk]

Switched firewalld to iptables and that solved the problem for now.

according to what I found here:
     https://lists.fedorahosted.org/archives/list/[e-mail]...
     https://firewalld.org/2020/09/policy-objects-introduction
I have to do something with policy objects if I want to use nftables. I'm going to investigate that route.

Thanks to all who gave me suggestions in this thread
                            Jouk

[Tim Evans]

On 5/4/21 8:33 AM, Jouk Jansen wrote:
> Hi All,
>
> I'm using one of my Fedora machines as a router between 2 networks. The two
> network devices on the machine are called enp0s25 and tun0. On F33 it worked
> as expected. However, after an upgrade to F34 It looks like it does not work
> anymore.

Jouk, when you say "upgrade to F34," by what means did you do the 
upgrade?  Specifically, did you:

# dnf system-upgrade download --releasever=34
# dnf system-upgrade reboot

Or did you use some other method?

[Jouk]

yes this method. However, I always have to add --allowerasing.

[Tim Evans]

On 5/4/21 8:33 AM, Jouk Jansen wrote:
> Hi All,
>
> I'm using one of my Fedora machines as a router between 2 networks. The two
> network devices on the machine are called enp0s25 and tun0. On F33 it worked
> as expected. However, after an upgrade to F34 It looks like it does not work
> anymore.

Jouk, have you resolved this?  Anyone else seen it?

Wanting to upgrade my F33 router/NAT/firewall system.

Thanks.

[Tim Evans]

On 5/29/21 12:54 PM, Tim Evans wrote:
> On 5/4/21 8:33 AM, Jouk Jansen wrote:
> > Hi All,
> >
> > I'm using one of my Fedora machines as a router between 2 networks. 
> > The two
> > network devices on the machine are called enp0s25 and tun0. On F33 it 
> > worked
> > as expected. However, after an upgrade to F34 It looks like it does 
> > not work
> > anymore.
>
> Jouk, have you resolved this?  Anyone else seen it?
>
> Wanting to upgrade my F33 router/NAT/firewall system.

This thread having fallen silent, I decided to go ahead and try my own 
upgrade today.

I think my network setup is less complex than the OP's, with just one 
internal and one external network, so my experience may or may not 
address the OP's situation. My router/firewall/NAT machine sits between 
internal and external networks.

[root@kestrel ~]# firewall-cmd --list-all --zone=external
external (active)
   target: default
   icmp-block-inversion: no
   interfaces: enp3s1
   sources:
   services: ssh
   ports:
   protocols:
   forward: no
   masquerade: yes
   forward-ports:
   source-ports:
   icmp-blocks:
   rich rules:

[root@kestrel ~]# firewall-cmd --list-all --zone=internal
internal (active)
   target: default
   icmp-block-inversion: no
   interfaces: enp2s0
   sources:
   services: dhcpv6-client mdns samba-client ssh
   ports:
   protocols:
   forward: no
   masquerade: no
   forward-ports:
   source-ports:
   icmp-blocks:
   rich rules:

[root@kestrel ~]# cat /proc/sys/net/ipv4/ip_forward
1

That having been said, my in-place upgrade went without incident, and 
routing is working the same afterwards as it was before.