Allowed source-port is being blocked (AdGuard Home)

[Koen Drai]

Hi,

I am running an AdGuard Home DNS server in my home network (on Debian 11, which is v0.9.3).

Works fine so far, but I am getting the following error messages in syslog:
Mar  6 20:48:33 fook kernel: [160941.334608] "filter_IN_knet_REJECT: "IN=enp2s0 OUT= MAC=<MAC> SRC=5.9.164.112 DST=192.168.1.1 LEN=40 TOS=0x00 PREC=0x00 TTL=53 ID=0 DF PROTO=TCP SPT=853 DPT=46170 WINDOW=0 RES=0x00 RST URGP=0

However, this request should not be rejected according to the configuration:

/etc/firewalld/zones/knet.xml
(...)
   <rule family="ipv4">
     <!-- 853 TCP AdGuard Home-->
     <source-port port="853" protocol="tcp"/>
     <accept/>
   </rule>
(...)

--list-all:
(...)
  rich rules:
(...)
         rule family="ipv4" source-port port="853" protocol="tcp" accept
(...)

Similar behavior for source-port 443 TCP.

Am I missing something?
Is this a known error (and fixed in a later version?)?

Thanks for any hints,

Koen

[Andrei Borzenkov]

On 07.03.2022 01:26, Koen Drai wrote:
> Hi,
>
> I am running an AdGuard Home DNS server in my home network (on Debian 11, which is v0.9.3).
>
> Works fine so far, but I am getting the following error messages in syslog:
> Mar  6 20:48:33 fook kernel: [160941.334608] "filter_IN_knet_REJECT: "IN=enp2s0 OUT= MAC=<MAC> SRC=5.9.164.112 DST=192.168.1.1 LEN=40 TOS=0x00 PREC=0x00 TTL=53 ID=0 DF PROTO=TCP SPT=853 DPT=46170 WINDOW=0 RES=0x00 RST URGP=0

Packet has TCP RST flag.
> However, this request should not be rejected according to the configuration:
>
> /etc/firewalld/zones/knet.xml
> (...)
>
>    <rule family="ipv4">
>      <!-- 853 TCP AdGuard Home-->
>      <source-port port="853" protocol="tcp"/>
>      <accept/>
>    </rule>
> (...)
>
> --list-all:
> (...)
>   rich rules:
> (...)
>          rule family="ipv4" source-port port="853" protocol="tcp" accept
> (...)
>
> Similar behavior for source-port 443 TCP.
>
> Am I missing something?

Firewalld configures netfilter to accept initial packets (SYN flag for
TCP) and related packets (for which connection was established).
Apparently host sends RST without previously established connection. It
may happen. You can check see currently tracked connections in
/proc/net/nf_conntrack.
> Is this a known error (and fixed in a later version?)?

There is no bug here.

[Koen Drai]

Thanks for the sharp eye and explanation!

Is there a way to make firewalld accept the RST flagged connections?

Thanks and regards,

Koen

On 07/03/2022 08:25, Andrei Borzenkov wrote:
> On 07.03.2022 01:26, Koen Drai wrote:
> > Hi,
> >
> > I am running an AdGuard Home DNS server in my home network (on Debian 11, which is v0.9.3).
> >
> > Works fine so far, but I am getting the following error messages in syslog:
> > Mar  6 20:48:33 fook kernel: [160941.334608] "filter_IN_knet_REJECT: "IN=enp2s0 OUT= MAC=<MAC> SRC=5.9.164.112 DST=192.168.1.1 LEN=40 TOS=0x00 PREC=0x00 TTL=53 ID=0 DF PROTO=TCP SPT=853 DPT=46170 WINDOW=0 RES=0x00 RST URGP=0
>
> Packet has TCP RST flag.
> > However, this request should not be rejected according to the configuration:
> >
> > /etc/firewalld/zones/knet.xml
> > (...)
> >     <rule family="ipv4">
> >       <!-- 853 TCP AdGuard Home-->
> >       <source-port port="853" protocol="tcp"/>
> >       <accept/>
> >     </rule>
> > (...)
> >
> > --list-all:
> > (...)
> >    rich rules:
> > (...)
> >           rule family="ipv4" source-port port="853" protocol="tcp" accept
> > (...)
> >
> > Similar behavior for source-port 443 TCP.
> >
> > Am I missing something?
>
> Firewalld configures netfilter to accept initial packets (SYN flag for
> TCP) and related packets (for which connection was established).
> Apparently host sends RST without previously established connection. It
> may happen. You can check see currently tracked connections in
> /proc/net/nf_conntrack.
> > Is this a known error (and fixed in a later version?)?
>
> There is no bug here.
> _______________________________________________
> firewalld-users mailing list -- [e-mail]
> To unsubscribe send an email to [e-mail]
> Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives: https://lists.fedorahosted.org/archives/list/[e-mail]...
> Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure

[Andrei Borzenkov]

On 08.03.2022 00:02, Koen Drai wrote:
> Thanks for the sharp eye and explanation!
>
> Is there a way to make firewalld accept the RST flagged connections?

The only way I can think of is to change backend to iptables and use
direct rules.
> Thanks and regards,
>
> Koen
>
> On 07/03/2022 08:25, Andrei Borzenkov wrote:
> > On 07.03.2022 01:26, Koen Drai wrote:
> > > Hi,
> > >
> > > I am running an AdGuard Home DNS server in my home network (on Debian 11, which is v0.9.3).
> > >
> > > Works fine so far, but I am getting the following error messages in syslog:
> > > Mar  6 20:48:33 fook kernel: [160941.334608] "filter_IN_knet_REJECT: "IN=enp2s0 OUT= MAC=<MAC> SRC=5.9.164.112 DST=192.168.1.1 LEN=40 TOS=0x00 PREC=0x00 TTL=53 ID=0 DF PROTO=TCP SPT=853 DPT=46170 WINDOW=0 RES=0x00 RST URGP=0
> >
> > Packet has TCP RST flag.
> > > However, this request should not be rejected according to the configuration:
> > >
> > > /etc/firewalld/zones/knet.xml
> > > (...)
> > >     <rule family="ipv4">
> > >       <!-- 853 TCP AdGuard Home-->
> > >       <source-port port="853" protocol="tcp"/>
> > >       <accept/>
> > >     </rule>
> > > (...)
> > >
> > > --list-all:
> > > (...)
> > >    rich rules:
> > > (...)
> > >           rule family="ipv4" source-port port="853" protocol="tcp" accept
> > > (...)
> > >
> > > Similar behavior for source-port 443 TCP.
> > >
> > > Am I missing something?
> >
> > Firewalld configures netfilter to accept initial packets (SYN flag for
> > TCP) and related packets (for which connection was established).
> > Apparently host sends RST without previously established connection. It
> > may happen. You can check see currently tracked connections in
> > /proc/net/nf_conntrack.
> > > Is this a known error (and fixed in a later version?)?
> >
> > There is no bug here.
> > _______________________________________________
> > firewalld-users mailing list -- [e-mail]
> > To unsubscribe send an email to [e-mail]
> > Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> > List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> > List Archives: https://lists.fedorahosted.org/archives/list/[e-mail]...
> > Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
>
> firewalld-users mailing list -- [e-mail]
> To unsubscribe send an email to [e-mail]
> Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives: https://lists.fedorahosted.org/archives/list/[e-mail]...
> Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure

[Eric Garver]

On Sun, Mar 06, 2022 at 11:26:38PM +0100, Koen Drai wrote:
> Hi,
>
> I am running an AdGuard Home DNS server in my home network (on Debian 11, which is v0.9.3).
>
> Works fine so far, but I am getting the following error messages in syslog:
> Mar  6 20:48:33 fook kernel: [160941.334608] "filter_IN_knet_REJECT: "IN=enp2s0 OUT= MAC=<MAC> SRC=5.9.164.112 DST=192.168.1.1 LEN=40 TOS=0x00 PREC=0x00 TTL=53 ID=0 DF PROTO=TCP SPT=853 DPT=46170 WINDOW=0 RES=0x00 RST URGP=0

A TCP RESET will be sent if the remote/server does not have a socket
listening on that port. This may indicate the packet passes through the
firewall, but no service is listening.
> However, this request should not be rejected according to the configuration:
>
> /etc/firewalld/zones/knet.xml
> (...)
>
>   <rule family="ipv4">
>     <!-- 853 TCP AdGuard Home-->
>     <source-port port="853" protocol="tcp"/>
>     <accept/>
>   </rule>
> (...)

This is using _source_ port. If this node is running a DNS-over-TLS
service then you want to use `<port ... />` instead.
> --list-all:
> (...)
>  rich rules:
> (...)
>         rule family="ipv4" source-port port="853" protocol="tcp" accept
> (...)
>
> Similar behavior for source-port 443 TCP.
>
> Am I missing something?
> Is this a known error (and fixed in a later version?)?

I think you should check that your application is actually listening.

# ss -n -l |grep 853

[Andrei Borzenkov]

On 08.03.2022 17:31, Eric Garver wrote:
> On Sun, Mar 06, 2022 at 11:26:38PM +0100, Koen Drai wrote:
> > Hi,
> >
> > I am running an AdGuard Home DNS server in my home network (on Debian 11, which is v0.9.3).
> >
> > Works fine so far, but I am getting the following error messages in syslog:
> > Mar  6 20:48:33 fook kernel: [160941.334608] "filter_IN_knet_REJECT: "IN=enp2s0 OUT= MAC=<MAC> SRC=5.9.164.112 DST=192.168.1.1 LEN=40 TOS=0x00 PREC=0x00 TTL=53 ID=0 DF PROTO=TCP SPT=853 DPT=46170 WINDOW=0 RES=0x00 RST URGP=0
>
> A TCP RESET will be sent if the remote/server does not have a socket
> listening on that port. This may indicate the packet passes through the
> firewall, but no service is listening.

In this case it should be sent in response to connection request so it
should be related packet. Why is it rejected then?
> > However, this request should not be rejected according to the configuration:
> >
> > /etc/firewalld/zones/knet.xml
> > (...)
> >
> >   <rule family="ipv4">
> >     <!-- 853 TCP AdGuard Home-->
> >     <source-port port="853" protocol="tcp"/>
> >     <accept/>
> >   </rule>
> > (...)
>
> This is using _source_ port. If this node is running a DNS-over-TLS
> service then you want to use `<port ... />` instead.
> > --list-all:
> > (...)
> >  rich rules:
> > (...)
> >         rule family="ipv4" source-port port="853" protocol="tcp" accept
> > (...)
> >
> > Similar behavior for source-port 443 TCP.
> >
> > Am I missing something?
> > Is this a known error (and fixed in a later version?)?
>
> I think you should check that your application is actually listening.
>
> # ss -n -l |grep 853
> _______________________________________________
> firewalld-users mailing list -- [e-mail]
> To unsubscribe send an email to [e-mail]
> Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives: https://lists.fedorahosted.org/archives/list/[e-mail]...
> Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure

[Eric Garver]

On Tue, Mar 08, 2022 at 05:35:54PM +0300, Andrei Borzenkov wrote:
> On 08.03.2022 17:31, Eric Garver wrote:
> > On Sun, Mar 06, 2022 at 11:26:38PM +0100, Koen Drai wrote:
> > > Hi,
> > >
> > > I am running an AdGuard Home DNS server in my home network (on Debian 11, which is v0.9.3).
> > >
> > > Works fine so far, but I am getting the following error messages in syslog:
> > > Mar  6 20:48:33 fook kernel: [160941.334608] "filter_IN_knet_REJECT: "IN=enp2s0 OUT= MAC=<MAC> SRC=5.9.164.112 DST=192.168.1.1 LEN=40 TOS=0x00 PREC=0x00 TTL=53 ID=0 DF PROTO=TCP SPT=853 DPT=46170 WINDOW=0 RES=0x00 RST URGP=0
> >
> > A TCP RESET will be sent if the remote/server does not have a socket
> > listening on that port. This may indicate the packet passes through the
> > firewall, but no service is listening.
>
> In this case it should be sent in response to connection request so it
> should be related packet. Why is it rejected then?

My mistake. My colleague says it will be established.
> > > However, this request should not be rejected according to the configuration:
> > >
> > > /etc/firewalld/zones/knet.xml
> > > (...)
> > >
> > >   <rule family="ipv4">
> > >     <!-- 853 TCP AdGuard Home-->
> > >     <source-port port="853" protocol="tcp"/>
> > >     <accept/>
> > >   </rule>
> > > (...)
> >
> > This is using _source_ port. If this node is running a DNS-over-TLS
> > service then you want to use `<port ... />` instead.

Can you describe your setup a bit more?
Is the AdGuard Home DNS server the same node running firewalld?
Is the tcpdump also from that machine?

[Koen Drai]

Gents,

Thanks again for your support!

On 08/03/2022 21:32, Eric Garver wrote:
> On Tue, Mar 08, 2022 at 05:35:54PM +0300, Andrei Borzenkov wrote:
> > On 08.03.2022 17:31, Eric Garver wrote:
> > > On Sun, Mar 06, 2022 at 11:26:38PM +0100, Koen Drai wrote:
> > > > Hi,
> > > >
> > > > I am running an AdGuard Home DNS server in my home network (on Debian 11, which is v0.9.3).
> > > >
> > > > Works fine so far, but I am getting the following error messages in syslog:
> > > > Mar  6 20:48:33 fook kernel: [160941.334608] "filter_IN_knet_REJECT: "IN=enp2s0 OUT= MAC=<MAC> SRC=5.9.164.112 DST=192.168.1.1 LEN=40 TOS=0x00 PREC=0x00 TTL=53 ID=0 DF PROTO=TCP SPT=853 DPT=46170 WINDOW=0 RES=0x00 RST URGP=0
> > >
> > > A TCP RESET will be sent if the remote/server does not have a socket
> > > listening on that port. This may indicate the packet passes through the
> > > firewall, but no service is listening.

I get the feeling that it is indeed the case "This may indicate the packet passes through the firewall, but no service is listening."

Got the following response from netstat:
# netstat -v -p -n | grep Ad
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
tcp        0      0 192.168.1.1:55252       5.9.164.112:853         ESTABLISHED 2486/AdGuardHome
tcp        0      0 192.168.1.1:55266       5.9.164.112:853         ESTABLISHED 2486/AdGuardHome

Log shows e.g. this:
Mar  8 21:56:40 fook kernel: [23772.652824] "filter_IN_knet_REJECT: "IN=enp2s0 OUT= MAC=<MAC> SRC=5.9.164.112 DST=192.168.1.1 LEN=40 TOS=0x00 PREC=0x00 TTL=53 ID=0 DF PROTO=TCP SPT=853 DPT=55166 WINDOW=0 RES=0x00 RST URGP=0

Seems as if AdGuard Home is changing ports frequently and/or not informing the queried server about the new port?
> > > > However, this request should not be rejected according to the configuration:
> > > >
> > > > /etc/firewalld/zones/knet.xml
> > > > (...)
> > > >
> > > >    <rule family="ipv4">
> > > >      <!-- 853 TCP AdGuard Home-->
> > > >      <source-port port="853" protocol="tcp"/>
> > > >      <accept/>
> > > >    </rule>
> > > > (...)
> > >
> > > This is using _source_ port. If this node is running a DNS-over-TLS
> > > service then you want to use `<port ... />` instead.
>
> Can you describe your setup a bit more?
> Is the AdGuard Home DNS server the same node running firewalld?
> Is the tcpdump also from that machine?

Yes, both on the same node.
Yes, tcpdump from that machine.

Happy to share more info. However, I think we have found the reason for the REJECT.
Will add a respective filter to logcheck.

[Eric Garver]

On Tue, Mar 08, 2022 at 10:16:46PM +0100, Koen Drai wrote:
> Gents,
>
> Thanks again for your support!
>
> On 08/03/2022 21:32, Eric Garver wrote:
> > On Tue, Mar 08, 2022 at 05:35:54PM +0300, Andrei Borzenkov wrote:
> > > On 08.03.2022 17:31, Eric Garver wrote:
> > > > On Sun, Mar 06, 2022 at 11:26:38PM +0100, Koen Drai wrote:
> > > > > Hi,
> > > > >
> > > > > I am running an AdGuard Home DNS server in my home network (on Debian 11, which is v0.9.3).
> > > > >
> > > > > Works fine so far, but I am getting the following error messages in syslog:
> > > > > Mar  6 20:48:33 fook kernel: [160941.334608] "filter_IN_knet_REJECT: "IN=enp2s0 OUT= MAC=<MAC> SRC=5.9.164.112 DST=192.168.1.1 LEN=40 TOS=0x00 PREC=0x00 TTL=53 ID=0 DF PROTO=TCP SPT=853 DPT=46170 WINDOW=0 RES=0x00 RST URGP=0
> > > >
> > > > A TCP RESET will be sent if the remote/server does not have a socket
> > > > listening on that port. This may indicate the packet passes through the
> > > > firewall, but no service is listening.
>
> I get the feeling that it is indeed the case "This may indicate the packet passes through the firewall, but no service is listening."
>
> Got the following response from netstat:
> # netstat -v -p -n | grep Ad
> Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
> tcp        0      0 192.168.1.1:55252       5.9.164.112:853         ESTABLISHED 2486/AdGuardHome
> tcp        0      0 192.168.1.1:55266       5.9.164.112:853         ESTABLISHED 2486/AdGuardHome
>
> Log shows e.g. this:
> Mar  8 21:56:40 fook kernel: [23772.652824] "filter_IN_knet_REJECT: "IN=enp2s0 OUT= MAC=<MAC> SRC=5.9.164.112 DST=192.168.1.1 LEN=40 TOS=0x00 PREC=0x00 TTL=53 ID=0 DF PROTO=TCP SPT=853 DPT=55166 WINDOW=0 RES=0x00 RST URGP=0

This log indicates the kernel thinks this TCP RST is not part of an
existing connection. I don't know why.
> Seems as if AdGuard Home is changing ports frequently and/or not informing the queried server about the new port?

The sport changing on the client (AdGuard Home) is normal. Each new
connection/request to your upstream DNS-over-TLS server may use a
different source port.
> > > > > However, this request should not be rejected according to the configuration:
> > > > >
> > > > > /etc/firewalld/zones/knet.xml
> > > > > (...)
> > > > >
> > > > >    <rule family="ipv4">
> > > > >      <!-- 853 TCP AdGuard Home-->
> > > > >      <source-port port="853" protocol="tcp"/>
> > > > >      <accept/>
> > > > >    </rule>
> > > > > (...)

I don't think you need this rich rule at all. What is the intent?

Outbound packets (e.g. DoT query) are allowed by default. Return path
(existing connection) packets are allowed by default.

[Andrei Borzenkov]

On 09.03.2022 18:19, Eric Garver wrote:
> > Log shows e.g. this:
> > Mar  8 21:56:40 fook kernel: [23772.652824] "filter_IN_knet_REJECT: "IN=enp2s0 OUT= MAC=<MAC> SRC=5.9.164.112 DST=192.168.1.1 LEN=40 TOS=0x00 PREC=0x00 TTL=53 ID=0 DF PROTO=TCP SPT=853 DPT=55166 WINDOW=0 RES=0x00 RST URGP=0
>
> This log indicates the kernel thinks this TCP RST is not part of an
> existing connection. I don't know why.

As the local address is private, there must be some NAT going on. If
external address is dynamic, this could have been addressed to previous
owner. One needs to see the whole picture in this case.