https://fedoraproject.org/wiki/Changes/systemd-resolved
== Summary ==
Enable systemd-resolved by default. glibc will perform name resolution using nss-resolve rather than nss-dns.
== Owner == * Name: [[User:catanzaro| Michael Catanzaro]] * Email: mcatanzaro@redhat.com
== Detailed Description ==
We will enable systemd-resolved by default.
# We will change the [https://src.fedoraproject.org/rpms/fedora-release/blob/f4cc5b6ce095bb4233e5e... fedora-release presets] to enable systemd-resolved instead of disable it. # systemd-libs currently has [https://src.fedoraproject.org/rpms/systemd/blob/bb79fb73875f8e71841a1ee8ede5... a %post scriplet] to enable nss-myhostname and nss-systemd by either (a) modifying authselect's user-nsswitch.conf template, if authselect is in use, or (b) directly modifying /etc/nsswitch.conf otherwise. We will work with the systemd maintainers to enable nss-resolve here as well. # We will work with the authselect maintainers to update authselect's minimal and nis profiles to enforce nss-resolve. These profiles modify the hosts line of /etc/resolv.conf. (By default, Fedora uses authselect's sssd profile, which does not modify the hosts line and therefore does not have this problem.) # We will remove our downstream patch to systemd that prevents systemd from symlinking /etc/resolv.conf to /run/systemd/resolve/stub-resolv.conf on new installs. For system upgrades, a systemd-libs %post scriptlet will be used to reassign the symlink during upgrade. Upon detecting this symlink, NetworkManager will automatically enable its systemd-resolved support and configure split DNS as appropriate.
systemd-resolved has been enabled by default in Ubuntu since Ubuntu 16.10, but please note we are doing this differently than Ubuntu has. Ubuntu does not use nss-resolve. Instead, Ubuntu uses the traditional nss-dns provided by glibc upstream, so glibc on Ubuntu continues to read /etc/resolv.conf, as is traditional. This extra step is not useful and not recommended by upstream. We want to follow upstream recommendations in using nss-resolve instead.
If you do not wish to use systemd-resolved, then manual intervention will be required to disable it:
* Modify /etc/authselect/user-nsswitch.conf and remove `resolve [!UNAVAIL=return]` from the hosts line. Run `authselect apply-changes`. (If you have disabled authselect, then edit /etc/nsswitch.conf directly.) * Disable and stop systemd-resolved.service. * Restart the NetworkManager service. NetworkManager will then create a traditional /etc/resolv.conf. (If you are not using NetworkManager, you may create your own /etc/resolv.conf.)
== Benefit to Fedora ==
=== Standardization ===
Fedora will continue its history of enabling new systemd-provided services whenever it makes sense to do so. Standardizing on upstream systemd services is beneficial to the broader Linux ecosystem in addition to Fedora, since standardizing reduces behavior differences between different Linux distributions. Sadly, Fedora is no longer leading in this area. Ubuntu has enabled systemd-resolved by default since Ubuntu 16.10, so by the time Fedora 33 is released, we will be three years behind Ubuntu here.
=== resolvectl ===
`resolvectl` has several useful functions (e.g. `resolvectl status` or `resolvectl query`) that will be enabled out-of-the-box.
=== Caching ===
systemd-resolved caches DNS queries for a short while. This can [https://gitlab.gnome.org/GNOME/glib/-/merge_requests/682#note_441846 dramatically] improve performance for applications that do not already manually cache their own DNS results. (Generally, only web browsers bother with manually caching DNS results.)
=== Split DNS ===
When systemd-resolved is enabled, users who use multiple VPNs at the same time will notice that DNS requests are now sent to the correct DNS server by default. Previously, this scenario would result in embarrassing "DNS leaks" and, depending on the order that the VPN connections were established, possible failure to resolve private resources. These scenarios will now work properly. For example, consider the scenario of Distrustful Denise, who (quite reasonably) does not trust her ISP. Denise uses a public VPN service, public-vpn.example.com, to hide her internet activity from her ISP, but she also needs to use her employer's corporate VPN, corporation.example.com, in order to access internal company resources while working from home. Using the Network panel in System Settings, Denise has configured her employer's VPN to "use this connection only for resources on its network." Distrustful Denise expects that her employer's VPN will receive all traffic for corporation.example.com, and no other traffic. And while this mostly works in Fedora 32, she discovers that it does not work properly for DNS:
* If Denise connects to public-vpn.example.com first and corporation.example.com second, she is unable to access internal company resources. All DNS requests are sent to public-vpn.example.com's DNS server, so she is unable to resolve names for internal company websites. * If Denise connects to corporation.example.com first and public-vpn.example.com second, then she is able to access internal company resources. However, it only works because ''all'' her DNS requests are sent to corporation.example.com's DNS server. Sadly for Distrustful Denise, her employer discovers that she has been making some embarrassing DNS requests that she had expected to go through public-vpn.example.com instead.
Whichever VPN Denise connects to first receives all DNS requests because glibc's nss-dns module is not smart enough to split the requests. /etc/resolv.conf is just a list of DNS servers to try in order, and NetworkManager has no plausible way to decide which to list first, because both ways are broken, so it just prefers whichever was connected first. And if one server fails to respond, then the next server in the list will be tried, likely resulting in a DNS leak. In contrast, when systemd-resolved is enabled, it will send each DNS request only to the correct DNS server. The DNS server that will be used for each tun interface can be inspected using the resolvectl tool.
Migrating away from /etc/resolv.conf will also avoid an annoying footgun with this legacy file: only the first three listed nameservers are respected. All further nameservers are silently ignored. NetworkManager adds a warning comment when writing more than three nameservers to this file, but it cannot do any better than that.
=== DNS over TLS ===
systemd-resolved supports DNS over TLS (different from DNS over HTTPS). Although this feature will not initially be enabled by default, using systemd-resolved will enable us to turn on DNS over TLS in a future Fedora release, providing improved security if the user's DNS server supports DNS over TLS.
== Scope == * Proposal owners: We will update Fedora presets to enable systemd-resolved by default.
* Other developers: This change requires coordination with the systemd and authselect maintainers.
* Release engineering: [https://pagure.io/releng/issue/9367 #9367]
* Policies and guidelines: none required
* Trademark approval: N/A (not needed for this Change)
== Upgrade/compatibility impact ==
systemd-resolved will be enabled automatically when upgrading to Fedora 33. After upgrade, /etc/resolv.conf will be managed by systemd and symlinked to /run/systemd/resolve/stub-resolv.conf. '''glibc will no longer look at /etc/resolv.conf when performing name resolution.''' Instead, glibc will communicate directly with systemd-resolved via nss-resolve. systemd adds a large warning comment to the top of /etc/resolv.conf to warn system administrators that changes to this file will be ignored; however, scripts that edit this file manually will break. Because this file is usually managed by NetworkManager, impact to Fedora users will be limited to users who have manually disabled NetworkManager; such users are expected to be experienced system administrators who should be comfortable adapting to the change (or disabling systemd-resolved).
Any applications that bypass glibc and read /etc/resolv.conf directly will still work because /etc/resolv.conf will point to systemd-resolved's stub resolver running on 127.0.0.53. Nevertheless, /etc/resolv.conf is provided only for compatibility purposes, and applications should prefer to use either glibc or the systemd-resolved D-Bus API instead; see systemd-resolved(8) for details.
In short, '''applications that read /etc/resolv.conf will continue to work as before.''' Applications that write to it will no longer work as expected, but this only previously worked if NetworkManager is disabled, a non-default configuration. It remains possible to disable systemd-resolved if desired. Otherwise, any custom system administration scripts that manage /etc/resolv.conf will need to be updated.
=== DNSSEC ===
systemd-resolved's DNSSEC support is known to cause compatibility problems with certain network access points. Per recommendation from the systemd developers, we will change the default value of this setting in Fedora from the upstream default `DNSSEC=allow-downgrade` to `DNSSEC=no` by building systemd with the build option `-Ddefault-dnssec=no`. The upstream default attempts to use DNSSEC if it is working, but automatically disable it otherwise, allowing man-in-the-middle attackers to disable DNSSEC. Sadly, even the allow-downgrade setting suffers known compatibility problems. Because Fedora is not prepared to handle an influx of DNSSEC-related bug reports, we will disable this feature altogether. We anticipate that enabling DNSSEC by default will not be possible in the foreseeable future, or perhaps ever. Instead, enabling DNS over TLS (when supported by the DNS server) seems likely in the near future.
=== Multicast DNS ===
systemd-resolved's multicast DNS support conflicts with Avahi. Per recommendation from the systemd developers, we will change the default value of this setting in Fedora from the upstream default `MulticastDNS=yes` to `MulticastDNS=resolve`. Multicast DNS resolving will be enabled, but responding will be disabled. This will require adding a new systemd build option to control the default value of the MulticastDNS setting, similar to the existing `default-dnssec` and `default-dns-over-tls` build options.
== How To Test ==
Load any website in a web browser. If you succeed, then name resolution probably works.
Try using `resolvectl status` and, for example, `resolvectl query fedoraproject.org`, to see how they work and sanity-check their output.
Users who use multiple VPNs at the same time are encouraged to test DNS in a multiple VPN scenario, to ensure that DNS requests are sent to the expected DNS server.
== User Experience ==
See the Benefit to Fedora section, above, for direct benefits to users who use multiple VPNs at the same time.
Users will be able to use the resolvectl tool and the functionality it provides.
/etc/resolv.conf will now be managed by systemd rather than by NetworkManager. As before, this file must not be modified directly when it is managed.
== Dependencies ==
In Fedora, /etc/nsswitch.conf is managed by authselect. By default, authselect uses the sssd profile to generate configuration compatible with sssd. In this mode of operation, it does not modify the hosts line in /etc/nsswitch.conf. This is also true if using the winbind profile instead of the sssd profile. However, authselect's minimal and nis profiles do modify the hosts line. These authselect profiles must be updated to enable nss-resolved. If you are using authselect in one of these modes, it will not be possible to cleanly disable systemd-resolved because the hosts line in /etc/nsswitch.conf will be clobbered whenever 'authselect apply-changes' is run. If you wish to disable systemd-resolved and you are using authselect in one of these modes, then you should stop using authselect. This is not expected to cause many problems because virtually all Fedora users will be using the default sssd profile.
We do not need to directly make any changes to the /etc/nsswitch.conf shipped by glibc. Changes will be applied in the systemd-libs %post scriptlet.
== Contingency Plan ==
The contingency plan, in the unlikely event of unexpected problems, is simply to revert any changes and not enable systemd-resolved.
* Contingency deadline: beta freeze * Blocks release? No * Blocks product? No
== Documentation ==
* systemd-resolved is documented in several manpages: resolvectl(1), resolved.conf(5), nss-resolve(8), systemd-resolved(8). * [https://wiki.archlinux.org/index.php/Systemd-resolved Arch Wiki documentation] * [https://wiki.gnome.org/Projects/NetworkManager/DNS NetworkManager DNS documentation]
== Release Notes ==
systemd-resolved is now enabled by default. systemd-resolved provides a system-level DNS cache that can substantially improve performance for applications that do not cache their own DNS results, allows correct handling of split DNS scenarios such as when multiple VPNs are in use, and will allow Fedora to enable DNS over TLS in the future. /etc/resolv.conf will now be managed by systemd-resolved rather than by NetworkManager. /etc/resolv.conf will no longer be read when performing name resolution using glibc; however, it is still provided for compatibility with applications that manually read this file to perform name resolution. Writing to /etc/resolv.conf will no longer work as expected.
On 4/14/20 2:23 PM, Ben Cotton wrote:
=== DNS over TLS ===
systemd-resolved supports DNS over TLS (different from DNS over HTTPS). Although this feature will not initially be enabled by default, using systemd-resolved will enable us to turn on DNS over TLS in a future Fedora release, providing improved security if the user's DNS server supports DNS over TLS.
Why wait?
This is something I've been interested in and was interested in implementing in Fedora.
On Tue, Apr 14, 2020 at 2:33 pm, Michael Cronenworth mike@cchtml.com wrote:
Why wait?
This is something I've been interested in and was interested in implementing in Fedora.
Caution mainly, so that we only make one major change at a time instead of two. The goal is to do this without generating too many new bug reports for the systemd developers all at the same time. My thinking was that if this change goes smoothly in F33, then it should be possible to enable DNS over TLS by default in F34.
That said, there are not currently any known compatibility problems with the DNS over TLS support as far as I know, so I would *expect* it to go smoothly regardless.
Of course, once systemd-resolved is enabled, then enabling or disabling DNS over TLS will be a one-line config file change in /etc/systemd/resolved.conf. :)
On Tue, Apr 14, 2020 at 02:40:08PM -0500, Michael Catanzaro wrote:
On Tue, Apr 14, 2020 at 2:33 pm, Michael Cronenworth mike@cchtml.com wrote:
Why wait?
This is something I've been interested in and was interested in implementing in Fedora.
Caution mainly, so that we only make one major change at a time instead of two. The goal is to do this without generating too many new bug reports for the systemd developers all at the same time. My thinking was that if this change goes smoothly in F33, then it should be possible to enable DNS over TLS by default in F34.
Can you expand on what that means?
Does it mean:
a) systemd-resolved will use DNS over TLS if it detects that the nameservers it is querying can do so (ie, it would do a query to port 853 of the nameservers dhcp or static config gave it)
b) systemd-resolved will use DNS over TLS and always use some 'well known' public dns servers for queries, ignoring locally configured servers.
I'm very much in favor of a, but not in favor of b. :)
That said, there are not currently any known compatibility problems with the DNS over TLS support as far as I know, so I would *expect* it to go smoothly regardless.
Of course, once systemd-resolved is enabled, then enabling or disabling DNS over TLS will be a one-line config file change in /etc/systemd/resolved.conf. :)
Is that going to be to set it to 'opportunistic' or 'true' ?
kevin
On Tue, Apr 14, 2020 at 12:57 pm, Kevin Fenzi kevin@scrye.com wrote:
Can you expand on what that means?
Does it mean:
a) systemd-resolved will use DNS over TLS if it detects that the nameservers it is querying can do so (ie, it would do a query to port 853 of the nameservers dhcp or static config gave it)
b) systemd-resolved will use DNS over TLS and always use some 'well known' public dns servers for queries, ignoring locally configured servers.
I'm very much in favor of a, but not in favor of b. :)
It would do (a). (But as part of a future change, not part of this change.)
I think (b) would be too controversial for Fedora.
That said, there are not currently any known compatibility problems with the DNS over TLS support as far as I know, so I would *expect* it to go smoothly regardless.
Of course, once systemd-resolved is enabled, then enabling or disabling DNS over TLS will be a one-line config file change in /etc/systemd/resolved.conf. :)
Is that going to be to set it to 'opportunistic' or 'true' ?
It would be "opportunistic". (But again, that would be a future change, not this change.)
On Di, 14.04.20 12:57, Kevin Fenzi (kevin@scrye.com) wrote:
Can you expand on what that means?
Does it mean:
a) systemd-resolved will use DNS over TLS if it detects that the nameservers it is querying can do so (ie, it would do a query to port 853 of the nameservers dhcp or static config gave it)
b) systemd-resolved will use DNS over TLS and always use some 'well known' public dns servers for queries, ignoring locally configured servers.
Nah. We will only talk to configured DNS servers. If no DNS servers are configured at all we'll try to use a default set of DNS servers however, which can be specified when building systemd. it's a fallback to make things more robust, i.e. making sure DNS works if possible.
Lennart
-- Lennart Poettering, Berlin
On Wednesday, April 15, 2020 6:34:56 AM MST Lennart Poettering wrote:
On Di, 14.04.20 12:57, Kevin Fenzi (kevin@scrye.com) wrote:
Can you expand on what that means?
Does it mean:
a) systemd-resolved will use DNS over TLS if it detects that the nameservers it is querying can do so (ie, it would do a query to port 853 of the nameservers dhcp or static config gave it)
b) systemd-resolved will use DNS over TLS and always use some 'well known' public dns servers for queries, ignoring locally configured servers.
Nah. We will only talk to configured DNS servers. If no DNS servers are configured at all we'll try to use a default set of DNS servers however, which can be specified when building systemd. it's a fallback to make things more robust, i.e. making sure DNS works if possible.
Lennart
-- Lennart Poettering, Berlin
If there are no servers configured... Shouldn't it use no servers?
On Thu, Apr 16, 2020 at 07:27:29PM +0200, Lennart Poettering wrote:
If there are no servers configured... Shouldn't it use no servers?
Well, our assumption is that working DNS is better than DNS that doesn't work.
I hope no one is using lack of configured DNS as a security measure! But I can see a case for wanting lack of configuration to be a failure rather than working but non-obviously not in the way you expected.
On Do, 16.04.20 14:07, Matthew Miller (mattdm@fedoraproject.org) wrote:
On Thu, Apr 16, 2020 at 07:27:29PM +0200, Lennart Poettering wrote:
If there are no servers configured... Shouldn't it use no servers?
Well, our assumption is that working DNS is better than DNS that doesn't work.
I hope no one is using lack of configured DNS as a security measure! But I can see a case for wanting lack of configuration to be a failure rather than working but non-obviously not in the way you expected.
Use FallbackDNS= to override the built-in defaults, and set it to the empty string if you want lookups to fail.
Lennart
-- Lennart Poettering, Berlin
* Matthew Miller:
On Thu, Apr 16, 2020 at 07:27:29PM +0200, Lennart Poettering wrote:
If there are no servers configured... Shouldn't it use no servers?
Well, our assumption is that working DNS is better than DNS that doesn't work.
I hope no one is using lack of configured DNS as a security measure! But I can see a case for wanting lack of configuration to be a failure rather than working but non-obviously not in the way you expected.
At the lowest level, lack of configuration (i.e., no name servers in /etc/resolv.conf or no such file at all) currently means that 127.0.0.1 is used as the DNS resolver. This has been the case for such a long time that I think it would not be a good idea to configure different servers instead.
Thanks, Florian
Once upon a time, Lennart Poettering mzerqung@0pointer.de said:
On Do, 16.04.20 07:45, John M. Harris Jr (johnmh@splentity.com) wrote:
If there are no servers configured... Shouldn't it use no servers?
Well, our assumption is that working DNS is better than DNS that doesn't work.
Talking to servers the admin didn't configure could be an information leak and IMHO should not be on by default. You have no idea why there are no servers, so trying to second-guess it is a bad idea.
DNS over TLS is offered already by package called stubby. But DNS over TLS does not bring you more privacy usually. It only allows moving (some) queries away from your ISP to somewhere else, but always someone can read them.
On 4/14/20 9:33 PM, Michael Cronenworth wrote:
On 4/14/20 2:23 PM, Ben Cotton wrote:
=== DNS over TLS ===
systemd-resolved supports DNS over TLS (different from DNS over HTTPS). Although this feature will not initially be enabled by default, using systemd-resolved will enable us to turn on DNS over TLS in a future Fedora release, providing improved security if the user's DNS server supports DNS over TLS.
Why wait?
This is something I've been interested in and was interested in implementing in Fedora.
On Thu, Oct 1, 2020 at 11:45 AM Vitaly Zaitsev via devel devel@lists.fedoraproject.org wrote:
On 01.10.2020 16:54, Petr Menšík wrote:
But DNS over TLS does not bring you more privacy usually.
It does. DoT and DoH encrypt all DNS traffic. Your ISP can no longer spy on you and sell the collected data to third parties.
They also completely break most VPNs and corporate network setups, so...
On Thu, Oct 1, 2020 at 3:48 PM Neal Gompa ngompa13@gmail.com wrote:
On Thu, Oct 1, 2020 at 11:45 AM Vitaly Zaitsev via devel devel@lists.fedoraproject.org wrote:
On 01.10.2020 16:54, Petr Menšík wrote:
But DNS over TLS does not bring you more privacy usually.
It does. DoT and DoH encrypt all DNS traffic. Your ISP can no longer spy on you and sell the collected data to third parties.
They also completely break most VPNs and corporate network setups, so...
Well, as much else, whether it breaks a VPN or corporate network, or even a countries attempt to use DNS for blocking purposes depends on the details of implementation, and what tools are available at the various levels to influence the resolver behaviours to implement whatever the entities are trying to accomplish.
It should come to no real surprise to those on this list that newer tech can certainly require existing practices to be forced to adopt, nor that those that want things to stay the same need to get used to disappointment.
Am 01.10.20 um 19:45 schrieb Vitaly Zaitsev via devel:
On 01.10.2020 17:46, Neal Gompa wrote:
They also completely break most VPNs and corporate network setups, so...
It will not, because Fedora will use opportunistic mode by default.
DoT won't break anything, it's blockable by an admin in the firewall, unlike DoH, which uses port 443.
Marius
Am 01.10.20 um 17:44 schrieb Vitaly Zaitsev via devel:
On 01.10.2020 16:54, Petr Menšík wrote:
But DNS over TLS does not bring you more privacy usually.
It does. DoT and DoH encrypt all DNS traffic. Your ISP can no longer spy on you and sell the collected data to third parties.
... if you changed the dns servers manually to other != isp caches. ;)
Marius
On 10/1/20 5:44 PM, Vitaly Zaitsev via devel wrote:
On 01.10.2020 16:54, Petr Menšík wrote:
But DNS over TLS does not bring you more privacy usually.
It does. DoT and DoH encrypt all DNS traffic. Your ISP can no longer spy on you and sell the collected data to third parties.
No, it doesn't. Unless you operate the remote resolver as well, it moves ability to see your name lookups from your ISP to DoT or DoH provider. They can see it instead. It makes you able to choose different receiver, but it does not 'protect' it from other parties in general.
Now your DoH provider can sell the collected data to third parties. Living in the EU, I expect GDPR protects me better than DoH in the US. My ISP needs legal consent to sell such information.
Cheers, Petr
On Tue, 2020-04-14 at 15:23 -0400, Ben Cotton wrote:
=== Caching ===
systemd-resolved caches DNS queries for a short while. This can [https://gitlab.gnome.org/GNOME/glib/-/merge_requests/682#note_441846 dramatically] improve performance for applications that do not already manually cache their own DNS results. (Generally, only web browsers bother with manually caching DNS results.)
=== Split DNS ===
Doesn't NetworkManager already broadly address both of these on all installations where it's used (which is all Fedora installs by default)?
On Tue, Apr 14, 2020 at 12:45 pm, Adam Williamson adamwill@fedoraproject.org wrote:
Doesn't NetworkManager already broadly address both of these on all installations where it's used (which is all Fedora installs by default)?
I don't think so, no.
As far as I know, NetworkManager does not have a DNS cache. The only way to implement one systemwide would be to write a glibc NSS plugin. Otherwise, how would glibc be able to talk to NetworkManager to use the cached results?
Then the description of multi-VPN scenario is written based on the status quo with NetworkManager already installed and enabled. NetworkManager has three DNS backends: default (nss-dns, what we use currently), dnsmasq, and systemd-resolved. The default backend just does the wrong thing and cannot be fixed. When either dnsmasq or systemd-resolved is in use, NetworkManager will go ahead and do the right thing by telling dnsmasq/systemd-resolved which network interfaces should be used to resolve which hostnames. I consulted with the NetworkManager developers and they recommended systemd-resolved over dnsmasq, although I understand that dnsmasq is good too.
Michael
On Tue, 2020-04-14 at 16:18 -0500, Michael Catanzaro wrote:
On Tue, Apr 14, 2020 at 12:45 pm, Adam Williamson adamwill@fedoraproject.org wrote:
Doesn't NetworkManager already broadly address both of these on all installations where it's used (which is all Fedora installs by default)?
I don't think so, no.
As far as I know, NetworkManager does not have a DNS cache. The only way to implement one systemwide would be to write a glibc NSS plugin. Otherwise, how would glibc be able to talk to NetworkManager to use the cached results?
Then the description of multi-VPN scenario is written based on the status quo with NetworkManager already installed and enabled. NetworkManager has three DNS backends: default (nss-dns, what we use currently), dnsmasq, and systemd-resolved. The default backend just does the wrong thing and cannot be fixed. When either dnsmasq or systemd-resolved is in use, NetworkManager will go ahead and do the right thing by telling dnsmasq/systemd-resolved which network interfaces should be used to resolve which hostnames. I consulted with the NetworkManager developers and they recommended systemd-resolved over dnsmasq, although I understand that dnsmasq is good too.
I thought we'd made the dnsmasq config default at some point (that implements both caching and split DNS). I guess I was remembering wrong.
On Tue, 14 Apr 2020 16:18:02 -0500 Michael Catanzaro mcatanzaro@gnome.org wrote:
NetworkManager has three DNS backends: default (nss-dns, what we use currently), dnsmasq, and systemd-resolved. The default backend just does the wrong thing and cannot be fixed. When either dnsmasq or systemd-resolved is in use, NetworkManager will go ahead and do the right thing by telling dnsmasq/systemd-resolved which network interfaces should be used to resolve which hostnames. I consulted with the NetworkManager developers and they recommended systemd-resolved over dnsmasq, although I understand that dnsmasq is good too.
Will the ability to turn off NetworkManager involvement in DNS in the configuration file (None) still remain? I use a local caching DNS server, and had to do that in order to allow it to run without interference / override by NetworkManager.
On Tue, 14 Apr 2020 15:52:55 -0700 stan via devel devel@lists.fedoraproject.org wrote:
On Tue, 14 Apr 2020 16:18:02 -0500 Michael Catanzaro mcatanzaro@gnome.org wrote:
NetworkManager has three DNS backends: default (nss-dns, what we use currently), dnsmasq, and systemd-resolved. The default backend just does the wrong thing and cannot be fixed. When either dnsmasq or systemd-resolved is in use, NetworkManager will go ahead and do the right thing by telling dnsmasq/systemd-resolved which network interfaces should be used to resolve which hostnames. I consulted with the NetworkManager developers and they recommended systemd-resolved over dnsmasq, although I understand that dnsmasq is good too.
Will the ability to turn off NetworkManager involvement in DNS in the configuration file (None) still remain? I use a local caching DNS server, and had to do that in order to allow it to run without interference / override by NetworkManager.
Just a further note. I tried both dnsmasq and systemd-resolved, but neither seemed to work. I still saw my browser saying Looking up blah-blah.com ... and timing for seconds even if I just visited the page a few minutes before. Once I set up my own caching DNS server, that went away except when I visit a new site.
On Tue, Apr 14, 2020 at 3:52 pm, stan via devel devel@lists.fedoraproject.org wrote:
Will the ability to turn off NetworkManager involvement in DNS in the configuration file (None) still remain? I use a local caching DNS server, and had to do that in order to allow it to run without interference / override by NetworkManager.
Of course. You just have to set dns=None in your /etc/NetworkManager.conf, as before. Also: 'systemctl disable systemd-resolved.service'.
On Tue, 14 Apr 2020 18:39:05 -0500 Michael Catanzaro mcatanzaro@gnome.org wrote:
On Tue, Apr 14, 2020 at 3:52 pm, stan via devel devel@lists.fedoraproject.org wrote:
Will the ability to turn off NetworkManager involvement in DNS in the configuration file (None) still remain? I use a local caching DNS server, and had to do that in order to allow it to run without interference / override by NetworkManager.
Of course. You just have to set dns=None in your /etc/NetworkManager.conf, as before. Also: 'systemctl disable systemd-resolved.service'.
Excellent! Thanks.
On Di, 14.04.20 15:52, Fedora Development ML (devel@lists.fedoraproject.org) wrote:
On Tue, 14 Apr 2020 16:18:02 -0500 Michael Catanzaro mcatanzaro@gnome.org wrote:
NetworkManager has three DNS backends: default (nss-dns, what we use currently), dnsmasq, and systemd-resolved. The default backend just does the wrong thing and cannot be fixed. When either dnsmasq or systemd-resolved is in use, NetworkManager will go ahead and do the right thing by telling dnsmasq/systemd-resolved which network interfaces should be used to resolve which hostnames. I consulted with the NetworkManager developers and they recommended systemd-resolved over dnsmasq, although I understand that dnsmasq is good too.
Will the ability to turn off NetworkManager involvement in DNS in the configuration file (None) still remain? I use a local caching DNS server, and had to do that in order to allow it to run without interference / override by NetworkManager.
resolved has three modes:
1. If /etc/resolv.conf is a regular file, resolved will *consume* it for DNS configuration, and never change it or modify it or replace it. If this mode is selected arbitrary other programs that do DNS will talk directly to the provided DNS servers, and resolved is out of the loop.
2. Alternatvely, /etc/resolv.conf can be made a symlink to /run/systemd/resolve/stub-resolv.conf. That file is updated by resolved whenever DNS configuration changes. In it you'll find as DNS server the local host configured, where resolved is listening. In this mode other programs that do DNS will talk to resolved, and resolved propagates it to other DNS servers.
3. Alternatively /etc/resolv.conf can be made a symlink to /run/systemd/resolve/resolv.conf. That file is also updated by resolved whenever DNS configuration changes. In it you find a merged set of configured upstream DNS servers. In this mode programs that do DNS will talk directly to the upstream DNS servers, but resolved will still tell them which ones, based on all the information it has.
In mode #1 resolved neither manages /etc/resolv.conf nor inserts itself into DNS resolution in any way. In mode #2 it manages *and* inserts itself into DNS resolution. In mode #3 it manages DNS resolution but will not insert itself to.
NetworkManager natively supports informing resolved about DNS configuration changes, hence whenever NM discovers a new DNS server it tells resolved, which resolved then uses itself and writes it to /run/systemd/resolve/stub-resolv.conf and /run/systemd/resolve/resolv.conf.
Long story short: we want to default to mode #2. But if you can chose mode #1 or #3 if you like, depending on whether you want to kick resolved out of managing resolv.conf or out of doing any DNS lookups for you at all.
Lennart
-- Lennart Poettering, Berlin
* Lennart Poettering:
- If /etc/resolv.conf is a regular file, resolved will *consume* it for DNS configuration, and never change it or modify it or replace it. If this mode is selected arbitrary other programs that do DNS will talk directly to the provided DNS servers, and resolved is out of the loop.
In mode #1 resolved neither manages /etc/resolv.conf nor inserts itself into DNS resolution in any way.
What will nss_resolve do in this case? Nothing?
Does this mean we need both resolve and dns in nsswitch.conf?
Thanks, Florian
On Mi, 15.04.20 15:50, Florian Weimer (fweimer@redhat.com) wrote:
- Lennart Poettering:
- If /etc/resolv.conf is a regular file, resolved will *consume* it for DNS configuration, and never change it or modify it or replace it. If this mode is selected arbitrary other programs that do DNS will talk directly to the provided DNS servers, and resolved is out of the loop.
In mode #1 resolved neither manages /etc/resolv.conf nor inserts itself into DNS resolution in any way.
What will nss_resolve do in this case? Nothing?
The nss_resolve module is just a wrapper around resolved's bus API. And the bus API uses resolved's own DNS resolution code. And resolved is smart enough to automatically become a *consumer* of /etc/nsswitch.conf (instead of a *manager* of it) if it is a regular file instead of a symlink to resolved's own files in /run.
Does this mean we need both resolve and dns in nsswitch.conf?
Yeah, that's the idea. resolve takes control if resolved runs, and otherwise we fall back to classic nss-dns.
Lennart
-- Lennart Poettering, Berlin
On Mi, 15.04.20 16:30, Lennart Poettering (mzerqung@0pointer.de) wrote:
On Mi, 15.04.20 15:50, Florian Weimer (fweimer@redhat.com) wrote:
- Lennart Poettering:
- If /etc/resolv.conf is a regular file, resolved will *consume* it for DNS configuration, and never change it or modify it or replace it. If this mode is selected arbitrary other programs that do DNS will talk directly to the provided DNS servers, and resolved is out of the loop.
In mode #1 resolved neither manages /etc/resolv.conf nor inserts itself into DNS resolution in any way.
What will nss_resolve do in this case? Nothing?
The nss_resolve module is just a wrapper around resolved's bus API. And the bus API uses resolved's own DNS resolution code. And resolved is smart enough to automatically become a *consumer* of /etc/nsswitch.conf (instead of a *manager* of it) if it is a regular file instead of a symlink to resolved's own files in /run.
Meh. I mean /etc/resolv.conf here, of course, not /etc/nsswitch.conf.
Lennart
-- Lennart Poettering, Berlin
* Lennart Poettering:
On Mi, 15.04.20 16:30, Lennart Poettering (mzerqung@0pointer.de) wrote:
On Mi, 15.04.20 15:50, Florian Weimer (fweimer@redhat.com) wrote:
- Lennart Poettering:
- If /etc/resolv.conf is a regular file, resolved will *consume* it for DNS configuration, and never change it or modify it or replace it. If this mode is selected arbitrary other programs that do DNS will talk directly to the provided DNS servers, and resolved is out of the loop.
In mode #1 resolved neither manages /etc/resolv.conf nor inserts itself into DNS resolution in any way.
What will nss_resolve do in this case? Nothing?
The nss_resolve module is just a wrapper around resolved's bus API. And the bus API uses resolved's own DNS resolution code. And resolved is smart enough to automatically become a *consumer* of /etc/nsswitch.conf (instead of a *manager* of it) if it is a regular file instead of a symlink to resolved's own files in /run.
Meh. I mean /etc/resolv.conf here, of course, not /etc/nsswitch.conf.
So if /etc/resolv.conf comes from somewhere else, then nss_resolve will still forward queries to the daemon, which contacts the upstream server on nss_resolve's behave (possibly with some caching), and eventually return the data to the application?
Or does nss_resolve fail with UNAVAIL and expects nss_dns to fetch the data?
I'd prefer the first approach, but it really means that resolved is out of the loop only for queries submitted over the DNS transport (so res_query and the like, or direct use of UDP & TCP). Hence my confusion. 8-)
Thanks, Florian
On Thu, Apr 16, 2020 at 12:53:48PM +0200, Florian Weimer wrote:
- Lennart Poettering:
On Mi, 15.04.20 16:30, Lennart Poettering (mzerqung@0pointer.de) wrote:
On Mi, 15.04.20 15:50, Florian Weimer (fweimer@redhat.com) wrote:
- Lennart Poettering:
- If /etc/resolv.conf is a regular file, resolved will *consume* it for DNS configuration, and never change it or modify it or replace it. If this mode is selected arbitrary other programs that do DNS will talk directly to the provided DNS servers, and resolved is out of the loop.
In mode #1 resolved neither manages /etc/resolv.conf nor inserts itself into DNS resolution in any way.
What will nss_resolve do in this case? Nothing?
The nss_resolve module is just a wrapper around resolved's bus API. And the bus API uses resolved's own DNS resolution code. And resolved is smart enough to automatically become a *consumer* of /etc/nsswitch.conf (instead of a *manager* of it) if it is a regular file instead of a symlink to resolved's own files in /run.
Meh. I mean /etc/resolv.conf here, of course, not /etc/nsswitch.conf.
So if /etc/resolv.conf comes from somewhere else, then nss_resolve will still forward queries to the daemon, which contacts the upstream server on nss_resolve's behave (possibly with some caching), and eventually return the data to the application?
nss-resolve is enabled/disabled through nsswitch.conf. It always talks to systemd-resolved using local IPC. It doesn't care about /etc/resolv.conf in any way.
What Lennart wrote above applies to systemd-resolved and to things which look at /etc/resolv.conf for some reason. If nss-resolve is enabled, then only things which do not use nss at all would fall into this category.
Or does nss_resolve fail with UNAVAIL and expects nss_dns to fetch the data?
nss_resolve fails with UNAVAIL when systemd-resolved is not running. So yeah, we use want to use nss_dns as a fallback for that case. I'm not sure if that is what you are asking about.
I'd prefer the first approach, but it really means that resolved is out of the loop only for queries submitted over the DNS transport (so res_query and the like, or direct use of UDP & TCP). Hence my confusion. 8-)
Zbyszek
* Zbigniew Jędrzejewski-Szmek:
On Thu, Apr 16, 2020 at 12:53:48PM +0200, Florian Weimer wrote:
- Lennart Poettering:
On Mi, 15.04.20 16:30, Lennart Poettering (mzerqung@0pointer.de) wrote:
On Mi, 15.04.20 15:50, Florian Weimer (fweimer@redhat.com) wrote:
- Lennart Poettering:
- If /etc/resolv.conf is a regular file, resolved will *consume* it for DNS configuration, and never change it or modify it or replace it. If this mode is selected arbitrary other programs that do DNS will talk directly to the provided DNS servers, and resolved is out of the loop.
In mode #1 resolved neither manages /etc/resolv.conf nor inserts itself into DNS resolution in any way.
What will nss_resolve do in this case? Nothing?
The nss_resolve module is just a wrapper around resolved's bus API. And the bus API uses resolved's own DNS resolution code. And resolved is smart enough to automatically become a *consumer* of /etc/nsswitch.conf (instead of a *manager* of it) if it is a regular file instead of a symlink to resolved's own files in /run.
Meh. I mean /etc/resolv.conf here, of course, not /etc/nsswitch.conf.
So if /etc/resolv.conf comes from somewhere else, then nss_resolve will still forward queries to the daemon, which contacts the upstream server on nss_resolve's behave (possibly with some caching), and eventually return the data to the application?
nss-resolve is enabled/disabled through nsswitch.conf. It always talks to systemd-resolved using local IPC. It doesn't care about /etc/resolv.conf in any way.
What Lennart wrote above applies to systemd-resolved and to things which look at /etc/resolv.conf for some reason. If nss-resolve is enabled, then only things which do not use nss at all would fall into this category.
Or does nss_resolve fail with UNAVAIL and expects nss_dns to fetch the data?
nss_resolve fails with UNAVAIL when systemd-resolved is not running. So yeah, we use want to use nss_dns as a fallback for that case. I'm not sure if that is what you are asking about.
Let me rephrase:
If /etc/resolv.conf is a regular file, will systemd-resolved deactivate itself? Or use the name server configuration found there instead?
Thanks, Florian
On Thu, Apr 16, 2020, at 9:26 AM, Florian Weimer wrote:
- Zbigniew Jędrzejewski-Szmek:
On Thu, Apr 16, 2020 at 12:53:48PM +0200, Florian Weimer wrote:
- Lennart Poettering:
On Mi, 15.04.20 16:30, Lennart Poettering (mzerqung@0pointer.de) wrote:
On Mi, 15.04.20 15:50, Florian Weimer (fweimer@redhat.com) wrote:
- Lennart Poettering:
> 1. If /etc/resolv.conf is a regular file, resolved will *consume* it > for DNS configuration, and never change it or modify it or replace > it. If this mode is selected arbitrary other programs that do DNS > will talk directly to the provided DNS servers, and resolved is out > of the loop.
> In mode #1 resolved neither manages /etc/resolv.conf nor inserts > itself into DNS resolution in any way.
What will nss_resolve do in this case? Nothing?
The nss_resolve module is just a wrapper around resolved's bus API. And the bus API uses resolved's own DNS resolution code. And resolved is smart enough to automatically become a *consumer* of /etc/nsswitch.conf (instead of a *manager* of it) if it is a regular file instead of a symlink to resolved's own files in /run.
Meh. I mean /etc/resolv.conf here, of course, not /etc/nsswitch.conf.
So if /etc/resolv.conf comes from somewhere else, then nss_resolve will still forward queries to the daemon, which contacts the upstream server on nss_resolve's behave (possibly with some caching), and eventually return the data to the application?
nss-resolve is enabled/disabled through nsswitch.conf. It always talks to systemd-resolved using local IPC. It doesn't care about /etc/resolv.conf in any way.
What Lennart wrote above applies to systemd-resolved and to things which look at /etc/resolv.conf for some reason. If nss-resolve is enabled, then only things which do not use nss at all would fall into this category.
Or does nss_resolve fail with UNAVAIL and expects nss_dns to fetch the data?
nss_resolve fails with UNAVAIL when systemd-resolved is not running. So yeah, we use want to use nss_dns as a fallback for that case. I'm not sure if that is what you are asking about.
Let me rephrase:
If /etc/resolv.conf is a regular file, will systemd-resolved deactivate itself? Or use the name server configuration found there instead?
Based on earlier replies in the thread, resolved will use the nameservers from the file. There's no mention of it disabling itself.
V/r, James Cassell
On Do, 16.04.20 15:26, Florian Weimer (fweimer@redhat.com) wrote:
If /etc/resolv.conf is a regular file, will systemd-resolved deactivate itself? Or use the name server configuration found there instead?
It will use it.
It's smart on this: if it finds a symlink there that points to one of the files resolved manages it will assume it will not read the files, and assume it's the *provider*, not the *consumer* of them. But if it finds a regular file there it understands that and becomes a *consumer* of that file, like any other tool, and knows it doesn't provide the file anymore.
Lennart
-- Lennart Poettering, Berlin
On Do, 16.04.20 12:53, Florian Weimer (fweimer@redhat.com) wrote:
Meh. I mean /etc/resolv.conf here, of course, not /etc/nsswitch.conf.
So if /etc/resolv.conf comes from somewhere else, then nss_resolve will still forward queries to the daemon, which contacts the upstream server on nss_resolve's behave (possibly with some caching), and eventually return the data to the application?
correct.
Or does nss_resolve fail with UNAVAIL and expects nss_dns to fetch the data?
no.
Lennart
-- Lennart Poettering, Berlin
* Lennart Poettering:
On Do, 16.04.20 12:53, Florian Weimer (fweimer@redhat.com) wrote:
Meh. I mean /etc/resolv.conf here, of course, not /etc/nsswitch.conf.
So if /etc/resolv.conf comes from somewhere else, then nss_resolve will still forward queries to the daemon, which contacts the upstream server on nss_resolve's behave (possibly with some caching), and eventually return the data to the application?
correct.
Or does nss_resolve fail with UNAVAIL and expects nss_dns to fetch the data?
no.
Okay, this behavior is nice and what I expect.
Thanks, Florian
On Wed, 15 Apr 2020 15:46:02 +0200 Lennart Poettering mzerqung@0pointer.de wrote:
resolved has three modes:
[Snipped for brevity.]
Thanks. Saved for future reference.
migrating a couple of local servers to, atm, F32, i noted this planned/accepted 'systemd-wide change' for switching to systemd-resolved as default in >= F33, here, and
https://fedoraproject.org/wiki/Changes/systemd-resolved
again, i understand it's default @ >= F33.
iiuc, tho, it should already be usable with F32?
attempting to enable 'mode #2' on a F32 install, my override configs are being ignored.
reading more closely, i've managed to miss any comment re: config splitting into "/etc/systemd/resolved.conf.d", either currently or planned.
looking at current install @ F32, there's only "/etc/systemd/resolved.conf".
both docs referenced
https://www.freedesktop.org/software/systemd/man/resolved.conf.html https://wiki.archlinux.org/index.php/Systemd-resolved
make mention of
/etc/systemd/resolved.conf.d/*.conf
config support.
in my F32 install, i've enabled
systemctl status systemd-resolved.service -l ● systemd-resolved.service - Network Name Resolution Loaded: loaded (/usr/lib/systemd/system/systemd-resolved.service; enabled; vendor preset: disabled) Active: active (running) since Sun 2020-07-26 13:58:26 PDT; 18min ago
created the necessary symlink,
ls -al /etc/resolv.conf lrwxrwxrwx 1 root root 37 Jul 26 13:43 /etc/resolv.conf -> /run/systemd/resolve/stub-resolv.conf
and, in the 'supported' /etc/systemd/resolved.conf.d dir, config'd my local resolver
cat /etc/systemd/resolved.conf.d/dns_servers.conf [Resolve] DNS=10.0.1.53 Domains=.
and DISabled fallbacks
cat /etc/systemd/resolved.conf.d/fallback_dns.conf [Resolve] FallbackDNS=
but, after restart
systemctl restart systemd-resolved
i still see
resolvectl status Global LLMNR setting: yes MulticastDNS setting: yes DNSOverTLS setting: no DNSSEC setting: allow-downgrade DNSSEC supported: yes Fallback DNS Servers: 1.1.1.1 8.8.8.8 1.0.0.1 8.8.4.4 2606:4700:4700::1111 2001:4860:4860::8888 2606:4700:4700::1001 2001:4860:4860::8844 ...
which are still/only the defaults,
grep -i fallback /etc/systemd/resolved.conf #FallbackDNS=1.1.1.1 8.8.8.8 1.0.0.1 8.8.4.4 2606:4700:4700::1111 2001:4860:4860::8888 2606:4700:4700::1001 2001:4860:4860::8844
assuming that conf.d/* configs _are_ to be supported for F33, and already work with F32,
what's the correct method/mechanism for overrides of the defaults?
On Tue, Apr 14, 2020, at 3:23 PM, Ben Cotton wrote:
https://fedoraproject.org/wiki/Changes/systemd-resolved
== Summary ==
Enable systemd-resolved by default. glibc will perform name resolution using nss-resolve rather than nss-dns.
== Owner ==
- Name: [[User:catanzaro| Michael Catanzaro]]
- Email: mcatanzaro@redhat.com
== Detailed Description ==
We will enable systemd-resolved by default.
Does this require systemd to be running? How does this affect DNS resolution on a Fedora 33 container?
V/r James Cassell
On Tue, Apr 14, 2020 at 03:57:50PM -0400, James Cassell wrote:
On Tue, Apr 14, 2020, at 3:23 PM, Ben Cotton wrote:
https://fedoraproject.org/wiki/Changes/systemd-resolved
== Summary ==
Enable systemd-resolved by default. glibc will perform name resolution using nss-resolve rather than nss-dns.
== Owner ==
- Name: [[User:catanzaro| Michael Catanzaro]]
- Email: mcatanzaro@redhat.com
== Detailed Description ==
We will enable systemd-resolved by default.
Does this require systemd to be running? How does this affect DNS resolution on a Fedora 33 container?
That's a good point. With systemd-resolved not running, resolution might not work properly.
There's two parts to this: - whether a fallback is included in the nss stack - whether dns servers are appropriately configured
For the first part: there should be no issue. Upstream recommends nss-resolve(8) the following:
hosts: ... resolve [!UNAVAIL=return] dns ...
Assuming that the same is done in Fedora, the nss stack will automatically fall back to nss-dns when resolved is not running.
I guess the lesson here is the nsswitch.conf change should be clarified in the proposal.
For the second part: the answer is complicated. When /etc/resolv.conf is a symlink to /run/systemd/resolve/stub-resolv.conf, nss-dns does not work when systemd-resolved stops. In the case of a container without systemd running, this will be a broken symlink, and nss-dns will not work either.
But we seem to already have this problem to some extent. NetworkManager allows /etc/resolv.conf to be a symlink to /run/NetworkManager/resolv.conf too, to support name servers configured at run time with a read-only root, and with systemd not running, NM won't either, and this will be a dangling symlink.
I'm not sure what the best path option here is. The path of least resistance would be to simply leave /etc/resolv.conf out of this change. nss-resolve doesn't care, and the effect is only on things which don't use the nss stack, or read /etc/resolv.conf for other purposes.
Zbyszek
On Tue, Apr 14, 2020 at 8:48 pm, Zbigniew Jędrzejewski-Szmek zbyszek@in.waw.pl wrote:
I guess the lesson here is the nsswitch.conf change should be clarified in the proposal.
OK, I've just added it at the end of this part here:
"systemd-libs currently has [https://src.fedoraproject.org/rpms/systemd/blob/bb79fb73875f8e71841a1ee8ede5... a %post scriplet] to enable nss-myhostname and nss-systemd by either (a) modifying authselect's user-nsswitch.conf template, if authselect is in use, or (b) directly modifying /etc/nsswitch.conf otherwise. We will work with the systemd maintainers to enable nss-resolve here as well by adding `resolve [!UNAVAIL=return]` to the hosts line."
Then the instructions in the change proposal for disabling systemd-resolved say:
"Modify /etc/authselect/user-nsswitch.conf and remove resolve [!UNAVAIL=return] from the hosts line. Run authselect apply-changes. (If you have disabled authselect, then edit /etc/nsswitch.conf directly.)"
I guess I should delete that from the proposal, since it's not needed?
I'm not sure what the best path option here is. The path of least resistance would be to simply leave /etc/resolv.conf out of this change. nss-resolve doesn't care, and the effect is only on things which don't use the nss stack, or read /etc/resolv.conf for other purposes.
NetworkManager only enables its systemd-resolved backend if /etc/resolv.conf is symlinked appropriately. So that needs to happen.
I didn't consider cases where systemd is not running because Fedora hasn't supported booting without systemd in about a decade. But I guess the problem here is for containers where systemd is not running inside the container, but is running on the host system? I hadn't considered this scenario. What do Ubuntu containers do? I guess those are not all broken. :)
* Michael Catanzaro:
On Tue, Apr 14, 2020 at 8:48 pm, Zbigniew Jędrzejewski-Szmek zbyszek@in.waw.pl wrote:
I guess the lesson here is the nsswitch.conf change should be clarified in the proposal.
OK, I've just added it at the end of this part here:
"systemd-libs currently has [https://src.fedoraproject.org/rpms/systemd/blob/bb79fb73875f8e71841a1ee8ede5... a %post scriplet] to enable nss-myhostname and nss-systemd by either (a) modifying authselect's user-nsswitch.conf template, if authselect is in use, or (b) directly modifying /etc/nsswitch.conf otherwise. We will work with the systemd maintainers to enable nss-resolve here as well by adding `resolve [!UNAVAIL=return]` to the hosts line."
At which position? After files?
Does systemd-resolved cache /etc/hosts?
And we really need to move /etc/nsswitch.conf out of glibc. We spend some time on maintaining that file, when in fact it doesn't matter because too many scriptlets and programs patch it.
Thanks, Florian
On Mi, 15.04.20 09:36, Florian Weimer (fweimer@redhat.com) wrote:
- Michael Catanzaro:
On Tue, Apr 14, 2020 at 8:48 pm, Zbigniew Jędrzejewski-Szmek zbyszek@in.waw.pl wrote:
I guess the lesson here is the nsswitch.conf change should be clarified in the proposal.
OK, I've just added it at the end of this part here:
"systemd-libs currently has [https://src.fedoraproject.org/rpms/systemd/blob/bb79fb73875f8e71841a1ee8ede5... a %post scriplet] to enable nss-myhostname and nss-systemd by either (a) modifying authselect's user-nsswitch.conf template, if authselect is in use, or (b) directly modifying /etc/nsswitch.conf otherwise. We will work with the systemd maintainers to enable nss-resolve here as well by adding `resolve [!UNAVAIL=return]` to the hosts line."
At which position? After files?
The suggested line in nsswitch.conf is:
hosts: files mymachines resolve [!UNAVAIL=return] dns myhostname
See https://www.freedesktop.org/software/systemd/man/nss-resolve.html
So currently we leave "files" the way it is, taking precendence.
That said, resolved has a bus API for resolving hosts too, which gives a bit richer an API to do things, instead of using gethostbyname(). resolved parses and caches /etc/hosts for that natively, so that we can server the same set of names when going via the bus API or via NSS.
Does systemd-resolved cache /etc/hosts?
Yes.
Lennart
-- Lennart Poettering, Berlin
* Lennart Poettering:
On Mi, 15.04.20 09:36, Florian Weimer (fweimer@redhat.com) wrote:
- Michael Catanzaro:
On Tue, Apr 14, 2020 at 8:48 pm, Zbigniew Jędrzejewski-Szmek zbyszek@in.waw.pl wrote:
I guess the lesson here is the nsswitch.conf change should be clarified in the proposal.
OK, I've just added it at the end of this part here:
"systemd-libs currently has [https://src.fedoraproject.org/rpms/systemd/blob/bb79fb73875f8e71841a1ee8ede5... a %post scriplet] to enable nss-myhostname and nss-systemd by either (a) modifying authselect's user-nsswitch.conf template, if authselect is in use, or (b) directly modifying /etc/nsswitch.conf otherwise. We will work with the systemd maintainers to enable nss-resolve here as well by adding `resolve [!UNAVAIL=return]` to the hosts line."
At which position? After files?
The suggested line in nsswitch.conf is:
hosts: files mymachines resolve [!UNAVAIL=return] dns myhostnameSee https://www.freedesktop.org/software/systemd/man/nss-resolve.html
So currently we leave "files" the way it is, taking precendence.
That said, resolved has a bus API for resolving hosts too, which gives a bit richer an API to do things, instead of using gethostbyname(). resolved parses and caches /etc/hosts for that natively, so that we can server the same set of names when going via the bus API or via NSS.
Does systemd-resolved cache /etc/hosts?
Yes.
Then I don't understand why we are listing files first, before resolve.
If we can handle /etc/hosts through nss_resolve, with caching, we make progress towards replacing nscd. /etc/hosts is one of the things not handled by ssssd, so one of the remaining gaps would be covered.
Thanks, Florian
On Mi, 15.04.20 16:27, Florian Weimer (fweimer@redhat.com) wrote:
That said, resolved has a bus API for resolving hosts too, which gives a bit richer an API to do things, instead of using gethostbyname(). resolved parses and caches /etc/hosts for that natively, so that we can server the same set of names when going via the bus API or via NSS.
Does systemd-resolved cache /etc/hosts?
Yes.
Then I don't understand why we are listing files first, before resolve.
It should be fine to swap this around I guess. This hasn't come up before. The /etc/hosts support in resolved was added primarily for full compat when queries come in via dbus. It wasn't supposed to be a better implementation of nss-files (though it effectively is, since we parse on when the file changes and cache in memory), but I guess we can decide it's not just compat feature now, but also and performance improvement feature.
Lennart
-- Lennart Poettering, Berlin
On Wed, Apr 15, 2020 at 4:12 pm, Lennart Poettering mzerqung@0pointer.de wrote:
The suggested line in nsswitch.conf is:
hosts: files mymachines resolve [!UNAVAIL=return] dns myhostname
My plan is to use:
hosts: files resolve [!UNAVAIL=return] mdns4_minimal [NOTFOUND=return] dns myhostname
Apparently there is some problem with mymachines, so we don't use it currently, and adding it would require a separate change proposal. Then we need to keep avahi as well.
I guess we can swap files and resolve if Florian thinks that's best. Let's update the upstream manpage's recommendation if we do so, though.
On Wed, Apr 15, 2020 at 10:02:17AM -0500, Michael Catanzaro wrote:
On Wed, Apr 15, 2020 at 4:12 pm, Lennart Poettering mzerqung@0pointer.de wrote:
The suggested line in nsswitch.conf is:
hosts: files mymachines resolve [!UNAVAIL=return] dns myhostname
My plan is to use:
hosts: files resolve [!UNAVAIL=return] mdns4_minimal [NOTFOUND=return] dns myhostname
Apparently there is some problem with mymachines, so we don't use it currently, and adding it would require a separate change proposal. Then we need to keep avahi as well.
I guess we can swap files and resolve if Florian thinks that's best. Let's update the upstream manpage's recommendation if we do so, though.
https://github.com/systemd/systemd/pull/15437
Zbyszek
On Wed, Apr 15, 2020 at 4:33 pm, Zbigniew Jędrzejewski-Szmek zbyszek@in.waw.pl wrote:
To change this for existing Fedora systems is going to require some scriptlet hackery... somewhere (systemd package, maybe?).
On Wed, Apr 15, 2020 at 12:05:40PM -0500, Michael Catanzaro wrote:
On Wed, Apr 15, 2020 at 4:33 pm, Zbigniew Jędrzejewski-Szmek zbyszek@in.waw.pl wrote:
To change this for existing Fedora systems is going to require some scriptlet hackery... somewhere (systemd package, maybe?).
Yeah. We already have such hackery in systemd's %post macros, so while this is not something enjoyable, we can add more... Alternatively, the same rules might be moved to some other package, most likely the one that ends up owning /etc/nsswitch.conf.
Zbyszek
On Wed, Apr 15, 2020 at 9:36 am, Florian Weimer fweimer@redhat.com wrote:
And we really need to move /etc/nsswitch.conf out of glibc. We spend some time on maintaining that file, when in fact it doesn't matter because too many scriptlets and programs patch it.
Moving it to authselect might be sensible.
BTW: I have https://src.fedoraproject.org/rpms/glibc/pull-request/17 still outstanding to fix a mistake in glibc's version.
On Mi, 15.04.20 10:02, Michael Catanzaro (mcatanzaro@gnome.org) wrote:
On Wed, Apr 15, 2020 at 9:36 am, Florian Weimer fweimer@redhat.com wrote:
And we really need to move /etc/nsswitch.conf out of glibc. We spend some time on maintaining that file, when in fact it doesn't matter because too many scriptlets and programs patch it.
Moving it to authselect might be sensible.
Why not setup.rpm? /etc/hosts is owned by setup.rpm.
Lennart
-- Lennart Poettering, Berlin
* Michael Catanzaro:
On Wed, Apr 15, 2020 at 9:36 am, Florian Weimer fweimer@redhat.com wrote:
And we really need to move /etc/nsswitch.conf out of glibc. We spend some time on maintaining that file, when in fact it doesn't matter because too many scriptlets and programs patch it.
Moving it to authselect might be sensible.
I came up with setup:
https://bugzilla.redhat.com/show_bug.cgi?id=1717384
If authselect wants to take it, that's okay with me too. We can reassign the bug.
BTW: I have https://src.fedoraproject.org/rpms/glibc/pull-request/17 still outstanding to fix a mistake in glibc's version.
I've merged it, sorry that it fell through the cracks.
Thanks, Florian
On 4/14/20 17:26, Michael Catanzaro wrote:
On Tue, Apr 14, 2020 at 8:48 pm, Zbigniew Jędrzejewski-Szmek zbyszek@in.waw.pl wrote:
I guess the lesson here is the nsswitch.conf change should be clarified in the proposal.
OK, I've just added it at the end of this part here:
"systemd-libs currently has [https://src.fedoraproject.org/rpms/systemd/blob/bb79fb73875f8e71841a1ee8ede5... a %post scriplet] to enable nss-myhostname and nss-systemd by either (a) modifying authselect's user-nsswitch.conf template, if authselect is in use, or (b) directly modifying /etc/nsswitch.conf otherwise. We will work with the systemd maintainers to enable nss-resolve here as well by adding `resolve [!UNAVAIL=return]` to the hosts line."
Then the instructions in the change proposal for disabling systemd-resolved say:
"Modify /etc/authselect/user-nsswitch.conf and remove resolve [!UNAVAIL=return] from the hosts line. Run authselect apply-changes. (If you have disabled authselect, then edit /etc/nsswitch.conf directly.)"
I guess I should delete that from the proposal, since it's not needed?
I'm not sure what the best path option here is. The path of least resistance would be to simply leave /etc/resolv.conf out of this change. nss-resolve doesn't care, and the effect is only on things which don't use the nss stack, or read /etc/resolv.conf for other purposes.
NetworkManager only enables its systemd-resolved backend if /etc/resolv.conf is symlinked appropriately. So that needs to happen.
I didn't consider cases where systemd is not running because Fedora hasn't supported booting without systemd in about a decade. But I guess the problem here is for containers where systemd is not running inside the container, but is running on the host system? I hadn't considered this scenario. What do Ubuntu containers do? I guess those are not all broken. :)
devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
We can change container engines (podman, Buildah, CRI-O) to handle this but they need to have a location of a properly configured resolv.conf file, somewhere on the system to be used without systemd.
On Mi, 15.04.20 09:01, Daniel J Walsh (dwalsh@redhat.com) wrote:
I didn't consider cases where systemd is not running because Fedora hasn't supported booting without systemd in about a decade. But I guess the problem here is for containers where systemd is not running inside the container, but is running on the host system? I hadn't considered this scenario. What do Ubuntu containers do? I guess those are not all broken. :)
devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
We can change container engines (podman, Buildah, CRI-O) to handle this but they need to have a location of a properly configured resolv.conf file, somewhere on the system to be used without systemd.
My suggestion for those container managers: if /etc/resolv.conf is a regular file, always use that as copy source. If it is a symlinkt to /run/systemd/resolve/stub-resolv.conf or /run/systemd/resolve/resolv.conf then use /run/systemd/resolve/resolv.conf as copy source.
Yes, I mean that, even if /run/systemd/resolve/stub-resolv.conf is the symlink destination use the file without "stub-" in the name as copy source. Because that file always contains the literal upstream DNS servers, and does not redirect DNS traffic to 127.0.0.53 like the file with "-stub" in the name does. Since containers are typically run inside their own network namespace it's wise to use the upstream DNS servers directly, instead of trying to the DNS forwarder called resolved on localhost that is likely not going to be there in a netns container.
The algorithm above in C-ish pseudo-code:
for (;;) { fd = open("/etc/resolv.conf", O_RDONLY|O_CLOEXEC|O_NOFOLLOW); if (fd >= 0) break; /* success! it's a regular file */ if (errno != ELOOP) break; /* failure! something unexpected */
/* It's a symlink */ r = readlink("/etc/resolv.conf", &dest); if (r < 0) { if (errno == EINVAL) continue; /* Oh, it's not a symlink anymore? somebody must just have replace the file, let's try again */
/* failure! some unexpected error */ break; }
/* Check where the symlink points. Check by absolute and by relative paths, just in case. If this points to any of the three files provided by systemd-resolved, use the one that encodes upstream DNS info */ if (strcmp(dest, "/run/systemd/resolve/stub-resolv.conf") == 0 || strcmp(dest, "../run/systemd/resolve/stub-resolv.conf") == 0 || strcmp(dest, "/run/systemd/resolve/resolv.conf") == 0 || strcmp(dest, "../run/systemd/resolve/resolv.conf") == 0 || strcmp(dest, "/usr/lib/systemd/resolv.conf") == 0 || strcmp(dest, "../usr/lib/systemd/resolv.conf") == 0)) {
fd = open("/run/systemd/resolve/resolv.conf", O_RDONLY|O_CLOEXEC); else fd = open("/run/systemd/resolve/resolv.conf", O_RDONLY|O_CLOEXEC);
break; }
You get the idea: use O_LOOP to check if it's a symlink and then use readlink() to see if the file points to something managed by resolved.
Lennart
-- Lennart Poettering, Berlin
On Di, 14.04.20 15:57, James Cassell (fedoraproject@cyberpear.com) wrote:
On Tue, Apr 14, 2020, at 3:23 PM, Ben Cotton wrote:
https://fedoraproject.org/wiki/Changes/systemd-resolved
== Summary ==
Enable systemd-resolved by default. glibc will perform name resolution using nss-resolve rather than nss-dns.
== Owner ==
- Name: [[User:catanzaro| Michael Catanzaro]]
- Email: mcatanzaro@redhat.com
== Detailed Description ==
We will enable systemd-resolved by default.
Does this require systemd to be running? How does this affect DNS resolution on a Fedora 33 container?
Depends.
If a container manager copies in /etc/resolv.conf from the host into the container on container *start*, it might be wise to copy in /run/systemd/resolve/resolv.conf instead of /etc/resolv.conf, if it exists. That file in /run contains the currently up-to-date upstream DNS info literally.
If a container builder copies in /etc/resolv.conf during build time, it probably should insert something like 8.8.8.8 as DNS servers there, also replacing whatever is there.
Note that the logic in systemd and resolved is very defensive: if /etc/resolv.conf is not a symlink to /run/systemd/resolve/{stub-,}resolv.conf then resolved will consume /etc/resolv.conf instead of managing it (see other mail), hence a container mgr/builder that wants to direct DNS traffic somewhere should just override the file to whatever it wants, and things will just work, regarldess if resolved runs in the container or not, and resolved -- if used -- will honour whatever the container mgr/builder put there.
Lennart
-- Lennart Poettering, Berlin
On 4/15/20 10:07, Lennart Poettering wrote:
On Di, 14.04.20 15:57, James Cassell (fedoraproject@cyberpear.com) wrote:
On Tue, Apr 14, 2020, at 3:23 PM, Ben Cotton wrote:
https://fedoraproject.org/wiki/Changes/systemd-resolved
== Summary ==
Enable systemd-resolved by default. glibc will perform name resolution using nss-resolve rather than nss-dns.
== Owner ==
- Name: [[User:catanzaro| Michael Catanzaro]]
- Email: mcatanzaro@redhat.com
== Detailed Description ==
We will enable systemd-resolved by default.
Does this require systemd to be running? How does this affect DNS resolution on a Fedora 33 container?
Depends.
If a container manager copies in /etc/resolv.conf from the host into the container on container *start*, it might be wise to copy in /run/systemd/resolve/resolv.conf instead of /etc/resolv.conf, if it exists. That file in /run contains the currently up-to-date upstream DNS info literally.
Containers copy the /etc/resolv.conf. /etc/hosts on creation, that way they can modify it internally,
It looks like podman will just follow the link. I setup this simple test
# ls -l /etc/resolv.conf lrwxrwxrwx. 1 root root 16 Apr 15 13:25 /etc/resolv.conf -> /run/resolv.conf # cat /etc/resolv.conf # Generated by NetworkManager search redhat.com nameserver 10.5.30.160 nameserver 10.11.5.19 nameserver 192.168.1.1 # podman run fedora cat /etc/resolv.conf search redhat.com nameserver 10.5.30.160 nameserver 10.11.5.19 nameserver 192.168.1.1
So as long as the
/run/systemd/resolve/resolv.conf
file is properly formated, our container engines will just work.
If a container builder copies in /etc/resolv.conf during build time, it probably should insert something like 8.8.8.8 as DNS servers there, also replacing whatever is there.
Note that the logic in systemd and resolved is very defensive: if /etc/resolv.conf is not a symlink to /run/systemd/resolve/{stub-,}resolv.conf then resolved will consume /etc/resolv.conf instead of managing it (see other mail), hence a container mgr/builder that wants to direct DNS traffic somewhere should just override the file to whatever it wants, and things will just work, regarldess if resolved runs in the container or not, and resolved -- if used -- will honour whatever the container mgr/builder put there.
Lennart
-- Lennart Poettering, Berlin _______________________________________________ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
On Wed, Apr 15, 2020, at 1:27 PM, Daniel Walsh wrote:
On 4/15/20 10:07, Lennart Poettering wrote:
On Di, 14.04.20 15:57, James Cassell (fedoraproject@cyberpear.com) wrote:
On Tue, Apr 14, 2020, at 3:23 PM, Ben Cotton wrote:
https://fedoraproject.org/wiki/Changes/systemd-resolved
== Summary ==
Enable systemd-resolved by default. glibc will perform name resolution using nss-resolve rather than nss-dns.
== Owner ==
- Name: [[User:catanzaro| Michael Catanzaro]]
- Email: mcatanzaro@redhat.com
== Detailed Description ==
We will enable systemd-resolved by default.
Does this require systemd to be running? How does this affect DNS resolution on a Fedora 33 container?
Depends.
If a container manager copies in /etc/resolv.conf from the host into the container on container *start*, it might be wise to copy in /run/systemd/resolve/resolv.conf instead of /etc/resolv.conf, if it exists. That file in /run contains the currently up-to-date upstream DNS info literally.
Containers copy the /etc/resolv.conf. /etc/hosts on creation, that way they can modify it internally,
It looks like podman will just follow the link. I setup this simple test
# ls -l /etc/resolv.conf lrwxrwxrwx. 1 root root 16 Apr 15 13:25 /etc/resolv.conf -> /run/resolv.conf # cat /etc/resolv.conf # Generated by NetworkManager search redhat.com nameserver 10.5.30.160 nameserver 10.11.5.19 nameserver 192.168.1.1 # podman run fedora cat /etc/resolv.conf search redhat.com nameserver 10.5.30.160 nameserver 10.11.5.19 nameserver 192.168.1.1
So as long as the
/run/systemd/resolve/resolv.conf
file is properly formated, our container engines will just work.
I think there's some existing black magic to handle the case when resolv.conf references 127.0.0.1... maybe it already also works for 127.0.0.53. Otherwise, maybe it could be patched to handle 127.0.0.0/8 in the same way. Then no special casing for resolved would be needed.
V/r, James Cassell
If a container builder copies in /etc/resolv.conf during build time, it probably should insert something like 8.8.8.8 as DNS servers there, also replacing whatever is there.
Note that the logic in systemd and resolved is very defensive: if /etc/resolv.conf is not a symlink to /run/systemd/resolve/{stub-,}resolv.conf then resolved will consume /etc/resolv.conf instead of managing it (see other mail), hence a container mgr/builder that wants to direct DNS traffic somewhere should just override the file to whatever it wants, and things will just work, regarldess if resolved runs in the container or not, and resolved -- if used -- will honour whatever the container mgr/builder put there.
Lennart
On 4/15/20 17:06, James Cassell wrote:
On Wed, Apr 15, 2020, at 1:27 PM, Daniel Walsh wrote:
On 4/15/20 10:07, Lennart Poettering wrote:
On Di, 14.04.20 15:57, James Cassell (fedoraproject@cyberpear.com) wrote:
On Tue, Apr 14, 2020, at 3:23 PM, Ben Cotton wrote:
https://fedoraproject.org/wiki/Changes/systemd-resolved
== Summary ==
Enable systemd-resolved by default. glibc will perform name resolution using nss-resolve rather than nss-dns.
== Owner ==
- Name: [[User:catanzaro| Michael Catanzaro]]
- Email: mcatanzaro@redhat.com
== Detailed Description ==
We will enable systemd-resolved by default.
Does this require systemd to be running? How does this affect DNS resolution on a Fedora 33 container?
Depends.
If a container manager copies in /etc/resolv.conf from the host into the container on container *start*, it might be wise to copy in /run/systemd/resolve/resolv.conf instead of /etc/resolv.conf, if it exists. That file in /run contains the currently up-to-date upstream DNS info literally.
Containers copy the /etc/resolv.conf. /etc/hosts on creation, that way they can modify it internally,
It looks like podman will just follow the link. I setup this simple test
# ls -l /etc/resolv.conf lrwxrwxrwx. 1 root root 16 Apr 15 13:25 /etc/resolv.conf -> /run/resolv.conf # cat /etc/resolv.conf # Generated by NetworkManager search redhat.com nameserver 10.5.30.160 nameserver 10.11.5.19 nameserver 192.168.1.1 # podman run fedora cat /etc/resolv.conf search redhat.com nameserver 10.5.30.160 nameserver 10.11.5.19 nameserver 192.168.1.1
So as long as the
/run/systemd/resolve/resolv.conf
file is properly formated, our container engines will just work.
I think there's some existing black magic to handle the case when resolv.conf references 127.0.0.1... maybe it already also works for 127.0.0.53. Otherwise, maybe it could be patched to handle 127.0.0.0/8 in the same way. Then no special casing for resolved would be needed.
V/r, James Cassell
Yes I believe the container engines see 127.0.0.1 and switch it to 8.8.8.8
If a container builder copies in /etc/resolv.conf during build time, it probably should insert something like 8.8.8.8 as DNS servers there, also replacing whatever is there.
Note that the logic in systemd and resolved is very defensive: if /etc/resolv.conf is not a symlink to /run/systemd/resolve/{stub-,}resolv.conf then resolved will consume /etc/resolv.conf instead of managing it (see other mail), hence a container mgr/builder that wants to direct DNS traffic somewhere should just override the file to whatever it wants, and things will just work, regarldess if resolved runs in the container or not, and resolved -- if used -- will honour whatever the container mgr/builder put there.
Lennart
devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
On Mi, 15.04.20 13:27, Daniel J Walsh (dwalsh@redhat.com) wrote:
If a container manager copies in /etc/resolv.conf from the host into the container on container *start*, it might be wise to copy in /run/systemd/resolve/resolv.conf instead of /etc/resolv.conf, if it exists. That file in /run contains the currently up-to-date upstream DNS info literally.
Containers copy the /etc/resolv.conf. /etc/hosts on creation, that way they can modify it internally,
It looks like podman will just follow the link. I setup this simple test
# ls -l /etc/resolv.conf lrwxrwxrwx. 1 root root 16 Apr 15 13:25 /etc/resolv.conf -> /run/resolv.conf # cat /etc/resolv.conf # Generated by NetworkManager search redhat.com nameserver 10.5.30.160 nameserver 10.11.5.19 nameserver 192.168.1.1 # podman run fedora cat /etc/resolv.conf search redhat.com nameserver 10.5.30.160 nameserver 10.11.5.19 nameserver 192.168.1.1
So as long as the
/run/systemd/resolve/resolv.conf
file is properly formated, our container engines will just work.
Yes, /run/systemd/resolve/resolv.conf is properly formatted, the way glibc expects it. It only contains IPv4 + IPv6 "nameserver" stanzas as well as "search" stanzas.
The files systemd-resolved generates there look something like this:
``` nameserver 172.31.0.1 nameserver fd00::3a10:d5ff:fe78:6bbe search fritz.box ```
(with some additional explanatory comments at the top, which I stripped here)
Key is to access it under its proper path instead of via the symlink, for the aforementioned reasons.
Lennart
-- Lennart Poettering, Berlin
On Tuesday, April 14, 2020 9:23:27 PM CEST Ben Cotton wrote:
https://fedoraproject.org/wiki/Changes/systemd-resolved
== Summary ==
Enable systemd-resolved by default. ...
We had serious headaches because racy systemd-resolved got enabled for some unknown reasons on copr builders before (and my host as well, for example). I filled bug [1] and then I was told that nobody tests/maintains systemd-resolved... why on earth I'd opt-in to use that.
Can we fix the bug, so the generated resolv.conf pointing to local server does work _immediately_ inside mock, and isn't it port-triggered (or what it is :-))?
[1] https://bugzilla.redhat.com/show_bug.cgi?id=1710699
Pavel
On Mi, 15.04.20 07:10, Pavel Raiskup (praiskup@redhat.com) wrote:
On Tuesday, April 14, 2020 9:23:27 PM CEST Ben Cotton wrote:
https://fedoraproject.org/wiki/Changes/systemd-resolved
== Summary ==
Enable systemd-resolved by default. ...
We had serious headaches because racy systemd-resolved got enabled for some unknown reasons on copr builders before (and my host as well, for example). I filled bug [1] and then I was told that nobody tests/maintains systemd-resolved... why on earth I'd opt-in to use that.
Can we fix the bug, so the generated resolv.conf pointing to local server does work _immediately_ inside mock, and isn't it port-triggered (or what it is :-))?
For the sake of the archives, let's follow-up the discussion on this specific issue on the bug report, instead of the ML.
Lennart
-- Lennart Poettering, Berlin
On 4/14/20 9:23 PM, Ben Cotton wrote:
=== Multicast DNS ===
systemd-resolved's multicast DNS support conflicts with Avahi. Per recommendation from the systemd developers, we will change the default value of this setting in Fedora from the upstream default `MulticastDNS=yes` to `MulticastDNS=resolve`. Multicast DNS resolving will be enabled, but responding will be disabled. This will require adding a new systemd build option to control the default value of the MulticastDNS setting, similar to the existing `default-dnssec` and `default-dns-over-tls` build options.
Hi Michael,
would you mind telling me more about the change's impact on MDNS support provided by Avahi and nss-mdns package, since you mention Avahi conflicts with systemd-resolved?
CUPS relies on Avahi/nss-mdns for its MDNS functionality (resolving MDNS addresses, browsing services, registering services...) because it is essential for automatic printer discovery and driverless printing functionality, which is supported by devices since 2010.
According https://github.com/apple/cups/issues/5452 systemd-resolved was not the replacement for Avahi at the time, so is there a way how to make Avahi work with the change, hopefully as default?
Non-working MDNS via Avahi would put us back in <2010.
Thank you in advance for response!
Zdenek
On Wed, Apr 15, 2020 at 07:42:12AM +0200, Zdenek Dohnal wrote:
On 4/14/20 9:23 PM, Ben Cotton wrote:
=== Multicast DNS ===
systemd-resolved's multicast DNS support conflicts with Avahi. Per recommendation from the systemd developers, we will change the default value of this setting in Fedora from the upstream default `MulticastDNS=yes` to `MulticastDNS=resolve`. Multicast DNS resolving will be enabled, but responding will be disabled. This will require adding a new systemd build option to control the default value of the MulticastDNS setting, similar to the existing `default-dnssec` and `default-dns-over-tls` build options.
Hi Michael,
would you mind telling me more about the change's impact on MDNS support provided by Avahi and nss-mdns package, since you mention Avahi conflicts with systemd-resolved?
It conflicts in the default settings of "yes". With that changed to "resolve" as described, there should be no interference with Avahi. resolved will function as a client, without announcing hostnames or services.
Zbyszek
* Ben Cotton:
Enable systemd-resolved by default. glibc will perform name resolution using nss-resolve rather than nss-dns.
Is this intended for Fedora Server and others as well, or just Workstation? I assume it's for everywhere.
systemd-resolved has been enabled by default in Ubuntu since Ubuntu 16.10, but please note we are doing this differently than Ubuntu has. Ubuntu does not use nss-resolve. Instead, Ubuntu uses the traditional nss-dns provided by glibc upstream, so glibc on Ubuntu continues to read /etc/resolv.conf, as is traditional. This extra step is not useful and not recommended by upstream. We want to follow upstream recommendations in using nss-resolve instead.
I cannot find documentation of the systemd stub resolver behavior: how it handles search list processing, and how it decides which upstream name servers to query.
Most Kubernetes/OKD clusters assume that both single-label and multi-label query names are forwarded over DNS (contrary to ICANN recommendations), and that DNS servers are processed in listed order for all queries (no parallel queries, no randomized server selection). If systemd-resolved behaves differently, it can make Fedora incompatible with Kubernetes clusters. (This is one reason why glibc still not follows the ICANN recommendations.)
With the Ubuntu approach, search list processing still resides within glibc, so their Kubernetes experience would not necessarily match ours.
=== Split DNS ===
When systemd-resolved is enabled, users who use multiple VPNs at the same time will notice that DNS requests are now sent to the correct DNS server by default. Previously, this scenario would result in embarrassing "DNS leaks" and, depending on the order that the VPN connections were established, possible failure to resolve private resources. These scenarios will now work properly. For example, consider the scenario of Distrustful Denise, who (quite reasonably) does not trust her ISP. Denise uses a public VPN service, public-vpn.example.com, to hide her internet activity from her ISP, but she also needs to use her employer's corporate VPN, corporation.example.com, in order to access internal company resources while working from home. Using the Network panel in System Settings, Denise has configured her employer's VPN to "use this connection only for resources on its network." Distrustful Denise expects that her employer's VPN will receive all traffic for corporation.example.com, and no other traffic. And while this mostly works in Fedora 32, she discovers that it does not work properly for DNS:
Will Fedora treat such cross-VPN leaks as security bugs going and forward and fix them? (I suspect it would require constant work, and lots of it.)
Is this expected to work with the Red Hat VPN out of the box, or do we have to disable all this and use a custom configuration? Has this been discussed with Infosec? It looks like this will break their DNS sinkholing for domains such as REDHAT[.]CO (not COM).
== Scope ==
- Other developers: This change requires coordination with the systemd
and authselect maintainers.
It may also need to changes to Postfix (for DANE/TLSA handling in particular, depending on the level of DNSSEC support, see below).
libvirt may need changes as well, for its internal DNS relay.
The manual page resolv.conf(5) needs to be updated.
Since libnss_resolve.so.2 is still linked against libpthread and we might not be able to remove libpthread from glibc as a separate library in time for Fedora 33, some applications may have to start linking against libpthread explicitly even though they do not actually need it. (This is due to a long-standing limitation of loading libpthread via dlopen.)
=== DNSSEC ===
systemd-resolved's DNSSEC support is known to cause compatibility problems with certain network access points. Per recommendation from the systemd developers, we will change the default value of this setting in Fedora from the upstream default `DNSSEC=allow-downgrade` to `DNSSEC=no` by building systemd with the build option `-Ddefault-dnssec=no`. The upstream default attempts to use DNSSEC if it is working, but automatically disable it otherwise, allowing man-in-the-middle attackers to disable DNSSEC. Sadly, even the allow-downgrade setting suffers known compatibility problems. Because Fedora is not prepared to handle an influx of DNSSEC-related bug reports, we will disable this feature altogether. We anticipate that enabling DNSSEC by default will not be possible in the foreseeable future, or perhaps ever. Instead, enabling DNS over TLS (when supported by the DNS server) seems likely in the near future.
Will the built-in DNS server still support DNSSEC without validation, passing through the records if they are requested by the client over the DNS interface? The section above is not clear.
Thanks, Florian
On 15/04/2020 09:08, Florian Weimer wrote:
I cannot find documentation of the systemd stub resolver behavior: how it handles search list processing, and how it decides which upstream name servers to query.
As I understand the terminology the "stub resolver" in systemd-resolved refers to the thing that listens on 127.0.0.53 and that won't do anything clever with single label queries because it will expect it is answering DNS queries - that is exactly the way that Ubuntu uses it.
Most Kubernetes/OKD clusters assume that both single-label and multi-label query names are forwarded over DNS (contrary to ICANN recommendations), and that DNS servers are processed in listed order for all queries (no parallel queries, no randomized server selection). If systemd-resolved behaves differently, it can make Fedora incompatible with Kubernetes clusters. (This is one reason why glibc still not follows the ICANN recommendations.)
When accessed via nss-resolve single label queries will be subject to search list processing but I don't believe multi label ones will.
Each interface can have DNS servers and search domains as well as there being global ones. There can also be search domains with a prefix of "~" which force queries for that domain to be sent to a specific interface (this is how the VPN thing works).
I'm not sure what happens if there are multiple interfaces with no specific routing but I think it may try them all?
It definitely doesn't use the servers on an interface in any particular order, but I didn't think glibc did either? It has a current server but will sometimes change it based on how well they are responding or something.
Is this expected to work with the Red Hat VPN out of the box, or do we have to disable all this and use a custom configuration? Has this been discussed with Infosec? It looks like this will break their DNS sinkholing for domains such as REDHAT[.]CO (not COM).
I think so long as the VPN interface has ~redhat.co in it's search list then queries for that domain will be forced to the servers for that interface if that's what is required?
Will the built-in DNS server still support DNSSEC without validation, passing through the records if they are requested by the client over the DNS interface? The section above is not clear.
I don't think so no. I happen to have a query to hand which fails validation due to a bug in systemd-resolved and I get SERVFAIL when querying 127.0.0.53 even with +cdflag on the dig command.
Tom
On 15/04/2020 09:29, Tom Hughes via devel wrote:
I'm not sure what happens if there are multiple interfaces with no specific routing but I think it may try them all?
Found the documentation now - it does try them all. Full details from systemd-resolved(8) are:
Lookup requests are routed to the available DNS servers, LLMNR and MulticastDNS interfaces according to the following rules:
· Lookups for the special hostname "localhost" are never routed to the network. (A few other, special domains are handled the same way.)
· Single-label names are routed to all local interfaces capable of IP multicasting, using the LLMNR protocol. Lookups for IPv4 addresses are only sent via LLMNR on IPv4, and lookups for IPv6 addresses are only sent via LLMNR on IPv6. Lookups for the locally configured host name and the "_gateway" host name are never routed to LLMNR.
· Multi-label names with the domain suffix ".local" are routed to all local interfaces capable of IP multicasting, using the MulticastDNS protocol. As with LLMNR IPv4 address lookups are sent via IPv4 and IPv6 address lookups are sent via IPv6.
· Other multi-label names are routed to all local interfaces that have a DNS server configured, plus the globally configured DNS server if there is one. Address lookups from the link-local address range are never routed to DNS. Note that by default lookups for domains with the ".local" suffix are not routed to DNS servers, unless the domain is specified explicitly as routing or search domain for the DNS server and interface. This means that on networks where the ".local" domain is defined in a site-specific DNS server, explicit search or routing domains need to be configured to make lookups within this DNS domain work. Note that today it's generally recommended to avoid defining ".local" in a DNS server, as RFC6762[3] reserves this domain for exclusive MulticastDNS use.
If lookups are routed to multiple interfaces, the first successful response is returned (thus effectively merging the lookup zones on all matching interfaces). If the lookup failed on all interfaces, the last failing response is returned.
Routing of lookups may be influenced by configuring per-interface domain names and other settings. See systemd.network(5) and resolvectl(1) for details. The following query routing logic applies for unicast DNS traffic:
· If a name to look up matches (that is: is equal to or has as suffix) any of the configured search or route-only domains of any link (or the globally configured DNS settings), the "best matching" search/route-only domain is determined: the matching one with the most labels. The query is then sent to all DNS servers of any links or the globally configured DNS servers associated with this "best matching" search/route-only domain. (Note that more than one link might have this same "best matching" search/route-only domain configured, in which case the query is sent to all of them in parallel).
· If a query does not match any configured search/route-only domain (neither per-link nor global), it is sent to all DNS servers that are configured on links with the "DNS default route" option set, as well as the globally configured DNS server.
· If there is no link configured as "DNS default route" and no global DNS server configured, the compiled-in fallback DNS server is used.
· Otherwise the query is failed as no suitable DNS servers could be determined.
The "DNS default route" option is a boolean setting configurable with resolvectl or in .network files. If not set, it is implicitly determined based on the configured DNS domains for a link: if there's any route-only domain (not matching "~.") it defaults to false, otherwise to true.
Effectively this means: in order to preferably route all DNS queries not explicitly matched by search/route-only domain configuration to a specific link, configure a "~." route-only domain on it. This will ensure that other links will not be considered for the queries (unless they too carry such a route-only domain). In order to route all such DNS queries to a specific link only in case no other link is preferable, then set the "DNS default route" option for the link to true, and do not configure a "~." route-only domain on it. Finally, in order to ensure that a specific link never receives any DNS traffic not matching any of its configured search/route-only domains, set the "DNS default route" option for it to false.
Tom
* Tom Hughes:
· Single-label names are routed to all local interfaces capable of IP multicasting, using the LLMNR protocol. Lookups for IPv4 addresses are only sent via LLMNR on IPv4, and lookups for IPv6 addresses are only sent via LLMNR on IPv6. Lookups for the locally configured host name and the "_gateway" host name are never routed to LLMNR.
Routing of lookups may be influenced by configuring per-interface domain names and other settings. See systemd.network(5) and resolvectl(1) for details. The following query routing logic applies for unicast DNS traffic: · If a name to look up matches (that is: is equal to or has as suffix) any of the configured search or route-only domains of any link (or the globally configured DNS settings), the "best matching" search/route-only domain is determined: the matching one with the most labels. The query is then sent to all DNS servers of any links or the globally configured DNS servers associated with this "best matching" search/route-only domain. (Note that more than one link might have this same "best matching" search/route-only domain configured, in which case the query is sent to all of them in parallel). · If a query does not match any configured search/route-only domain (neither per-link nor global), it is sent to all DNS servers that are configured on links with the "DNS default route" option set, as well as the globally configured DNS server. · If there is no link configured as "DNS default route" and no global DNS server configured, the compiled-in fallback DNS server is used. · Otherwise the query is failed as no suitable DNS servers could be determined.
Thanks. Does this mean that no search list processing happens, for neither single-label names (per for the first paragraph), nor for multi-label names (per the routing description)? Or is this process described in some other context?
Thanks, Florian
On 15/04/2020 09:53, Florian Weimer wrote:
Thanks. Does this mean that no search list processing happens, for neither single-label names (per for the first paragraph), nor for multi-label names (per the routing description)? Or is this process described in some other context?
That text doesn't mention search list processing because I assume it is describing what happens to each name after any domain from the search list has been added but resolved.conf(5) says:
Domains= A space-separated list of domains. These domains are used as search suffixes when resolving single-label host names (domain names which contain no dot), in order to qualify them into fully-qualified domain names (FQDNs). Search domains are strictly processed in the order they are specified, until the name with the suffix appended is found. For compatibility reasons, if this setting is not specified, the search domains listed in /etc/resolv.conf are used instead, if that file exists and any domains are configured in it. This setting defaults to the empty list.
Specified domain names may optionally be prefixed with "~". In this case they do not define a search path, but preferably direct DNS queries for the indicated domains to the DNS servers configured with the system DNS= setting (see above), in case additional, suitable per-link DNS servers are known. If no per-link DNS servers are known using the "~" syntax has no effect. Use the construct "~." (which is composed of "~" to indicate a routing domain and "." to indicate the DNS root domain that is the implied suffix of all DNS domains) to use the system DNS server defined with DNS= preferably for all domains.
That is about the global option but the same logic applies to per-interface search lists I assume.
Tom
On Mi, 15.04.20 10:53, Florian Weimer (fweimer@redhat.com) wrote:
Thanks. Does this mean that no search list processing happens, for neither single-label names (per for the first paragraph), nor for multi-label names (per the routing description)? Or is this process described in some other context?
We never suffix search domains to multi-label names. We do traditional suffixing however for single-label names.
Lennart
-- Lennart Poettering, Berlin
* Tom Hughes:
On 15/04/2020 09:08, Florian Weimer wrote:
I cannot find documentation of the systemd stub resolver behavior: how it handles search list processing, and how it decides which upstream name servers to query.
As I understand the terminology the "stub resolver" in systemd-resolved refers to the thing that listens on 127.0.0.53 and that won't do anything clever with single label queries because it will expect it is answering DNS queries - that is exactly the way that Ubuntu uses it.
Well, the whole thing is a stub resolver externally, even if it doesn't speak DNS with applications. It has to perform upstream server selection and search list processing in some way. The search list processing turns into the responsibility of the local client (application) only on the DNS interface.
To be clear here, glibc does not perform generic search list processing for all NSS modules, that behavior is specific to nss_dns. Any DNS lookalike that aims to replace nss_dns has to implement some search list processing from scratch.
Most Kubernetes/OKD clusters assume that both single-label and multi-label query names are forwarded over DNS (contrary to ICANN recommendations), and that DNS servers are processed in listed order for all queries (no parallel queries, no randomized server selection). If systemd-resolved behaves differently, it can make Fedora incompatible with Kubernetes clusters. (This is one reason why glibc still not follows the ICANN recommendations.)
When accessed via nss-resolve single label queries will be subject to search list processing but I don't believe multi label ones will.
That is reasonable (and matches ICANN recommendations), but it looks like it will break many Kubernetes deployments unfortunatelly.
It definitely doesn't use the servers on an interface in any particular order, but I didn't think glibc did either? It has a current server but will sometimes change it based on how well they are responding or something.
No, glibc does not do this by default. (There is a non-default “rotate” option that provides some randomness in the server selection.)
Actually, Fedora's current NetworkManager/OpenVPN behavior depends on this because it lists the original system name server in /etc/resolv.conf after the ones received over OpenVPN. This means that with randomized/parallel queries, you might try to resolve internal domains over the public Internet, and get unwanted/unusable result.
The second Kubernetes issue I worry about [1] is that the CoreDNS name server is installed first, and it does additional rule-based processing for in-cluster names. External DNS servers are listed later. Parallel queries and random server selection could bypass the CoreDNS service for queries that need to be handled by it.
[1] I have no real Kubernetes experience, I'm relaying (perhaps imperfectly) the results of discussions I had with people who actually know something about it.
Is this expected to work with the Red Hat VPN out of the box, or do we have to disable all this and use a custom configuration? Has this been discussed with Infosec? It looks like this will break their DNS sinkholing for domains such as REDHAT[.]CO (not COM).
I think so long as the VPN interface has ~redhat.co in it's search list then queries for that domain will be forced to the servers for that interface if that's what is required?
Does OpenVPN log the list of these domains somewhere? Or do they have to be configured manually?
Will the built-in DNS server still support DNSSEC without validation, passing through the records if they are requested by the client over the DNS interface? The section above is not clear.
I don't think so no. I happen to have a query to hand which fails validation due to a bug in systemd-resolved and I get SERVFAIL when querying 127.0.0.53 even with +cdflag on the dig command.
Ugh. That will have to be fixed, otherwise it will break DANE/TLSA and other DNSSEC-mandatory functionality on upgrades: the system used to have a DNSSEC-clean path to the outside world, and after the switch to systemd-resolved, it won't.
Thanks, Florian
On 15/04/2020 09:48, Florian Weimer wrote:
Is this expected to work with the Red Hat VPN out of the box, or do we have to disable all this and use a custom configuration? Has this been discussed with Infosec? It looks like this will break their DNS sinkholing for domains such as REDHAT[.]CO (not COM).
I think so long as the VPN interface has ~redhat.co in it's search list then queries for that domain will be forced to the servers for that interface if that's what is required?
Does OpenVPN log the list of these domains somewhere? Or do they have to be configured manually?
I think a lot will depend on exactly how it is setup. My openvpn setups on linux tend to use an up script to configure DNS things so my VPN to home just has an up script that does:
resolvectl dns $1 172.16.15.1 172.16.15.2 172.16.15.5 resolvectl domain $1 ~compton.nu ~15.16.172.in-addr.arpa ~d.b.0.0.0.b.8.0.1.0.0.2.ip6.arpa resolvectl flush-caches
To set the DNS servers on the interface and force routing of certain domains to it.
I'm not sure OpenVPN itself has any way to do DNS setup automatically on linux but the NetworkManager integration might, I don't use that though.
Tom
* Tom Hughes:
I'm not sure OpenVPN itself has any way to do DNS setup automatically on linux but the NetworkManager integration might, I don't use that though.
Yes, the NetworkManager integration seems to mirror what happens on Windows, by looking at the foreign_option_* environment variables and processing those which start with "dhcp-option ".
Not sure if that's compatible with the new split DNS model because VPN1 could simply start pushing longer names in the scope of VPN2, thus hijacking internal traffic there (and this sort of hijacking is exactly what a DNS sinkhole against typosquatting would need).
Thanks, Florian
On Wed, Apr 15, 2020 at 1:38 pm, Florian Weimer fweimer@redhat.com wrote:
Not sure if that's compatible with the new split DNS model because VPN1 could simply start pushing longer names in the scope of VPN2, thus hijacking internal traffic there (and this sort of hijacking is exactly what a DNS sinkhole against typosquatting would need).
You deserve bonus points for thinking like an attacker and exploring the security model, but let's assume the configured VPNs are trusted. Otherwise the user is screwed no matter what. ;)
On Wed, 2020-04-15 at 10:02 -0500, Michael Catanzaro wrote:
On Wed, Apr 15, 2020 at 1:38 pm, Florian Weimer fweimer@redhat.com wrote:
Not sure if that's compatible with the new split DNS model because VPN1 could simply start pushing longer names in the scope of VPN2, thus hijacking internal traffic there (and this sort of hijacking is exactly what a DNS sinkhole against typosquatting would need).
You deserve bonus points for thinking like an attacker and exploring the security model, but let's assume the configured VPNs are trusted. Otherwise the user is screwed no matter what. ;)
Trusted for what? I would expect corporate VPNs doing such tricks to monitor the user's internet traffic. Which does not mean the user is fully screwed with such VPN if he for example uses hardcoded configuration of a caching nameserver.
On Thu, Apr 16, 2020 at 4:18 pm, Tomas Mraz tmraz@redhat.com wrote:
Trusted for what? I would expect corporate VPNs doing such tricks to monitor the user's internet traffic. Which does not mean the user is fully screwed with such VPN if he for example uses hardcoded configuration of a caching nameserver.
In Florian's scenario, one of the VPNs is actively malicious. E.g. public-vpn.example.com tries to hijack DNS for subdomain.corporation.example.com. It might actually be a realistic attack scenario, but it's not something we should attempt to mitigate.
Anyway this goes both ways. As explained many times already, without systemd-resolved, the VPN you connect to first gets all the DNS queries currently. Normally users connect to public VPN first, then corporate VPN second. That's broken. Splitting the DNS is just the right thing to do. If you want the corporate VPN to see everything, then do not check "use this VPN only for resources on its network" and it will get everything (but then it needs to have capacity to really handle everything!).
Michael
* Tomas Mraz:
On Wed, 2020-04-15 at 10:02 -0500, Michael Catanzaro wrote:
On Wed, Apr 15, 2020 at 1:38 pm, Florian Weimer fweimer@redhat.com wrote:
Not sure if that's compatible with the new split DNS model because VPN1 could simply start pushing longer names in the scope of VPN2, thus hijacking internal traffic there (and this sort of hijacking is exactly what a DNS sinkhole against typosquatting would need).
You deserve bonus points for thinking like an attacker and exploring the security model, but let's assume the configured VPNs are trusted. Otherwise the user is screwed no matter what. ;)
Trusted for what? I would expect corporate VPNs doing such tricks to monitor the user's internet traffic. Which does not mean the user is fully screwed with such VPN if he for example uses hardcoded configuration of a caching nameserver.
Yes, what I described was given as a motivation for this change.
I find the response puzzling. I would definitely like to see greater robustness to hostile networks, but it is a lot of work. Really a lot. Lots of code to review, and quite a few shell scripts as well.
Thanks, Florian
On Wed, Apr 15, 2020 at 10:48 am, Florian Weimer fweimer@redhat.com wrote:
The second Kubernetes issue I worry about [1] is that the CoreDNS name server is installed first, and it does additional rule-based processing for in-cluster names. External DNS servers are listed later. Parallel queries and random server selection could bypass the CoreDNS service for queries that need to be handled by it.
Hm, CoreDNS might need to construct its own nss module, or you might need to use /etc/resolv.conf in "mode 1" or "mode 3" described by Lennart. (Or disable systemd-resolved, but that shouldn't be necessary.) We'll default to Lennart's "mode 2" so it sounds like that might be a problem indeed.
Does OpenVPN log the list of these domains somewhere? Or do they have to be configured manually?
This managed by NetworkManager and systemd-resolved. You can inspect with 'resolvectl status'. I don't think OpenVPN knows anything about it.
Ugh. That will have to be fixed, otherwise it will break DANE/TLSA and other DNSSEC-mandatory functionality on upgrades: the system used to have a DNSSEC-clean path to the outside world, and after the switch to systemd-resolved, it won't.
I think that, if you need DNSSEC, you will just need to enable it manually. I think >99% of users won't need to do this, and it's a one-line config file change for power users who do need it, just edit /etc/systemd/resolved.conf and then restart systemd-resolved service. Problem is that DNSSEC is just not safe to enable by default. :(
* Michael Catanzaro:
On Wed, Apr 15, 2020 at 10:48 am, Florian Weimer fweimer@redhat.com wrote:
The second Kubernetes issue I worry about [1] is that the CoreDNS name server is installed first, and it does additional rule-based processing for in-cluster names. External DNS servers are listed later. Parallel queries and random server selection could bypass the CoreDNS service for queries that need to be handled by it.
Hm, CoreDNS might need to construct its own nss module,
This is not possible. You cannot realistically inject binary code into the container (see the fun with GPU userspace driver parts).
or you might need to use /etc/resolv.conf in "mode 1" or "mode 3" described by Lennart. (Or disable systemd-resolved, but that shouldn't be necessary.) We'll default to Lennart's "mode 2" so it sounds like that might be a problem indeed.
Yeah.
Does OpenVPN log the list of these domains somewhere? Or do they have to be configured manually?
This managed by NetworkManager and systemd-resolved. You can inspect with 'resolvectl status'. I don't think OpenVPN knows anything about it.
As explained elsewhere, NetworkManager-openvpn extracts the search list from OpenVPN parameters, passes that to NetworkManager, which I expect will pass ito to systemd-resolved in the future.
Ugh. That will have to be fixed, otherwise it will break DANE/TLSA and other DNSSEC-mandatory functionality on upgrades: the system used to have a DNSSEC-clean path to the outside world, and after the switch to systemd-resolved, it won't.
I think that, if you need DNSSEC, you will just need to enable it manually. I think >99% of users won't need to do this, and it's a one-line config file change for power users who do need it, just edit /etc/systemd/resolved.conf and then restart systemd-resolved service. Problem is that DNSSEC is just not safe to enable by default. :(
See my message to Lennart about separate DO/CD query caching.
My point is that these users *have* enabled DNSSEC in their infrastructure, and we break what they have during an update (assuming that DNSSEC=off means that systemd-resolved turns DNSSEC-unware, rather than just disabling validation).
Thanks, Florian
On Do, 16.04.20 12:49, Florian Weimer (fweimer@redhat.com) wrote:
As explained elsewhere, NetworkManager-openvpn extracts the search list from OpenVPN parameters, passes that to NetworkManager, which I expect will pass ito to systemd-resolved in the future.
Ugh. That will have to be fixed, otherwise it will break DANE/TLSA and other DNSSEC-mandatory functionality on upgrades: the system used to have a DNSSEC-clean path to the outside world, and after the switch to systemd-resolved, it won't.
I think that, if you need DNSSEC, you will just need to enable it manually. I think >99% of users won't need to do this, and it's a one-line config file change for power users who do need it, just edit /etc/systemd/resolved.conf and then restart systemd-resolved service. Problem is that DNSSEC is just not safe to enable by default. :(
See my message to Lennart about separate DO/CD query caching.
My point is that these users *have* enabled DNSSEC in their infrastructure, and we break what they have during an update (assuming that DNSSEC=off means that systemd-resolved turns DNSSEC-unware, rather than just disabling validation).
Maybe a safer bet might be to leave resolved off during upgrades on the server edition?
I don't think we can reliably determine whether people have deployed things in a way that relies on /etc/resolv.conf only listing a fully blown DNS server or who are fine with it being a more restricted stub like systemd-resolved.
I'd claim it's reasonably safe to declare that on workstations having a restrictive stub between local clients and a real DNS server is OK, but maybe for servers we don't want to make such a claim, dunno, and just enable this for newly deployed stuff but not on upgraded stuff.
Lennart
-- Lennart Poettering, Berlin
* Lennart Poettering:
On Do, 16.04.20 12:49, Florian Weimer (fweimer@redhat.com) wrote:
As explained elsewhere, NetworkManager-openvpn extracts the search list from OpenVPN parameters, passes that to NetworkManager, which I expect will pass ito to systemd-resolved in the future.
Ugh. That will have to be fixed, otherwise it will break DANE/TLSA and other DNSSEC-mandatory functionality on upgrades: the system used to have a DNSSEC-clean path to the outside world, and after the switch to systemd-resolved, it won't.
I think that, if you need DNSSEC, you will just need to enable it manually. I think >99% of users won't need to do this, and it's a one-line config file change for power users who do need it, just edit /etc/systemd/resolved.conf and then restart systemd-resolved service. Problem is that DNSSEC is just not safe to enable by default. :(
See my message to Lennart about separate DO/CD query caching.
My point is that these users *have* enabled DNSSEC in their infrastructure, and we break what they have during an update (assuming that DNSSEC=off means that systemd-resolved turns DNSSEC-unware, rather than just disabling validation).
Maybe a safer bet might be to leave resolved off during upgrades on the server edition?
A Fedora upgrade can also mean reprovision from start via kickstart/ansible, so I assume that this isn't a proper mitigation, sorry.
I don't think we can reliably determine whether people have deployed things in a way that relies on /etc/resolv.conf only listing a fully blown DNS server or who are fine with it being a more restricted stub like systemd-resolved.
Unfortunately, I see something similar to what Tom Hughes reported earlier: dig +dnssec responses are not even correctly formatted. The CD query flag is not handled, either. The AD bit is not set on validated responses. I also see some really strange stability issues. It seems that resolved is incorrectly blacklisting upstream servers with an incompatible-server error after a validation failure.
This is with systemd-245.4-1.fc33.x86_64 in rawhide. Is this approximately what will land in Fedora 33? Or is this old code, long replaced upstream?
Thanks, Florian
On Do, 16.04.20 17:14, Florian Weimer (fweimer@redhat.com) wrote:
I don't think we can reliably determine whether people have deployed things in a way that relies on /etc/resolv.conf only listing a fully blown DNS server or who are fine with it being a more restricted stub like systemd-resolved.
Unfortunately, I see something similar to what Tom Hughes reported earlier: dig +dnssec responses are not even correctly formatted. The CD query flag is not handled, either. The AD bit is not set on validated responses. I also see some really strange stability issues. It seems that resolved is incorrectly blacklisting upstream servers with an incompatible-server error after a validation failure.
Again, we do not support DNSSEC from client to the stub. If you set CD we'll return NOTIMP as rcode, indicating that. We do not implement a full DNS server, but just enough for local stub clients (such as the one implemented in glibc or Java) to work. If you want the full DNSSEC client stuff, talk directly to the upstream DNS server.
We set AD only if we managed to authenticate ourselves, which can either be via DNSSEC if that's enabled to the upstream DNS server. We also set it for hosts we read from /etc/hosts (i.e. a source owned by root). If you saw incompatible server this looks like you left DNSSEC on between resolved and upstream DNS server? Again, this is not what we intend to do in Fedora.
Lennart
-- Lennart Poettering, Berlin
* Lennart Poettering:
On Do, 16.04.20 17:14, Florian Weimer (fweimer@redhat.com) wrote:
I don't think we can reliably determine whether people have deployed things in a way that relies on /etc/resolv.conf only listing a fully blown DNS server or who are fine with it being a more restricted stub like systemd-resolved.
Unfortunately, I see something similar to what Tom Hughes reported earlier: dig +dnssec responses are not even correctly formatted. The CD query flag is not handled, either. The AD bit is not set on validated responses. I also see some really strange stability issues. It seems that resolved is incorrectly blacklisting upstream servers with an incompatible-server error after a validation failure.
Again, we do not support DNSSEC from client to the stub.
I don't think this change is ready for Fedora, then.
If you set CD we'll return NOTIMP as rcode, indicating that. We do not implement a full DNS server, but just enough for local stub clients (such as the one implemented in glibc or Java) to work.
Sorry? RES_USE_DNSSEC is part of the glibc stub resolver. It does not work anymore.
The libunbound validator is broken by this, too.
If you want the full DNSSEC client stuff, talk directly to the upstream DNS server.
How? The address is no longer in /etc/resolv.conf. According to the change proposal, this also endangers Denise, who relies on the request routing in systemd-resolved.
Thanks, Florian
On Thu, Apr 16, 2020 at 8:07 pm, Florian Weimer fweimer@redhat.com wrote:
I don't think this change is ready for Fedora, then.
Florian, isn't this a quite specialized use-case of the sort where you can simply write your own /etc/resolv.conf and let resolved consume that? Or just disable resolved?
How? The address is no longer in /etc/resolv.conf. According to the change proposal, this also endangers Denise, who relies on the request routing in systemd-resolved.
Well, I guess you're right, you can't have all of the above. But how many users really need to use Postfix or something else that needs DNSSEC on a machine with multiple VPNs enabled...?
I think resolved is a sane default for Workstation users. And for Server users, if you need more control, you can just create your own /etc/resolv.conf and things will work as before, but you lose the split DNS feature. That's OK, right?
Once upon a time, Lennart Poettering mzerqung@0pointer.de said:
Again, we do not support DNSSEC from client to the stub. If you set CD we'll return NOTIMP as rcode, indicating that. We do not implement a full DNS server, but just enough for local stub clients (such as the one implemented in glibc or Java) to work. If you want the full DNSSEC client stuff, talk directly to the upstream DNS server.
If you want to go in /etc/resolv.conf, you need to be a full resolver. There's no telling what programs expect to be able to talk the full DNS protocol to the "nameserver" lines from /etc/resolv.conf (for example, the perl Net::DNS module gets its servers from there by default, so all kinds of perl scripts could break). dnsmasq defaults to using resolvers from /etc/resolv.conf too IIRC.
If you want to be a resolver, be an actual resolver, and in 2020, that includes implementing EDNS0, DNSSEC, etc.
On Do, 16.04.20 19:53, Chris Adams (linux@cmadams.net) wrote:
Once upon a time, Lennart Poettering mzerqung@0pointer.de said:
Again, we do not support DNSSEC from client to the stub. If you set CD we'll return NOTIMP as rcode, indicating that. We do not implement a full DNS server, but just enough for local stub clients (such as the one implemented in glibc or Java) to work. If you want the full DNSSEC client stuff, talk directly to the upstream DNS server.
If you want to go in /etc/resolv.conf, you need to be a full resolver. There's no telling what programs expect to be able to talk the full DNS protocol to the "nameserver" lines from /etc/resolv.conf (for example, the perl Net::DNS module gets its servers from there by default, so all kinds of perl scripts could break). dnsmasq defaults to using resolvers from /etc/resolv.conf too IIRC.
If you want to be a resolver, be an actual resolver, and in 2020, that includes implementing EDNS0, DNSSEC, etc.
resolved implements edns0 just fine.
And then there's DNSSEC support and DNSSEC support.
There's DNSSEC support as in "AD bit". Which is a silly thing if you ask me. It is a bit you can set in the DNS response which tells the client the data was "authenticated". It has no value however, because anyone can claim that, it's not signed or nothing. This is the DNSSEC support glibc implementss. It's silly, fake security.
And then there's DNSSEC support as in "full validation". In this case the client checks all the signatures, and the AD bit isn't relevant. This one actually is useful, but isn't what glibc implement. Few clients implement that, and even fewer are deployed that do this regularly. (My guess is that most clients that do this are probably tools like "dig", that are used to debug this.)
The DNS servers in edge routers are awful at supporting either. i.e. the DNS servers you usually get informed about in DHCP leases are typically too crap at supporting either kind of DNSSEC (and that for a reason actually, these devices generally define their own private, local DNS names (e.g. "fritz.box"), which couldn't possibly be validated with DNSSEC, because they are made up and local.)
systemd-resolved implements the full validation when talking to its upstream servers. Because edge routers are so awful, we have trouble making this work reliably enough in the general case. In specific cases with a nice DNS server upstream everything is fine, but in the general case it's not fun. systemd-reslolved does not care for the "AD" bit stuff when talking to its upstream servers, because it's useless.
When clients talk to systemd-resolved's DNS stub they will get errors back if they themselves try to do the full validation, because we don't cache that information suitably. We return clean errors in this case, so that the client understands it cannot talk DNSSEC to us. We intend to implement the "AD" stuff however correctly for this, but this isn't tested much since pretty much noone except for a few DNS devs actually set this, hence there might be issues, which might be what Florian found. At least in our history of having been enabled in Ubuntu this came up only sporadingly.
I think it#s worth tweaking the "AD" support in the stub, but I also don't think it's really as important as people might think, because there are so few clients that actually want it, and conceptually it's kinda silly anyway (OK, admittedly, it's a bit less silly if a you trust the "AD" bit if it's only sent between a local client to a local system stub via the loopback device, but still...).
Lennart
-- Lennart Poettering, Berlin
Once upon a time, Lennart Poettering mzerqung@0pointer.de said:
The DNS servers in edge routers are awful at supporting either. i.e. the DNS servers you usually get informed about in DHCP leases are typically too crap at supporting either kind of DNSSEC (and that for a reason actually, these devices generally define their own private, local DNS names (e.g. "fritz.box"), which couldn't possibly be validated with DNSSEC, because they are made up and local.)
That might be true if you are just considering residential users with cheap gateways as your only use case (but even then, most of those run dnsmasq, which has gotten a lot better). However, there are lots of other use cases.
We intend to implement the "AD" stuff however correctly for this, but this isn't tested much since pretty much noone except for a few DNS devs actually set this, hence there might be issues, which might be what Florian found.
Advertising yourself as a "nameserver" in /etc/resolv.conf means you get to handle all the requests, including the ones you didn't think about or want to just dismiss as only of interest "a few DNS devs". That's the only standard way for software to find DNS servers to use for any purpose. Returning errors to clients for things you don't care about is basically useless, because they have no other way to get that information when actual DNS servers aren't in /etc/resolv.conf.
If systemd-resolved is not going to implement a standards-compliant DNS server (and not just "we return errors to things we don't care about", but actual current DNS standards), then it does not belong in /etc/resolv.conf. Listening on the system bus as an alternative, to implment gethostbyname/getaddrinfo/etc., is fine, but don't pretend to be an actual DNS server and go in /etc/resolv.conf.
On Fri, Apr 17, 2020 at 8:34 AM Chris Adams linux@cmadams.net wrote:
Once upon a time, Lennart Poettering mzerqung@0pointer.de said:
The DNS servers in edge routers are awful at supporting either. i.e. the DNS servers you usually get informed about in DHCP leases are typically too crap at supporting either kind of DNSSEC (and that for a reason actually, these devices generally define their own private, local DNS names (e.g. "fritz.box"), which couldn't possibly be validated with DNSSEC, because they are made up and local.)
That might be true if you are just considering residential users with cheap gateways as your only use case (but even then, most of those run dnsmasq, which has gotten a lot better). However, there are lots of other use cases.
We intend to implement the "AD" stuff however correctly for this, but this isn't tested much since pretty much noone except for a few DNS devs actually set this, hence there might be issues, which might be what Florian found.
Advertising yourself as a "nameserver" in /etc/resolv.conf means you get to handle all the requests, including the ones you didn't think about or want to just dismiss as only of interest "a few DNS devs". That's the only standard way for software to find DNS servers to use for any purpose. Returning errors to clients for things you don't care about is basically useless, because they have no other way to get that information when actual DNS servers aren't in /etc/resolv.conf.
If systemd-resolved is not going to implement a standards-compliant DNS server (and not just "we return errors to things we don't care about", but actual current DNS standards), then it does not belong in /etc/resolv.conf. Listening on the system bus as an alternative, to implment gethostbyname/getaddrinfo/etc., is fine, but don't pretend to be an actual DNS server and go in /etc/resolv.conf.
I tend to agree. Right now, a client-side resolving validator works on Fedora. This change proposal breaks that. Could resolved be extended to pass DNSSEC data through correctly so that client side validation will continue to work?
On Mi, 15.04.20 10:48, Florian Weimer (fweimer@redhat.com) wrote:
As I understand the terminology the "stub resolver" in systemd-resolved refers to the thing that listens on 127.0.0.53 and that won't do anything clever with single label queries because it will expect it is answering DNS queries - that is exactly the way that Ubuntu uses it.
Well, the whole thing is a stub resolver externally, even if it doesn't speak DNS with applications. It has to perform upstream server selection and search list processing in some way. The search list processing turns into the responsibility of the local client (application) only on the DNS interface.
Yes, resolved is a stub resolver only. And it is so regardless which interface you use to talk to it.
The local listener on 127.0.0.53:53 is not a fully featured DNS server either because of that. It does stuff real DNS servers don't do (such as LLMNR, merging in /etc/hosts and stuff, such as the scope routing and so on), and we will say NOTIMPL to a vareity of requests that you can send to a real DNS server but that make no sense for a local resolving stub.
Yes, this means if you have some special "dig" command line with all kinds of special options there's a good chance systemd-resolved will refuse your request, because that option is not implemented. It tries to make this discoverable as much as that is possible, by returned the most appropriate error (the error vocabulary isn't too verbose howerver) and by logging about it to syslog.
Note that when you talk to resolved via the DNS protocol it will not do search list processing for you, it leaves that to the client as this was always handled, in order to not do it twice. If you talk to resolved via the bus API (i.e. over nss-resolve or so) then it will to the search list stuff server side. Beaviour in these two cases is slightly different, as mentioned elsewhere, as resolved never leaks unqualified names to upstream servers as they are.
Actually, Fedora's current NetworkManager/OpenVPN behavior depends on this because it lists the original system name server in /etc/resolv.conf after the ones received over OpenVPN. This means that with randomized/parallel queries, you might try to resolve internal domains over the public Internet, and get unwanted/unusable result.
NM can tell resolved the full DNS info with per-interface info, so that resolved can merge the view of the world on each (see other mail)
I think so long as the VPN interface has ~redhat.co in it's search list then queries for that domain will be forced to the servers for that interface if that's what is required?
Does OpenVPN log the list of these domains somewhere? Or do they have to be configured manually?
Most other distributions (and even BSDs) provide a tool called "resolvconf" that various subsystems that provide DNS server info call when they learn some new DNS severs or DNS servers known go away. OpenVPN is a subsystem with DNS info, hence on most distros it will already be hooked up to "resolvconf" to provide the DNS data at the right moment. The various existing implementations of "resolvconf" usually take this info, sort it somehow and write it to /etc/resolv.conf. systemd now implements its own version of the tool which uses the information in a smarter way.
Long story short: if OpenVPN is configured like it already is on other distros then we should have excellent behaviour as resolved will learn about the DNS servers correctly with the right metadata we need to do our per-interface magic.
Will the built-in DNS server still support DNSSEC without validation, passing through the records if they are requested by the client over the DNS interface? The section above is not clear.
I don't think so no. I happen to have a query to hand which fails validation due to a bug in systemd-resolved and I get SERVFAIL when querying 127.0.0.53 even with +cdflag on the dig command.
Ugh. That will have to be fixed, otherwise it will break DANE/TLSA and other DNSSEC-mandatory functionality on upgrades: the system used to have a DNSSEC-clean path to the outside world, and after the switch to systemd-resolved, it won't.
Do we do DANE/TLSA by default anywhere in Fedora? I mean, we realistically cannot because DNSSEC sucks in most DNS servers...
But yeah, clients that want to do full DNSSEC need to talk directly to upstream DNS servers.
There's a feature request to switch to a special pass-through-proxy mode in resolved as soon as some of the more special DNSSEC features are turned on in an incoming DNS client packet on the stub. it's something to think about, but having been on the other end of DNSSEC I am a bit afraid it just makes things worse: i.e. writing a DNS client that can deal with servers that expose completely different behaviour depending on some flags in the headers is extremely hard. i.e. think about this: if you send a regular DNS packet to resolved it would process it natively, answer from its cache, do its LLMNR, mDNS, /etc/hosts magic and whatever else it does. But as soon as you turn on that DNSSEC bit resolved would just proxy through without any of this, without caching, any processing, without the other information sources. A client trying to make sense of such a server will be utterly confused since without the bit set it will be exposed to resolved's behaviour and quirks and its view of the world and with the bit set it will be exposed to some upstream server's view of the world and its quirks and behaviours, which are likely very very different... Hence so far my take on it was: if you want real, fully featured DNS with all weird, strange options then talk to upstream directly, don't bother with resolved's internal stub, it's not built for that.
Lennart
-- Lennart Poettering, Berlin
On Mi, 15.04.20 09:29, Fedora Development ML (devel@lists.fedoraproject.org) wrote:
Most Kubernetes/OKD clusters assume that both single-label and multi-label query names are forwarded over DNS (contrary to ICANN recommendations), and that DNS servers are processed in listed order for all queries (no parallel queries, no randomized server selection). If systemd-resolved behaves differently, it can make Fedora incompatible with Kubernetes clusters. (This is one reason why glibc still not follows the ICANN recommendations.)
When accessed via nss-resolve single label queries will be subject to search list processing but I don't believe multi label ones will.
correct.
Each interface can have DNS servers and search domains as well as there being global ones. There can also be search domains with a prefix of "~" which force queries for that domain to be sent to a specific interface (this is how the VPN thing works).
Small addition: the "~" thing will declare a domain as suitable for routing only. i.e. classic search domains are used both for suffixing and for routing, but those prefixed with "~" are used for routing only and not for suffixing.
Classic DNS doesn't know the routing concept, it thus only uses search domains for suffixing. We are hence using this information provided to us for more than it was originally intended, but this should be an OK thing to do.
I'm not sure what happens if there are multiple interfaces with no specific routing but I think it may try them all?
Exactly. If our routing info doesn't help us our logic is to route queries to all scopes in parallel.
Lennart
-- Lennart Poettering, Berlin
On Mi, 15.04.20 10:08, Florian Weimer (fweimer@redhat.com) wrote:
systemd-resolved has been enabled by default in Ubuntu since Ubuntu 16.10, but please note we are doing this differently than Ubuntu has. Ubuntu does not use nss-resolve. Instead, Ubuntu uses the traditional nss-dns provided by glibc upstream, so glibc on Ubuntu continues to read /etc/resolv.conf, as is traditional. This extra step is not useful and not recommended by upstream. We want to follow upstream recommendations in using nss-resolve instead.
I cannot find documentation of the systemd stub resolver behavior: how it handles search list processing, and how it decides which upstream name servers to query.
systemd-resolved generally manages DNS servers and search lists per interface, if possible. Basically, each interface that is up and has DNS info associated with it gets a "scope" associated with it. If there's global DNS configuration (i.e. read from /etc/resolv.conf if that's a regular file), then that's considered an additional "scope".
Whenever looking up a hostname systemd-resolved will match the name against all current scopes and issue queries on all matching ones in parallel. Scopes are considered "matching" if the hostname being looked up has a suffix of one of the search domains configured for the scope. If no scopes match given the search domains, the query is sent to all scopes in parallel.
This has some nice effects. Let's say you have one wifi interface and one vpn interface. The latter has "redhat.com" as search domain, the former has "fritz.box" as search domain configured. Now you try to look up a name "quux.redhat.com". Because the vpn has "redhat.com" as search domain the query will only be sent onto the vpn. If you look up "wuff.fritz.box" otoh it will be sent only to the local wifi. if you look up "systemd.io" otoh (i.e a domain for which no search domain is configured on any local interface), the query is sent to both scopes in parallel.
If a query is sent to multiple scopes in parallel then the the first positive answer wins. If there are no positive answers than the last negative one wins. This if you are connected to two different networks with private DNS zones, the view is merged.
Search domains are also used to suffix unqualified names. Note that resolved is a lot more strict than traditional DNS clients: single-label names are considered unqualified, and multi-label names are considered qualified. Unqualified names are never queried as they are on public DNS, but require a search domain of some kind to be configured with which they can be suffixed. Qualified names are never subject to search domain suffixing however.
Search domains/query routing are a bit more complex than what i describe above. See the man page for further details:
https://www.freedesktop.org/software/systemd/man/systemd-resolved.html#Proto...
resolved will start using the first DNS server configured for a scope, and stick to it for subsequent lookups. When an error is seen from it, it will switch to the next DNS server and then stick to that for future lookups. i.e. it basically "caches" whether the first lookup worked or not to speed up subsequent lookups. This means if the first listed DNS server is borked and doesn't respond you'll pay for it during the first lookup only, but all subsequent ones would use the working one immediately. This is of course quite different from nss-dns where on every single lookup things begin with the first DNS server configured.
resolved's behaviour on this hence assumes that all DNS servers configured on a scope are equivalent in the zones they provide. Or to turn this around, if you have multiple DNS servers offering different views of the world, then configure each set of them on a separate interface, so that they become different scopes and thus can be queried in parallel.
Most Kubernetes/OKD clusters assume that both single-label and multi-label query names are forwarded over DNS (contrary to ICANN recommendations), and that DNS servers are processed in listed order for all queries (no parallel queries, no randomized server selection).
resolved never sends single-label names to public DNS, under the assumption they have local meaning only, and should not leak onto the internet. They will end up on the Internet only if a search domain is defined that can be used to qualify them.
systemd-resolved starts with the first listed DNS server initially, but as mentioned if it fails it will switch to the next DNS server and then stay with it for subsequent lookups, unlike nss-dns which starts from the beginning on every single query.
If passing single-label un-suffixed to the DNS servers is mandatory for OKD/kubernetes, and if starting with the first DNS server on every single lookup is too, then such images should probably override /etc/resolv.conf with their own.
I am pretty sure that both these behaviours of kubernetes/OKD only make sense in a very specific environment, and are not approproate for general purpose systems.
systemd-resolved behaves differently, it can make Fedora incompatible with Kubernetes clusters. (This is one reason why glibc still not follows the ICANN recommendations.)
With the Ubuntu approach, search list processing still resides within glibc, so their Kubernetes experience would not necessarily match ours.
Hmm, containers usually don#t boot up systemd inside, and thus usually don't get resolved inside them anyway either. Hence my guess would be that ubuntu just copies in upstream DNS info directly to /etc/resolv.conf when starting the container, at which point what resolved does doesn't matter at all anymore.
Will Fedora treat such cross-VPN leaks as security bugs going and forward and fix them? (I suspect it would require constant work, and lots of it.)
Is this expected to work with the Red Hat VPN out of the box, or do we have to disable all this and use a custom configuration? Has this been discussed with Infosec? It looks like this will break their DNS sinkholing for domains such as REDHAT[.]CO (not COM).
If RH VPN configures "redhat.com" as search domain for their VPN then this means all redhat.com traffic is automatically pulled over to the VPN and will not be routed elsewhere anymore.
Will the built-in DNS server still support DNSSEC without validation, passing through the records if they are requested by the client over the DNS interface? The section above is not clear.
depends on the record type. A number of DNSSEC RR types are magic, and we'll refuse sending those to DNS servers that we don't think speak DNSSEC.
Lennart
-- Lennart Poettering, Berlin
On Wed, Apr 15, 2020 at 5:06 pm, Lennart Poettering mzerqung@0pointer.de wrote:
If RH VPN configures "redhat.com" as search domain for their VPN then this means all redhat.com traffic is automatically pulled over to the VPN and will not be routed elsewhere anymore.
In particular: current behavior is that redhat.com queries will leak to public DNS if the user connects to the public VPN first, which is the usual case, especially for anyone who configures public VPN to autoconnect on startup. So the status quo is really not secure at all. Yes, it will break the sinkholing for lookalike domains, but on balance I would say that getting the right DNS queries to the right servers is more important for security overall.
On Wed, Apr 15, 2020 at 10:08 am, Florian Weimer fweimer@redhat.com wrote:
- Ben Cotton:
Enable systemd-resolved by default. glibc will perform name resolution using nss-resolve rather than nss-dns.
Is this intended for Fedora Server and others as well, or just Workstation? I assume it's for everywhere.
Well I've proposed it for everywhere, because it seems desirable to have everywhere. But if Server or another product doesn't want it, we could do Workstation-only. It's already been approved by the Workstation WG.
Most Kubernetes/OKD clusters assume that both single-label and multi-label query names are forwarded over DNS (contrary to ICANN recommendations), and that DNS servers are processed in listed order for all queries (no parallel queries, no randomized server selection). If systemd-resolved behaves differently, it can make Fedora incompatible with Kubernetes clusters. (This is one reason why glibc still not follows the ICANN recommendations.)
With the Ubuntu approach, search list processing still resides within glibc, so their Kubernetes experience would not necessarily match ours.
Hm, it sounds like this is the main outstanding issue from this discussion. It is beyond my expertise. I guess we'll need a bug report where the relevant experts can figure out whether we need to change Kubernetes or systemd here.
You're right that continuing to use nss-dns would avoid any such problems while maintaining the other benefits of systemd-resolved. That could be a fallback plan if needed.
Will Fedora treat such cross-VPN leaks as security bugs going and forward and fix them? (I suspect it would require constant work, and lots of it.)
Well they were always serious security bugs, just bugs that we were unable to solve without a local resolver. I don't think it's going to require constant work. It's a problem already solved by NetworkManager as long as you use either dnsmasq and systemd-resolved for DNS.
Is this expected to work with the Red Hat VPN out of the box, or do we have to disable all this and use a custom configuration? Has this been discussed with Infosec? It looks like this will break their DNS sinkholing for domains such as REDHAT[.]CO (not COM).
It works out of the box, yes, and should work with any other VPN plugin as well, at least for VPNs that create separate tun interfaces for each connection like OpenVPN does. I think some VPN plugins might not do that (AnyConnect?).
I hadn't considered discussing a Fedora OS change with infosec. Anyway, the current behavior is insecure, and the proposed behavior is correct. ;) A sinkholing hack used by one company is nice to avoid some phishing, but it's still just a hack, not something we should design our OS around. And certainly not worth compromising the privacy of Fedora users. That said, any company's IT department can change whatever defaults they want on managed systems, configure systemd-resolved however they please, disable it entirely, whatever.
It may also need to changes to Postfix (for DANE/TLSA handling in particular, depending on the level of DNSSEC support, see below).
Based on Tom's mail, it sounds like this will be a problem. I guess the simple solution is to mention this in the release notes and suggest that users enable DNSSEC if running Postfix. We can also warn the Postfix developers to document this problem. I guess that should suffice?
libvirt may need changes as well, for its internal DNS relay.
I'm not familiar with this; what would require changes in libvirt? Are you concerned about nss-libvirt?
The manual page resolv.conf(5) needs to be updated.
Good catch, I would have forgotten that.
Since libnss_resolve.so.2 is still linked against libpthread and we might not be able to remove libpthread from glibc as a separate library in time for Fedora 33, some applications may have to start linking against libpthread explicitly even though they do not actually need it. (This is due to a long-standing limitation of loading libpthread via dlopen.)
I don't understand this. Why would an application have to link to pthread to dlopen a shared object that uses it?
Michael
On Wed, Apr 15, 2020 at 10:09 am, Michael Catanzaro mcatanzaro@gnome.org wrote:
Hm, it sounds like this is the main outstanding issue from this discussion. It is beyond my expertise. I guess we'll need a bug report where the relevant experts can figure out whether we need to change Kubernetes or systemd here.
You're right that continuing to use nss-dns would avoid any such problems while maintaining the other benefits of systemd-resolved. That could be a fallback plan if needed.
As you see, I just lost an email race to Lennart. I like his suggestion to manage /etc/resolv.conf manually on systems that need this behavior.
On Mi, 15.04.20 10:09, Michael Catanzaro (mcatanzaro@gnome.org) wrote:
You're right that continuing to use nss-dns would avoid any such problems while maintaining the other benefits of systemd-resolved. That could be a fallback plan if needed.
So, it is my understanding that containers as deployed with kubernetes generally don't boot up with systemd as PID 1 inside them, no?
If that's the case things should just work: if a container manager copies in their /etc/resolv.conf, and resolved is not running in the container, then nss-dns with traditional configuration is in effect as before.
If otoh containers are now started with systemd as PID 1 inside them, then this would also mean resolved is started inside, and yes in that case the single-label thing and the DNS-server-order thing might conflict with kubernetes' expectations. In that case it should be sufficient to "systemctl disable" systemd-resolved however, and nss-dns will take front seat again.
Lennart
-- Lennart Poettering, Berlin
* Lennart Poettering:
On Mi, 15.04.20 10:09, Michael Catanzaro (mcatanzaro@gnome.org) wrote:
You're right that continuing to use nss-dns would avoid any such problems while maintaining the other benefits of systemd-resolved. That could be a fallback plan if needed.
So, it is my understanding that containers as deployed with kubernetes generally don't boot up with systemd as PID 1 inside them, no?
If that's the case things should just work: if a container manager copies in their /etc/resolv.conf, and resolved is not running in the container, then nss-dns with traditional configuration is in effect as before.
As far as I know, the Kubernetes DNS hacks are used on the infrastructure layer, not just within containers.
I guess we can prepare a Fedora compose with all this implemented, and ask someone with the expertise to use it to deploy a Kubernetes cluster, and see what happens?
Thanks, Florian
On 14.04.2020 21:23, Ben Cotton wrote:
Enable systemd-resolved by default. glibc will perform name resolution using nss-resolve rather than nss-dns.
I've tested systemd-resolved on my laptop for a month. It worked very, very unstable. Sometimes it stopped responding and I needed to manually restart its service.
I think we need to stay on current solution. It works stable for ages.
On 15/04/2020 10:14, Vitaly Zaitsev via devel wrote:
On 14.04.2020 21:23, Ben Cotton wrote:
Enable systemd-resolved by default. glibc will perform name resolution using nss-resolve rather than nss-dns.
I've tested systemd-resolved on my laptop for a month. It worked very, very unstable. Sometimes it stopped responding and I needed to manually restart its service.
I too have been using it increasingly both on Fedora and on a fleet of Ubuntu 18.04 machines and don't recognise that experience at all.
I also (modulo one bug which is now fixed upstream) find it to be much better at handling DNSSEC than either bind or unbound.
Tom
On Wed, Apr 15, 2020 at 5:31 AM Tom Hughes via devel devel@lists.fedoraproject.org wrote:
On 15/04/2020 10:14, Vitaly Zaitsev via devel wrote:
On 14.04.2020 21:23, Ben Cotton wrote:
Enable systemd-resolved by default. glibc will perform name resolution using nss-resolve rather than nss-dns.
I've tested systemd-resolved on my laptop for a month. It worked very, very unstable. Sometimes it stopped responding and I needed to manually restart its service.
I too have been using it increasingly both on Fedora and on a fleet of Ubuntu 18.04 machines and don't recognise that experience at all.
I have seen the stability issues that Vitaly described also, requiring manual restart occasionally.
I also (modulo one bug which is now fixed upstream) find it to be much better at handling DNSSEC than either bind or unbound.
I occasionally have to disable DNSSEC, because it doesn't always work... though this might be due to improper DNS records for some sites, and I haven't compared with anything else.
Tom
-- Tom Hughes (tom@compton.nu) http://compton.nu/ _______________________________________________ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
On Mi, 15.04.20 11:14, Fedora Development ML (devel@lists.fedoraproject.org) wrote:
On 14.04.2020 21:23, Ben Cotton wrote:
Enable systemd-resolved by default. glibc will perform name resolution using nss-resolve rather than nss-dns.
I've tested systemd-resolved on my laptop for a month. It worked very, very unstable. Sometimes it stopped responding and I needed to manually restart its service.
I think we need to stay on current solution. It works stable for ages.
Note that resolved so far defaulted to DNSSEC mode by default (in opportunistic mode). DNSSEC support in DNS servers provided by edge routers (i.e. the DNS servers typically supplied in the DHCP leases you get at home) is usually crap, and very hard to detect properly. So far we have not been able to fine tune it in the myriad ways DNS servers are creatively broken; and even for the many cases where we detect things correctly it might still take a while after DNS config became available for us to completed our learning.
DNSSEC with resolved should work fine if your local DNS server is well behaving, but unfortunately that's generally not the case. THis means DNSSEC support will be switched from opt-out to opt-in with resolved becoming default to Fedora, to avoid this breakage.
Note that Ubuntu has been turning on resolved for quite some time by default now (with DNSSEC off). They did a lot of testing for us ultimately, so I#d expect few issues if we turn resolved on in Fedora by default, as long as we also keep DNSSEC off.
Long story short: if you experienced issues with DNSSEC on with resolved today, then be assured that with DNSSEC off things are much much better, and that's how we'd ship it in Fedora if it becomes the default.
Lennart
-- Lennart Poettering, Berlin
* Lennart Poettering:
Long story short: if you experienced issues with DNSSEC on with resolved today, then be assured that with DNSSEC off things are much much better, and that's how we'd ship it in Fedora if it becomes the default.
Would you please clarify what switching DNSSEC off means? Just no validation, or no DNSSEC support at all?
I'm worried that the following scenario will break: A Fedora system on a uses a DNSSEC-capable resolver (validating or not) and performs its own DNSSEC validation, using data obtained by contacting the name servers in /etc/resolv.conf. (/etc/resolv.conf is managed by NetworkManager or cloud-init in this scenario.)
Since /etc/resolv.conf is already managed, I expect that after the upgrade, systemd-resolved will be active, with the same upstream recursive resolvers as before. The new /etc/resolv.conf contents will point to the local systemd-resolved DNS service, though.
If systemd-resolved is not DNSSEC-aware with DNSSEC=off on the DNS interface, this will break DNSSEC validation in the application. It requires an explicit configuration change to fix.
In the past, caching resolvers have dealt with this situation by having separate caches for DO (DNSSEC answer OK) or CD (Checking Disabled) queries. This allows non-DNSSEC operations to continue even if the DNSSEC side is broken, so it is safe to enable it by default. It would also ensure that the configuration sketched above would not break (at least not for this reason).
Thanks, Florian
On 16/04/2020 11:46, Florian Weimer wrote:
- Lennart Poettering:
Long story short: if you experienced issues with DNSSEC on with resolved today, then be assured that with DNSSEC off things are much much better, and that's how we'd ship it in Fedora if it becomes the default.
Would you please clarify what switching DNSSEC off means? Just no validation, or no DNSSEC support at all?
If I'm understanding what is expected correctly then it looks to me like it is actually broken regardless of whether or not DNSSEC is switched on...
Adding +dnssec to the dig flags results in some additional flags being set in the OPT section of the response but it does not cause RRSIG records to be returned and whether DNSSEC is on or off makes no difference to that.
Tom
On Do, 16.04.20 12:46, Florian Weimer (fweimer@redhat.com) wrote:
- Lennart Poettering:
Long story short: if you experienced issues with DNSSEC on with resolved today, then be assured that with DNSSEC off things are much much better, and that's how we'd ship it in Fedora if it becomes the default.
Would you please clarify what switching DNSSEC off means? Just no validation, or no DNSSEC support at all?
It means we'd not attempt to validate DNS response we get with DNSSEC and just trust them blindly, i.e. like this always worked.
It would still be compiled in, and be opt-in. And it works fine with a well-behaving uptsream DNS servers, but given that so many public networks I know have no well behaved upstream DNS servers it would be opt-in.
I'm worried that the following scenario will break: A Fedora system on a uses a DNSSEC-capable resolver (validating or not) and performs its own DNSSEC validation, using data obtained by contacting the name servers in /etc/resolv.conf. (/etc/resolv.conf is managed by NetworkManager or cloud-init in this scenario.)
So, yes, if you attempt to use a client-side validating resolver against resolved's DNS stub you will not be happy. But you'll get a clean error back, and you will find something about it in syslog. it's not ideal, but it's usually OK. i.e. It's going to be like you talk to a DNS server that simply cannot do DNSSEC, except better, because you get helpful logging in syslog.
If you want a client-side validating resolver to work you need to bypass resolved, for example using the DNS server data in /run/systemd/resolve/resolv.conf. Or by using 8.8.8.8 or so directly...
Since /etc/resolv.conf is already managed, I expect that after the upgrade, systemd-resolved will be active, with the same upstream recursive resolvers as before. The new /etc/resolv.conf contents will point to the local systemd-resolved DNS service, though.
Exactly.
Lennart
-- Lennart Poettering, Berlin
On Tuesday, April 14, 2020 12:23:27 PM MST Ben Cotton wrote:
https://fedoraproject.org/wiki/Changes/systemd-resolved
== Summary ==
Enable systemd-resolved by default. glibc will perform name resolution using nss-resolve rather than nss-dns.
== Owner ==
- Name: [[User:catanzaro| Michael Catanzaro]]
- Email: mcatanzaro@redhat.com
== Detailed Description ==
We will enable systemd-resolved by default.
# We will change the [https://src.fedoraproject.org/rpms/fedora-release/blob/f4cc5b6ce095bb4233e5 e984a487e385def0ddca/f/90-default.preset fedora-release presets] to enable systemd-resolved instead of disable it. # systemd-libs currently has [https://src.fedoraproject.org/rpms/systemd/blob/bb79fb73875f8e71841a1ee8ede 5d234b7878753/f/systemd.spec#_606 a %post scriplet] to enable nss-myhostname and nss-systemd by either (a) modifying authselect's user-nsswitch.conf template, if authselect is in use, or (b) directly modifying /etc/nsswitch.conf otherwise. We will work with the systemd maintainers to enable nss-resolve here as well. # We will work with the authselect maintainers to update authselect's minimal and nis profiles to enforce nss-resolve. These profiles modify the hosts line of /etc/resolv.conf. (By default, Fedora uses authselect's sssd profile, which does not modify the hosts line and therefore does not have this problem.) # We will remove our downstream patch to systemd that prevents systemd from symlinking /etc/resolv.conf to /run/systemd/resolve/stub-resolv.conf on new installs. For system upgrades, a systemd-libs %post scriptlet will be used to reassign the symlink during upgrade. Upon detecting this symlink, NetworkManager will automatically enable its systemd-resolved support and configure split DNS as appropriate.
systemd-resolved has been enabled by default in Ubuntu since Ubuntu 16.10, but please note we are doing this differently than Ubuntu has. Ubuntu does not use nss-resolve. Instead, Ubuntu uses the traditional nss-dns provided by glibc upstream, so glibc on Ubuntu continues to read /etc/resolv.conf, as is traditional. This extra step is not useful and not recommended by upstream. We want to follow upstream recommendations in using nss-resolve instead.
If you do not wish to use systemd-resolved, then manual intervention will be required to disable it:
- Modify /etc/authselect/user-nsswitch.conf and remove `resolve
[!UNAVAIL=return]` from the hosts line. Run `authselect apply-changes`. (If you have disabled authselect, then edit /etc/nsswitch.conf directly.)
- Disable and stop systemd-resolved.service.
- Restart the NetworkManager service. NetworkManager will then create
a traditional /etc/resolv.conf. (If you are not using NetworkManager, you may create your own /etc/resolv.conf.)
== Benefit to Fedora ==
=== Standardization ===
Fedora will continue its history of enabling new systemd-provided services whenever it makes sense to do so. Standardizing on upstream systemd services is beneficial to the broader Linux ecosystem in addition to Fedora, since standardizing reduces behavior differences between different Linux distributions. Sadly, Fedora is no longer leading in this area. Ubuntu has enabled systemd-resolved by default since Ubuntu 16.10, so by the time Fedora 33 is released, we will be three years behind Ubuntu here.
=== resolvectl ===
`resolvectl` has several useful functions (e.g. `resolvectl status` or `resolvectl query`) that will be enabled out-of-the-box.
=== Caching ===
systemd-resolved caches DNS queries for a short while. This can [https://gitlab.gnome.org/GNOME/glib/-/merge_requests/682#note_441846 dramatically] improve performance for applications that do not already manually cache their own DNS results. (Generally, only web browsers bother with manually caching DNS results.)
=== Split DNS ===
When systemd-resolved is enabled, users who use multiple VPNs at the same time will notice that DNS requests are now sent to the correct DNS server by default. Previously, this scenario would result in embarrassing "DNS leaks" and, depending on the order that the VPN connections were established, possible failure to resolve private resources. These scenarios will now work properly. For example, consider the scenario of Distrustful Denise, who (quite reasonably) does not trust her ISP. Denise uses a public VPN service, public-vpn.example.com, to hide her internet activity from her ISP, but she also needs to use her employer's corporate VPN, corporation.example.com, in order to access internal company resources while working from home. Using the Network panel in System Settings, Denise has configured her employer's VPN to "use this connection only for resources on its network." Distrustful Denise expects that her employer's VPN will receive all traffic for corporation.example.com, and no other traffic. And while this mostly works in Fedora 32, she discovers that it does not work properly for DNS:
- If Denise connects to public-vpn.example.com first and
corporation.example.com second, she is unable to access internal company resources. All DNS requests are sent to public-vpn.example.com's DNS server, so she is unable to resolve names for internal company websites.
- If Denise connects to corporation.example.com first and
public-vpn.example.com second, then she is able to access internal company resources. However, it only works because ''all'' her DNS requests are sent to corporation.example.com's DNS server. Sadly for Distrustful Denise, her employer discovers that she has been making some embarrassing DNS requests that she had expected to go through public-vpn.example.com instead.
Whichever VPN Denise connects to first receives all DNS requests because glibc's nss-dns module is not smart enough to split the requests. /etc/resolv.conf is just a list of DNS servers to try in order, and NetworkManager has no plausible way to decide which to list first, because both ways are broken, so it just prefers whichever was connected first. And if one server fails to respond, then the next server in the list will be tried, likely resulting in a DNS leak. In contrast, when systemd-resolved is enabled, it will send each DNS request only to the correct DNS server. The DNS server that will be used for each tun interface can be inspected using the resolvectl tool.
Migrating away from /etc/resolv.conf will also avoid an annoying footgun with this legacy file: only the first three listed nameservers are respected. All further nameservers are silently ignored. NetworkManager adds a warning comment when writing more than three nameservers to this file, but it cannot do any better than that.
=== DNS over TLS ===
systemd-resolved supports DNS over TLS (different from DNS over HTTPS). Although this feature will not initially be enabled by default, using systemd-resolved will enable us to turn on DNS over TLS in a future Fedora release, providing improved security if the user's DNS server supports DNS over TLS.
== Scope ==
- Proposal owners: We will update Fedora presets to enable
systemd-resolved by default.
- Other developers: This change requires coordination with the systemd
and authselect maintainers.
Release engineering: [https://pagure.io/releng/issue/9367 #9367]
Policies and guidelines: none required
Trademark approval: N/A (not needed for this Change)
== Upgrade/compatibility impact ==
systemd-resolved will be enabled automatically when upgrading to Fedora 33. After upgrade, /etc/resolv.conf will be managed by systemd and symlinked to /run/systemd/resolve/stub-resolv.conf. '''glibc will no longer look at /etc/resolv.conf when performing name resolution.''' Instead, glibc will communicate directly with systemd-resolved via nss-resolve. systemd adds a large warning comment to the top of /etc/resolv.conf to warn system administrators that changes to this file will be ignored; however, scripts that edit this file manually will break. Because this file is usually managed by NetworkManager, impact to Fedora users will be limited to users who have manually disabled NetworkManager; such users are expected to be experienced system administrators who should be comfortable adapting to the change (or disabling systemd-resolved).
Any applications that bypass glibc and read /etc/resolv.conf directly will still work because /etc/resolv.conf will point to systemd-resolved's stub resolver running on 127.0.0.53. Nevertheless, /etc/resolv.conf is provided only for compatibility purposes, and applications should prefer to use either glibc or the systemd-resolved D-Bus API instead; see systemd-resolved(8) for details.
In short, '''applications that read /etc/resolv.conf will continue to work as before.''' Applications that write to it will no longer work as expected, but this only previously worked if NetworkManager is disabled, a non-default configuration. It remains possible to disable systemd-resolved if desired. Otherwise, any custom system administration scripts that manage /etc/resolv.conf will need to be updated.
=== DNSSEC ===
systemd-resolved's DNSSEC support is known to cause compatibility problems with certain network access points. Per recommendation from the systemd developers, we will change the default value of this setting in Fedora from the upstream default `DNSSEC=allow-downgrade` to `DNSSEC=no` by building systemd with the build option `-Ddefault-dnssec=no`. The upstream default attempts to use DNSSEC if it is working, but automatically disable it otherwise, allowing man-in-the-middle attackers to disable DNSSEC. Sadly, even the allow-downgrade setting suffers known compatibility problems. Because Fedora is not prepared to handle an influx of DNSSEC-related bug reports, we will disable this feature altogether. We anticipate that enabling DNSSEC by default will not be possible in the foreseeable future, or perhaps ever. Instead, enabling DNS over TLS (when supported by the DNS server) seems likely in the near future.
=== Multicast DNS ===
systemd-resolved's multicast DNS support conflicts with Avahi. Per recommendation from the systemd developers, we will change the default value of this setting in Fedora from the upstream default `MulticastDNS=yes` to `MulticastDNS=resolve`. Multicast DNS resolving will be enabled, but responding will be disabled. This will require adding a new systemd build option to control the default value of the MulticastDNS setting, similar to the existing `default-dnssec` and `default-dns-over-tls` build options.
== How To Test ==
Load any website in a web browser. If you succeed, then name resolution probably works.
Try using `resolvectl status` and, for example, `resolvectl query fedoraproject.org`, to see how they work and sanity-check their output.
Users who use multiple VPNs at the same time are encouraged to test DNS in a multiple VPN scenario, to ensure that DNS requests are sent to the expected DNS server.
== User Experience ==
See the Benefit to Fedora section, above, for direct benefits to users who use multiple VPNs at the same time.
Users will be able to use the resolvectl tool and the functionality it provides.
/etc/resolv.conf will now be managed by systemd rather than by NetworkManager. As before, this file must not be modified directly when it is managed.
== Dependencies ==
In Fedora, /etc/nsswitch.conf is managed by authselect. By default, authselect uses the sssd profile to generate configuration compatible with sssd. In this mode of operation, it does not modify the hosts line in /etc/nsswitch.conf. This is also true if using the winbind profile instead of the sssd profile. However, authselect's minimal and nis profiles do modify the hosts line. These authselect profiles must be updated to enable nss-resolved. If you are using authselect in one of these modes, it will not be possible to cleanly disable systemd-resolved because the hosts line in /etc/nsswitch.conf will be clobbered whenever 'authselect apply-changes' is run. If you wish to disable systemd-resolved and you are using authselect in one of these modes, then you should stop using authselect. This is not expected to cause many problems because virtually all Fedora users will be using the default sssd profile.
We do not need to directly make any changes to the /etc/nsswitch.conf shipped by glibc. Changes will be applied in the systemd-libs %post scriptlet.
== Contingency Plan ==
The contingency plan, in the unlikely event of unexpected problems, is simply to revert any changes and not enable systemd-resolved.
- Contingency deadline: beta freeze
- Blocks release? No
- Blocks product? No
== Documentation ==
- systemd-resolved is documented in several manpages: resolvectl(1),
resolved.conf(5), nss-resolve(8), systemd-resolved(8).
documentation]
- [https://wiki.gnome.org/Projects/NetworkManager/DNS NetworkManager
DNS documentation]
== Release Notes ==
systemd-resolved is now enabled by default. systemd-resolved provides a system-level DNS cache that can substantially improve performance for applications that do not cache their own DNS results, allows correct handling of split DNS scenarios such as when multiple VPNs are in use, and will allow Fedora to enable DNS over TLS in the future. /etc/resolv.conf will now be managed by systemd-resolved rather than by NetworkManager. /etc/resolv.conf will no longer be read when performing name resolution using glibc; however, it is still provided for compatibility with applications that manually read this file to perform name resolution. Writing to /etc/resolv.conf will no longer work as expected.
-- Ben Cotton He / Him / His Senior Program Manager, Fedora & CentOS Stream Red Hat TZ=America/Indiana/Indianapolis
Really, it may be best to go about this in the same way as Ubuntu, with nss- dns instead of nss-resolve.. Editing /etc/resolv.conf is still commonly done on Fedora, especially on servers. In fact, I never knew that NetworkManager would clobber that until this thread. If this isn't mean to wreck everyone's systems, backwards compatibility is key.
If not by using nss-dns, could systemd-resolved be modified such that it would read /etc/resolv.conf?
On Thursday, April 16, 2020 7:41:07 AM MST John M. Harris Jr wrote:
Really, it may be best to go about this in the same way as Ubuntu, with nss- dns instead of nss-resolve.. Editing /etc/resolv.conf is still commonly done on Fedora, especially on servers. In fact, I never knew that NetworkManager would clobber that until this thread. If this isn't mean to wreck everyone's systems, backwards compatibility is key.
If not by using nss-dns, could systemd-resolved be modified such that it would read /etc/resolv.conf?
Correcting what I said above, perhaps it'd be best to use what Lennart mentions as "mode 1" of systemd-resolved, such that /etc/resolv.conf is read, while using nss-resolve.
On Thu, Apr 16, 2020 at 7:55 am, John M. Harris Jr johnmh@splentity.com wrote:
Correcting what I said above, perhaps it'd be best to use what Lennart mentions as "mode 1" of systemd-resolved, such that /etc/resolv.conf is read, while using nss-resolve.
If you want to do that, you can. You just need to make /etc/resolv.conf a regular file, not a symlink. The text of the change proposal is wrong; I'll fix it.
On Thu, Apr 16, 2020 at 07:41:07AM -0700, John M. Harris Jr wrote:
Really, it may be best to go about this in the same way as Ubuntu, with nss- dns instead of nss-resolve.. Editing /etc/resolv.conf is still commonly done on Fedora, especially on servers. In fact, I never knew that NetworkManager would clobber that until this thread. If this isn't mean to wreck everyone's systems, backwards compatibility is key.
Relying on nss_dns causes bugs like https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/1624320 (systemd-resolved appends 127.0.0.53 to resolv.conf alongside existing entries). It still gets (angry) comments, years after it was filled.
On 4/16/20 5:43 PM, Tomasz Torcz wrote:
On Thu, Apr 16, 2020 at 07:41:07AM -0700, John M. Harris Jr wrote:
Really, it may be best to go about this in the same way as Ubuntu, with nss- dns instead of nss-resolve.. Editing /etc/resolv.conf is still commonly done on Fedora, especially on servers. In fact, I never knew that NetworkManager would clobber that until this thread. If this isn't mean to wreck everyone's systems, backwards compatibility is key.
Relying on nss_dns causes bugs like https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/1624320 (systemd-resolved appends 127.0.0.53 to resolv.conf alongside existing entries). It still gets (angry) comments, years after it was filled.
It is not nss_dns use, which makes angry comments from people. It should not be pushed into nameservers list when its broken. It should be possible to use resolved just for mdns and llmnr and leave dns for proper servers. If it were opt-in and not opt-out, no annoyed comments would be required.
On Tuesday, April 14, 2020 12:23:27 PM MST Ben Cotton wrote:
https://fedoraproject.org/wiki/Changes/systemd-resolved
== Summary ==
Enable systemd-resolved by default. glibc will perform name resolution using nss-resolve rather than nss-dns.
== Owner ==
- Name: [[User:catanzaro| Michael Catanzaro]]
- Email: mcatanzaro@redhat.com
== Detailed Description ==
We will enable systemd-resolved by default.
# We will change the [https://src.fedoraproject.org/rpms/fedora-release/blob/f4cc5b6ce095bb4233e5 e984a487e385def0ddca/f/90-default.preset
fedora-release presets] to enable
systemd-resolved instead of disable it. # systemd-libs currently has [https://src.fedoraproject.org/rpms/systemd/blob/bb79fb73875f8e71841a1ee8ede 5d234b7878753/f/systemd.spec#_606
a %post scriplet] to enable
nss-myhostname and nss-systemd by either (a) modifying authselect's user-nsswitch.conf template, if authselect is in use, or (b) directly modifying /etc/nsswitch.conf otherwise. We will work with the systemd maintainers to enable nss-resolve here as well. # We will work with the authselect maintainers to update authselect's minimal and nis profiles to enforce nss-resolve. These profiles modify the hosts line of /etc/resolv.conf. (By default, Fedora uses authselect's sssd profile, which does not modify the hosts line and therefore does not have this problem.) # We will remove our downstream patch to systemd that prevents systemd from symlinking /etc/resolv.conf to /run/systemd/resolve/stub-resolv.conf on new installs. For system upgrades, a systemd-libs %post scriptlet will be used to reassign the symlink during upgrade. Upon detecting this symlink, NetworkManager will automatically enable its systemd-resolved support and configure split DNS as appropriate.
systemd-resolved has been enabled by default in Ubuntu since Ubuntu 16.10, but please note we are doing this differently than Ubuntu has. Ubuntu does not use nss-resolve. Instead, Ubuntu uses the traditional nss-dns provided by glibc upstream, so glibc on Ubuntu continues to read /etc/resolv.conf, as is traditional. This extra step is not useful and not recommended by upstream. We want to follow upstream recommendations in using nss-resolve instead.
If you do not wish to use systemd-resolved, then manual intervention will be required to disable it:
- Modify /etc/authselect/user-nsswitch.conf and remove `resolve
[!UNAVAIL=return]` from the hosts line. Run `authselect apply-changes`. (If you have disabled authselect, then edit /etc/nsswitch.conf directly.)
- Disable and stop systemd-resolved.service.
- Restart the NetworkManager service. NetworkManager will then create
a traditional /etc/resolv.conf. (If you are not using NetworkManager, you may create your own /etc/resolv.conf.)
== Benefit to Fedora ==
=== Standardization ===
Fedora will continue its history of enabling new systemd-provided services whenever it makes sense to do so. Standardizing on upstream systemd services is beneficial to the broader Linux ecosystem in addition to Fedora, since standardizing reduces behavior differences between different Linux distributions. Sadly, Fedora is no longer leading in this area. Ubuntu has enabled systemd-resolved by default since Ubuntu 16.10, so by the time Fedora 33 is released, we will be three years behind Ubuntu here.
=== resolvectl ===
`resolvectl` has several useful functions (e.g. `resolvectl status` or `resolvectl query`) that will be enabled out-of-the-box.
=== Caching ===
systemd-resolved caches DNS queries for a short while. This can [https://gitlab.gnome.org/GNOME/glib/-/merge_requests/682#note_441846 dramatically] improve performance for applications that do not already manually cache their own DNS results. (Generally, only web browsers bother with manually caching DNS results.)
=== Split DNS ===
When systemd-resolved is enabled, users who use multiple VPNs at the same time will notice that DNS requests are now sent to the correct DNS server by default. Previously, this scenario would result in embarrassing "DNS leaks" and, depending on the order that the VPN connections were established, possible failure to resolve private resources. These scenarios will now work properly. For example, consider the scenario of Distrustful Denise, who (quite reasonably) does not trust her ISP. Denise uses a public VPN service, public-vpn.example.com, to hide her internet activity from her ISP, but she also needs to use her employer's corporate VPN, corporation.example.com, in order to access internal company resources while working from home. Using the Network panel in System Settings, Denise has configured her employer's VPN to "use this connection only for resources on its network." Distrustful Denise expects that her employer's VPN will receive all traffic for corporation.example.com, and no other traffic. And while this mostly works in Fedora 32, she discovers that it does not work properly for DNS:
- If Denise connects to public-vpn.example.com first and
corporation.example.com second, she is unable to access internal company resources. All DNS requests are sent to public-vpn.example.com's DNS server, so she is unable to resolve names for internal company websites.
- If Denise connects to corporation.example.com first and
public-vpn.example.com second, then she is able to access internal company resources. However, it only works because ''all'' her DNS requests are sent to corporation.example.com's DNS server. Sadly for Distrustful Denise, her employer discovers that she has been making some embarrassing DNS requests that she had expected to go through public-vpn.example.com instead.
Whichever VPN Denise connects to first receives all DNS requests because glibc's nss-dns module is not smart enough to split the requests. /etc/resolv.conf is just a list of DNS servers to try in order, and NetworkManager has no plausible way to decide which to list first, because both ways are broken, so it just prefers whichever was connected first. And if one server fails to respond, then the next server in the list will be tried, likely resulting in a DNS leak. In contrast, when systemd-resolved is enabled, it will send each DNS request only to the correct DNS server. The DNS server that will be used for each tun interface can be inspected using the resolvectl tool.
Migrating away from /etc/resolv.conf will also avoid an annoying footgun with this legacy file: only the first three listed nameservers are respected. All further nameservers are silently ignored. NetworkManager adds a warning comment when writing more than three nameservers to this file, but it cannot do any better than that.
=== DNS over TLS ===
systemd-resolved supports DNS over TLS (different from DNS over HTTPS). Although this feature will not initially be enabled by default, using systemd-resolved will enable us to turn on DNS over TLS in a future Fedora release, providing improved security if the user's DNS server supports DNS over TLS.
== Scope ==
- Proposal owners: We will update Fedora presets to enable
systemd-resolved by default.
- Other developers: This change requires coordination with the systemd
and authselect maintainers.
Release engineering: [https://pagure.io/releng/issue/9367 #9367]
Policies and guidelines: none required
Trademark approval: N/A (not needed for this Change)
== Upgrade/compatibility impact ==
systemd-resolved will be enabled automatically when upgrading to Fedora 33. After upgrade, /etc/resolv.conf will be managed by systemd and symlinked to /run/systemd/resolve/stub-resolv.conf. '''glibc will no longer look at /etc/resolv.conf when performing name resolution.''' Instead, glibc will communicate directly with systemd-resolved via nss-resolve. systemd adds a large warning comment to the top of /etc/resolv.conf to warn system administrators that changes to this file will be ignored; however, scripts that edit this file manually will break. Because this file is usually managed by NetworkManager, impact to Fedora users will be limited to users who have manually disabled NetworkManager; such users are expected to be experienced system administrators who should be comfortable adapting to the change (or disabling systemd-resolved).
Any applications that bypass glibc and read /etc/resolv.conf directly will still work because /etc/resolv.conf will point to systemd-resolved's stub resolver running on 127.0.0.53. Nevertheless, /etc/resolv.conf is provided only for compatibility purposes, and applications should prefer to use either glibc or the systemd-resolved D-Bus API instead; see systemd-resolved(8) for details.
In short, '''applications that read /etc/resolv.conf will continue to work as before.''' Applications that write to it will no longer work as expected, but this only previously worked if NetworkManager is disabled, a non-default configuration. It remains possible to disable systemd-resolved if desired. Otherwise, any custom system administration scripts that manage /etc/resolv.conf will need to be updated.
=== DNSSEC ===
systemd-resolved's DNSSEC support is known to cause compatibility problems with certain network access points. Per recommendation from the systemd developers, we will change the default value of this setting in Fedora from the upstream default `DNSSEC=allow-downgrade` to `DNSSEC=no` by building systemd with the build option `-Ddefault-dnssec=no`. The upstream default attempts to use DNSSEC if it is working, but automatically disable it otherwise, allowing man-in-the-middle attackers to disable DNSSEC. Sadly, even the allow-downgrade setting suffers known compatibility problems. Because Fedora is not prepared to handle an influx of DNSSEC-related bug reports, we will disable this feature altogether. We anticipate that enabling DNSSEC by default will not be possible in the foreseeable future, or perhaps ever. Instead, enabling DNS over TLS (when supported by the DNS server) seems likely in the near future.
=== Multicast DNS ===
systemd-resolved's multicast DNS support conflicts with Avahi. Per recommendation from the systemd developers, we will change the default value of this setting in Fedora from the upstream default `MulticastDNS=yes` to `MulticastDNS=resolve`. Multicast DNS resolving will be enabled, but responding will be disabled. This will require adding a new systemd build option to control the default value of the MulticastDNS setting, similar to the existing `default-dnssec` and `default-dns-over-tls` build options.
== How To Test ==
Load any website in a web browser. If you succeed, then name resolution probably works.
Try using `resolvectl status` and, for example, `resolvectl query fedoraproject.org`, to see how they work and sanity-check their output.
Users who use multiple VPNs at the same time are encouraged to test DNS in a multiple VPN scenario, to ensure that DNS requests are sent to the expected DNS server.
== User Experience ==
See the Benefit to Fedora section, above, for direct benefits to users who use multiple VPNs at the same time.
Users will be able to use the resolvectl tool and the functionality it provides.
/etc/resolv.conf will now be managed by systemd rather than by NetworkManager. As before, this file must not be modified directly when it is managed.
== Dependencies ==
In Fedora, /etc/nsswitch.conf is managed by authselect. By default, authselect uses the sssd profile to generate configuration compatible with sssd. In this mode of operation, it does not modify the hosts line in /etc/nsswitch.conf. This is also true if using the winbind profile instead of the sssd profile. However, authselect's minimal and nis profiles do modify the hosts line. These authselect profiles must be updated to enable nss-resolved. If you are using authselect in one of these modes, it will not be possible to cleanly disable systemd-resolved because the hosts line in /etc/nsswitch.conf will be clobbered whenever 'authselect apply-changes' is run. If you wish to disable systemd-resolved and you are using authselect in one of these modes, then you should stop using authselect. This is not expected to cause many problems because virtually all Fedora users will be using the default sssd profile.
We do not need to directly make any changes to the /etc/nsswitch.conf shipped by glibc. Changes will be applied in the systemd-libs %post scriptlet.
== Contingency Plan ==
The contingency plan, in the unlikely event of unexpected problems, is simply to revert any changes and not enable systemd-resolved.
- Contingency deadline: beta freeze
- Blocks release? No
- Blocks product? No
== Documentation ==
- systemd-resolved is documented in several manpages: resolvectl(1),
resolved.conf(5), nss-resolve(8), systemd-resolved(8).
documentation]
- [https://wiki.gnome.org/Projects/NetworkManager/DNS NetworkManager
DNS documentation]
== Release Notes ==
systemd-resolved is now enabled by default. systemd-resolved provides a system-level DNS cache that can substantially improve performance for applications that do not cache their own DNS results, allows correct handling of split DNS scenarios such as when multiple VPNs are in use, and will allow Fedora to enable DNS over TLS in the future. /etc/resolv.conf will now be managed by systemd-resolved rather than by NetworkManager. /etc/resolv.conf will no longer be read when performing name resolution using glibc; however, it is still provided for compatibility with applications that manually read this file to perform name resolution. Writing to /etc/resolv.conf will no longer work as expected.
-- Ben Cotton He / Him / His Senior Program Manager, Fedora & CentOS Stream Red Hat TZ=America/Indiana/Indianapolis
Please do not disable reading from /etc/resolv.conf. If you do so, please limit that to the Spins that it won't affect people on, such as Workstation, if you believe people there don't set their own DNS servers.
The impact section of this is flat out wrong. Removing that file's contents and replacing it with a symlink to some systemd think will break users' systems, leaving them unable to make DNS queries. It's more than possible to make use of the benefits of systemd-resolved without removing this critical functionality. Ubuntu has already done so, and Lennart pointed out that you can keep resolv.conf working without going about it the way Ubuntu did.
If this Change goes through as it's currently written, this will wreck users' systems.
On Sun, Jul 26, 2020 at 6:15 pm, John M. Harris Jr johnmh@splentity.com wrote:
Please do not disable reading from /etc/resolv.conf. If you do so, please limit that to the Spins that it won't affect people on, such as Workstation, if you believe people there don't set their own DNS servers.
Except:
* /etc/resolv.conf is broken by design, as you would know if you read the section on split DNS that you just quoted * There's no value in reading from /etc/resolv.conf unless you have written something custom to it * /etc/resolv.conf is managed by NetworkManager in Fedora, so you cannot safely write to it anyway in our default configuration
Fact is that unless you have done custom work to allow manual modifications to /etc/resolv.conf, you're not going to notice this change at all. And if you have, then surely you'll be able to figure out the very, very simple steps to get back to the original behavior. In fact, it should actually be *easier* than before to get traditional behavior. Remove the symlink. Create your own /etc/resolv.conf. Hey presto! systemd will read it....
On Sunday, July 26, 2020 7:06:48 PM MST Michael Catanzaro wrote:
On Sun, Jul 26, 2020 at 6:15 pm, John M. Harris Jr johnmh@splentity.com wrote:
Please do not disable reading from /etc/resolv.conf. If you do so, please limit that to the Spins that it won't affect people on, such as Workstation, if you believe people there don't set their own DNS servers.
Except:
- /etc/resolv.conf is broken by design, as you would know if you read
the section on split DNS that you just quoted
/etc/resolv.conf is not broken. It's the standard way of defining DNS servers for systems, and has worked for well over a decade.
- There's no value in reading from /etc/resolv.conf unless you have
written something custom to it
The value is actually getting DNS lookup to work on users' systems. Unless they've only used NetworkManager, and never touched /etc/resolv.conf, their system *will not be able to resolve hostnames after this forced removal*. There's a clean way to prevent that. Do not remove the file upon update.
- /etc/resolv.conf is managed by NetworkManager in Fedora, so you
cannot safely write to it anyway in our default configuration
/etc/resolv.conf is managed by NetworkManager, but it only gets updated if you use NetworkManager to manage DNS.
Fact is that unless you have done custom work to allow manual modifications to /etc/resolv.conf, you're not going to notice this change at all.
This is literally removing the file upon upgrade. That wasn't there originally, and it's a horrible addition. See your original response when I brought up this concern..
And if you have, then surely you'll be able to figure out the very, very simple steps to get back to the original behavior. In fact, it should actually be *easier* than before to get traditional behavior. Remove the symlink. Create your own /etc/resolv.conf. Hey presto! systemd will read it....
Is that what will actually happen, or will systemd still continue to ignore it? That's not made clear, because we've decided to go with something other than what Lennart calls "mode 1".
* Michael Catanzaro:
On Sun, Jul 26, 2020 at 6:15 pm, John M. Harris Jr johnmh@splentity.com wrote:
Please do not disable reading from /etc/resolv.conf. If you do so, please limit that to the Spins that it won't affect people on, such as Workstation, if you believe people there don't set their own DNS servers.
Except:
- /etc/resolv.conf is broken by design, as you would know if you read
the section on split DNS that you just quoted
It works for the things it's meant to do.
Split DNS does not exist as a concept. Some web browser concepts, such as the canary domain for DoH are explicitly incompatible with it:
https://support.mozilla.org/en-US/kb/canary-domain-use-application-dnsnet
Incompatible in the sense that when connecting to a VPN, DNS traffic will now be sent to a third party, when it would not before.
- There's no value in reading from /etc/resolv.conf unless you have
written something custom to it
Any DNS client library has to read /etc/resolv.conf to determine the system DNS configuration.
The format is about as stable than _res, and from languages which are not C, much easier to access.
This isn't an obscure use case, this is something that really has to work. Even C programs use alternative DNS clients for asynchronous name resolution and similar things.
Fact is that unless you have done custom work to allow manual modifications to /etc/resolv.conf, you're not going to notice this change at all.
It depends on the quality of the DNS implementation whose address is given in /etc/resolv.conf.
And if you have, then surely you'll be able to figure out the very, very simple steps to get back to the original behavior. In fact, it should actually be *easier* than before to get traditional behavior. Remove the symlink. Create your own /etc/resolv.conf. Hey presto! systemd will read it....
What if I want to manage name servers via DHCP (and Network Manager), but still retain DNSSEC support for local applications?
Thanks, Florian
On Sun, 2020-07-26 at 21:06 -0500, Michael Catanzaro wrote:
On Sun, Jul 26, 2020 at 6:15 pm, John M. Harris Jr johnmh@splentity.com wrote:
Please do not disable reading from /etc/resolv.conf. If you do so, please limit that to the Spins that it won't affect people on, such as Workstation, if you believe people there don't set their own DNS servers.
Except:
- /etc/resolv.conf is broken by design, as you would know if you read
the section on split DNS that you just quoted
- There's no value in reading from /etc/resolv.conf unless you have
written something custom to it
- /etc/resolv.conf is managed by NetworkManager in Fedora, so you
cannot safely write to it anyway in our default configuration
Fact is that unless you have done custom work to allow manual modifications to /etc/resolv.conf, you're not going to notice this change at all. And if you have, then surely you'll be able to figure out the very, very simple steps to get back to the original behavior. In fact, it should actually be *easier* than before to get traditional behavior. Remove the symlink. Create your own /etc/resolv.conf. Hey presto! systemd will read it....
Sorry Michael, but there are truckloads of programs that read resolv.conf directly, if all this change is doing is moving the file elsewhere but providing the same data in it, fine. If it is removing the pointers to DNS servers this change is absolutely not something you can do covertly in an update, it *will* be seen, but normal users will have a hard time figuring out what broke in the first place, let alone figure out how to resolve it (pun intended :).
Moreover if glibc is changed to use a different resolver then why change resolv.conf away? Only non-glibc querying programs would use it, and they intentionally do so anyway. Case in point one of the core components of the identity system in Fedora (SSSD) MUST be able to query directly DNS in scenarios were it is joined to an AD or IPA domain. Because it queries for stuff like SRV and TXT records concurrently (we use an async library) and needs to see those fields, as well as receive multiple IP addresses in order to set up fallbacks.
Please do not break DNS this way, if you really want to force everything through a single system resolver (I think that is a worthy goal) then it needs to be a thing that speaks DNS on UDP port 53 on 127.0.0.X and that address needs to be in resolv.conf as the nameserver address.
Any other "solution" will just break a bunch of stuff unnecessarily.
Simo.
On Mon, Jul 27, 2020 at 9:07 AM Simo Sorce simo@redhat.com wrote:
On Sun, 2020-07-26 at 21:06 -0500, Michael Catanzaro wrote:
On Sun, Jul 26, 2020 at 6:15 pm, John M. Harris Jr johnmh@splentity.com wrote:
Please do not disable reading from /etc/resolv.conf. If you do so, please limit that to the Spins that it won't affect people on, such as Workstation, if you believe people there don't set their own DNS servers.
Except:
- /etc/resolv.conf is broken by design, as you would know if you read
the section on split DNS that you just quoted
- There's no value in reading from /etc/resolv.conf unless you have
written something custom to it
- /etc/resolv.conf is managed by NetworkManager in Fedora, so you
cannot safely write to it anyway in our default configuration
Fact is that unless you have done custom work to allow manual modifications to /etc/resolv.conf, you're not going to notice this change at all. And if you have, then surely you'll be able to figure out the very, very simple steps to get back to the original behavior. In fact, it should actually be *easier* than before to get traditional behavior. Remove the symlink. Create your own /etc/resolv.conf. Hey presto! systemd will read it....
Sorry Michael, but there are truckloads of programs that read resolv.conf directly, if all this change is doing is moving the file elsewhere but providing the same data in it, fine. If it is removing the pointers to DNS servers this change is absolutely not something you can do covertly in an update, it *will* be seen, but normal users will have a hard time figuring out what broke in the first place, let alone figure out how to resolve it (pun intended :).
Moreover if glibc is changed to use a different resolver then why change resolv.conf away? Only non-glibc querying programs would use it, and they intentionally do so anyway. Case in point one of the core components of the identity system in Fedora (SSSD) MUST be able to query directly DNS in scenarios were it is joined to an AD or IPA domain. Because it queries for stuff like SRV and TXT records concurrently (we use an async library) and needs to see those fields, as well as receive multiple IP addresses in order to set up fallbacks.
Please do not break DNS this way, if you really want to force everything through a single system resolver (I think that is a worthy goal) then it needs to be a thing that speaks DNS on UDP port 53 on 127.0.0.X and that address needs to be in resolv.conf as the nameserver address.
Any other "solution" will just break a bunch of stuff unnecessarily.
That *is* what will happen. In this scenario, systemd-resolved creates a file in /run that is populated with the required information for applications to request name resolution from resolved through the standard DNS protocol. The /etc/resolv.conf file becomes a symlink to the file in /run so that the file in /etc is stable and regenerating the file in /run won't cause issues for package management. This system has been in use *already* for a while now in other distributions (see Debian and resolvconf(8), which systemd-resolved replaced in Ubuntu).
The only thing this mechanism breaks is applications trying to *write* to the resolv.conf file, because systemd-resolved will just blow away those changes right after. If you want to modify DNS settings, you need to configure systemd-resolved itself, either through NetworkManager (as we will recommend) or directly through systemd-resolved's configuration interface (if not using NetworkManager).
-- 真実はいつも一つ！/ Always, there's only one truth!
On Mon, 2020-07-27 at 09:20 -0400, Neal Gompa wrote:
On Mon, Jul 27, 2020 at 9:07 AM Simo Sorce simo@redhat.com wrote:
On Sun, 2020-07-26 at 21:06 -0500, Michael Catanzaro wrote:
On Sun, Jul 26, 2020 at 6:15 pm, John M. Harris Jr johnmh@splentity.com wrote:
Please do not disable reading from /etc/resolv.conf. If you do so, please limit that to the Spins that it won't affect people on, such as Workstation, if you believe people there don't set their own DNS servers.
Except:
- /etc/resolv.conf is broken by design, as you would know if you read
the section on split DNS that you just quoted
- There's no value in reading from /etc/resolv.conf unless you have
written something custom to it
- /etc/resolv.conf is managed by NetworkManager in Fedora, so you
cannot safely write to it anyway in our default configuration
Fact is that unless you have done custom work to allow manual modifications to /etc/resolv.conf, you're not going to notice this change at all. And if you have, then surely you'll be able to figure out the very, very simple steps to get back to the original behavior. In fact, it should actually be *easier* than before to get traditional behavior. Remove the symlink. Create your own /etc/resolv.conf. Hey presto! systemd will read it....
Sorry Michael, but there are truckloads of programs that read resolv.conf directly, if all this change is doing is moving the file elsewhere but providing the same data in it, fine. If it is removing the pointers to DNS servers this change is absolutely not something you can do covertly in an update, it *will* be seen, but normal users will have a hard time figuring out what broke in the first place, let alone figure out how to resolve it (pun intended :).
Moreover if glibc is changed to use a different resolver then why change resolv.conf away? Only non-glibc querying programs would use it, and they intentionally do so anyway. Case in point one of the core components of the identity system in Fedora (SSSD) MUST be able to query directly DNS in scenarios were it is joined to an AD or IPA domain. Because it queries for stuff like SRV and TXT records concurrently (we use an async library) and needs to see those fields, as well as receive multiple IP addresses in order to set up fallbacks.
Please do not break DNS this way, if you really want to force everything through a single system resolver (I think that is a worthy goal) then it needs to be a thing that speaks DNS on UDP port 53 on 127.0.0.X and that address needs to be in resolv.conf as the nameserver address.
Any other "solution" will just break a bunch of stuff unnecessarily.
That *is* what will happen. In this scenario, systemd-resolved creates a file in /run that is populated with the required information for applications to request name resolution from resolved through the standard DNS protocol. The /etc/resolv.conf file becomes a symlink to the file in /run so that the file in /etc is stable and regenerating the file in /run won't cause issues for package management. This system has been in use *already* for a while now in other distributions (see Debian and resolvconf(8), which systemd-resolved replaced in Ubuntu).
The only thing this mechanism breaks is applications trying to *write* to the resolv.conf file, because systemd-resolved will just blow away those changes right after. If you want to modify DNS settings, you need to configure systemd-resolved itself, either through NetworkManager (as we will recommend) or directly through systemd-resolved's configuration interface (if not using NetworkManager).
Thanks, I guess I misunderstood because of alarmist tones. This will work just fine, writing to resolv.conf is indeed something advanced, and a user that can set that up will likely know how to fix it as well.
Simo.
On Mon, Jul 27, 2020 at 09:43:23AM -0400, Simo Sorce wrote:
Thanks, I guess I misunderstood because of alarmist tones.
I think it's fair to say that there's a lesson to be learned here.
This will work just fine, writing to resolv.conf is indeed something advanced, and a user that can set that up will likely know how to fix it as well.
Additionally, Fedora's default configuration (for many, many, many years now) has had /etc/resolv.conf dynamically generated by NetworkManager. Local changes made directly to resolv.conf could (and would) get blown away at any time.
So users needing a non-volatile resolv.conf already have to take extra steps to achieve that. Under this proposal, different (but simpler!) steps will be necessary to achieve the same result.
- Solomon
On Monday, July 27, 2020 6:57:07 AM MST Solomon Peachy wrote:
Additionally, Fedora's default configuration (for many, many, many years now) has had /etc/resolv.conf dynamically generated by NetworkManager. Local changes made directly to resolv.conf could (and would) get blown away at any time.
Yes, but only IF NetworkManager is being used for DNS. If you don't set DNS servers there, or use a static configuration (as I do both at home and at work, and many others do, even though NetworkManager), this is just going to break users' systems.
So users needing a non-volatile resolv.conf already have to take extra steps to achieve that. Under this proposal, different (but simpler!) steps will be necessary to achieve the same result.
Yes, it's already there right now! And, upon upgrade, their file will be removed without even checking to see if it's set by NetworkManager or the end user, and replaced with a symlink. This will needlessly break users' systems upon upgrade. Lennart described another mode of operation for systmed- resolved, where it continues to use /etc/resolv.conf exactly as you'd expect.
On Monday, July 27, 2020 6:43:23 AM MST Simo Sorce wrote:
On Mon, 2020-07-27 at 09:20 -0400, Neal Gompa wrote:
On Mon, Jul 27, 2020 at 9:07 AM Simo Sorce simo@redhat.com wrote:
On Sun, 2020-07-26 at 21:06 -0500, Michael Catanzaro wrote:
On Sun, Jul 26, 2020 at 6:15 pm, John M. Harris Jr johnmh@splentity.com wrote:
Please do not disable reading from /etc/resolv.conf. If you do so, please limit that to the Spins that it won't affect people on, such as Workstation, if you believe people there don't set their own DNS servers.
Except:
- /etc/resolv.conf is broken by design, as you would know if you
read
the section on split DNS that you just quoted
- There's no value in reading from /etc/resolv.conf unless you have
written something custom to it
- /etc/resolv.conf is managed by NetworkManager in Fedora, so you
cannot safely write to it anyway in our default configuration
Fact is that unless you have done custom work to allow manual modifications to /etc/resolv.conf, you're not going to notice this change at all. And if you have, then surely you'll be able to figure out the very, very simple steps to get back to the original behavior. In fact, it should actually be *easier* than before to get traditional behavior. Remove the symlink. Create your own /etc/resolv.conf. Hey presto! systemd will read it....
Sorry Michael, but there are truckloads of programs that read resolv.conf directly, if all this change is doing is moving the file elsewhere but providing the same data in it, fine. If it is removing the pointers to DNS servers this change is absolutely not something you can do covertly in an update, it *will* be seen, but normal users will have a hard time figuring out what broke in the first place, let alone figure out how to resolve it (pun intended :).
Moreover if glibc is changed to use a different resolver then why change resolv.conf away? Only non-glibc querying programs would use it, and they intentionally do so anyway. Case in point one of the core components of the identity system in Fedora (SSSD) MUST be able to query directly DNS in scenarios were it is joined to an AD or IPA domain. Because it queries for stuff like SRV and TXT records concurrently (we use an async library) and needs to see those fields, as well as receive multiple IP addresses in order to set up fallbacks.
Please do not break DNS this way, if you really want to force everything through a single system resolver (I think that is a worthy goal) then it needs to be a thing that speaks DNS on UDP port 53 on 127.0.0.X and that address needs to be in resolv.conf as the nameserver address.
Any other "solution" will just break a bunch of stuff unnecessarily.
That *is* what will happen. In this scenario, systemd-resolved creates a file in /run that is populated with the required information for applications to request name resolution from resolved through the standard DNS protocol. The /etc/resolv.conf file becomes a symlink to the file in /run so that the file in /etc is stable and regenerating the file in /run won't cause issues for package management. This system has been in use *already* for a while now in other distributions (see Debian and resolvconf(8), which systemd-resolved replaced in Ubuntu).
The only thing this mechanism breaks is applications trying to *write* to the resolv.conf file, because systemd-resolved will just blow away those changes right after. If you want to modify DNS settings, you need to configure systemd-resolved itself, either through NetworkManager (as we will recommend) or directly through systemd-resolved's configuration interface (if not using NetworkManager).
Thanks, I guess I misunderstood because of alarmist tones. This will work just fine, writing to resolv.conf is indeed something advanced, and a user that can set that up will likely know how to fix it as well.
Please note that the issue with this isn't replacing resolv.conf with a symlink to a file in the same format that provides a local DNS server. The issue is ignoring users' configuration and effectively doing this upon upgrade:
rm -f /etc/resolv.conf
then putting a symlink there.
There's no reason for this. systemd-resolved has what's called "mode 1", where it reads from /etc/resolv.conf and everything continues to work just as you'd expect.
Once upon a time, Neal Gompa ngompa13@gmail.com said:
That *is* what will happen. In this scenario, systemd-resolved creates a file in /run that is populated with the required information for applications to request name resolution from resolved through the standard DNS protocol.
I modify that to say: through a *subset* of the standard DNS protocol; systemd-resolved intentionally doesn't implement the full protocol.
On Mo, 27.07.20 09:20, Neal Gompa (ngompa13@gmail.com) wrote:
That *is* what will happen. In this scenario, systemd-resolved creates a file in /run that is populated with the required information for applications to request name resolution from resolved through the standard DNS protocol. The /etc/resolv.conf file becomes a symlink to the file in /run so that the file in /etc is stable and regenerating the file in /run won't cause issues for package management. This system has been in use *already* for a while now in other distributions (see Debian and resolvconf(8), which systemd-resolved replaced in Ubuntu).
The only thing this mechanism breaks is applications trying to *write* to the resolv.conf file, because systemd-resolved will just blow away those changes right after. If you want to modify DNS settings, you need to configure systemd-resolved itself, either through NetworkManager (as we will recommend) or directly through systemd-resolved's configuration interface (if not using NetworkManager).
That's not quit true: if you replace th /etc/resolv.conf symlink with a file of your own choosing, then resolved will not muck around with that, and not modify it anymore. instead, it will start to *read* it and use the data. i.e. depending on symlinks vs. file it either provides or consumes the data in it.
Thus admin-provided configuration in /etc/resolv.conf takes precedence over the stuff systemd-resolved puts there, as long as the admin properly replaces the symlink. If the admin doesn't replace the symlink and writes to it naively, i.e. where it points then it will make changes to files in /run/systemd/ (because that's where the symlink points to), i.e. files clearly owned by systemd, and systemd-resolved will brutally overwrite them whenever it feels the need to.
Lennart
-- Lennart Poettering, Berlin
On Tuesday, July 28, 2020 5:11:31 AM MST Lennart Poettering wrote:
On Mo, 27.07.20 09:20, Neal Gompa (ngompa13@gmail.com) wrote:
That *is* what will happen. In this scenario, systemd-resolved creates a file in /run that is populated with the required information for applications to request name resolution from resolved through the standard DNS protocol. The /etc/resolv.conf file becomes a symlink to the file in /run so that the file in /etc is stable and regenerating the file in /run won't cause issues for package management. This system has been in use *already* for a while now in other distributions (see Debian and resolvconf(8), which systemd-resolved replaced in Ubuntu).
The only thing this mechanism breaks is applications trying to *write* to the resolv.conf file, because systemd-resolved will just blow away those changes right after. If you want to modify DNS settings, you need to configure systemd-resolved itself, either through NetworkManager (as we will recommend) or directly through systemd-resolved's configuration interface (if not using NetworkManager).
That's not quit true: if you replace th /etc/resolv.conf symlink with a file of your own choosing, then resolved will not muck around with that, and not modify it anymore. instead, it will start to *read* it and use the data. i.e. depending on symlinks vs. file it either provides or consumes the data in it.
Thus admin-provided configuration in /etc/resolv.conf takes precedence over the stuff systemd-resolved puts there, as long as the admin properly replaces the symlink. If the admin doesn't replace the symlink and writes to it naively, i.e. where it points then it will make changes to files in /run/systemd/ (because that's where the symlink points to), i.e. files clearly owned by systemd, and systemd-resolved will brutally overwrite them whenever it feels the need to.
To prevent brutally overwriting configuration, it would be best not to replace /etc/resolv.conf with a symlink on upgrade, ignoring user configuration, but to do so on all new installs.
On Tue, Jul 28, 2020 at 10:58 AM John M. Harris Jr johnmh@splentity.com wrote:
On Tuesday, July 28, 2020 5:11:31 AM MST Lennart Poettering wrote:
On Mo, 27.07.20 09:20, Neal Gompa (ngompa13@gmail.com) wrote:
That *is* what will happen. In this scenario, systemd-resolved creates a file in /run that is populated with the required information for applications to request name resolution from resolved through the standard DNS protocol. The /etc/resolv.conf file becomes a symlink to the file in /run so that the file in /etc is stable and regenerating the file in /run won't cause issues for package management. This system has been in use *already* for a while now in other distributions (see Debian and resolvconf(8), which systemd-resolved replaced in Ubuntu).
The only thing this mechanism breaks is applications trying to *write* to the resolv.conf file, because systemd-resolved will just blow away those changes right after. If you want to modify DNS settings, you need to configure systemd-resolved itself, either through NetworkManager (as we will recommend) or directly through systemd-resolved's configuration interface (if not using NetworkManager).
That's not quit true: if you replace th /etc/resolv.conf symlink with a file of your own choosing, then resolved will not muck around with that, and not modify it anymore. instead, it will start to *read* it and use the data. i.e. depending on symlinks vs. file it either provides or consumes the data in it.
Thus admin-provided configuration in /etc/resolv.conf takes precedence over the stuff systemd-resolved puts there, as long as the admin properly replaces the symlink. If the admin doesn't replace the symlink and writes to it naively, i.e. where it points then it will make changes to files in /run/systemd/ (because that's where the symlink points to), i.e. files clearly owned by systemd, and systemd-resolved will brutally overwrite them whenever it feels the need to.
To prevent brutally overwriting configuration, it would be best not to replace /etc/resolv.conf with a symlink on upgrade, ignoring user configuration, but to do so on all new installs.
We can be smart here and replace the file when we detect that it's managed by NetworkManager. Otherwise we won't replace it.
-- 真実はいつも一つ！/ Always, there's only one truth!
On 28 July 2020 17:07:14 CEST, Neal Gompa ngompa13@gmail.com wrote:
To prevent brutally overwriting configuration, it would be best not to replace /etc/resolv.conf with a symlink on upgrade, ignoring user configuration, but to do so on all new installs.
We can be smart here and replace the file when we detect that it's managed by NetworkManager. Otherwise we won't replace it.
This sounds like a good way to do it. I very much like the smooth major version upgrades and this would help. Given that some changes DNS servers, for whatever reason, we have to assume that those who have did so because of a reason. Not breaking DNS for them is important.
Br M
-- 真実はいつも一つ！/ Always, there's only one truth! _______________________________________________ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
I agree we should not replace /etc/resolv.conf if it's not already a symlink (although this will not be a common case, since by default it is a symlink managed by NetworkManager).
On Tue, Jul 28, 2020 at 11:07 am, Neal Gompa ngompa13@gmail.com wrote:
We can be smart here and replace the file when we detect that it's managed by NetworkManager. Otherwise we won't replace it.
Currently we actually don't touch it at all, which is not good because it means systemd-resolved will not be managing resolv.conf on upgraded systems. We should replace it with a symlink to systemd if (and only if) it's managed by NetworkManager.
On Tue, Jul 28, 2020 at 11:22:58AM -0500, Michael Catanzaro wrote:
I agree we should not replace /etc/resolv.conf if it's not already a symlink (although this will not be a common case, since by default it is a symlink managed by NetworkManager).
On Tue, Jul 28, 2020 at 11:07 am, Neal Gompa ngompa13@gmail.com wrote:
We can be smart here and replace the file when we detect that it's managed by NetworkManager. Otherwise we won't replace it.
Currently we actually don't touch it at all, which is not good because it means systemd-resolved will not be managing resolv.conf on upgraded systems. We should replace it with a symlink to systemd if (and only if) it's managed by NetworkManager.
Right now there's the following scriptlet:
grep -q 'Generated by NetworkManager' /etc/resolv.conf 2>/dev/null && \
echo -e '/etc/resolv.conf was generated by NetworkManager.\nConsider removing it to let systemd-resolved manage this file.' \
|| :
I.e. only a hint is emitted. I'm open to suggestions how to improve it.
Zbyszek
On Tue, Jul 28, 2020 at 5:23 pm, Zbigniew Jędrzejewski-Szmek zbyszek@in.waw.pl wrote:
I.e. only a hint is emitted. I'm open to suggestions how to improve it.
I think we should remove it if it's generated by NetworkManager, and leave it untouched otherwise. If NetworkManager is managing DNS then it will just push all its settings to systemd-resolved anyway after upgrade, right?
On 7/28/20 12:31 PM, Michael Catanzaro wrote:
I think we should remove it if it's generated by NetworkManager, and leave it untouched otherwise. If NetworkManager is managing DNS then it will just push all its settings to systemd-resolved anyway after upgrade, right?
Don't assume that the presence of the "Generated by NetworkManager" comment means that NetworkManager is *currently* managing that file; it just means that it did so when the file was generated.
On Tuesday, July 28, 2020 11:47:27 AM MST Ian Pilcher wrote:
On 7/28/20 12:31 PM, Michael Catanzaro wrote:
I think we should remove it if it's generated by NetworkManager, and leave it untouched otherwise. If NetworkManager is managing DNS then it will just push all its settings to systemd-resolved anyway after upgrade, right?
Don't assume that the presence of the "Generated by NetworkManager" comment means that NetworkManager is *currently* managing that file; it just means that it did so when the file was generated.
Agreed, I don't think I ever removed that from my /etc/resolv.conf on any of my systems.
On Tue, Jul 28, 2020 at 5:23 pm, Zbigniew Jędrzejewski-Szmek zbyszek@in.waw.pl wrote:
Right now there's the following scriptlet:
grep -q 'Generated by NetworkManager' /etc/resolv.conf 2>/dev/null && \
echo -e '/etc/resolv.conf was generated by NetworkManager.\nConsider removing it to let systemd-resolved manage this file.' \
|| :
I.e. only a hint is emitted. I'm open to suggestions how to improve it.
Zbyszek
Zbigniew, do you agree that we should remove the script if and only if it is generated by NetworkManager? Otherwise, the change is only partially-implemented for users upgrading from F32 and earlier.
On Tue, Aug 4, 2020 at 9:08 am, Michael Catanzaro mcatanzaro@gnome.org wrote:
Zbigniew, do you agree that we should remove the script if and only if it is generated by NetworkManager? Otherwise, the change is only partially-implemented for users upgrading from F32 and earlier.
We agreed to go with this approach. /etc/resolv.conf will be moved to /etc/resolv.conf.orig-with-nm on upgrade to F33 if the 'Generated by NetworkManager' comment is present. If you've gone to the trouble to prevent NetworkManager from managing this file, it's likely you've also removed the comment. If not, you can either remove the comment before upgrade, or recover your previous configuration from the /etc/resolv.conf.orig-with-nm backup file after upgrade, and hopefully have only a small blip in your upgrade experience. This will make the upgrade work properly for the 99% of users who don't mess with the defaults and should not be too difficult for those who do.
Michael
On Monday, August 10, 2020 9:52:42 AM MST Michael Catanzaro wrote:
On Tue, Aug 4, 2020 at 9:08 am, Michael Catanzaro mcatanzaro@gnome.org wrote:
Zbigniew, do you agree that we should remove the script if and only if it is generated by NetworkManager? Otherwise, the change is only partially-implemented for users upgrading from F32 and earlier.
We agreed to go with this approach. /etc/resolv.conf will be moved to /etc/resolv.conf.orig-with-nm on upgrade to F33 if the 'Generated by NetworkManager' comment is present. If you've gone to the trouble to prevent NetworkManager from managing this file, it's likely you've also removed the comment. If not, you can either remove the comment before upgrade, or recover your previous configuration from the /etc/resolv.conf.orig-with-nm backup file after upgrade, and hopefully have only a small blip in your upgrade experience. This will make the upgrade work properly for the 99% of users who don't mess with the defaults and should not be too difficult for those who do.
That comment being there doesn't mean that is still the case. I haven't removed that comment, I just fixed the file. It would be best to just not mess with this file on upgrade, and, instead, change the behavior of new installs. This wouldn't break anything, and would have all newly installed systems with the configuration you seem to want.
There's no reason to break peoples' systems here, we can easily plan for this. I also don't know where you're getting this estimate of 99% of users not changing this file.
On Saturday, August 29, 2020, John M. Harris Jr johnmh@splentity.com wrote:
On Monday, August 10, 2020 9:52:42 AM MST Michael Catanzaro wrote:
On Tue, Aug 4, 2020 at 9:08 am, Michael Catanzaro mcatanzaro@gnome.org wrote:
Zbigniew, do you agree that we should remove the script if and only if it is generated by NetworkManager? Otherwise, the change is only partially-implemented for users upgrading from F32 and earlier.
We agreed to go with this approach. /etc/resolv.conf will be moved to /etc/resolv.conf.orig-with-nm on upgrade to F33 if the 'Generated by NetworkManager' comment is present. If you've gone to the trouble to prevent NetworkManager from managing this file, it's likely you've also removed the comment. If not, you can either remove the comment before upgrade, or recover your previous configuration from the /etc/resolv.conf.orig-with-nm backup file after upgrade, and hopefully have only a small blip in your upgrade experience. This will make the upgrade work properly for the 99% of users who don't mess with the defaults and should not be too difficult for those who do.
That comment being there doesn't mean that is still the case. I haven't removed that comment, I just fixed the file. It would be best to just not mess with this file on upgrade, and, instead, change the behavior of new installs. This wouldn't break anything, and would have all newly installed systems with the configuration you seem to want.
There's no reason to break peoples' systems here, we can easily plan for this. I also don't know where you're getting this estimate of 99% of users not changing this file.
People expect to get the new features on upgrades without having to reinstall.
There is no reason to upgrades differently - if you have customized configuration you should be reading the changes before doing an upgrade.
-- John M. Harris, Jr.
devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject. org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists. fedoraproject.org
On Friday, August 28, 2020 9:55:18 PM MST drago01 wrote:
On Saturday, August 29, 2020, John M. Harris Jr johnmh@splentity.com
wrote:
On Monday, August 10, 2020 9:52:42 AM MST Michael Catanzaro wrote:
On Tue, Aug 4, 2020 at 9:08 am, Michael Catanzaro
mcatanzaro@gnome.org wrote:
Zbigniew, do you agree that we should remove the script if and only if it is generated by NetworkManager? Otherwise, the change is only partially-implemented for users upgrading from F32 and earlier.
We agreed to go with this approach. /etc/resolv.conf will be moved to /etc/resolv.conf.orig-with-nm on upgrade to F33 if the 'Generated by NetworkManager' comment is present. If you've gone to the trouble to prevent NetworkManager from managing this file, it's likely you've also removed the comment. If not, you can either remove the comment before upgrade, or recover your previous configuration from the /etc/resolv.conf.orig-with-nm backup file after upgrade, and hopefully have only a small blip in your upgrade experience. This will make the upgrade work properly for the 99% of users who don't mess with the defaults and should not be too difficult for those who do.
That comment being there doesn't mean that is still the case. I haven't removed that comment, I just fixed the file. It would be best to just not mess with this file on upgrade, and, instead, change the behavior of new installs. This wouldn't break anything, and would have all newly installed systems with the configuration you seem to want.
There's no reason to break peoples' systems here, we can easily plan for this. I also don't know where you're getting this estimate of 99% of users not changing this file.
People expect to get the new features on upgrades without having to reinstall.
This isn't a new feature, it's breaking existing functionality.
There is no reason to upgrades differently - if you have customized configuration you should be reading the changes before doing an upgrade.
The major reason that we've had different functionality on upgrade in the past is precisely what I described: to prevent needlessly breaking the users' systems. Regardless of the changes coming, updating should never break your system, especially when it's easily avoided, as it is in this case.
On Tuesday, July 28, 2020 9:22:58 AM MST Michael Catanzaro wrote:
I agree we should not replace /etc/resolv.conf if it's not already a symlink (although this will not be a common case, since by default it is a symlink managed by NetworkManager).
I haven't found any systems where that's the case, though my systems have been upgraded all along through from ~F24. Looking at my latest install, which has been upgraded from F30, /etc/resolv.conf is still just a file by default.
On Tue, Jul 28, 2020 at 11:07 am, Neal Gompa ngompa13@gmail.com wrote:
We can be smart here and replace the file when we detect that it's managed by NetworkManager. Otherwise we won't replace it.
Currently we actually don't touch it at all, which is not good because it means systemd-resolved will not be managing resolv.conf on upgraded systems.
That's not necessarily a bad thing. Regardless of what we do here, systemd- resolved is a consumer of NetworkManager's configuration.
We should replace it with a symlink to systemd if (and only if) it's managed by NetworkManager.
Sounds good.
On Tue, Jul 28, 2020 at 4:35 pm, John M. Harris Jr johnmh@splentity.com wrote:
I haven't found any systems where that's the case, though my systems have been upgraded all along through from ~F24. Looking at my latest install, which has been upgraded from F30, /etc/resolv.conf is still just a file by default.
Sorry I'm wrong, it's managed by NetworkManager but *not* as a symlink. It's just edited in-place.
On Tue, Jul 28, 2020 at 06:45:06PM -0500, Michael Catanzaro wrote:
On Tue, Jul 28, 2020 at 4:35 pm, John M. Harris Jr johnmh@splentity.com wrote:
I haven't found any systems where that's the case, though my systems have been upgraded all along through from ~F24. Looking at my latest install, which has been upgraded from F30, /etc/resolv.conf is still just a file by default.
Sorry I'm wrong, it's managed by NetworkManager but *not* as a symlink. It's just edited in-place.
Hmm. There are different solutions in the wild: $ exa -l /etc/resolv.conf lrwxrwxrwx@ 35 root 28 wrz 2019 /etc/resolv.conf -> /var/run/NetworkManager/resolv.conf
This is Fedora 32, initially installed a decade or so ago, and upgraded.
Observations based on Fedora-Workstation-Live-x86_64-33-20200828.n.0.iso
1. The ext4+squashfs image on media has no /etc/resolv.conf at all; this is what forms the basis of /dev/mapper/live-base, and is the source for installations (copied by rsync).
2. The installation live environment uses /dev/mapper/live-rw as system root; it has an /etc/resolv.conf symlink pointed to /run/systemd/resolve/stub-resolv.conf
3. The installed system has an /etc/resolv.conf file following installation (while still in the installation environment). First line is: # This file is managed by man:systemd-resolved(8). Do not edit.
4. Following a reboot, the system has an /etc/resolv.conf file. First line is: # Generated by NetworkManager
Are these the expected behavior?
-- Chris Murphy
On my F33 test system (new install yesterday) I can see my NAS but I can't connect to it. Could that be due to this change?
Best regards Andreas
On Sun, Aug 30, 2020 at 12:30 pm, Andreas Tunek andreas.tunek@gmail.com wrote:
On my F33 test system (new install yesterday) I can see my NAS but I can't connect to it. Could that be due to this change?
Possibly! Try using 'resolvectl query' and see what it says....
Den mån 31 aug. 2020 kl 00:55 skrev Michael Catanzaro <mcatanzaro@gnome.org
:
On Sun, Aug 30, 2020 at 12:30 pm, Andreas Tunek andreas.tunek@gmail.com wrote:
On my F33 test system (new install yesterday) I can see my NAS but I can't connect to it. Could that be due to this change?
Possibly! Try using 'resolvectl query' and see what it says....
I can't get that command to do anything useful in either F32 or F33. However, if I write "http://nas-name.local in F32 in either Firefox or Gnome Web I can connect to my NAS, but if I write the same in F33 in the same programs I get an error.
Best regards Andreas
_______________________________________________
devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
On Mon, Aug 31, 2020 at 10:19 pm, Andreas Tunek andreas.tunek@gmail.com wrote:
I can't get that command to do anything useful in either F32 or F33.
You should see something like:
$ resolvectl query example.com example.com: 2606:2800:220:1:248:1893:25c8:1946 -- link: tun0 93.184.216.34 -- link: tun0
-- Information acquired via protocol DNS in 86.4ms. -- Data is authenticated: no
And that should be working out-of-the-box in F33.
However, if I write "http://nas-name.local in F32 in either Firefox or Gnome Web I can connect to my NAS, but if I write the same in F33 in the same programs I get an error.
Ah, I bet this your NAS is mDNS, which is broken right now: https://bugzilla.redhat.com/show_bug.cgi?id=1867830
Until the root cause is diagnosed, here is a workaround you can try. As root, open up /etc/authselect/user-nsswitch.conf and look at the hosts line. In a freshly-installed F33 system, it should look like this:
hosts: resolve [!UNAVAIL=return] myhostname files mdns4_minimal [NOTFOUND=return] dns
This says: "if resolved is running, and it doesn't return a hostname, then STOP and don't try anything else." Everything else is listed only for the case where resolved is not running. But since resolved is currently not resolving mDNS as expected, let's change it to check with avahi first, then check with resolved second, like this:
hosts: mdns4_minimal [NOTFOUND=return] resolve [!UNAVAIL=return] myhostname files dns
Then, as root, run:
# authselect apply-changes
and then restart your browser. That's not the configuration we want to use in F33, but hopefully it will "fix" your problem. Please let me know if it works!
Michael
On Monday, August 31, 2020 1:39:37 PM MST Michael Catanzaro wrote:
On Mon, Aug 31, 2020 at 10:19 pm, Andreas Tunek andreas.tunek@gmail.com wrote:
I can't get that command to do anything useful in either F32 or F33.
You should see something like:
$ resolvectl query example.com example.com: 2606:2800:220:1:248:1893:25c8:1946 -- link: tun0 93.184.216.34 -- link: tun0
-- Information acquired via protocol DNS in 86.4ms. -- Data is authenticated: no
And that should be working out-of-the-box in F33.
However, if I write "http://nas-name.local in F32 in either Firefox or Gnome Web I can connect to my NAS, but if I write the same in F33 in the same programs I get an error.
Ah, I bet this your NAS is mDNS, which is broken right now: https://bugzilla.redhat.com/show_bug.cgi?id=1867830
Until the root cause is diagnosed, here is a workaround you can try. As root, open up /etc/authselect/user-nsswitch.conf and look at the hosts line. In a freshly-installed F33 system, it should look like this:
hosts: resolve [!UNAVAIL=return] myhostname files mdns4_minimal [NOTFOUND=return] dns
This says: "if resolved is running, and it doesn't return a hostname, then STOP and don't try anything else." Everything else is listed only for the case where resolved is not running. But since resolved is currently not resolving mDNS as expected, let's change it to check with avahi first, then check with resolved second, like this:
hosts: mdns4_minimal [NOTFOUND=return] resolve [!UNAVAIL=return] myhostname files dns
Then, as root, run:
# authselect apply-changes
and then restart your browser. That's not the configuration we want to use in F33, but hopefully it will "fix" your problem. Please let me know if it works!
Michael
Michael,
The file is /etc/nsswitch.conf.
On Mon, Aug 31, 2020 at 11:49 pm, John M. Harris Jr johnmh@splentity.com wrote:
Michael,
The file is /etc/nsswitch.conf.
You're wasting everyone's time with these low-effort spam posts. Lest anyone become confused, there is a big warning at the top of that file warning you that it is managed by authselect, and that manual changes will be overwritten.
Anyway, it seems this problem was not related to mDNS at all, so doesn't matter.
On Tuesday, September 1, 2020 7:14:35 AM MST Michael Catanzaro wrote:
On Mon, Aug 31, 2020 at 11:49 pm, John M. Harris Jr johnmh@splentity.com wrote:
Michael,
The file is /etc/nsswitch.conf.
You're wasting everyone's time with these low-effort spam posts.
I don't see how this could possibly be spam. This is where the file is, is it not?
Lest anyone become confused, there is a big warning at the top of that file warning you that it is managed by authselect, and that manual changes will be overwritten.
I don't know what you're talking about here. Am I missing something? Is this a F33 Change? Exact content of my /etc/nsswitch:
# # /etc/nsswitch.conf # # An example Name Service Switch config file. This file should be # sorted with the most-used services at the beginning. # # The entry '[NOTFOUND=return]' means that the search for an # entry should stop if the search in the previous entry turned # up nothing. Note that if the search failed due to some other reason # (like no NIS server responding) then the search continues with the # next entry. # # Valid entries include: # # nisplus Use NIS+ (NIS version 3) # nis Use NIS (NIS version 2), also called YP # dns Use DNS (Domain Name Service) # files Use the local files in /etc # db Use the pre-processed /var/db files # compat Use /etc files plus *_compat pseudo-databases # hesiod Use Hesiod (DNS) for user lookups # sss Use sssd (System Security Services Daemon) # [NOTFOUND=return] Stop searching if not found so far # # 'sssd' performs its own 'files'-based caching, so it should # generally come before 'files'.
# To use 'db', install the nss_db package, and put the 'db' in front # of 'files' for entries you want to be looked up first in the # databases, like this: # # passwd: db files # shadow: db files # group: db files
passwd: sss files shadow: sss files group: sss files
hosts: files mdns4_minimal [NOTFOUND=return] dns myhostname
bootparams: files
ethers: files netmasks: files networks: files protocols: files rpc: files services: sss files
netgroup: sss
publickey: files
automount: sss files aliases: files
On 2020-09-02 10:21, John M. Harris Jr wrote:
I don't know what you're talking about here. Am I missing something? Is this a F33 Change? Exact content of my /etc/nsswitch:
Is your system an upgrade of an earlier version?
In my case I have an F32 system which has been around for a few years. It has the text you cite.
I also have an F32 system which was a fresh install as well as a F31 fresh install. Both have
# Generated by authselect on Fri Jul 31 09:40:00 2020 # Do not modify this file manually.
# If you want to make changes to nsswitch.conf please modify # /etc/authselect/user-nsswitch.conf and run 'authselect apply-changes'.
In /etc/nsswitch.conf. So, the default use of authselect has been around since at least F31.
On Tuesday, September 1, 2020 7:29:44 PM MST Ed Greshko wrote:
On 2020-09-02 10:21, John M. Harris Jr wrote:
I don't know what you're talking about here. Am I missing something? Is this a F33 Change? Exact content of my /etc/nsswitch:
Is your system an upgrade of an earlier version?
In my case I have an F32 system which has been around for a few years. It has the text you cite.
I also have an F32 system which was a fresh install as well as a F31 fresh install. Both have
# Generated by authselect on Fri Jul 31 09:40:00 2020 # Do not modify this file manually.
# If you want to make changes to nsswitch.conf please modify # /etc/authselect/user-nsswitch.conf and run 'authselect apply-changes'.
In /etc/nsswitch.conf. So, the default use of authselect has been around since at least F31.
My system was originally F24. So, which of these is the actual configuration file at this point?
On 2020-09-02 11:06, John M. Harris Jr wrote:
On Tuesday, September 1, 2020 7:29:44 PM MST Ed Greshko wrote:
On 2020-09-02 10:21, John M. Harris Jr wrote:
I don't know what you're talking about here. Am I missing something? Is this a F33 Change? Exact content of my /etc/nsswitch:
Is your system an upgrade of an earlier version?
In my case I have an F32 system which has been around for a few years. It has the text you cite.
I also have an F32 system which was a fresh install as well as a F31 fresh install. Both have
# Generated by authselect on Fri Jul 31 09:40:00 2020 # Do not modify this file manually.
# If you want to make changes to nsswitch.conf please modify # /etc/authselect/user-nsswitch.conf and run 'authselect apply-changes'.
In /etc/nsswitch.conf. So, the default use of authselect has been around since at least F31.
My system was originally F24. So, which of these is the actual configuration file at this point?
Upgrades didn't change how things are done. I think it may have been F29 or F30 when authselect was introduced and made the default method to manage nsswitch.conf.
FWIW, on this old system of mine I did just run
authselect select --force sssd
to bring it forward to the current practice.
On Tue, Sep 1, 2020 at 10:21 PM John M. Harris Jr johnmh@splentity.com wrote:
On Tuesday, September 1, 2020 7:14:35 AM MST Michael Catanzaro wrote:
On Mon, Aug 31, 2020 at 11:49 pm, John M. Harris Jr johnmh@splentity.com wrote:
Michael,
The file is /etc/nsswitch.conf.
You're wasting everyone's time with these low-effort spam posts.
I don't see how this could possibly be spam. This is where the file is, is it not?
Lest anyone become confused, there is a big warning at the top of that file warning you that it is managed by authselect, and that manual changes will be overwritten.
I don't know what you're talking about here. Am I missing something? Is this a F33 Change? Exact content of my /etc/nsswitch:
Stated in the file or not, it is in fact edited by authconfig, sometimes as part of RPM installation. Manual editing of it is not and has never been stable without setting up some kind of configuration management to restore RPM based modifications. Been there, done that, with one of those 10-year solo admins who decided to hand-edit tweaks but refused to permit management of the file.
And oh, "files" always comes first because local config files should always take priority over upstream network based services.
On Tue, Sep 1, 2020 at 10:51 PM Nico Kadel-Garcia nkadel@gmail.com wrote:
On Tue, Sep 1, 2020 at 10:21 PM John M. Harris Jr johnmh@splentity.com wrote:
On Tuesday, September 1, 2020 7:14:35 AM MST Michael Catanzaro wrote:
On Mon, Aug 31, 2020 at 11:49 pm, John M. Harris Jr johnmh@splentity.com wrote:
Michael,
The file is /etc/nsswitch.conf.
You're wasting everyone's time with these low-effort spam posts.
I don't see how this could possibly be spam. This is where the file is, is it not?
Lest anyone become confused, there is a big warning at the top of that file warning you that it is managed by authselect, and that manual changes will be overwritten.
I don't know what you're talking about here. Am I missing something? Is this a F33 Change? Exact content of my /etc/nsswitch:
Stated in the file or not, it is in fact edited by authconfig, sometimes as part of RPM installation. Manual editing of it is not and has never been stable without setting up some kind of configuration management to restore RPM based modifications. Been there, done that, with one of those 10-year solo admins who decided to hand-edit tweaks but refused to permit management of the file.
And oh, "files" always comes first because local config files should always take priority over upstream network based services.
authconfig is dead. But yes, nsswitch is managed by authselect.
On 2020-09-02 11:00, Neal Gompa wrote:
On Tue, Sep 1, 2020 at 10:51 PM Nico Kadel-Garcia nkadel@gmail.com wrote:
On Tue, Sep 1, 2020 at 10:21 PM John M. Harris Jr johnmh@splentity.com wrote:
On Tuesday, September 1, 2020 7:14:35 AM MST Michael Catanzaro wrote:
On Mon, Aug 31, 2020 at 11:49 pm, John M. Harris Jr johnmh@splentity.com wrote:
Michael,
The file is /etc/nsswitch.conf.
You're wasting everyone's time with these low-effort spam posts.
I don't see how this could possibly be spam. This is where the file is, is it not?
Lest anyone become confused, there is a big warning at the top of that file warning you that it is managed by authselect, and that manual changes will be overwritten.
I don't know what you're talking about here. Am I missing something? Is this a F33 Change? Exact content of my /etc/nsswitch:
Stated in the file or not, it is in fact edited by authconfig, sometimes as part of RPM installation. Manual editing of it is not and has never been stable without setting up some kind of configuration management to restore RPM based modifications. Been there, done that, with one of those 10-year solo admins who decided to hand-edit tweaks but refused to permit management of the file.
And oh, "files" always comes first because local config files should always take priority over upstream network based services.
authconfig is dead. But yes, nsswitch is managed by authselect.
I would state that slightly differently.
nsswitch.conf is now managed by authselect by default.
I say that since, as I've already mentioned, I have a F32 system which has been upgraded over time from versions without authselect. The upgrade didn't changed that.
So, to become more conforming to current practice I did just run...
authselect select --force sssd
On Tue, Sep 1, 2020 at 10:50 pm, Nico Kadel-Garcia nkadel@gmail.com wrote:
And oh, "files" always comes first because local config files should always take priority over upstream network based services.
Zbigniew wound up moving files first in the end, but his justification for putting resolve first is here: https://github.com/authselect/authselect/pull/222#discussion_r454460745. It actually caches the local files so they only have to be read once globally, rather than once for each process.
But that failed due to our mDNS bug: https://bugzilla.redhat.com/show_bug.cgi?id=1867830#c34.
This is just a data point:
F31 Workstation clean installed (Live ISO), then updated to current F31, then upgraded to F33 (no interim F32).
/etc/nsswitch.conf hosts: files mdns4_minimal [NOTFOUND=return] resolve [!UNAVAIL=return] myhostname dns
And lrwxrwxrwx. 1 root root 39 Sep 10 19:42 resolv.conf -> ../run/systemd/resolve/stub-resolv.conf
Chris Murphy
Subject: Re: Fedora 33 System-Wide Change proposal: systemd-resolved
I was just hit by the first bug in systemd-resolved 4 days after I upgraded to fedora33. I will file a bug report for that, but I wanted to discuss something more fundamental.
systemd-resolved has a number of architectural flaws. When these were pointed out, bugs are not accepted and closed or ignored. Worse, I was told that systemd-resolved would not become the system DNS resolver, so I could just choose to not use it.
Unfortunately, with my upgrade to fedora 33 I was unwittingly upgraded to systemd-resolved. I want to remove it from my system, but I cannot because it is not even a sub-package of systemd, it is part of the core systemd package. This goes against promises made in the past.
Not only that, this change apparently "obsoletes" /etc/resolv.conf, which is just not acceptable.
It is my opinion as a long time DNS RFC author and package maintainer that systemd-resolved is not a suitable DNS resolver. Downgrading DNSSEC allowing local DNS to bypass DNSSEC is bad enough, but I read on:
https://fedoraproject.org/wiki/Changes/systemd-resolved
that DNSSEC is now completely disabled. Again, this is completely unacceptable. DNSSEC is part of the core of DNS protocols. My fedora mail server uses DNSSEC based TLSA records to prevent MITM attacks on the STARTTLS layer, which is now completely broken. My IPsec VPN server uses dnssec validation using the specified nameserves in /etc/resolve.conf that now point to systemd-resolvd that does not return DNSSEC records and is completely broken:
paul@thinkpad:~$ dig +dnssec vpn.nohats.ca @127.0.0.53
; <<>> DiG 9.11.22-RedHat-9.11.22-1.fc33 <<>> +dnssec vpn.nohats.ca @127.0.0.53 ;; global options: +cmd ;; Got answer: ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 51669 ;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1
;; OPT PSEUDOSECTION: ; EDNS: version: 0, flags: do; udp: 65494 ; OPT=5: 05 07 08 0a 0d 0e 0f (".......") ; OPT=6: 01 02 04 ("...") ; OPT=7: 01 (".") ;; QUESTION SECTION: ;vpn.nohats.ca. IN A
;; ANSWER SECTION: vpn.nohats.ca. 10 IN A 193.110.157.148
;; Query time: 143 msec ;; SERVER: 127.0.0.53#53(127.0.0.53) ;; WHEN: Mon Sep 28 00:18:32 EDT 2020 ;; MSG SIZE rcvd: 81
libreswan will see this result as an attack, and fail to resolve DNS names in its configuration. My postfix daemon will hold on to mail because it cannot get a DNSSEC proof of denial of existence of TLSA records if all DNSSEC records are filtered - even for domains that don't use DNSSEC because the denial of existence of DNSSEC for a specific domain requires the return of DNSSEC records that systemd now does not return.
I am sorry that I did not follow the fedora list carefully enough to notice this feature when it was proposed.
This change is harmful to network security, impacts existing installations depending on DNSSEC security, and leaks private queries for VPN/internal domains to the open internet, and prefers faster non-dnssec answers over dnssec validated answers. It fails various types of queries, misimplements part of the DNS protocol. Not only according to me, but to 20years+ developers of the bind software as well.
The first mandatory step is to not disable DNSSEC. If I put on my security hat, I would actually have to look into filing a CVE for Fedora 33.
A further discussion should happen on the future of systemd-resolved as the default Fedora (and later RHEL/CentOS) system resolver.
Paul
On 28.09.2020 06:44, Paul Wouters wrote:
Unfortunately, with my upgrade to fedora 33 I was unwittingly upgraded to systemd-resolved. I want to remove it from my system, but I cannot because it is not even a sub-package of systemd, it is part of the core systemd package.
You can always easily restore the previous behavior with Network Manager:
sudo rm -f /etc/resolv.conf sudo ln -sf /run/NetworkManager/resolv.conf /etc/resolv.conf sudo systemctl disable --now systemd-resolved.service sudo systemctl mask systemd-resolved.service sudo systemctl reboot
On Mon, Sep 28, 2020 at 12:44:13AM -0400, Paul Wouters wrote:
Subject: Re: Fedora 33 System-Wide Change proposal: systemd-resolved
I was just hit by the first bug in systemd-resolved 4 days after I upgraded to fedora33. I will file a bug report for that, but I wanted to discuss something more fundamental.
systemd-resolved has a number of architectural flaws. When these were pointed out, bugs are not accepted and closed or ignored. Worse, I was told that systemd-resolved would not become the system DNS resolver, so I could just choose to not use it.
Unfortunately, with my upgrade to fedora 33 I was unwittingly upgraded to systemd-resolved. I want to remove it from my system, but I cannot because it is not even a sub-package of systemd, it is part of the core systemd package. This goes against promises made in the past.
Hi,
in the end we decided to do a one-time "upgrade" of /etc/resolv.conf through a scriptlet. Two reasons: one, the configuration is split between /etc/resolv.conf, /etc/nsswitch.conf, and the enablement state of systemd-resolved.service. We were already adjusting the other two, and leaving old /etc/resolv.conf would likely lead to user confusion. Also, without adjusting of /etc/resolv.conf, newly installed systems would be different than upgraded ones in this fundamental regard. Overall, we think it'll be better for users who don't care about the details of the dns stack to fully swich to the new default. Power users and special setups can undo the changes.
Instructions were already posted by Vitaly, so I won't repeat that here. I'll just note that the scriptlet in systemd.rpm looks for 'Generated by NetworkManager' in /etc/resolv.conf as an indicator that the file is autogenerated.
Not only that, this change apparently "obsoletes" /etc/resolv.conf, which is just not acceptable.
I'm not sure what you mean by that. It is true that /etc/resolv.conf is not able to express split DNS. But it is still in place, with contents that try to express the actual DNS configuration to the extent possible. /etc/resolv.conf points to 127.0.0.53 as the resolver so that programs which don't use the nss stack also go through systemd-resolved. The list of remote dns servers can be queried from /run/systemd/resolve/resolv.conf in the classic resolv.conf syntax, or over dbus.
It is my opinion as a long time DNS RFC author and package maintainer that systemd-resolved is not a suitable DNS resolver. Downgrading DNSSEC allowing local DNS to bypass DNSSEC is bad enough, but I read on:
https://fedoraproject.org/wiki/Changes/systemd-resolved
that DNSSEC is now completely disabled. Again, this is completely unacceptable. DNSSEC is part of the core of DNS protocols. My fedora mail server uses DNSSEC based TLSA records to prevent MITM attacks on the STARTTLS layer, which is now completely broken. My IPsec VPN server uses dnssec validation using the specified nameserves in /etc/resolve.conf that now point to systemd-resolvd that does not return DNSSEC records and is completely broken:
paul@thinkpad:~$ dig +dnssec vpn.nohats.ca @127.0.0.53
; <<>> DiG 9.11.22-RedHat-9.11.22-1.fc33 <<>> +dnssec vpn.nohats.ca @127.0.0.53 ;; global options: +cmd ;; Got answer: ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 51669 ;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1
;; OPT PSEUDOSECTION: ; EDNS: version: 0, flags: do; udp: 65494 ; OPT=5: 05 07 08 0a 0d 0e 0f (".......") ; OPT=6: 01 02 04 ("...") ; OPT=7: 01 (".") ;; QUESTION SECTION: ;vpn.nohats.ca. IN A
;; ANSWER SECTION: vpn.nohats.ca. 10 IN A 193.110.157.148
;; Query time: 143 msec ;; SERVER: 127.0.0.53#53(127.0.0.53) ;; WHEN: Mon Sep 28 00:18:32 EDT 2020 ;; MSG SIZE rcvd: 81
libreswan will see this result as an attack, and fail to resolve DNS names in its configuration. My postfix daemon will hold on to mail because it cannot get a DNSSEC proof of denial of existence of TLSA records if all DNSSEC records are filtered - even for domains that don't use DNSSEC because the denial of existence of DNSSEC for a specific domain requires the return of DNSSEC records that systemd now does not return.
I am sorry that I did not follow the fedora list carefully enough to notice this feature when it was proposed.
This change is harmful to network security, impacts existing installations depending on DNSSEC security, and leaks private queries for VPN/internal domains to the open internet, and prefers faster non-dnssec answers over dnssec validated answers. It fails various types of queries, misimplements part of the DNS protocol. Not only according to me, but to 20years+ developers of the bind software as well.
You're mixing a few different things here. We decided to not enable DNSSEC in resolved with this change, at least initially. For most users, DNSSEC is problematic because various intermediary DNS servers found in hotspots and routers don't support DNSSEC properly, leading to hard-to-debug validation failures. DNSSEC support in resolved can be enabled through resolved.conf. This may be a reasonable thing to do in an environment where the configured dns servers are known to support dnssec properly.
leaks private queries for VPN/internal domains to the open internet
It's a complicated subject, but that statement is not true in general. systemd-resolved uses the servers it is told to use. Sometimes we're not sure what to tell it, see https://bugzilla.redhat.com/show_bug.cgi?id=1863041 for a long discussion.
and prefers faster non-dnssec answers over dnssec validated answers
Not exactly. It uses a single server, unless the server is deemed non-responsive and then it switches to the next one one, round-robin. Whether the server does dnssec or not does not directly factor into this. (Except that if resolved is configured to require dnssec, it won't want to talk to non-dnssec servers.) If dnssec is enabled, it shall only accept validated answers.
The first mandatory step is to not disable DNSSEC. If I put on my security hat, I would actually have to look into filing a CVE for Fedora 33.
A further discussion should happen on the future of systemd-resolved as the default Fedora (and later RHEL/CentOS) system resolver.
Please file bugs if something is not working as it should. But please be detailed, because there are a lot of unstated expectations based on how things worked in the past that don't necessarily makes sense now. (Like with the name servers accessible over vpn: some people think a server on vpn should be used by default for *everything*, while others say that it shouldn't be used unless configured so. Both options make sense depending on whom you trust more, but resolved cannot guess by itself.)
Zbyszek
On 28/09/2020 12:47, Zbigniew Jędrzejewski-Szmek wrote:
You're mixing a few different things here. We decided to not enable DNSSEC in resolved with this change, at least initially. For most users, DNSSEC is problematic because various intermediary DNS servers found in hotspots and routers don't support DNSSEC properly, leading to hard-to-debug validation failures. DNSSEC support in resolved can be enabled through resolved.conf. This may be a reasonable thing to do in an environment where the configured dns servers are known to support dnssec properly.
Well you're not just "not enabling it" really, for people like me that have already made the switch to systemd-resolved (in large part in search of better DNSSEC support) you're actually disabling it...
Having as I said experienced the trauma of trying to get DNSSEC working reliably I do understand how hard a problem is it however. I just need to remember to start adding a dropin to enable it again ;-)
Tom
On Mon, Sep 28, 2020 at 01:45:02PM +0100, Tom Hughes wrote:
On 28/09/2020 12:47, Zbigniew Jędrzejewski-Szmek wrote:
You're mixing a few different things here. We decided to not enable DNSSEC in resolved with this change, at least initially. For most users, DNSSEC is problematic because various intermediary DNS servers found in hotspots and routers don't support DNSSEC properly, leading to hard-to-debug validation failures. DNSSEC support in resolved can be enabled through resolved.conf. This may be a reasonable thing to do in an environment where the configured dns servers are known to support dnssec properly.
Well you're not just "not enabling it" really, for people like me that have already made the switch to systemd-resolved (in large part in search of better DNSSEC support) you're actually disabling it...
Having as I said experienced the trauma of trying to get DNSSEC working reliably I do understand how hard a problem is it however. I just need to remember to start adding a dropin to enable it again ;-)
What was the setup you were using? If this is something that we can reliably detect, I think it it would make sense to adjust the scriptlet that enables systemd-resolved to print a hint about needing to set DNSSEC=yes. (Or maybe even set that itself?).
Zbyszek
On Monday, 28 September 2020 at 15:30, Zbigniew Jędrzejewski-Szmek wrote: [...]
What was the setup you were using? If this is something that we can reliably detect, I think it it would make sense to adjust the scriptlet that enables systemd-resolved to print a hint about needing to set DNSSEC=yes. (Or maybe even set that itself?).
Having dnssec-trigger package installed would be an indicator in my case. Also, it does write its own comment in /etc/resolv.conf: $ rpm -qa dnssec* dnssec-trigger-0.15-12.fc32.x86_64 dnssec-trigger-panel-0.15-12.fc32.x86_64 $ cat /etc/resolv.conf # Generated by dnssec-trigger-script nameserver 127.0.0.1
It uses a local unbound instance for local DNSSEC resolution.
Regards, Dominik
On 28/09/2020 14:30, Zbigniew Jędrzejewski-Szmek wrote:
What was the setup you were using? If this is something that we can reliably detect, I think it it would make sense to adjust the scriptlet that enables systemd-resolved to print a hint about needing to set DNSSEC=yes. (Or maybe even set that itself?).
Well on most systems I have replaced NetworkManager with systemd-networkd though I don't think that's relevant here.
Other than that I have enable resolved and linked /etc/resolv.conf to /run/systemd/resolve/stub-resolv.conf, so basically what this change is doing.
Tom
On 9/28/20 6:47 AM, Zbigniew Jędrzejewski-Szmek wrote:
Instructions were already posted by Vitaly, so I won't repeat that here. I'll just note that the scriptlet in systemd.rpm looks for 'Generated by NetworkManager' in /etc/resolv.conf as an indicator that the file is autogenerated.
Which is a terrible idea, as has been previously mentioned. It really only indicates that the file was once touched my NetworkManager, not that it is currently managed.
If often let Anaconda set up a new system witha NetworkManager-managed DHCP and then convert to a legacy network scripts-managed static IP later. This doesn't change the DNS server or domain, so I don't bother editing resolv.conf to remove this comment. I'm relatively certain that this is a common pattern.
On Mon, Sep 28, 2020 at 07:57:13AM -0500, Ian Pilcher wrote:
On 9/28/20 6:47 AM, Zbigniew Jędrzejewski-Szmek wrote:
Instructions were already posted by Vitaly, so I won't repeat that here. I'll just note that the scriptlet in systemd.rpm looks for 'Generated by NetworkManager' in /etc/resolv.conf as an indicator that the file is autogenerated.
Which is a terrible idea, as has been previously mentioned. It really only indicates that the file was once touched my NetworkManager, not that it is currently managed.
If often let Anaconda set up a new system witha NetworkManager-managed DHCP and then convert to a legacy network scripts-managed static IP later. This doesn't change the DNS server or domain, so I don't bother editing resolv.conf to remove this comment. I'm relatively certain that this is a common pattern.
Yeah, that test is far from ideal, but we need something. If you have a constructive proposal how to improve it, I'm all ears.
Zbyszek
On 9/28/20 8:32 AM, Zbigniew Jędrzejewski-Szmek wrote:
Yeah, that test is far from ideal, but we need something. If you have a constructive proposal how to improve it, I'm all ears.
I anticipated this question. I don't have a good proposal for you ... but I believe that it's up to the people advocating/implementing this change to come up with that. If it isn't possible to automate this change in a reliable way, maybe it shouldn't be automated.
On Mon, Sep 28, 2020 at 10:51 am, Ian Pilcher arequipeno@gmail.com wrote:
I anticipated this question. I don't have a good proposal for you ... but I believe that it's up to the people advocating/implementing this change to come up with that. If it isn't possible to automate this change in a reliable way, maybe it shouldn't be automated.
Of course it's impossible. :P The only way to guess whether NetworkManager generated the file is to check if it says "NetworkManager" at the top. There's no way to be certain.
Anyway, if you don't like this heuristic, we could decide to always delete /etc/resolv.conf. The only other alternative I can think of would be to leave it unchanged, such that upgraded systems don't get fully migrated to systemd-resolved, but that's not a good option.
On Mon, 28 Sep 2020, Michael Catanzaro wrote:
Anyway, if you don't like this heuristic, we could decide to always delete /etc/resolv.conf.
You will break all software linked against libunbound that uses the ub_ctx_resolvconf() function. Most users of libunbound will use this, because firewalls might prevent UDP 53 packets going out from anything but the configured system resolver. It also then uses and gets use of the system's DNS cache.
The only other alternative I can think of would be to leave it unchanged, such that upgraded systems don't get fully migrated to systemd-resolved, but that's not a good option.
I do not think systemd-resolved is ready for prime time, even unrelated to the specific split DNS and DNSSEC case. A number of bugs have been closed that affect DNS resolving despite DNS experts reporting this as violating RFC standards and breaking things. For example:
https://github.com/systemd/systemd/issues/8967
Not migrating everything to systemd-resolved per default would not be the worst solution.
Paul
Dne 28. 09. 20 v 18:03 Michael Catanzaro napsal(a):
On Mon, Sep 28, 2020 at 10:51 am, Ian Pilcher arequipeno@gmail.com wrote:
I anticipated this question. I don't have a good proposal for you ... but I believe that it's up to the people advocating/implementing this change to come up with that. If it isn't possible to automate this change in a reliable way, maybe it shouldn't be automated.
Of course it's impossible. :P The only way to guess whether NetworkManager generated the file is to check if it says "NetworkManager" at the top. There's no way to be certain.
I think that for the future generations, it could help if NM included also checksum of the content.
Vít
On Mon, 2020-09-28 at 13:32 +0000, Zbigniew Jędrzejewski-Szmek wrote:
On Mon, Sep 28, 2020 at 07:57:13AM -0500, Ian Pilcher wrote:
On 9/28/20 6:47 AM, Zbigniew Jędrzejewski-Szmek wrote:
Instructions were already posted by Vitaly, so I won't repeat that here. I'll just note that the scriptlet in systemd.rpm looks for 'Generated by NetworkManager' in /etc/resolv.conf as an indicator that the file is autogenerated.
Which is a terrible idea, as has been previously mentioned. It really only indicates that the file was once touched my NetworkManager, not that it is currently managed.
If often let Anaconda set up a new system witha NetworkManager-managed DHCP and then convert to a legacy network scripts-managed static IP later. This doesn't change the DNS server or domain, so I don't bother editing resolv.conf to remove this comment. I'm relatively certain that this is a common pattern.
Yeah, that test is far from ideal, but we need something. If you have a constructive proposal how to improve it, I'm all ears.
Check if Network Manager is actually enabled as well ?
Simo.
On Mon, Sep 28, 2020 at 02:20:46PM -0400, Simo Sorce wrote:
On Mon, 2020-09-28 at 13:32 +0000, Zbigniew Jędrzejewski-Szmek wrote:
On Mon, Sep 28, 2020 at 07:57:13AM -0500, Ian Pilcher wrote:
On 9/28/20 6:47 AM, Zbigniew Jędrzejewski-Szmek wrote:
Instructions were already posted by Vitaly, so I won't repeat that here. I'll just note that the scriptlet in systemd.rpm looks for 'Generated by NetworkManager' in /etc/resolv.conf as an indicator that the file is autogenerated.
Which is a terrible idea, as has been previously mentioned. It really only indicates that the file was once touched my NetworkManager, not that it is currently managed.
If often let Anaconda set up a new system witha NetworkManager-managed DHCP and then convert to a legacy network scripts-managed static IP later. This doesn't change the DNS server or domain, so I don't bother editing resolv.conf to remove this comment. I'm relatively certain that this is a common pattern.
Yeah, that test is far from ideal, but we need something. If you have a constructive proposal how to improve it, I'm all ears.
Check if Network Manager is actually enabled as well ?
Makes sense: https://src.fedoraproject.org/rpms/systemd/c/f3f602da25b51caaa188e02003f3db9... https://src.fedoraproject.org/rpms/systemd/c/39055121170430fa599f454533543ce...
Zbyszek
On Tue, 2020-09-29 at 16:35 +0000, Zbigniew Jędrzejewski-Szmek wrote:
On Mon, Sep 28, 2020 at 02:20:46PM -0400, Simo Sorce wrote:
On Mon, 2020-09-28 at 13:32 +0000, Zbigniew Jędrzejewski-Szmek wrote:
On Mon, Sep 28, 2020 at 07:57:13AM -0500, Ian Pilcher wrote:
On 9/28/20 6:47 AM, Zbigniew Jędrzejewski-Szmek wrote:
Instructions were already posted by Vitaly, so I won't repeat that here. I'll just note that the scriptlet in systemd.rpm looks for 'Generated by NetworkManager' in /etc/resolv.conf as an indicator that the file is autogenerated.
Which is a terrible idea, as has been previously mentioned. It really only indicates that the file was once touched my NetworkManager, not that it is currently managed.
If often let Anaconda set up a new system witha NetworkManager-managed DHCP and then convert to a legacy network scripts-managed static IP later. This doesn't change the DNS server or domain, so I don't bother editing resolv.conf to remove this comment. I'm relatively certain that this is a common pattern.
Yeah, that test is far from ideal, but we need something. If you have a constructive proposal how to improve it, I'm all ears.
Check if Network Manager is actually enabled as well ?
Makes sense: https://src.fedoraproject.org/rpms/systemd/c/f3f602da25b51caaa188e02003f3db9... https://src.fedoraproject.org/rpms/systemd/c/39055121170430fa599f454533543ce...
Zbyszek
Nice job, thanks Zbyszek!
Simo.
On Mon, Sep 28, 2020 at 07:57:13AM -0500, Ian Pilcher wrote:
On 9/28/20 6:47 AM, Zbigniew Jędrzejewski-Szmek wrote:
Instructions were already posted by Vitaly, so I won't repeat that here. I'll just note that the scriptlet in systemd.rpm looks for 'Generated by NetworkManager' in /etc/resolv.conf as an indicator that the file is autogenerated.
Which is a terrible idea, as has been previously mentioned. It really only indicates that the file was once touched my NetworkManager, not that it is currently managed.
If often let Anaconda set up a new system witha NetworkManager-managed DHCP and then convert to a legacy network scripts-managed static IP later. This doesn't change the DNS server or domain, so I don't bother editing resolv.conf to remove this comment. I'm relatively certain that this is a common pattern.
Having read your email, I've checked my /etc/resolv.conf and _removed_ the NM comment. Never bothered to touch the comment before, as I'm using systemd-networkd for networking on the server. I would say that leaving the comment is a common pattern. There's not much there actually, only search domain and 'nameserver ::1', where dnsmasq listens, providing split-horizon for my network and pihole blacklisting of advertisment networks. But yeah, there are no better ideas how to handle resolv.conf on upgrades.
On Mon, 28 Sep 2020, Zbigniew Jędrzejewski-Szmek wrote:
This change is harmful to network security, impacts existing installations depending on DNSSEC security, and leaks private queries for VPN/internal domains to the open internet, and prefers faster non-dnssec answers over dnssec validated answers. It fails various types of queries, misimplements part of the DNS protocol. Not only according to me, but to 20years+ developers of the bind software as well.
You're mixing a few different things here. We decided to not enable DNSSEC in resolved with this change, at least initially. For most users, DNSSEC is problematic because various intermediary DNS servers found in hotspots and routers don't support DNSSEC properly, leading to hard-to-debug validation failures. DNSSEC support in resolved can be enabled through resolved.conf. This may be a reasonable thing to do in an environment where the configured dns servers are known to support dnssec properly.
If you want to do innovate new things with hotspot detection, the place to do the protocol work is at the IETF Captive Portal Working Group:
https://datatracker.ietf.org/wg/capport/charter/
Work done there include an architecture docoument:
https://datatracker.ietf.org/doc/draft-ietf-capport-architecture/
Captive Portal API: https://tools.ietf.org/html/rfc8908
DHCP and RA options: https://tools.ietf.org/html/rfc8910
These are efforts with big vendor signon. These will be compatible with new hotspots and work similar to other network devices. This is preferred over homegrown solutions.
leaks private queries for VPN/internal domains to the open internet
It's a complicated subject, but that statement is not true in general.
It was true when we had a DNSSEC systemd meeting in Brno about five years ago when I raised it as a privacy issue and was told my use case was "not valid". And it still seems to be true.
With Tommy Pauly of Apple, I co-write Split DNS for IKEv2 VPNs, that saw a major discusison in the security area of IETF (specifically SAAG and TLS) to ensure every one (including browser vendors) were okay with the resulting DNS reconfiguration requirements of VPN servers. This is especially important now that we are storing certificates as TLSA records in DNS, storing encrypted SNI in DNS, and the current draft SVCB and HTTPS service binding DNS records that are used in Apple's IOS14. These records contain information that must not be vulnerale to spoofing or rogue DNS server redirection and should be DNSSEC signed.
The designers of systemd-resolved will find it a useful read:
https://tools.ietf.org/html/rfc8598
systemd-resolved uses the servers it is told to use. Sometimes we're not sure what to tell it, see https://bugzilla.redhat.com/show_bug.cgi?id=1863041 for a long discussion.
If that worked, I wouldn't have even found out that my system got upgraded to systemd-resolved. Clearly this is broken. Furthermore, I doubt the rhbz will take into account the various risks of reconfiguring DNS and DNSSEC trust anchors or how to limit certain forwarders to certain domains. We are not talking about bugs that need fixing. We are talking about design decisions that I objected to five years ago that are now starting to cause damage.
and prefers faster non-dnssec answers over dnssec validated answers
Not exactly. It uses a single server, unless the server is deemed non-responsive and then it switches to the next one one, round-robin. Whether the server does dnssec or not does not directly factor into this. (Except that if resolved is configured to require dnssec, it won't want to talk to non-dnssec servers.) If dnssec is enabled, it shall only accept validated answers.
This is better thant it was five years ago. I'm glad some things were at least successfully conveyed in the Brno meeting. However, this still leaks queries meant for the LAN or VPN onto the wide internet and is still a privacy and security concern. Some of these issues might look like minor unimportant bugs, but could be life changing for some people. I recommend the systemd-resolved people look into the Human Rights Protocol Considerations Research Group (https://irtf.org/hrpc) and its draft documents.
Please file bugs if something is not working as it should. But please be detailed, because there are a lot of unstated expectations based on how things worked in the past that don't necessarily makes sense now.
My servers had functional DNSSEC. After an upgrade they don't. No more detailed reports are needed. You know what you need to do to address this bug. I see Florian had already filed a rhbz a few days ago: https://bugzilla.redhat.com/show_bug.cgi?id=1879028
systemd-resolved should not be a required base system package. You know what you need to do to fix this as well. If you want to make it part of the graphical desktop install that is okay. But it must not be included in server installs.
(Like with the name servers accessible over vpn: some people think a server on vpn should be used by default for *everything*, while others say that it shouldn't be used unless configured so. Both options make sense depending on whom you trust more, but resolved cannot guess by itself.)
See RFC 8598 for a detailed instruction on how you must implement this for IKEv2 / IPsec VPNs. Other VPN technologies should do something similar. It is very clear how you should act. And yes, some of that might depend on the VPN provisioning configuration.
libreswan already implements support for the unbound DNS server to reconfigure DNS limited to the domains covered by the VPN. This can be extended to support systemd-resolved if you point me to a documentation API.
Paul
On Mon, Sep 28, 2020 at 10:28 am, Paul Wouters paul@nohats.ca wrote:
This is better thant it was five years ago. I'm glad some things were at least successfully conveyed in the Brno meeting. However, this still leaks queries meant for the LAN or VPN onto the wide internet and is still a privacy and security concern.
systemd-resolved is designed to prevent DNS leaks that are unfixable with nss-mdns, not create them. DNS requests go *exactly* where you tell systemd-resolved to send them. If you've found some case where requests are not going where they're supposed to, then please report a bug.
Michael
On Mo, 28.09.20 10:28, Paul Wouters (paul@nohats.ca) wrote:
This is better thant it was five years ago. I'm glad some things were at least successfully conveyed in the Brno meeting. However, this still leaks queries meant for the LAN or VPN onto the wide internet and is
Classic resolv.conf cannot communicate to us where what should be routed. There's no per-domain syntax.
People have different usecases when they have multiple interfaces with multiple sets of DNS data around. We cannot guess which one of the two ifaces is the one to trust without explicit help.
Thing is: your LAN connection might be your trusted home network, or some untrusted cafe wifi. your VPN connection might be your trusted home VPN or the VPN to your company where you'd rather not send all DNS traffic. resolved cannot possibly guess what the trust level of ifaces and their DNS servers is, and if we'd priorize one clearly over the other we are likely getting it wrong: if we send all DNS traffic exclusively to the VPN then you lose resolveability of your local LAN names, such as the router or print config UI. If we send all DNS traffic exclusively to the local router's DHCP lease configured DNS server you'd of course lose the ability to resolve company private names.
So what did we opt to do? We think that working DNS is better than non-working DNS: so in absence of any further config info we'll look up names on all interfaces and use the first positive or the last negative answer. This behaviour doesn't bother too much with the security issue, i.e. everyting outside the local laptop is consider equally trusted or untrusted. But it maximizes the chance that things just work. And that's a inherently a *good* thing, the thing i care about the most.
Now, of coruse it would be better to only send to the VPN what really needs to be sent to the VPN, and conversely send to the local DHCP-supplied DNS only what shall be sent there. For that we need routing info. We'll synthesize some from search domains, if they are configured, but beyond that we require you to configure them manually.
Summary: we support routing if queries, you can configure that explicitly now, and if you don't you at least have the biggest chance that things "just work".
Lennart
-- Lennart Poettering, Berlin
Hi Paul,
is there any generic protocol exchanging what (sub)domains should be targetted to specific DNS server? I know dnssec-trigger/unbound is able to send queries only to specified search domains received by DHCP server.
Are you aware of any implementation independent way to store domains for each interface?
I think there should be just single way for connection provider to specify, which domains should go to its DNS server. Then any split-DNS capable software (unbound, dnsmasq, systemd-resolved) should just implement its layer to execute such redirection in runtime. Without special hooks in connection provider to running DNS stub.
I doubt there is already generic interface, but it seems it SHOULD be created. Because connection should not be pushing anything to resolved. It should push it to (whatever) network manager and resolved should pull it from there.
Can you point me to your support for unbound?
Regards, Petr
On 9/28/20 4:28 PM, Paul Wouters wrote:
On Mon, 28 Sep 2020, Zbigniew Jędrzejewski-Szmek wrote:
This change is harmful to network security, impacts existing installations depending on DNSSEC security, and leaks private queries for VPN/internal domains to the open internet, and prefers faster non-dnssec answers over dnssec validated answers. It fails various types of queries, misimplements part of the DNS protocol. Not only according to me, but to 20years+ developers of the bind software as well.
You're mixing a few different things here. We decided to not enable DNSSEC in resolved with this change, at least initially. For most users, DNSSEC is problematic because various intermediary DNS servers found in hotspots and routers don't support DNSSEC properly, leading to hard-to-debug validation failures. DNSSEC support in resolved can be enabled through resolved.conf. This may be a reasonable thing to do in an environment where the configured dns servers are known to support dnssec properly.
If you want to do innovate new things with hotspot detection, the place to do the protocol work is at the IETF Captive Portal Working Group:
https://datatracker.ietf.org/wg/capport/charter/
Work done there include an architecture docoument:
https://datatracker.ietf.org/doc/draft-ietf-capport-architecture/
Captive Portal API: https://tools.ietf.org/html/rfc8908
DHCP and RA options: https://tools.ietf.org/html/rfc8910
These are efforts with big vendor signon. These will be compatible with new hotspots and work similar to other network devices. This is preferred over homegrown solutions.
leaks private queries for VPN/internal domains to the open internet
It's a complicated subject, but that statement is not true in general.
It was true when we had a DNSSEC systemd meeting in Brno about five years ago when I raised it as a privacy issue and was told my use case was "not valid". And it still seems to be true.
With Tommy Pauly of Apple, I co-write Split DNS for IKEv2 VPNs, that saw a major discusison in the security area of IETF (specifically SAAG and TLS) to ensure every one (including browser vendors) were okay with the resulting DNS reconfiguration requirements of VPN servers. This is especially important now that we are storing certificates as TLSA records in DNS, storing encrypted SNI in DNS, and the current draft SVCB and HTTPS service binding DNS records that are used in Apple's IOS14. These records contain information that must not be vulnerale to spoofing or rogue DNS server redirection and should be DNSSEC signed.
The designers of systemd-resolved will find it a useful read:
https://tools.ietf.org/html/rfc8598
systemd-resolved uses the servers it is told to use. Sometimes we're not sure what to tell it, see https://bugzilla.redhat.com/show_bug.cgi?id=1863041 for a long discussion.
If that worked, I wouldn't have even found out that my system got upgraded to systemd-resolved. Clearly this is broken. Furthermore, I doubt the rhbz will take into account the various risks of reconfiguring DNS and DNSSEC trust anchors or how to limit certain forwarders to certain domains. We are not talking about bugs that need fixing. We are talking about design decisions that I objected to five years ago that are now starting to cause damage.
and prefers faster non-dnssec answers over dnssec validated answers
Not exactly. It uses a single server, unless the server is deemed non-responsive and then it switches to the next one one, round-robin. Whether the server does dnssec or not does not directly factor into this. (Except that if resolved is configured to require dnssec, it won't want to talk to non-dnssec servers.) If dnssec is enabled, it shall only accept validated answers.
This is better thant it was five years ago. I'm glad some things were at least successfully conveyed in the Brno meeting. However, this still leaks queries meant for the LAN or VPN onto the wide internet and is still a privacy and security concern. Some of these issues might look like minor unimportant bugs, but could be life changing for some people. I recommend the systemd-resolved people look into the Human Rights Protocol Considerations Research Group (https://irtf.org/hrpc) and its draft documents.
Please file bugs if something is not working as it should. But please be detailed, because there are a lot of unstated expectations based on how things worked in the past that don't necessarily makes sense now.
My servers had functional DNSSEC. After an upgrade they don't. No more detailed reports are needed. You know what you need to do to address this bug. I see Florian had already filed a rhbz a few days ago: https://bugzilla.redhat.com/show_bug.cgi?id=1879028
systemd-resolved should not be a required base system package. You know what you need to do to fix this as well. If you want to make it part of the graphical desktop install that is okay. But it must not be included in server installs.
(Like with the name servers accessible over vpn: some people think a server on vpn should be used by default for *everything*, while others say that it shouldn't be used unless configured so. Both options make sense depending on whom you trust more, but resolved cannot guess by itself.)
See RFC 8598 for a detailed instruction on how you must implement this for IKEv2 / IPsec VPNs. Other VPN technologies should do something similar. It is very clear how you should act. And yes, some of that might depend on the VPN provisioning configuration.
libreswan already implements support for the unbound DNS server to reconfigure DNS limited to the domains covered by the VPN. This can be extended to support systemd-resolved if you point me to a documentation API.
Paul
On Tue, 29 Sep 2020, Petr Menšík wrote:
is there any generic protocol exchanging what (sub)domains should be targetted to specific DNS server?
The search domains are usually the only signal available and used for this. RFC 7296 (IKEv2) and split-DNS (RFC 8598) defines the sent domain name list as having a meaning of INTERNAL_DNS_DOMAIN. So it is no longer a 'search domain' but officially a "name to resolve via the nameserver's specified by the VPN". Whether to accept these or not is a local policy, but usually there is a trust relationship that trusts the VPN server.
I know dnssec-trigger/unbound is able to send queries only to specified search domains received by DHCP server.
Yes, dnssec-trigger tests the nameservers and when they pass, it calls calls "unbound-control forward_add". This is similar to what systemd-resolved has adopted via resolvectl.
Are you aware of any implementation independent way to store domains for each interface?
There is none that I know of.
I think there should be just single way for connection provider to specify, which domains should go to its DNS server. Then any split-DNS capable software (unbound, dnsmasq, systemd-resolved) should just implement its layer to execute such redirection in runtime. Without special hooks in connection provider to running DNS stub.
I think using the 'search domain' from DHCP is fine. And the VPN use cases are clear too. As long as "public network" connections never target specific domains (eg avoid reconfiguring paypal.com)
I doubt there is already generic interface, but it seems it SHOULD be created.
These discussions are now happening at the IETF ADD and DPRIVE working groups. While focused on DoT and DoH, the problem is identical. We might see a "list of domains to resolve via XXX nameserver" in the near future. Once that happens, it should be trivial to use that with things like resolvectl or unbound-control.
Can you point me to your support for unbound?
The support for unbound in libreswan is also really easy, as all the lifting is done by the unbound daemon/library code. We just relay the domain list and nameserver IPs to forward_add/delete and flush the related zone names:
https://github.com/libreswan/libreswan/blob/main/programs/_updown.xfrm/_updo...
If we find a running unbound, we reconfigure it. If we don't find it, we rewrite resolv.conf and send all queries over the VPN. As I said, I'll work on adding systemd-resolved support here for the next version of libreswan.
Paul
Hi,
Am 28.09.20 um 13:47 schrieb Zbigniew Jędrzejewski-Szmek:
I'm not sure what you mean by that. It is true that /etc/resolv.conf is not able to express split DNS. But it is still in place, with contents that try to express the actual DNS configuration to the extent possible.
as one who had split horizone dns setups, it's not the client who splits, it's the server.
If you meant to have round robin, use nscd.
Best regards, Marius
On Mon, Sep 28, 2020 at 4:32 pm, Marius Schwarz fedoradev@cloud-foo.de wrote:
as one who had split horizone dns setups, it's not the client who splits, it's the server.
It's really the client... or the server running on the client:
nss-dns (traditional): split DNS impossible. No way to ever split DNS. dnsmasq, systemd-resolved: split DNS is possible
Yes, dnsmasq and systemd-resolved are local DNS servers, but point is, they split your DNS so that requests for different networks are ultimately sent to the DNS server configured for that network. With nss-dns, that is impossible: all your requests go to one server and that's that. See [1] for an example of how this would work in practice.
[1] https://fedoraproject.org/wiki/Changes/systemd-resolved#Split_DNS
Am 28.09.20 um 13:47 schrieb Zbigniew Jędrzejewski-Szmek:
DNSSEC support in resolved can be enabled through resolved.conf.
Why isn't that the default, if this resolver can do it?
Best regards, Marius
On 28/09/2020 15:57, Marius Schwarz wrote:
Am 28.09.20 um 13:47 schrieb Zbigniew Jędrzejewski-Szmek:
DNSSEC support in resolved can be enabled through resolved.conf.
Why isn't that the default, if this resolver can do it?
Because DNSSEC is a disaster area and if you try and use it on random networks you're going to get failed lookups on a reasonable number - it's fine if you're on a known network with decent upstream servers but once you start going out and using random WiFi hotspots and things it's a very different story.
Tom
On Mon, 28 Sep 2020, Tom Hughes via devel wrote:
On 28/09/2020 15:57, Marius Schwarz wrote:
Am 28.09.20 um 13:47 schrieb Zbigniew Jędrzejewski-Szmek:
DNSSEC support in resolved can be enabled through resolved.conf.
Why isn't that the default, if this resolver can do it?
Because DNSSEC is a disaster area and if you try and use it on random networks you're going to get failed lookups on a reasonable number - it's fine if you're on a known network with decent upstream servers but once you start going out and using random WiFi hotspots and things it's a very different story.
And that's why DNS-Over-TLS (DoT) and DNS-over-HTTPS (DoH) are now being deployed. And why browsers are, contrary to Michael Catanzaro's wrong claim, overriding the system DNS already. See Mozilla's TRR program https://wiki.mozilla.org/Trusted_Recursive_Resolver and Google's chrome https://www.chromium.org/developers/dns-over-https
There have been very vocal discussion at the IETF of browsers vendors vs enterprise vendors on how good/bad DNS reconfigurations are. Some discovery methods, and a protocol for Adaptive DNS by Apple have been discussed at the IETF ADD working https://datatracker.ietf.org/wg/add/documents/
The real solution to the requirement for following spoofed DNS answers to login to captive portals is to have a container with the "uplink" and a sandboxed browser inside it handling the captive portal. Once the captive portal part is done, you either use the network's DNS server, or the user provided one. And maybe change it based on testing the given DNS server in some way, using one of the ADD IETF protocols. And only then mark the network as "up" to all other applications. Or if the user prefers, only use the local DNS for portal authentication and once there is a clean internet link, use DoH or DoT to a known truted (non-coffeeshop) DNS server.
What we do not need is systemd-resolved making up its own incompatible and unsuspected protocols.
Paul
On Mon, Sep 28, 2020 at 11:57 AM Paul Wouters paul@nohats.ca wrote:
On Mon, 28 Sep 2020, Tom Hughes via devel wrote:
On 28/09/2020 15:57, Marius Schwarz wrote:
Am 28.09.20 um 13:47 schrieb Zbigniew Jędrzejewski-Szmek:
DNSSEC support in resolved can be enabled through resolved.conf.
Why isn't that the default, if this resolver can do it?
Because DNSSEC is a disaster area and if you try and use it on random networks you're going to get failed lookups on a reasonable number - it's fine if you're on a known network with decent upstream servers but once you start going out and using random WiFi hotspots and things it's a very different story.
And that's why DNS-Over-TLS (DoT) and DNS-over-HTTPS (DoH) are now being deployed. And why browsers are, contrary to Michael Catanzaro's wrong claim, overriding the system DNS already. See Mozilla's TRR program https://wiki.mozilla.org/Trusted_Recursive_Resolver and Google's chrome https://www.chromium.org/developers/dns-over-https
Michael is not wrong. We are, in fact, forcing Firefox to respect system DNS settings in Fedora. But if you use Mozilla's builds, you will have this problem. Same goes for Chrome/Chromium.
On Mon, Sep 28, 2020 at 11:56:28AM -0400, Paul Wouters wrote:
And that's why DNS-Over-TLS (DoT) and DNS-over-HTTPS (DoH) are now being deployed. And why browsers are, contrary to Michael Catanzaro's wrong claim, overriding the system DNS already. See Mozilla's TRR program https://wiki.mozilla.org/Trusted_Recursive_Resolver and Google's chrome https://www.chromium.org/developers/dns-over-https
Can we please be careful to not make this personal? It's getting a little heated. I'm all for technical discussion, but let's not make it about the people please?
On Mon, Sep 28, 2020 at 11:56 am, Paul Wouters paul@nohats.ca wrote:
And that's why DNS-Over-TLS (DoT) and DNS-over-HTTPS (DoH) are now being deployed. And why browsers are, contrary to Michael Catanzaro's wrong claim, overriding the system DNS already. See Mozilla's TRR program https://wiki.mozilla.org/Trusted_Recursive_Resolver and Google's chrome https://www.chromium.org/developers/dns-over-https
Florian just linked to that same chromium.org page as evidence that Chrome is not ignoring system DNS. :) Indeed, if you read the page, they're only using DNS over HTTPS (DoH) if system DNS matches a hardcoded list of providers that support DoH. So I believe I'm correct to say that only Firefox is doing that... and we have already patched Firefox to not do that.
Similarly, system-resolved will allow us to enable DNS over TLS (DoT) systemwide for supported providers. That's not enabled in F33, but I think we should flip the default for F34.
What we do not need is systemd-resolved making up its own incompatible and unsuspected protocols.
Now I'm lost. What are you talking about...?
Better standardization for captive portals seems good, but I'm not sure what this has to do with the systemd-resolved change?
On Mon, Sep 28, 2020 at 11:11 am, Michael Catanzaro mcatanzaro@gnome.org wrote:
Florian just linked to that same chromium.org page as evidence that Chrome is not ignoring system DNS. :) Indeed, if you read the page, they're only using DNS over HTTPS (DoH) if system DNS matches a hardcoded list of providers that support DoH.
Hm, I guess your point remains though, because if Chrome does decide to do its own DNS after it sees that your default DNS matches one of the whitelisted providers, then of course trying to resolve hostnames that need to be resolved by a different DNS is going to fail. E.g. if your corporate VPN is configured to be used only for resources on its network, I imagine it would fail.
Anyway, nothing we can do about that at the system level, other than promote secure system DNS so applications don't have to do it themselves with these hacks. I'll start typing up a change proposal to enable DNS over TLS.
Michael
On Mon, Sep 28, 2020 at 11:11:31AM -0500, Michael Catanzaro wrote:
hardcoded list of providers that support DoH. So I believe I'm correct to say that only Firefox is doing that... and we have already patched Firefox to not do that.
Just for clarity since it confused me: configured by default, not patched.
On 28.09.2020 18:11, Michael Catanzaro wrote:
Similarly, system-resolved will allow us to enable DNS over TLS (DoT) systemwide for supported providers. That's not enabled in F33, but I think we should flip the default for F34.
Btw, Russian Federation is going to completely block DoT and DoH. Forcing these technologies to end users will disrupt Internet access for people from such countries.
On Mo, 28.09.20 19:51, Fedora Development ML (devel@lists.fedoraproject.org) wrote:
On 28.09.2020 18:11, Michael Catanzaro wrote:
Similarly, system-resolved will allow us to enable DNS over TLS (DoT) systemwide for supported providers. That's not enabled in F33, but I think we should flip the default for F34.
Btw, Russian Federation is going to completely block DoT and DoH. Forcing these technologies to end users will disrupt Internet access for people from such countries.
I doubt we can force that even if we wanted, even in places that aren't Russia. The vast majority of DNS servers you see in public wifi DHCP leases or company DHCP leases can't do DoT.
And then I am pretty sure we should not bypass local DNS server info willy-nilly.
That said, the "opportunistic" mode we have might be something we want to turn on by default: in that mode you get DoT if we can but if not you don't. In Russia you thus typically wouldn't get DoT, but everyone else would.
Opportunistic mode means vulnerability to downgrade attacks, but I guess that's still better than nothing, given that the downgrade attack surface is probably mostly limited to local networks.
Lennart
-- Lennart Poettering, Berlin
On Mon, Sep 28, 2020 at 7:51 pm, Vitaly Zaitsev via devel devel@lists.fedoraproject.org wrote:
Btw, Russian Federation is going to completely block DoT and DoH. Forcing these technologies to end users will disrupt Internet access for people from such countries.
We can't require it, because most ISPs don't offer it, and Windows and macOS don't require it. If other platforms decide to require it, then we could do so as well and maybe Russia's evil plan will hopefully be subverted? But in the real world, we can't require it. So instead we'll do it opportunistic: try DoT if available, and fall back to plaintext if not. This provides no protection against an active network attacker, but it does protect against passive attackers. And if you want, you can change a line in resolved.conf to make it mandatory. (You can try this today in F33.)
On 28/09/2020 16:56, Paul Wouters wrote:
On Mon, 28 Sep 2020, Tom Hughes via devel wrote:
On 28/09/2020 15:57, Marius Schwarz wrote:
Am 28.09.20 um 13:47 schrieb Zbigniew Jędrzejewski-Szmek:
DNSSEC support in resolved can be enabled through resolved.conf.
Why isn't that the default, if this resolver can do it?
Because DNSSEC is a disaster area and if you try and use it on random networks you're going to get failed lookups on a reasonable number - it's fine if you're on a known network with decent upstream servers but once you start going out and using random WiFi hotspots and things it's a very different story.
And that's why DNS-Over-TLS (DoT) and DNS-over-HTTPS (DoH) are now being deployed. And why browsers are, contrary to Michael Catanzaro's wrong claim, overriding the system DNS already. See Mozilla's TRR program https://wiki.mozilla.org/Trusted_Recursive_Resolver and Google's chrome https://www.chromium.org/developers/dns-over-https
They solve very different problems though - they secure that single hop but they don't verify the actual contents of the records which are then resolved. They also break rather important things like split horizon DNS on corporate networks. Hell even my home network has split horizon, which is why I publish the special record to stop firefox doing DoH.
I mean excuse me for being sceptical about the wisdom of the likes of google trying to capture the world's DNS traffic by baking their resolvers into chrome...
The real solution to the requirement for following spoofed DNS answers to login to captive portals is to have a container with the "uplink" and a sandboxed browser inside it handling the captive portal. Once the captive portal part is done, you either use the network's DNS server, or the user provided one. And maybe change it based on testing the given DNS server in some way, using one of the ADD IETF protocols. And only then mark the network as "up" to all other applications. Or if the user prefers, only use the local DNS for portal authentication and once there is a clean internet link, use DoH or DoT to a known truted (non-coffeeshop) DNS server.
Spoofed DNS is one thing, but even once you're connected they often manage to filter out critical data.
What I do know is that even on the personal and corporate networks where I manage DNS and where I control everything systemd-resolved is the first thing I've found where I've been happy to turn on validation of DNSSEC without expecting things to break regularly.
I spent literally years periodically trying to turn on validation in bind and quickly giving up again because it's so shockingly bad.
That's not so say systemd-resolved is perfect, it's not that long since I provided a patch for it to fix a DNSSEC issue after all, but it's pragmatic approach seems to be doing better than any of the other things I have tried, be that bind, unbound, dnssec-trigger or whatever.
Tom
Am 28.09.20 um 17:56 schrieb Paul Wouters:
Because DNSSEC is a disaster area and if you try and use it on random networks you're going to get failed lookups on a reasonable number - it's fine if you're on a known network with decent upstream servers but once you start going out and using random WiFi hotspots and things it's a very different story.
And that's why DNS-Over-TLS (DoT) and DNS-over-HTTPS (DoH) are now being deployed. And why browsers are, contrary to Michael Catanzaro's wrong claim, overriding the system DNS already. See Mozilla's TRR program https://wiki.mozilla.org/Trusted_Recursive_Resolver and Google's chrome https://www.chromium.org/developers/dns-over-https
It's always a bad idea for a programm to do the dns itself, instead of using the dns anyone on the host does. You get a inconsistent behaviour at best, and a security nightmare at worse. DOx in a browser or any other programm is wrong anyhow.
best regards, Marius
On Mon, 28 Sep 2020, Marius Schwarz wrote:
It's always a bad idea for a programm to do the dns itself, instead of using the dns anyone on the host does. You get a inconsistent behaviour at best, and a security nightmare at worse. DOx in a browser or any other programm is wrong anyhow.
The software that decided that DNS answers are too security sensitive to not be validated is now blamed for not trusting the system resolver after it found said system resolver was untrustworthy.
As John Gilmore used to say, if you can't not trust your friends, who can you not trust? :)
Paul
On Mon, 2020-09-28 at 16:02 +0100, Tom Hughes via devel wrote:
On 28/09/2020 15:57, Marius Schwarz wrote:
Am 28.09.20 um 13:47 schrieb Zbigniew Jędrzejewski-Szmek:
DNSSEC support in resolved can be enabled through resolved.conf.
Why isn't that the default, if this resolver can do it?
Because DNSSEC is a disaster area and if you try and use it on random networks you're going to get failed lookups on a reasonable number - it's fine if you're on a known network with decent upstream servers but once you start going out and using random WiFi hotspots and things it's a very different story.
Surely this is better solved by using DoH toward known good servers for anything but the local resources ?
I mean the whole point of systemd-resolved should be to make things better including DNSSEC ?
As it was already pointed out it is also reasonably simple to detect if the local network have bad DNS servers ...
What am I missing ? Simo.
On Mo, 28.09.20 14:29, Simo Sorce (simo@redhat.com) wrote:
On Mon, 2020-09-28 at 16:02 +0100, Tom Hughes via devel wrote:
On 28/09/2020 15:57, Marius Schwarz wrote:
Am 28.09.20 um 13:47 schrieb Zbigniew Jędrzejewski-Szmek:
DNSSEC support in resolved can be enabled through resolved.conf.
Why isn't that the default, if this resolver can do it?
Because DNSSEC is a disaster area and if you try and use it on random networks you're going to get failed lookups on a reasonable number - it's fine if you're on a known network with decent upstream servers but once you start going out and using random WiFi hotspots and things it's a very different story.
Surely this is better solved by using DoH toward known good servers for anything but the local resources ?
Well, but how do you determine "local resources"?
Corporate networks tend to define local zones. Home wifi routers all do, too. There's no clear way to know what can go directly to well-known good DNS servers and what needs to be resolved locally.
Also, people would react very allergic if we'd start sending all DNS traffic to google or so. I mean, you can't believe how pissed people are that we have a fallback in place that if no DNS servers have been configured at all or acquired via DHCP we fall back to Cloudflare + Google DNS servers. Downstream distros (Debian…) tend to patch that fallback out even... I wouldn't want to know what happens if you start telling them we should now *prefer* them over local DNS servers, which is what you are saying here...
I mean the whole point of systemd-resolved should be to make things better including DNSSEC ?
Yes, resolved implements DNSSEC. But from my experience I can tell you it's very hard to do in a way resonably compatible with DNS servers deployed out there in particular edge ones. Things mostly work, but DNS servers are all broken in different ways, and we can't possibly test things on all possible cheap wifi hw...
(One thing I definitely want to add is an option to only do DNSSEC if DoT is also done, under the assumption that a DNS server that is good enough and new enough to implement the latter also should be able to do the former sanely.)
As it was already pointed out it is also reasonably simple to detect if the local network have bad DNS servers ...
No, it's not. It's extremely difficult. Cheap wifi router DNS servers are broken in so many ways. They return errors in some cases, freeze in others, return rubbish in others, or not at all in even others. If you ask the wrong questions anything can happen. We pretty carefully tests and probe DNS servers but this still comes at the price that on a particular bad implementation we might take a long time until we figure out that DNSSEC simply is not possible.
The simple fact that some DNS servers don't respond at all if you ask the "wrong" questions is already a problem: it means you have to wait for a timeout (which means super long lookups initially) or do queries in parallel. That however is a problem too since other DNS servers really don#t like it if you ask them multiple questions at once. Bombarding DNS servers with multiple questions all at once and see if one "sticks" isn't a workable strategy hence either.
And then, when you figured out that the local DNS server can do some thing but not others, and are happy you will eventually notice that many "edge" DNS servers have a split personality: for some domains they are just a proxy for the upstream DNS servers and other domains they implement things locally, which means the feature set you can probe differs vastly depending on which domains you query. That's particular awful for reverse lookups btw, since the domain separation is a lot more blurry there.
So I think we do quite well in resolved on the DNSSEC front actually, but people still are annoyed that one some local DNS servers DNSSEC doesn't work and or we take too long to detect that it doesn't.
Lennart
-- Lennart Poettering, Berlin
Am 29.09.20 um 10:19 schrieb Lennart Poettering:
Also, people would react very allergic if we'd start sending all DNS traffic to google or so. I mean, you can't believe how pissed people are that we have a fallback in place that if no DNS servers have been configured at all or acquired via DHCP we fall back to Cloudflare + Google DNS servers. Downstream distros (Debian…) tend to patch that fallback out even...
I hope they all patch it out. Google NS Service may be reliable, but with GDPR in place in Europe, you really can't sell your user base out to google and cloudflare like mozilla did.
DNS data, because a user IP is involved, is personal data per definition in Germany. You, of all others, should know that as the Kammergericht, Berlin stated it in their 2013 judgement. So, if my pc is sending a dns request to a google or CF dns server, and it's a company pc, it's a GDPR violation. And those are not the only data protection laws in the world.
The only valid way to handle a none working DNS is to NOT CIRCUMVENT IT. If the admin of that network has failed to set them up correctly, any other device will fail too or they too have hardcoded dns servers somewhere. I would prefer to be informed that dns is not setuped correctly to fix the mistake instead of silently working around it. (btw thats was the job of the replaced /etc/resolv.conf , to honour the nameservers the user wants, not some place to store the result of dhcp )
Linux is known to the public as a data protecting os and not one that rants out it's users to cloudflare. A few postings ago i thought, that reenabling dnssec would solve the issue, but now I tend to vote for a revert on the use of systemd-resolved in Fedora if I need to worry about my dns having privacy issues.
Marius Schwarz Germany EU
On Tue, 29 Sep 2020, Lennart Poettering wrote:
Well, but how do you determine "local resources"?
This is not the proper question. The proper question is "what are you trying to do". The .local domain discovery clearly is something meant to be local.
I assume the real question is: How to convey my custom local network domain to my local infrastructure. In the old days, it was what DHCP gave you as domain. If you do that with your own network, then it is pretty obvious. If you do it differently, you will have to coney this somehow via configuration. This is why 20 years ago Microsoft added "zones" to their network configuration. Is this a "home zone" or a "work zone" or a "public wifi".
So I expect the information of "what is local" to live in NetworkManager or systemd-networkd via configuration.
No further magic should be needed. The user selects this once when joining a new network.
Corporate networks tend to define local zones. Home wifi routers all do, too. There's no clear way to know what can go directly to well-known good DNS servers and what needs to be resolved locally.
Generally, resolve the names from the DHCP obtained domain name with the DHCP obtained name servers. Yes, this is limited to one domain name, which might not be ideal, but in general when you connect to a home or corporate network directly (no VPN) then you should use their DNS servers regardless. Enterprise is likely blocking port 53 (or DoT or trying to block DoH) for security reasons. And your home network you trust?
Most home routers these days allow configuration of a guest network along with the native home network. For those not requiring services on the home network, and who just need internet. It is the same as using a public wifi in a coffeeshop or guest network at an enterprise network. You might need to authenticate a captive portal and then you should not trust the network for anything else and ideally only give it encrypted packets (TLS, DoT to trusted DNS servers, VPN). If no trusted DNS servers are configured on your device, you have no choice but to trust their DNS servers.
For what the user deems is a "public wifi", there are simply never any "local resources" other than an internet uplink to your own remote resources.
In all the above scenario's, I see no ambiguity on which DNS servers to use, except when multiple domain names exist within only the LAN, which is rarely the case.
For the VPN scenario, it is just a little bit more complicated.
For those with proper standards, such as "Cisco IPsec", L2TP/IPsec", the VPN confiuration is dictated by the server to either send all or some traffic to the VPN server. If it is not "everything", then these VPNs convey 1 domain name and one or more IP's of DNS servers to use to resolve that domain.
For IKEv2 IPsec based VPNs, any number of domain names can be specified by the server to be used by the client. When doing split-DNS with DNSSEC trust anchors, these can be conveyed and there are strict rules on when to allow these to override public DNSSEC trust anchors as per RFC 8598.
For VPN protocols with no real standard, things are more complicated.
OpenVPN can do custom things. It all depends on the provisioning.
WireGuard has nothing related to DNS, it is all hidden in the per-vendor proprietary provisioning code. Perhaps the "wg-dynamic" userland protocol will address this. Let's hope they read RFC 8598 for inspiration to avoid the mistakes of IPsec 20 years ago.
What is important with all of the VPN cases is that you properly flush the cache when the VPN estalishes and terminates, so avoid having unreachable IP's in your DNS cache. It's important not to flush other DNS data to avoid DNS fingerprinting users across different networks.
It seems resolvectl is the API to support this with systemd-resolved.
In short, I don't understand the issue raised here of "How do you determine local resources".
For each and every domain name in the above scenario it is obvious what nameserver to send it to. There is never a need to broadcast this over a mix of public / private DNS servers.
Also, people would react very allergic if we'd start sending all DNS traffic to google or so.
So this feature has no purpose as far as I can see and is never ever a good idea, unless the user is specifically told their choice is to disconnect from a broken network or try to use the broken network with well known public DNS servers as a last resort.
Yes, resolved implements DNSSEC. But from my experience I can tell you it's very hard to do in a way resonably compatible with DNS servers deployed out there in particular edge ones. Things mostly work, but DNS servers are all broken in different ways, and we can't possibly test things on all possible cheap wifi hw...
Which is why the DNSSEC validation code should have been left to the large DNS teams at ISC, NLnetlabs, nic.cz, powerdns, IETF/ICANN communities etc. For any of the problems that systemd-resolved claims to have been written for - determing when and where to send which DNS queries to - is completely unrelated to DNSSEC and its deployment/implementation protocol interop issues and corner cases.
It was never required that systemd-resolved use its own DNSSEC validation code. I warned not to do this. The DNS community spends tens of millions of dollars a year on writing and maintaining DNS libraries and deamons and do protocol updates.
libreswan based its VPN DNS reconfiguration on the unbound daemon and libunbound. This work actually collaborated with NLnetlabs to extend unbound for all the VPN use cases to reconfigure the DNS server for all kinds of VPN domain scenario's.
FreeBSD has started using unbound with their own unbound reconfiguration tooling around it.
systemd-resolved resources were spend re-inventing DNSSEC implementations, making many of the same mistakes that the existing DNS libraries made, and is still buggy resolving certain complicated CNAME and wildcard scenario's with NSEC3. This is not because systemd-resolved programmers are bad. It is because implementing and maintaining DNSSEC is a million dollars a year operation. This money results in 3-4 production quality well maintained DNSSEC implementations that Linux can choose from. systemd-resolved simply does not have the resources to do this themselves, as is evident by the 1300 open bugs on github right now.
systemd-resolved should use an existing DNSSEC library. It can open a seperate DNS cache to each of the interface's supplied DNS servers. It can route DNS queries to the proper DNS cache. It will automatically get fixes and new record types supported by updates to thse DNS libraries.
systemd-resolved should focus on what it needs to do. Learn and reconfigure the stream of DNS queries to the right servers. It should get out of the DNS resolving and DNSSEC validation and DNS caching business.
(One thing I definitely want to add is an option to only do DNSSEC if DoT is also done, under the assumption that a DNS server that is good enough and new enough to implement the latter also should be able to do the former sanely.)
That assumption might be true now, but 5 years down the line there will be bugs and corner cases and not enough resources for systemd-resolved to track and handle this.
Also, the "only do DNSSEC if" is not a valid choice. Let's remember this whole thread started with my system getting broken because DNSSEC was silently dropped by systemd-resolved after a system upgrade.
No, it's not. It's extremely difficult. Cheap wifi router DNS servers are broken in so many ways. They return errors in some cases, freeze in others, return rubbish in others, or not at all in even others. If you ask the wrong questions anything can happen.
This is why systemd-resolved should use a DNS library and not invent its own thing. The teams at ISC, NLnetlabs, NIC.CZ, PowerDNS have spend the last 20 years dealing with this and solving it. Use their code. You don't have the resources to do this yourselves. Again, 1300 open bugs on github show you have never managed to dig out of this hole.
We pretty carefully tests and probe DNS servers but this still comes at the price that on a particular bad implementation we might take a long time until we figure out that DNSSEC simply is not possible.
See above. Also, the fix you applied now is to disable DNSSEC per default, damaging all the installed servers on enterprise networks that depend and receive completely valid DNSSEC traffic. So I am sorry if I strongly disagree with "pretty carefullt test and probe". That's not what happened to my laptop as VPN client and my mail server.
The simple fact that some DNS servers don't respond at all if you ask the "wrong" questions is already a problem: it means you have to wait for a timeout (which means super long lookups initially) or do queries in parallel. That however is a problem too since other DNS servers really don#t like it if you ask them multiple questions at once. Bombarding DNS servers with multiple questions all at once and see if one "sticks" isn't a workable strategy hence either.
Stop re-inventing the wheel. Bind, unbound, knot, powerdns do this with much more resources that you have, and for many more years than you have and they are far more aware of these issues then you are as they see a vastly larger audience with issues that the Linux desktop niche market. When systemd-resolved on github closes a bug reported and explained by Mark Andrews of Bind, the result is a bug in systemd-resolved.
So I think we do quite well in resolved on the DNSSEC front actually,
Compared to the dedicated DNS teams at the mentioned opensource DNS software, systemd-resolved is not doing quite well. It is doing poorly. Its developers are not attending the DNS conferences where issues are discussed. They are not at IETF, not at ICANN, not at DNS-OARC, not at RIPE. I have never seen systemd-resolvd people participating in the wider DNS community. A community of hundreds of DNS engineers.
So let me ExecSum what I wrote here. For systemd-resolved to become a high quality DNS solution:
1) Remove custom DNS/DNSSEC resolving code and use a well maintained DNS library. 2) Maintain a per interface DNS cache using these libraries 3) Use the above sketched out process to improve your process of deciding which interface to send the query to. This is the core of what systemd-resolved should give to the user. It is probably already pretty close to this when we work on integrating VPN supprt. 4) Deal with hotspots separately 5) Support user configured/prompted fallback using DoT and DoH to well known servers in case obtained DNS servers are too broken to work well (with DNSSEC)
No one else but systemd-resolved has item 2) and 3) and we only had a badly working dnssec-trigger that tried to do this. This is where systemd-resolved can shine.
I would seriously FALL IN LOVE with systemd-resolved for doing 2) and 3) even if I had to sometimes manually do 4) and 5)
I will work on extending 3) with VPN support in libreswan for IKEv1 and IKEv2 based IPsec VPNs.
But 1) is crucial to widespread voluntary adoption. Without 1) we have no choice to allow the user to completely disable/remove systemd-resolved from their system.
Paul
On Di, 29.09.20 11:21, Paul Wouters (paul@nohats.ca) wrote:
No further magic should be needed. The user selects this once when joining a new network.
This is terrible UI. It was on Windows, and it would be on Linux.
You shouldn't ask questions people cannot possibly answer correctly. There's a reason why Windows remained the only big OS doing that... (And do current version still do that even, I think they hid it in some secondary menu now, and do not prompt anymore?)
Corporate networks tend to define local zones. Home wifi routers all do, too. There's no clear way to know what can go directly to well-known good DNS servers and what needs to be resolved locally.
Generally, resolve the names from the DHCP obtained domain name with the DHCP obtained name servers. Yes, this is limited to one domain name, which might not be ideal, but in general when you connect to a home or corporate network directly (no VPN) then you should use their DNS servers regardless. Enterprise is likely blocking port 53 (or DoT or trying to block DoH) for security reasons. And your home network you trust?
resolved is doing just this. Note that with today's DHCP you can have many domains listed in the lease.
What is important with all of the VPN cases is that you properly flush the cache when the VPN estalishes and terminates, so avoid having unreachable IP's in your DNS cache. It's important not to flush other DNS data to avoid DNS fingerprinting users across different networks.
We maintain a per-interface cache anyway. If an interface goes down its cache ceases to exist altogether. There's no flushing necessary, since it just stops existing.
In short, I don't understand the issue raised here of "How do you determine local resources".
For each and every domain name in the above scenario it is obvious what nameserver to send it to. There is never a need to broadcast this over a mix of public / private DNS servers.
One example: As mentioned by someone else somewhere in this thread, apparently some private jboss domains are only resolvable via RH VPN but not listed as search domains in the server-provided VPN config.
So let me ExecSum what I wrote here. For systemd-resolved to become a high quality DNS solution:
- Remove custom DNS/DNSSEC resolving code and use a well maintained DNS library.
"Custom" is in the eye of the beholder. It appears to me you mean that in a derogatory way. I mean, given that Ubuntu has been enabling systemd-resolved since quite some time by default I have the suspicion our codebase is more often deployed IRL than the ones you listed? I mean, maybe I am wrong, but it's certainly not that this is exotic stuff as you imply.
I have the suspicion this is a territorial thing, no? It feels as if you believe that DNS clients that people wo are not core members of the DNS community are inherently a bad thing, and should go away, and leave the real work to the people who actually know how things work, right? I got that kind of behaviour once before when people told us to just leave service management to the real holy men of UNIX.
I mean, let's not forget: last time this came up, 5 years ago, I was so fed with dealing with this attitude of yours I just stepped away, and stopped pushing for systemd-resolved in Fedora. You and some other peeps then had your shot with dnssec-triggerd, but afaiu that didn't really go anywhere, we are still at square one, even though "millions of dollars" are behind things, as you say.
So we actually have a chance of delivering something now, but you just fight against it, just like 5y ago and nothing changed.
The reason systemd-resolved exists and that people have adopted it in some distros already and are now doing the same on Fedora is that is solves real problems, it improves things. It adds local caching, sane per-interface behaviour, makes VPN DNS mostly just work and so on, integrates LLMNR/mDNS and other sources of truth into one thing. If there was already something doing that we wouldn't have done resolved. But to my knowledge this doesn't exist. There are solutions to very specific problems, i.e. resolves for DNS with DNSSEC for example, but that is just one part of the problem and you cannot just ignore the rest.
- Maintain a per interface DNS cache using these libraries
We maintain per-interface DNS caches, but with our code.
- Use the above sketched out process to improve your process of deciding which interface to send the query to. This is the core of what systemd-resolved should give to the user. It is probably already pretty close to this when we work on integrating VPN supprt.
I understand you love the network "zone" concept. I am not a fan of the concept, but this doesn't really matter: we provide all the right IPC APIs to NM and friends to configure per-interface individually what to do. Hence, if you can convince GNOME and NM to ask people that zone question all the time (good luck!) then resolved is ready already, they just have to issue some bus calls to tell resolved what they want.
- Deal with hotspots separately
We don't deal with captive portals at all. This is up to NM and similar solutions: they should just switch the dnssec mode in resolved as they verified captive portal auth is complete.
- Support user configured/prompted fallback using DoT and DoH to well known servers in case obtained DNS servers are too broken to work well (with DNSSEC)
Again, prompting users about DoT/DoH use is — I am sorry for the strong words — simply a rubbish idea. Outside of a tiny circle of technical people noone could answer that question properly. There's no point in prompting users about things they most likely cannot answer correctly.
I mean, honestly: I for one don't even know if my local wifi router can do DoT or DNSSEC these days. It's a fritzbox, one of the fancier vendors, at least in Germany, and they keep adding and updating their firmware. It didn't use to support it, but maybe it does today, there very recently was a very big update of there firmware and I know they did some DNS love on it. But I haven't rechecked if one can now talk DNSSEC or DoT with the fritzbox. And if I can't answer that question correctly without much work, then who can (who's not Paul Wouters, that is... ;-))
BTW: that specific router model exposes its web UI on a non-internet domain "fritz.box". We can't automatically determine that for that domain we can't do DNSSEC hence. DHCP won't tell us. We can't bypass the local DNS server for it, and not do DNSSEC for it, but we don't know that this is the case — noone tells us. And that means doing DNSSEC by default or DoT to some known-good server bypassing the local wifi router isn't really an option — unless you accept that you cannot talk to your local devices anymore. Which sucks hard...
Lennart
-- Lennart Poettering, Berlin
On Tue, 29 Sep 2020, Lennart Poettering wrote:
"Custom" is in the eye of the beholder. It appears to me you mean that in a derogatory way.
I went out of my way to compare the systemd-resolved team to te DNS teams consisting of dozens of full time senior people working 20+ years on DNS with annual budgets of millions of dollars. I even pointed out these dedicated people spend weeks per year at dedicated DNS conferences and the IETF. I did this to specifically show that purely from a resource based point of view, the systemd-resolved team cannot ever match these resources. I did this specifically to prevent being seen as derogatory.
As we share the same employer, I encourage you to escalate my "derogatory behaviour" to our magenement.
I did not read the rest of your email. I will not read or respond to further emails from you on mailing lists.
Paul
On 9/29/20 6:18 PM, Lennart Poettering wrote:
On Di, 29.09.20 11:21, Paul Wouters (paul@nohats.ca) wrote:
No further magic should be needed. The user selects this once when joining a new network.
This is terrible UI. It was on Windows, and it would be on Linux.
You shouldn't ask questions people cannot possibly answer correctly. There's a reason why Windows remained the only big OS doing that... (And do current version still do that even, I think they hid it in some secondary menu now, and do not prompt anymore?)
Corporate networks tend to define local zones. Home wifi routers all do, too. There's no clear way to know what can go directly to well-known good DNS servers and what needs to be resolved locally.
Generally, resolve the names from the DHCP obtained domain name with the DHCP obtained name servers. Yes, this is limited to one domain name, which might not be ideal, but in general when you connect to a home or corporate network directly (no VPN) then you should use their DNS servers regardless. Enterprise is likely blocking port 53 (or DoT or trying to block DoH) for security reasons. And your home network you trust?
resolved is doing just this. Note that with today's DHCP you can have many domains listed in the lease.
What is important with all of the VPN cases is that you properly flush the cache when the VPN estalishes and terminates, so avoid having unreachable IP's in your DNS cache. It's important not to flush other DNS data to avoid DNS fingerprinting users across different networks.
We maintain a per-interface cache anyway. If an interface goes down its cache ceases to exist altogether. There's no flushing necessary, since it just stops existing.
In short, I don't understand the issue raised here of "How do you determine local resources".
For each and every domain name in the above scenario it is obvious what nameserver to send it to. There is never a need to broadcast this over a mix of public / private DNS servers.
One example: As mentioned by someone else somewhere in this thread, apparently some private jboss domains are only resolvable via RH VPN but not listed as search domains in the server-provided VPN config.
So let me ExecSum what I wrote here. For systemd-resolved to become a high quality DNS solution:
- Remove custom DNS/DNSSEC resolving code and use a well maintained DNS library.
"Custom" is in the eye of the beholder. It appears to me you mean that in a derogatory way. I mean, given that Ubuntu has been enabling systemd-resolved since quite some time by default I have the suspicion our codebase is more often deployed IRL than the ones you listed? I mean, maybe I am wrong, but it's certainly not that this is exotic stuff as you imply.
Is it relevant here who's the biggest? Ask any vendor of DNS software. All them would say, don't reinvent the wheel. It's harder than it seems. Yet you dont listen for 5 years.
I have the suspicion this is a territorial thing, no? It feels as if you believe that DNS clients that people wo are not core members of the DNS community are inherently a bad thing, and should go away, and leave the real work to the people who actually know how things work, right? I got that kind of behaviour once before when people told us to just leave service management to the real holy men of UNIX.
May it be because they struggle to make it all working well together, where you just say this is okay? They struggle with validation and privacy, you just throw queries to whomever would respond. There were already said countless issues. You don't want to hear them.
I mean, let's not forget: last time this came up, 5 years ago, I was so fed with dealing with this attitude of yours I just stepped away, and stopped pushing for systemd-resolved in Fedora. You and some other peeps then had your shot with dnssec-triggerd, but afaiu that didn't really go anywhere, we are still at square one, even though "millions of dollars" are behind things, as you say.
Because they cared for DNSSEC functionality. Most of implemented in resolved is already there in dnssec-trigger and unbound. It has to be admitted, resolved has better integration with NM.
So we actually have a chance of delivering something now, but you just fight against it, just like 5y ago and nothing changed.
They said their arguments 5 years ago and you are still refusing them. Problems which we failed to solve but you failed as well. But we just admit our implementation has limitations.
The reason systemd-resolved exists and that people have adopted it in some distros already and are now doing the same on Fedora is that is solves real problems, it improves things. It adds local caching, sane per-interface behaviour, makes VPN DNS mostly just work and so on, integrates LLMNR/mDNS and other sources of truth into one thing. If there was already something doing that we wouldn't have done resolved. But to my knowledge this doesn't exist. There are solutions to very specific problems, i.e. resolves for DNS with DNSSEC for example, but that is just one part of the problem and you cannot just ignore the rest.
I admit, LLMNR is the only thing resolved provides. We don't have even alternative package for that in Fedora. mdns usage had to be reduced recently, because issues with printing. Local caching were done ages ago with dnsmasq, not a new thing.
- Maintain a per interface DNS cache using these libraries
We maintain per-interface DNS caches, but with our code.
And your code cannot do "dig @127.0.0.53 . rrsig" or "delv @127.0.0.53". It is not finished.
- Use the above sketched out process to improve your process of deciding which interface to send the query to. This is the core of what systemd-resolved should give to the user. It is probably already pretty close to this when we work on integrating VPN supprt.
I understand you love the network "zone" concept. I am not a fan of the concept, but this doesn't really matter: we provide all the right IPC APIs to NM and friends to configure per-interface individually what to do. Hence, if you can convince GNOME and NM to ask people that zone question all the time (good luck!) then resolved is ready already, they just have to issue some bus calls to tell resolved what they want.
- Deal with hotspots separately
We don't deal with captive portals at all. This is up to NM and similar solutions: they should just switch the dnssec mode in resolved as they verified captive portal auth is complete.
Most of difficult things comes with strange changes during hotspot sign on. Because you don't care for DNSSEC, you don't find most of them. So any hosts used by hotspot avoids DNSSEC validation, even when it is enabled?
- Support user configured/prompted fallback using DoT and DoH to well known servers in case obtained DNS servers are too broken to work well (with DNSSEC)
Again, prompting users about DoT/DoH use is — I am sorry for the strong words — simply a rubbish idea. Outside of a tiny circle of technical people noone could answer that question properly. There's no point in prompting users about things they most likely cannot answer correctly.
Correction for that. Imagine dnssec-trigger. It discovers your broken router cannot pass DNSSEC validation, so any validation fails. Just tell user 'Hey, your box is broken. Get a replace. Continue here insecure'. And they can then ask what does it mean and how to fix it.
I mean, honestly: I for one don't even know if my local wifi router can do DoT or DNSSEC these days. It's a fritzbox, one of the fancier vendors, at least in Germany, and they keep adding and updating their firmware. It didn't use to support it, but maybe it does today, there very recently was a very big update of there firmware and I know they did some DNS love on it. But I haven't rechecked if one can now talk DNSSEC or DoT with the fritzbox. And if I can't answer that question correctly without much work, then who can (who's not Paul Wouters, that is... ;-))
I bet it uses dnsmasq inside. As its maintainer I can just say its code is ugly, but its DNSSEC support is far superrior to resolved. I am surprised, not in positive way :(
BTW: that specific router model exposes its web UI on a non-internet domain "fritz.box". We can't automatically determine that for that domain we can't do DNSSEC hence. DHCP won't tell us. We can't bypass the local DNS server for it, and not do DNSSEC for it, but we don't know that this is the case — noone tells us. And that means doing DNSSEC by default or DoT to some known-good server bypassing the local wifi router isn't really an option — unless you accept that you cannot talk to your local devices anymore. Which sucks hard...
And yet it were so simple. Just use a name that you own and can prove it has no signed denial of presence. fritz.box domain is not owned by anyone. Why do they set it there? Just bad design.
They just make broken designs, which you workaround. Let's get for example Turris MOX. Thing designed by DNS folks. It has nice local name out of box, turris.local. Only in local network, provided by mdns. Sure, resolved can find it! As can avahi. No problems with signatures, it just works. Because it was well designed by DNS aware people. Made in validating client in mind. Do you need it anywhere else but local network?
I think these cases have simple workaround. Trust unsigned answers from local routers, when validation only proves non-existent domains. That would work well with 'router.' name Microtik uses. It would still be a hack, but better than NOIMPL on dig +dnssec.
Lennart
-- Lennart Poettering, Berlin _______________________________________________ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
On 9/29/20 9:18 AM, Lennart Poettering wrote:
So let me ExecSum what I wrote here. For systemd-resolved to become a high quality DNS solution:
- Remove custom DNS/DNSSEC resolving code and use a well maintained DNS library.
"Custom" is in the eye of the beholder. It appears to me you mean that in a derogatory way. I mean, given that Ubuntu has been enabling systemd-resolved since quite some time by default I have the suspicion our codebase is more often deployed IRL than the ones you listed?
Ubuntu enables it by default, but we don't know how many people use it. My employer does not. Our AD domain has a LOT of controllers, due to a large number of offices around the world. systemd-resolved couldn't handle resolving the A record for our domain, so we had to turn it off.
I believe that was fixed in PR 11993, but that bug was enough to convince me very solidly that systemd-resolved should have re-used an existing protocol implementation rather than writing another one.
You're right that DNS has of quirks and compatibility issues, and that's exactly why writing another protocol implementation is such a poor decision.
On 29/09/2020 17:21, Paul Wouters wrote:
For the VPN scenario, it is just a little bit more complicated.
For those with proper standards, such as "Cisco IPsec", L2TP/IPsec", the VPN confiuration is dictated by the server to either send all or some traffic to the VPN server. If it is not "everything", then these VPNs convey 1 domain name and one or more IP's of DNS servers to use to resolve that domain.
For IKEv2 IPsec based VPNs, any number of domain names can be specified by the server to be used by the client. When doing split-DNS with DNSSEC trust anchors, these can be conveyed and there are strict rules on when to allow these to override public DNSSEC trust anchors as per RFC 8598.
For VPN protocols with no real standard, things are more complicated.
OpenVPN can do custom things. It all depends on the provisioning.
As an OpenVPN developer, I can't resist giving a few details here :)
OpenVPN 2.x using the openvpn-client@.service unit files depends entirely on the OpenVPN configuration. So you are right here.
OpenVPN 2.x using NetworkManager, will let NetworkManager pick up changes and apply them accordingly to the abilities of the NetworkManager-openvpn plugin.
OpenVPN 3 Linux can be enabled with systemd-resolved support [0], but out-of-the-box it will modify /etc/resolv.conf directly. Enabling systemd-resolved support, you will get fairly close to a split-DNS setup but not completely - but this integration is still considered tech-preview and we're using the v10_beta release to gain more experience with systemd-resolved across various distributions. Ubuntu 20.04 has also enabled systemd-resolved by default, but it seems it has not gone as far as Fedora 33.
Common to all of these alternatives, the VPN server must push DNS options or the client configuration file must include the appropriate --dhcp-options.
[0] https://www.mail-archive.com/openvpn-devel@lists.sourceforge.net/msg20607.html
On Tue, Sep 29, 2020 at 10:58 pm, David Sommerseth dazo@eurephia.org wrote:
Ubuntu 20.04 has also enabled systemd-resolved by default, but it seems it has not gone as far as Fedora 33.
Ubuntu has enabled systemd-resolved by default since Ubuntu 16.10, but it doesn't use nss-resolve, so getaddrinfo() uses traditional nss-dns that reads /etc/resolv.conf. In contrast, Fedora is using nss-resolve as recommended by upstream, so getaddrinfo() bypasses /etc/resolv.conf and instead talks directly to systemd-resolved.
On Tue, 2020-09-29 at 10:19 +0200, Lennart Poettering wrote:
On Mo, 28.09.20 14:29, Simo Sorce (simo@redhat.com) wrote:
On Mon, 2020-09-28 at 16:02 +0100, Tom Hughes via devel wrote:
On 28/09/2020 15:57, Marius Schwarz wrote:
Am 28.09.20 um 13:47 schrieb Zbigniew Jędrzejewski-Szmek:
DNSSEC support in resolved can be enabled through resolved.conf.
Why isn't that the default, if this resolver can do it?
Because DNSSEC is a disaster area and if you try and use it on random networks you're going to get failed lookups on a reasonable number - it's fine if you're on a known network with decent upstream servers but once you start going out and using random WiFi hotspots and things it's a very different story.
Surely this is better solved by using DoH toward known good servers for anything but the local resources ?
Well, but how do you determine "local resources"?
The same way you do now for "routing", if a name is in the search for an interface it is "local", of course other better methods can come in the future via better NM integration.
Corporate networks tend to define local zones. Home wifi routers all do, too. There's no clear way to know what can go directly to well-known good DNS servers and what needs to be resolved locally.
This is in contradiction with other mentions about routing here ?
Also, people would react very allergic if we'd start sending all DNS traffic to google or so. I mean, you can't believe how pissed people are that we have a fallback in place that if no DNS servers have been configured at all or acquired via DHCP we fall back to Cloudflare + Google DNS servers.
I would be quite pissed too if you disclosed my DNS lookups to parties I do not trust with that data, and I definitely do not trust Google DNS servers as those queries are often hijacked at the local ISP level exactly because those are known IP addresses.
Downstream distros (Debian…) tend to patch that fallback out even... I wouldn't want to know what happens if you start telling them we should now *prefer* them over local DNS servers, which is what you are saying here...
Nah, I am not saying you should use DoT to random servers, it should still be user configured or discovered through things like DHCP if that information will ever be populated there.
I mean the whole point of systemd-resolved should be to make things better including DNSSEC ?
Yes, resolved implements DNSSEC. But from my experience I can tell you it's very hard to do in a way resonably compatible with DNS servers deployed out there in particular edge ones. Things mostly work, but DNS servers are all broken in different ways, and we can't possibly test things on all possible cheap wifi hw...
(One thing I definitely want to add is an option to only do DNSSEC if DoT is also done, under the assumption that a DNS server that is good enough and new enough to implement the latter also should be able to do the former sanely.)
I think this is fine as an option. But let's not mix DNSSEC resolution with DNSSEC validation, they are very separate things.
As it was already pointed out it is also reasonably simple to detect if the local network have bad DNS servers ...
No, it's not. It's extremely difficult. Cheap wifi router DNS servers are broken in so many ways. They return errors in some cases, freeze in others, return rubbish in others, or not at all in even others. If you ask the wrong questions anything can happen. We pretty carefully tests and probe DNS servers but this still comes at the price that on a particular bad implementation we might take a long time until we figure out that DNSSEC simply is not possible.
I did not mean to trivialize the technical means, but current DNS libraries tend to do a reasonable job already, you could simply just use one of them to offload the problem from your shoulders. There is little value for systemd-resolved to re-implement all of DNS handling any way, the value is in the central caching and smart resolving features.
The simple fact that some DNS servers don't respond at all if you ask the "wrong" questions is already a problem: it means you have to wait for a timeout (which means super long lookups initially) or do queries in parallel. That however is a problem too since other DNS servers really don#t like it if you ask them multiple questions at once. Bombarding DNS servers with multiple questions all at once and see if one "sticks" isn't a workable strategy hence either.
And then, when you figured out that the local DNS server can do some thing but not others, and are happy you will eventually notice that many "edge" DNS servers have a split personality: for some domains they are just a proxy for the upstream DNS servers and other domains they implement things locally, which means the feature set you can probe differs vastly depending on which domains you query. That's particular awful for reverse lookups btw, since the domain separation is a lot more blurry there.
So I think we do quite well in resolved on the DNSSEC front actually, but people still are annoyed that one some local DNS servers DNSSEC doesn't work and or we take too long to detect that it doesn't.
I think you can do better by leveraging the existing mature code for these annoying aspects, and focusing on the things a resolver is uniquely positioned to do instead. DNSSEC is still evolving (I've seen NSEC5 proposal for example, and more), let the DNS experts chase those things and just reuse that work.
Simo.
On Mon, Sep 28, 2020 at 4:48 AM Zbigniew Jędrzejewski-Szmek < zbyszek@in.waw.pl> wrote:
On Mon, Sep 28, 2020 at 12:44:13AM -0400, Paul Wouters wrote:
Subject: Re: Fedora 33 System-Wide Change proposal: systemd-resolved
paul@thinkpad:~$ dig +dnssec vpn.nohats.ca @127.0.0.53
; <<>> DiG 9.11.22-RedHat-9.11.22-1.fc33 <<>> +dnssec vpn.nohats.ca @
127.0.0.53
;; global options: +cmd ;; Got answer: ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 51669 ;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1
;; OPT PSEUDOSECTION: ; EDNS: version: 0, flags: do; udp: 65494 ; OPT=5: 05 07 08 0a 0d 0e 0f (".......") ; OPT=6: 01 02 04 ("...") ; OPT=7: 01 (".") ;; QUESTION SECTION: ;vpn.nohats.ca. IN A
;; ANSWER SECTION: vpn.nohats.ca. 10 IN A 193.110.157.148
;; Query time: 143 msec ;; SERVER: 127.0.0.53#53(127.0.0.53) ;; WHEN: Mon Sep 28 00:18:32 EDT 2020 ;; MSG SIZE rcvd: 81
libreswan will see this result as an attack, and fail to resolve DNS
names
in its configuration. My postfix daemon will hold on to mail because it cannot get a DNSSEC proof of denial of existence of TLSA records if all DNSSEC records are filtered - even for domains that don't use DNSSEC because the denial of existence of DNSSEC for a specific domain requires the return of DNSSEC records that systemd now does not return.
I am sorry that I did not follow the fedora list carefully enough to notice this feature when it was proposed.
This change is harmful to network security, impacts existing
installations
depending on DNSSEC security, and leaks private queries for VPN/internal domains to the open internet, and prefers faster non-dnssec answers over dnssec validated answers. It fails various types of queries, misimplements part of the DNS protocol. Not only according to me, but to 20years+ developers of the bind software as well.
You're mixing a few different things here. We decided to not enable DNSSEC in resolved with this change, at least initially. For most users, DNSSEC is problematic because various intermediary DNS servers found in hotspots and routers don't support DNSSEC properly, leading to hard-to-debug validation failures. DNSSEC support in resolved can be enabled through resolved.conf. This may be a reasonable thing to do in an environment where the configured dns servers are known to support dnssec properly.
Paul may well have been mixing different things here, but I don't think you answered the one that seems like the most severe problem: systemd-resolved removed perfectly valid DNSSEC records that were supplied by the upstream server. One might reasonably debate whether Fedora's default DNS resolver configuration should validate DNSSEC, but I think it should honor the DO bit in client requests and return DNSSEC data.
Could the F33 default be changed to at least forward DNSSEC data to clients that ask for it?
(My personal view is that either this should happen or that systemd-resolved-as-default should be reverted for F33, but I'm not on FESCo.)
* Andrew Lutomirski:
Paul may well have been mixing different things here, but I don't think you answered the one that seems like the most severe problem: systemd-resolved removed perfectly valid DNSSEC records that were supplied by the upstream server. One might reasonably debate whether Fedora's default DNS resolver configuration should validate DNSSEC, but I think it should honor the DO bit in client requests and return DNSSEC data.
FWIW, this is https://bugzilla.redhat.com/show_bug.cgi?id=1879028.
Thanks, Florian
On Mon, Sep 28, 2020 at 06:36:02PM +0200, Florian Weimer wrote:
- Andrew Lutomirski:
Paul may well have been mixing different things here, but I don't think you answered the one that seems like the most severe problem: systemd-resolved removed perfectly valid DNSSEC records that were supplied by the upstream server. One might reasonably debate whether Fedora's default DNS resolver configuration should validate DNSSEC, but I think it should honor the DO bit in client requests and return DNSSEC data.
FWIW, this is https://bugzilla.redhat.com/show_bug.cgi?id=1879028.
In an ideal world, we would just implement this missing functionality. It's definitely on the TODO list, and there has been some preparatory work done, but so far nobody found the time. If this is judged necessary, we'll raise the priority of that work. Nevertheless, I don't think it is such high priority — the number of people using DNSSEC is not too large, and they are generally power-users who understand how to specify a different server. So while definitely annoying, I didn't consider this a deal-breaker.
Zbyszek
On Mon, Sep 28, 2020 at 04:59:17PM +0000, Zbigniew Jędrzejewski-Szmek wrote:
On Mon, Sep 28, 2020 at 06:36:02PM +0200, Florian Weimer wrote:
- Andrew Lutomirski:
Paul may well have been mixing different things here, but I don't think you answered the one that seems like the most severe problem: systemd-resolved removed perfectly valid DNSSEC records that were supplied by the upstream server. One might reasonably debate whether Fedora's default DNS resolver configuration should validate DNSSEC, but I think it should honor the DO bit in client requests and return DNSSEC data.
FWIW, this is https://bugzilla.redhat.com/show_bug.cgi?id=1879028.
In an ideal world, we would just implement this missing functionality. It's definitely on the TODO list, and there has been some preparatory work done, but so far nobody found the time. If this is judged necessary, we'll raise the priority of that work. Nevertheless, I don't think it is such high priority — the number of people using DNSSEC is not too large, and they are generally power-users who understand how to specify a different server. So while definitely annoying, I didn't consider this a deal-breaker.
DNSSEC is not meant for power-users, and it doesn't require specifying "a different server".
I thought Fedora was supposed to be First? How can it be if Fedora chooses to use/configure software by default that is missing critical DNSSEC functionality and breaks DNS standards?
On Mon, Sep 28, 2020 at 1:20 PM Chuck Anderson cra@alum.wpi.edu wrote:
On Mon, Sep 28, 2020 at 04:59:17PM +0000, Zbigniew Jędrzejewski-Szmek wrote:
On Mon, Sep 28, 2020 at 06:36:02PM +0200, Florian Weimer wrote:
- Andrew Lutomirski:
Paul may well have been mixing different things here, but I don't think you answered the one that seems like the most severe problem: systemd-resolved removed perfectly valid DNSSEC records that were supplied by the upstream server. One might reasonably debate whether Fedora's default DNS resolver configuration should validate DNSSEC, but I think it should honor the DO bit in client requests and return DNSSEC data.
FWIW, this is https://bugzilla.redhat.com/show_bug.cgi?id=1879028.
In an ideal world, we would just implement this missing functionality. It's definitely on the TODO list, and there has been some preparatory work done, but so far nobody found the time. If this is judged necessary, we'll raise the priority of that work. Nevertheless, I don't think it is such high priority — the number of people using DNSSEC is not too large, and they are generally power-users who understand how to specify a different server. So while definitely annoying, I didn't consider this a deal-breaker.
DNSSEC is not meant for power-users, and it doesn't require specifying "a different server".
I thought Fedora was supposed to be First? How can it be if Fedora chooses to use/configure software by default that is missing critical DNSSEC functionality and breaks DNS standards?
DNSSEC is broken by design for most users. For example, I literally cannot use it because my corporate VPN does not provide DNSSEC support because the DNS server software doesn't support it. If I didn't know what to look for, I'd just say Fedora is broken. Same goes for DoT and DoH.
On Mon, Sep 28, 2020 at 1:20 pm, Chuck Anderson cra@alum.wpi.edu wrote:
I thought Fedora was supposed to be First? How can it be if Fedora chooses to use/configure software by default that is missing critical DNSSEC functionality and breaks DNS standards?
Well, let's amend that to "first when it's smart to be first." We can't ever *require* DNSSEC validation, because Windows and macOS are not going to do so. They have to be first. I could just as well counter with "How can Fedora be first if it refuses to implement split DNS behavior by default that breaks user expectations and leaks queries to unexpected networks?"
As for just passing along records, see Zbigniew's responses; it's possible to do by default, just not a priority. This is really only interesting for specialized applications like mail servers that live on controlled networks where you know that DNSSEC is not broken, i.e. not relevant for 99% of users. If you're running such applications, it's a one-line change in resolved.conf to enable DNSSEC, not really a big deal. It's annoying to have to edit an extra config file, yes, and we should do better, but I don't think that should derail this change.
On Mon, 28 Sep 2020, Michael Catanzaro wrote:
Well, let's amend that to "first when it's smart to be first." We can't ever *require* DNSSEC validation, because Windows and macOS are not going to do so.
https://tools.ietf.org/id/draft-pauly-add-resolver-discovery-01.html
That draft has a Microsoft and Apple co-author on it.
It states for example:
There are several methods that can be used to discover and validate a resolver designation: * Discovery using SVCB DNS records (Section 3.1), and validation using DNSSEC
This document is precisely to discover DNSSEC (and DNS encryption) services reliably so that DNSSEC validation can be turned on by default.
Can you cite the documentation for your statement that these two vendors are not working on enabling DNSSEC validation?
They have to be first. I could just as well counter with "How can Fedora be first if it refuses to implement split DNS behavior by default that breaks user expectations and leaks queries to unexpected networks?"
How about systemd-resolved people join the IETF draft process, so that they can still influence the protocols while they are being designed, so that it can be made to work with systemd-resolved properly? There are a dozens of long time seasonsed DNS architects and programmers at the IETF working on this problem now. Join their effort.
As for just passing along records, see Zbigniew's responses; it's possible to do by default, just not a priority. This is really only interesting for specialized applications like mail servers that live on controlled networks where you know that DNSSEC is not broken, i.e. not relevant for 99% of users.
Please stop filtering out the use cases you don't like.
Besides that, what percentage of desktops / laptops uses Linux versus what percentage of servers use Linux? I would strongly argue the case is quite the reverse. Linux desktop uses are 0.000000% and Linux on servers is like 99.999999%
If you're running such applications, it's a one-line change in resolved.conf to enable DNSSEC, not really a big deal. It's annoying to have to edit an extra config file, yes, and we should do better, but I don't think that should derail this change.
If systemd-resolved was only installed on Linux desktops, you would have a much stronger argument. But right now it is part of the same package as /sbin/init.
Paul
On Mo, 28.09.20 14:11, Paul Wouters (paul@nohats.ca) wrote:
On Mon, 28 Sep 2020, Michael Catanzaro wrote:
Well, let's amend that to "first when it's smart to be first." We can't ever *require* DNSSEC validation, because Windows and macOS are not going to do so.
https://tools.ietf.org/id/draft-pauly-add-resolver-discovery-01.html
That draft has a Microsoft and Apple co-author on it.
It states for example:
There are several methods that can be used to discover and validate a resolver designation:
- Discovery using SVCB DNS records (Section 3.1), and validation using DNSSEC
This document is precisely to discover DNSSEC (and DNS encryption) services reliably so that DNSSEC validation can be turned on by default.
Can you cite the documentation for your statement that these two vendors are not working on enabling DNSSEC validation?
Let me emphasize that I primarily care about building something that works IRL. i.e. plans for the future of are absolutely secondary to the work we try to do with resolved. I mean, let's not forget: DNSSEC was the future 5 years already, and it was a dumpster fire back then already.
So, lofty plans, great future specs and stuff, is all great, but we try to match up with real implementations of the stuff that is out there, that's our focus.
RFCs become interesting if they are actually deployed. i.e. DoT became interesting and something we started to support the instance it was actually deployed on 8.8.8.8 for example. Before that it really didn't matter, sorry.
And yes, client-side DNSSEC still hasn't happen, regardless how many specs have been written for it. We support it nonetheless, in an opt-in kind of fashion. We'll revisit this and find ways to support it better out-of-the-box but a spec saying that you should do it doesn't influence us at all on this: real life does, and what is deployed on the real Internet.
Or to say this differently: resolved is *not* supposed to be a *pioneer* of any of this, but just an implementation of what is used in real life and has shown it works.
You come from a very different perspective, I totally understand that. You intend to *design* DNS, and hence have the luxury to not care as much about edge routers and how they actually implement DNS today.
Lennart
-- Lennart Poettering, Berlin
On Mon, 2020-09-28 at 12:30 -0500, Michael Catanzaro wrote:
On Mon, Sep 28, 2020 at 1:20 pm, Chuck Anderson cra@alum.wpi.edu wrote:
I thought Fedora was supposed to be First? How can it be if Fedora chooses to use/configure software by default that is missing critical DNSSEC functionality and breaks DNS standards?
Well, let's amend that to "first when it's smart to be first." We can't ever *require* DNSSEC validation, because Windows and macOS are not going to do so. They have to be first. I could just as well counter with "How can Fedora be first if it refuses to implement split DNS behavior by default that breaks user expectations and leaks queries to unexpected networks?"
Requiring Validation, and *passing through* requests for DNSSEC records are *entirely* different things.
As for just passing along records, see Zbigniew's responses; it's possible to do by default, just not a priority. This is really only interesting for specialized applications like mail servers that live on controlled networks where you know that DNSSEC is not broken, i.e. not relevant for 99% of users. If you're running such applications, it's a one-line change in resolved.conf to enable DNSSEC, not really a big deal. It's annoying to have to edit an extra config file, yes, and we should do better, but I don't think that should derail this change.
It is breaking working systems for people, at the very least it should be very high on priority, not downplayed to irrelevance (as that is the route for never fixing it, and having people always disable resolved as a matter of fact)
On Mo, 28.09.20 13:20, Chuck Anderson (cra@alum.wpi.edu) wrote:
On Mon, Sep 28, 2020 at 04:59:17PM +0000, Zbigniew Jędrzejewski-Szmek wrote:
On Mon, Sep 28, 2020 at 06:36:02PM +0200, Florian Weimer wrote:
- Andrew Lutomirski:
Paul may well have been mixing different things here, but I don't think you answered the one that seems like the most severe problem: systemd-resolved removed perfectly valid DNSSEC records that were supplied by the upstream server. One might reasonably debate whether Fedora's default DNS resolver configuration should validate DNSSEC, but I think it should honor the DO bit in client requests and return DNSSEC data.
FWIW, this is https://bugzilla.redhat.com/show_bug.cgi?id=1879028.
In an ideal world, we would just implement this missing functionality. It's definitely on the TODO list, and there has been some preparatory work done, but so far nobody found the time. If this is judged necessary, we'll raise the priority of that work. Nevertheless, I don't think it is such high priority — the number of people using DNSSEC is not too large, and they are generally power-users who understand how to specify a different server. So while definitely annoying, I didn't consider this a deal-breaker.
DNSSEC is not meant for power-users, and it doesn't require specifying "a different server".
I thought Fedora was supposed to be First? How can it be if Fedora chooses to use/configure software by default that is missing critical DNSSEC functionality and breaks DNS standards?
DNSSEC doesn't really work client-side IRL. The DNS servers typical clients talk to generally do not implement what you need, and if they do not correctly. This means if you have a great network admin who set everything up right it might work, but DNSSEC on a laptop that moves around and connects to a WLAN here, and another WLAN there and a third WLAN over there is just a nightmare.
If the other big OSes would enable DNSSEC client-side by default things might change, but neither Windows nor MacOS or Android do.
Lennart
-- Lennart Poettering, Berlin
On Mon, Sep 28, 2020 at 11:07 AM Lennart Poettering mzerqung@0pointer.de wrote:
On Mo, 28.09.20 13:20, Chuck Anderson (cra@alum.wpi.edu) wrote:
On Mon, Sep 28, 2020 at 04:59:17PM +0000, Zbigniew Jędrzejewski-Szmek
wrote:
On Mon, Sep 28, 2020 at 06:36:02PM +0200, Florian Weimer wrote:
- Andrew Lutomirski:
Paul may well have been mixing different things here, but I don't think you answered the one that seems like the most severe problem: systemd-resolved removed perfectly valid DNSSEC records that were supplied by the upstream server. One might reasonably debate
whether
Fedora's default DNS resolver configuration should validate DNSSEC, but I think it should honor the DO bit in client requests and
return
DNSSEC data.
FWIW, this is https://bugzilla.redhat.com/show_bug.cgi?id=1879028.
In an ideal world, we would just implement this missing functionality. It's definitely on the TODO list, and there has been some preparatory work done, but so far nobody found the time. If this is judged
necessary,
we'll raise the priority of that work. Nevertheless, I don't think it
is
such high priority — the number of people using DNSSEC is not too
large,
and they are generally power-users who understand how to specify a
different
server. So while definitely annoying, I didn't consider this a
deal-breaker.
DNSSEC is not meant for power-users, and it doesn't require specifying "a different server".
I thought Fedora was supposed to be First? How can it be if Fedora chooses to use/configure software by default that is missing critical DNSSEC functionality and breaks DNS standards?
DNSSEC doesn't really work client-side IRL. The DNS servers typical clients talk to generally do not implement what you need, and if they do not correctly. This means if you have a great network admin who set everything up right it might work, but DNSSEC on a laptop that moves around and connects to a WLAN here, and another WLAN there and a third WLAN over there is just a nightmare.
If the other big OSes would enable DNSSEC client-side by default things might change, but neither Windows nor MacOS or Android do.
The old unbound-resolveconf actually worked quite well when I played with it. The only problem I had was that I couldn't load google.com from one particular network. Upon a bit of investigation, I discovered that the ISP was maliciously replacing the A records for google.com with its own servers to inject JavaScript. So unbound-resolveconf's behavior was arguably correct. A better solution might have been to pop up some kind of notification like "your network is attempting to tamper with google.com. You can use the tampered version of google.com at your own risk by following these instructions, or you could try to access the real google.com by doing this other thing".
On Mo, 28.09.20 11:10, Andrew Lutomirski (luto@mit.edu) wrote:
If the other big OSes would enable DNSSEC client-side by default things might change, but neither Windows nor MacOS or Android do.
The old unbound-resolveconf actually worked quite well when I played with it. The only problem I had was that I couldn't load google.com from one particular network. Upon a bit of investigation, I discovered that the ISP was maliciously replacing the A records for google.com with its own servers to inject JavaScript. So unbound-resolveconf's behavior was arguably correct. A better solution might have been to pop up some kind of notification like "your network is attempting to tamper with google.com. You can use the tampered version of google.com at your own risk by following these instructions, or you could try to access the real google.com by doing this other thing".
That's terrible UI. The thing is: this stuff should just work and not pester users with questions they couldn#t possibly understand or answer properly.
I mean, let's face it. DNSSEC is great, but does it actually make your bank transfer safer? not really, SSL certs validate domains too in a way, so DNSSEC isn't strictly necessary because trusting a SSL CA isn't much different than trusting the DNSSEC root.
hence: client-side DNSSEC is certainly something we should support if we can: but it's not deployable as default as it stands now, simply because it breaks more stuff than it helps.
Lennart
-- Lennart Poettering, Berlin
On Mon, 2020-09-28 at 16:59 +0000, Zbigniew Jędrzejewski-Szmek wrote:
On Mon, Sep 28, 2020 at 06:36:02PM +0200, Florian Weimer wrote:
- Andrew Lutomirski:
Paul may well have been mixing different things here, but I don't think you answered the one that seems like the most severe problem: systemd-resolved removed perfectly valid DNSSEC records that were supplied by the upstream server. One might reasonably debate whether Fedora's default DNS resolver configuration should validate DNSSEC, but I think it should honor the DO bit in client requests and return DNSSEC data.
FWIW, this is https://bugzilla.redhat.com/show_bug.cgi?id=1879028.
In an ideal world, we would just implement this missing functionality. It's definitely on the TODO list, and there has been some preparatory work done, but so far nobody found the time. If this is judged necessary, we'll raise the priority of that work. Nevertheless, I don't think it is such high priority — the number of people using DNSSEC is not too large, and they are generally power-users who understand how to specify a different server. So while definitely annoying, I didn't consider this a deal-breaker.
Zbyszek
Sorry Zbyszek, but as other said, a *default* resolver that break the standard is definitely a problem.
As non-default, systemd-resolved may decide to break anything it wants, but once you decide you want to be the default in a system, then standard compliance becomes paramount.
Of course systemd-resolved can deviate *optionally*, but the default needs to be compliant, which means returning all records the client requests, and resolving all names as the standard requires, and supporting all record types, etc..
Simo.
On Mo, 28.09.20 18:36, Florian Weimer (fweimer@redhat.com) wrote:
- Andrew Lutomirski:
Paul may well have been mixing different things here, but I don't think you answered the one that seems like the most severe problem: systemd-resolved removed perfectly valid DNSSEC records that were supplied by the upstream server. One might reasonably debate whether Fedora's default DNS resolver configuration should validate DNSSEC, but I think it should honor the DO bit in client requests and return DNSSEC data.
FWIW, this is https://bugzilla.redhat.com/show_bug.cgi?id=1879028.
It's on the TODO list. But this creates probems of its own. Propagating DO stuff as is cannot work for LLMNR, mDNS, company-scope DNS and so on, i.e. anything that isn't the official Internet DNS.
systemd-resolved is not a traditional DNS server after all. It is a client to classic DNS, MulticastDNS, LLMNR, local definitions from /etc/hosts, synthetic records such as _gateway, localhost and the local hostname and similar, and then exposes the combination to clients. It also is capable of (limited) merging of DNS zones from different sources (think: you have a VPN connection with some zones the official internet doesn't have). Only one of these backends has a concept of DNSSEC + DO: classic DNS talking to upstream Internet DNS servers. Thus exposing DO to clients is a bit weird, it suggests clients could validate whatever we return with DNSSEC, but that isn't doable, stuff that doesn't come from classic Internet DNS cannot possibly be DNSSEC validated. So we'd have to have a weird split: for some domains we could propagate DO stuff, but for others we simply have to block out, because the requests simply doesn't make sense for it. What's worse, resolved would start having a split personality: for DO replies we'd propagate the 1:1 upstream responses, while for everything else we'd return our own stuff, from our own synthesis, sourcing and and son. A local DNS client talking to resolved would see a feature set that would be very different depending if you ask with or without DO, because if you ask with DO you effectively just talk to one of the upstream servers, while without you will speak to systemd-resolved. I can tell you from implementing much of systemd-resolved: dealing with a server that in some conditions acts like X and in other conditions like Y is super annoying. Many many home routers act like that: they synthesize records for the admin UI, which work very differently protocol-wise then talking to proper public Internet.
systemd-resolved is not supposed to be a real DNS *server*. It's supposed to be a good, combined client for the popular name resolution protocols, and the fact that we also listen on a port 53 is mostly to provide compat with local app code that doesn't go through glibc NSS for its name resolution needs. If you expect a full blown DNS server on port 53 then it's not what systemd-resolved is or strives to be.
So far we side-step the DO issue by returning a clean error when clients set DO: "not implemented", plus a log message in syslog with more info. I'd argue that for the vast majority of users this is perfectly enough. Because IRL client-side DNSSEC doesn't really exist outside of some very specific circles of DNS nerds, I guess. Yes, I dare to suggest that for your typical GNOME/Fedora Workstation it really doesn't matter.
I understand that some people want DNSSEC/DO stuff work client side just like that, and as mentioned we'll add that, but let's not pretend this was "obvious" and without pitfalls of its own. I understand some loud people here consider Internet DNS the true gospel, and mDNS and LLMNR and all kinds of other forms of popular name resolution, local and remote heresy, but I think we just have to agree to disagree here: i am interested in a *client* solution that speaks the big protocols deployed today, that supports talking to company-scope DNS servers, and that's not just DNS but also Microsoft-style LLMNR and Apple-style mDNS.
Until we implement the DO-bypass stuff, there's an easy way to bypass systemd-resolved for your DNSSEC resolver needs: just symlink /run/systemd/resolve/resolv.conf to /etc/resolv.conf (i.e. instead of /run/systemd/resolve/stub-resolv.conf). If you do that all local DNS clients will use the upstream DNS servers resolved picked up directly, bypassing resolved. But of course, if you do that then you also lose LLMNR, mdns, all local synthetic records, merging of VPN zones and so on. If you do things like this, then resolved will work like a re-implementation of the "resolvconf" tool that Debian and FreeBSD have, not more (it does provide a command line compatible binary for that btw). I have the strong suspicion that the same people who are able to deploy working DNSSEC client side and are educated enough in DNSSEC to know what that even means are also capable of replacing that one symlink in /etc.
Lennart
-- Lennart Poettering, Berlin
On Mon, Sep 28, 2020 at 11:04 AM Lennart Poettering mzerqung@0pointer.de wrote:
On Mo, 28.09.20 18:36, Florian Weimer (fweimer@redhat.com) wrote:
- Andrew Lutomirski:
Paul may well have been mixing different things here, but I don't think you answered the one that seems like the most severe problem: systemd-resolved removed perfectly valid DNSSEC records that were supplied by the upstream server. One might reasonably debate whether Fedora's default DNS resolver configuration should validate DNSSEC, but I think it should honor the DO bit in client requests and return DNSSEC data.
FWIW, this is https://bugzilla.redhat.com/show_bug.cgi?id=1879028.
It's on the TODO list. But this creates probems of its own. Propagating DO stuff as is cannot work for LLMNR, mDNS, company-scope DNS and so on, i.e. anything that isn't the official Internet DNS.
systemd-resolved is not a traditional DNS server after all. It is a client to classic DNS, MulticastDNS, LLMNR, local definitions from /etc/hosts, synthetic records such as _gateway, localhost and the local hostname and similar, and then exposes the combination to clients. It also is capable of (limited) merging of DNS zones from different sources (think: you have a VPN connection with some zones the official internet doesn't have). Only one of these backends has a concept of DNSSEC + DO: classic DNS talking to upstream Internet DNS servers. Thus exposing DO to clients is a bit weird, it suggests clients could validate whatever we return with DNSSEC, but that isn't doable, stuff that doesn't come from classic Internet DNS cannot possibly be DNSSEC validated. So we'd have to have a weird split: for some domains we could propagate DO stuff, but for others we simply have to block out, because the requests simply doesn't make sense for it. What's worse, resolved would start having a split personality: for DO replies we'd propagate the 1:1 upstream responses, while for everything else we'd return our own stuff, from our own synthesis, sourcing and and son. A local DNS client talking to resolved would see a feature set that would be very different depending if you ask with or without DO, because if you ask with DO you effectively just talk to one of the upstream servers, while without you will speak to systemd-resolved. I can tell you from implementing much of systemd-resolved: dealing with a server that in some conditions acts like X and in other conditions like Y is super annoying. Many many home routers act like that: they synthesize records for the admin UI, which work very differently protocol-wise then talking to proper public Internet.
Indeed, the problem you're trying to solve is hard.
systemd-resolved is not supposed to be a real DNS *server*. It's supposed to be a good, combined client for the popular name resolution protocols, and the fact that we also listen on a port 53 is mostly to provide compat with local app code that doesn't go through glibc NSS for its name resolution needs. If you expect a full blown DNS server on port 53 then it's not what systemd-resolved is or strives to be.
Then perhaps you should have a libsystemdresolvedclient and start convincing programs that want this behavior to use it.
On Mo, 28.09.20 11:06, Andrew Lutomirski (luto@mit.edu) wrote:
Indeed, the problem you're trying to solve is hard.
systemd-resolved is not supposed to be a real DNS *server*. It's supposed to be a good, combined client for the popular name resolution protocols, and the fact that we also listen on a port 53 is mostly to provide compat with local app code that doesn't go through glibc NSS for its name resolution needs. If you expect a full blown DNS server on port 53 then it's not what systemd-resolved is or strives to be.
Then perhaps you should have a libsystemdresolvedclient and start convincing programs that want this behavior to use it.
Oh, we did. It's called "glibc NSS". It's pretty popular, but not popular enough as name resolution API apparently... I doubt we could ever be more successful than glibc with any C library I guess.
I figure we come from different generations though: C libraries is not how you gonna convince Java or Rust or Go peope. In particular as this is an IPC question anyway, not a language binding question.
We offer our APIs via four ways these days:
1. Via D-Bus 2. Via Varlink 3. Via NSS (through the nss-resolve module, which is ultimately just a wrapper around the D-Bus/Varlink thing) 4. Via local DNS stub on 127.0.0.53
As it turns out the latter kinda works everywhere, it's hard to make a case for everyone to not use it if it works for this stuff. It uses DNS as local IPC. Which is pretty universal, and just works for almost everyone.
Lennart
-- Lennart Poettering, Berlin
On 9/28/20 11:03 AM, Lennart Poettering wrote:
I have the strong suspicion that the same people who are able to deploy working DNSSEC client side and are educated enough in DNSSEC to know what that even means are also capable of replacing that one symlink in /etc.
i'll start with: i'm generally a huge use-systemd-*-whenever-possible bigot. aka, NOT an anti-systemd'er.
but, this^ comment, though likely _true_, causes concern for those of us out here, in the peanut gallery.
<peanut-gallery hat>on</peanut-gallery hat>
as Paul Wouters has repeatedly pointed out ... others' use cases are not mine.
and statements such as "It's easy to do using resolvectl" make me ... antsy.
forcing use of, or switching by (coming) default, to solutions that cause significant breakage to working systems, is bad news. whether or not that breakage can be 'easily' worked around.
easy != zero effort / zero cost.
my typical 'small-office install' includes local split-horizon bind9 implementation, as well as instances of both NSD4/Unbound, multiple VPN links, and varied routing for IPv4 & IPv6 dns queries, as well as general & specific traffic. internal services/capabilities include mail, DNSSEC and instances of secure DNS (DoT/DoH), geoIP, etc etc.
'large-office' installs are correspondingly _more_ 'convoluted'.
that said, it all works. well.
(my) users see/use a static /etc/resolv.conf, with, generally, a single nameserver entry.
recent experiments (on F32, admittedly -- *not* yet F33) with NetworkManager &/or systemd-resolved here were nightmarish; a seemingly endless array of 'gotchas' ...
after trying, and failing, to chase down & completely resolve all the problems, the functional solution i landed on was
(1) disable NetworkManager everywhere (yes, causes some current pain with laptops) (2) enable/deploy systemd-networkd everywhere (3) disable systemd-resolved everywhere; reset to own-managed, /etc/resolv.conf (4) disabled DoH settings in all Firefox instances
it all works, again.
if/until a 'forced switch', &/or new default, works in _our_ use cases -- regardless of whether or not they fit into _others_ limited views -- then that^^ is my default.
here's hoping that turning "it" all OFF, without breaking 'the rest' of systemd*, or F33+, remains functionally doable ...
<peanut-gallery hat>off</peanut-gallery hat>
On Mon, Sep 28, 2020 at 11:19 AM PGNet Dev pgnet.dev@gmail.com wrote:
On 9/28/20 11:03 AM, Lennart Poettering wrote:
I have the strong suspicion that the same people who are able to deploy working DNSSEC client side and are educated enough in DNSSEC to know what that even means are also capable of replacing that one symlink in /etc.
i'll start with: i'm generally a huge use-systemd-*-whenever-possible bigot. aka, NOT an anti-systemd'er.
but, this^ comment, though likely _true_, causes concern for those of us out here, in the peanut gallery.
<peanut-gallery hat>on</peanut-gallery hat>
as Paul Wouters has repeatedly pointed out ... others' use cases are not mine.
and statements such as "It's easy to do using resolvectl" make me ... antsy.
forcing use of, or switching by (coming) default, to solutions that cause significant breakage to working systems, is bad news. whether or not that breakage can be 'easily' worked around.
easy != zero effort / zero cost.
my typical 'small-office install' includes local split-horizon bind9 implementation, as well as instances of both NSD4/Unbound, multiple VPN links, and varied routing for IPv4 & IPv6 dns queries, as well as general & specific traffic. internal services/capabilities include mail, DNSSEC and instances of secure DNS (DoT/DoH), geoIP, etc etc.
'large-office' installs are correspondingly _more_ 'convoluted'.
that said, it all works. well.
(my) users see/use a static /etc/resolv.conf, with, generally, a single nameserver entry.
recent experiments (on F32, admittedly -- *not* yet F33) with NetworkManager &/or systemd-resolved here were nightmarish; a seemingly endless array of 'gotchas' ...
after trying, and failing, to chase down & completely resolve all the problems, the functional solution i landed on was
(1) disable NetworkManager everywhere (yes, causes some current pain with laptops)
I would have expected NetworkManager to handle this kind of setup just fine. What went wrong?
On 9/28/20 11:21 AM, Andrew Lutomirski wrote:
I would have expected NetworkManager to handle this kind of setup just fine. What went wrong?
getting offtopic, but ... a laundry list.
including broken routes, missed existing unit-file interface dependencies particularly once bridges get involved, absolute disfunction with setups including shorewall & wireguard, blah blah blah.
and, generally, a stubborn, been-there-done-that view that NM has *no place* on my servers &/or workstations.
i'm sure _all_ of it was "bandaid-able" with enough monkeying around.
simply not worth the effort, cost, and uncertainty; i'd already moved OS/distros to get away from a flaky network stack (among other issues, of course; not the least of which was, ironically, old/dusty systemd version!) ... and intransigent 'policy' decisions that came with it.
systemd-networkd with enterprise-grade/fits-my-needs DNS servers/resolvers works. very nicely.
fingers-crossed that removing the unwanted/unwelcome cruft remains an easy option.
as always, YMMV.
On Mon, 28 Sep 2020, Lennart Poettering wrote:
stuff that doesn't come from classic Internet DNS cannot possibly be DNSSEC validated.
This statement is incorrect. Please read RFC 8598 and perhaps read up on the handling of Special Use Domain Names and DNSSEC validation. No one expects .local to be signed. This is not an actual problem. You are not unique. Participate at the IETF, write and implement new RFCs that fix your current issues.
If you expect a full blown DNS server on port 53 then it's not what systemd-resolved is or strives to be.
For non-local queries, that is exactly what applications expect and depend on.
So far we side-step the DO issue by returning a clean error when clients set DO: "not implemented"
That is not what I see:
paul@thinkpad:~)$ dig +dnssec nohats.ca @127.0.0.53
; <<>> DiG 9.11.22-RedHat-9.11.22-1.fc33 <<>> +dnssec nohats.ca @127.0.0.53 ;; global options: +cmd ;; Got answer: ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 6543 ;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1
I get NOERROR, so you should file another systemd-resolved bug report if the expected behaviour was NOTIMP.
But even so, DNS libraries getting NOTIMP will just try to go to the next server listed in resolv.conf but there is only 127.0.0.53, so it fails. So you do not "side step" this issue at all. Implementing NOTIMP will change nothing.
, plus a log message in syslog with more info. I'd argue that for the vast majority of users this is perfectly enough.
This is again redefining the use cases and pre-selecting the type of users you wish to support and punting everything not in your use case to the "you are the advanced user, you know how to work around this".
I was an enduser with a laptop dead in the water. It did not matter how advanced I am or not. It should not happen, and the fix is not for me to read syslog messages, google for documentation and then manually fix my system. My system should not break.
Because IRL client-side DNSSEC doesn't really exist outside of some very specific circles of DNS nerds, I guess. Yes, I dare to suggest that for your typical GNOME/Fedora Workstation it really doesn't matter.
This is a very outdated view.
I understand that some people want DNSSEC/DO stuff work client side just like that, and as mentioned we'll add that, but let's not pretend this was "obvious" and without pitfalls of its own.
I told you five years ago in Brno at a meeting organized to specifically talk about DNSSEC at the client side these exact same things. Let's not pretend I and others did not raise these issues then already.
I understand some loud people here consider Internet DNS the true gospel, and mDNS and LLMNR and all kinds of other forms of popular name resolution, local and remote heresy
As I suggested in previous emails, stop inventing your personal wheel and go get informed at the IETF about work being done in this space by the main vendors. I've listed the working groups, the drafts and the vendors. Raise any issues you think you have with respect to mDNS, LLMNR and what not. Work with others to define a functional protocol and implementation. Then push it first in fedora and I will happilly be dead in the water and report bugs and submit patches.
Until we implement the DO-bypass stuff
It is not acceptable to break all f33+ and rhel9/centos9 servers "until you implement" whatever it is you need to implement to violate 15+ year old DNS RFCs at the low priority you assign this bug baesd on your selective use cases.
This will be my last email in this thread.
Paul
On 9/28/20 11:03 AM, Lennart Poettering wrote:
So far we side-step the DO issue by returning a clean error when clients set DO: "not implemented", plus a log message in syslog with more info. I'd argue that for the vast majority of users this is perfectly enough. Because IRL client-side DNSSEC doesn't really exist outside of some very specific circles of DNS nerds, I guess.
Do you mean, "professionals for whom security is a primary job function?"
As other list members have already been asked to avoid making this conversation personal, can we *all* not make it about the people, please?
Lennart Poettering wrote:
On Mo, 28.09.20 18:36, Florian Weimer (fweimer@redhat.com) wrote:
- Andrew Lutomirski:
Paul may well have been mixing different things here, but I don't think you answered the one that seems like the most severe problem: systemd-resolved removed perfectly valid DNSSEC records that were supplied by the upstream server. One might reasonably debate whether Fedora's default DNS resolver configuration should validate DNSSEC, but I think it should honor the DO bit in client requests and return DNSSEC data.
FWIW, this is https://bugzilla.redhat.com/show_bug.cgi?id=1879028.
It's on the TODO list. But this creates probems of its own. Propagating DO stuff as is cannot work for LLMNR, mDNS, company-scope DNS and so on, i.e. anything that isn't the official Internet DNS.
It can work in company-scope if the company has competent network admins. My local DNS server at home resolves local hostnames to private IPv4 addresses in the 192.168/16 block. Clients on the Internet see another view. Both views are DNSsec-signed, and validation works fine. There's no reason why this setup wouldn't work on a corporate network. The key is to use a domain that is actually registered to the company, not some made-up TLD like "internal" or whatever the incompetent network admins come up with.
What's worse, resolved would start having a split personality: for DO replies we'd propagate the 1:1 upstream responses, while for everything else we'd return our own stuff, from our own synthesis, sourcing and and son. A local DNS client talking to resolved would see a feature set that would be very different depending if you ask with or without DO, because if you ask with DO you effectively just talk to one of the upstream servers, while without you will speak to systemd-resolved.
It would make more sense to select the "personality" based on what interface the client uses, than based on a DO flag in the query: Present actual standards-compliant DNS and nothing else on UDP and TCP port 53, and return your own synthesized stuff to programs that call getaddrinfo (and through the D-Bus interface I suppose).
Nobody connects to port 53 expecting to get entries from /etc/hosts or LLMNR. Programs that do this expect only DNS – and they're likely to expect advanced DNS features to work, because they would have just called getaddrinfo if they weren't interested in advanced features. It could even be argued that returning non-DNS data through the DNS protocol is wrong, but if you can do it without violating DNS standards, then I don't think it will hurt.
systemd-resolved is not supposed to be a real DNS *server*.
A real DNS server is however what existing programs expect to find in /etc/resolv.conf.
Until we implement the DO-bypass stuff, there's an easy way to bypass systemd-resolved for your DNSSEC resolver needs: just symlink /run/systemd/resolve/resolv.conf to /etc/resolv.conf (i.e. instead of /run/systemd/resolve/stub-resolv.conf). If you do that all local DNS clients will use the upstream DNS servers resolved picked up directly, bypassing resolved. But of course, if you do that then you also lose LLMNR, mdns, all local synthetic records, merging of VPN zones and so on.
When I'm speaking the DNS protocol, then I don't mind "losing" LLMNR, MDNS and synthetic records, which never existed in DNS to begin with. It would however be good to have the split DNS feature, and I see no reason why that wouldn't work with DNSsec. Of course, whether a DO query gets a useful response depends on how good the respective upstream DNS servers are.
Björn Persson
On Mo, 28.09.20 22:54, Björn Persson (Bjorn@rombobjörn.se) wrote:
It can work in company-scope if the company has competent network admins. My local DNS server at home resolves local hostnames to private IPv4 addresses in the 192.168/16 block. Clients on the Internet see another view. Both views are DNSsec-signed, and validation works fine. There's no reason why this setup wouldn't work on a corporate network. The key is to use a domain that is actually registered to the company, not some made-up TLD like "internal" or whatever the incompetent network admins come up with.
You never take your laptop outside to a cafe or so? You never connected it to something that is not your home or office network?
THing is most people do that, we need to have something that makes the best of such networks.
It would make more sense to select the "personality" based on what interface the client uses, than based on a DO flag in the query: Present actual standards-compliant DNS and nothing else on UDP and TCP port 53, and return your own synthesized stuff to programs that call getaddrinfo (and through the D-Bus interface I suppose).
Nah, that does not work. The majority of C programs might go via getaddrinfo(), but a major chunk of what we ship in this distro simply does not, and does DNS natively. And I think that's totally OK. There needs to be IPC involved anyway, to isolate the DNS stack (i.e. resolved) from the client issuing the query. Whether that's D-Bus or Varlink or just simple local DNS datagrams doesn't really matter.
Nobody connects to port 53 expecting to get entries from /etc/hosts or LLMNR. Programs that do this expect only DNS – and they're likely to expect advanced DNS features to work, because they would have just called getaddrinfo if they weren't interested in advanced features. It could even be argued that returning non-DNS data through the DNS protocol is wrong, but if you can do it without violating DNS standards, then I don't think it will hurt.
This is not the reality I live in though. New-style high level programming languages tend to avoid being just a wrapper around C APIs. And thus they implement minimal DNS clients themselves, ignoring the LLMNR, mDNS and so on.
Lennart
-- Lennart Poettering, Berlin
Le 2020-09-29 12:37, Lennart Poettering a écrit :
This is not the reality I live in though. New-style high level programming languages tend to avoid being just a wrapper around C APIs. And thus they implement minimal DNS clients themselves, ignoring the LLMNR, mDNS and so on.
Not just for DNS. For SMTP, HTTP, etc.
The modern way of coding apps is minimal marginally-compliant and secure built-in network client (so things sort of work on the dev system and in CI/CD unit tests), with the OS interposing a full-featured protocol proxy in “production” deployments.
The problem is that this software dev trend conflicts with the traditional end-to-end IETF view. When the end-to-end view wins you end up with a single canonical network client like chrome or firefox because coding a full-featured secure network client implementation is hard and devs want their apps to do network-like IPC not depend on libs that may or may not have the correct legal or language requirements
Regards,
Lennart Poettering wrote:
On Mo, 28.09.20 22:54, Björn Persson (Bjorn@rombobjörn.se) wrote:
It can work in company-scope if the company has competent network admins. My local DNS server at home resolves local hostnames to private IPv4 addresses in the 192.168/16 block. Clients on the Internet see another view. Both views are DNSsec-signed, and validation works fine. There's no reason why this setup wouldn't work on a corporate network. The key is to use a domain that is actually registered to the company, not some made-up TLD like "internal" or whatever the incompetent network admins come up with.
You never take your laptop outside to a cafe or so? You never connected it to something that is not your home or office network?
A cafe is company-scope? I'm not sure whether that counts as moving the goalposts or changing the subject, but neither is a constructive way to discuss a technical topic.
Björn Persson
On Tue, Sep 29, 2020 at 7:48 AM Björn Persson <Bjorn@rombobjörn.se> wrote:
Lennart Poettering wrote:
On Mo, 28.09.20 22:54, Björn Persson (Bjorn@rombobjörn.se) wrote:
It can work in company-scope if the company has competent network admins. My local DNS server at home resolves local hostnames to private IPv4 addresses in the 192.168/16 block. Clients on the Internet see another view. Both views are DNSsec-signed, and validation works fine. There's no reason why this setup wouldn't work on a corporate network. The key is to use a domain that is actually registered to the company, not some made-up TLD like "internal" or whatever the incompetent network admins come up with.
You never take your laptop outside to a cafe or so? You never connected it to something that is not your home or office network?
A cafe is company-scope? I'm not sure whether that counts as moving the goalposts or changing the subject, but neither is a constructive way to discuss a technical topic.
If you're a remote employee, it absolutely is. And especially in this pandemic, this kind of thing is now the *default* experience.
Am 29.09.20 um 14:38 schrieb Neal Gompa:
If you're a remote employee, it absolutely is. And especially in this pandemic, this kind of thing is now the *default* experience.
Company network - check Fedora 31 Laptops - check VPN users - check Androids - check Windows Laptops - check internal dns - check private domains - check
Not every company network is complicated.
best regards, Marius
Neal Gompa wrote:
On Tue, Sep 29, 2020 at 7:48 AM Björn Persson <Bjorn@rombobjörn.se> wrote:
Lennart Poettering wrote:
On Mo, 28.09.20 22:54, Björn Persson (Bjorn@rombobjörn.se) wrote:
It can work in company-scope if the company has competent network admins. My local DNS server at home resolves local hostnames to private IPv4 addresses in the 192.168/16 block. Clients on the Internet see another view. Both views are DNSsec-signed, and validation works fine. There's no reason why this setup wouldn't work on a corporate network. The key is to use a domain that is actually registered to the company, not some made-up TLD like "internal" or whatever the incompetent network admins come up with.
You never take your laptop outside to a cafe or so? You never connected it to something that is not your home or office network?
A cafe is company-scope? I'm not sure whether that counts as moving the goalposts or changing the subject, but neither is a constructive way to discuss a technical topic.
If you're a remote employee, it absolutely is. And especially in this pandemic, this kind of thing is now the *default* experience.
So we're assuming that we have successfully connected to the company VPN, as the company-scope DNS isn't involved unless we have access to the company network. The cafe's network may be crappy, but evidently not so bad that our VPN can't work. Now, how exactly does the cafe prevent us from sending queries with the DO bit set through the VPN tunnel to the company-scope DNS server and receiving security records in the response?
Lennart claimed that "propagating DO stuff as is cannot work for [...] company-scope DNS". I'm saying that claim is false. If you can't use the cafe scenario to prove me wrong, then the cafe is irrelevant. To have a constructive technical discussion it is necessary to keep separate issues separate.
Björn Persson
On Di, 29.09.20 13:47, Björn Persson (Bjorn@rombobjörn.se) wrote:
Lennart Poettering wrote:
On Mo, 28.09.20 22:54, Björn Persson (Bjorn@rombobjörn.se) wrote:
It can work in company-scope if the company has competent network admins. My local DNS server at home resolves local hostnames to private IPv4 addresses in the 192.168/16 block. Clients on the Internet see another view. Both views are DNSsec-signed, and validation works fine. There's no reason why this setup wouldn't work on a corporate network. The key is to use a domain that is actually registered to the company, not some made-up TLD like "internal" or whatever the incompetent network admins come up with.
You never take your laptop outside to a cafe or so? You never connected it to something that is not your home or office network?
A cafe is company-scope? I'm not sure whether that counts as moving the goalposts or changing the subject, but neither is a constructive way to discuss a technical topic.
I am just saying: Fedora cannot be focussed on just working for people who have a competent company admin and use their laptops in company networks only. We must have something that works well in company networks, as in home networks as in cafe wifis and suchlike.
Client-side DNSSEC only works in a subset of the "competent network admin" scenario, but not in the cafe wifi scenario or the home lan scenario.
Lennart
-- Lennart Poettering, Berlin
On 9/29/20 3:44 PM, Lennart Poettering wrote:
On Di, 29.09.20 13:47, Björn Persson (Bjorn@rombobjörn.se) wrote:
Lennart Poettering wrote:
On Mo, 28.09.20 22:54, Björn Persson (Bjorn@rombobjörn.se) wrote:
It can work in company-scope if the company has competent network admins. My local DNS server at home resolves local hostnames to private IPv4 addresses in the 192.168/16 block. Clients on the Internet see another view. Both views are DNSsec-signed, and validation works fine. There's no reason why this setup wouldn't work on a corporate network. The key is to use a domain that is actually registered to the company, not some made-up TLD like "internal" or whatever the incompetent network admins come up with.
You never take your laptop outside to a cafe or so? You never connected it to something that is not your home or office network?
A cafe is company-scope? I'm not sure whether that counts as moving the goalposts or changing the subject, but neither is a constructive way to discuss a technical topic.
I am just saying: Fedora cannot be focussed on just working for people who have a competent company admin and use their laptops in company networks only. We must have something that works well in company networks, as in home networks as in cafe wifis and suchlike.
Client-side DNSSEC only works in a subset of the "competent network admin" scenario, but not in the cafe wifi scenario or the home lan scenario.
Can you prove this claim somehow?
Is there list of cafe wifi scenarios and home lan scenarios, you are referring to? With explanation how resolved fixes them if possible?
Anyway, we might forgive working dnssec validation. What we cannot forgive is lack of DNSSEC information passtrough in 2020. For me, this would be blocker to Fedora release. Default installation cannot be supressing DNSSEC usability. It might not enforce it, but not disallow it.
If you want home lan to work, just accept local answers without signature, which then prove non-existing under DNSSEC. But do not allow changed addresses, other than localhost (for blocklist inclusion).
I am dnsmasq maintainer, which is found in most of cheap boxes you were referring to. It can proxy DNSSEC, unlike resolved with turned off support. Quite similar to resolved, it is not full-fledged DNS server, it just forwards (and optionally caches) queries forward. It fixed DNSSEC support some year back. Is your favourite café network broken so much?
Lennart
-- Lennart Poettering, Berlin
Thanks,
Petr
On Di, 29.09.20 16:51, Petr Menšík (pemensik@redhat.com) wrote:
I am just saying: Fedora cannot be focussed on just working for people who have a competent company admin and use their laptops in company networks only. We must have something that works well in company networks, as in home networks as in cafe wifis and suchlike.
Client-side DNSSEC only works in a subset of the "competent network admin" scenario, but not in the cafe wifi scenario or the home lan scenario.
Can you prove this claim somehow?
Is there list of cafe wifi scenarios and home lan scenarios, you are referring to?
I can give you an address of a local Cafe here with a non-working DNSSEC. I am pretty sure where you live they have plenty of those cafes too.
Or German ICE trains public wifi doesn't allow DNSSEC.
With explanation how resolved fixes them if possible?
Our fix: we do not do DNSSEC by default.
Lennart
-- Lennart Poettering, Berlin
On 9/29/20 5:23 PM, Lennart Poettering wrote:
On Di, 29.09.20 16:51, Petr Menšík (pemensik@redhat.com) wrote:
I am just saying: Fedora cannot be focussed on just working for people who have a competent company admin and use their laptops in company networks only. We must have something that works well in company networks, as in home networks as in cafe wifis and suchlike.
Client-side DNSSEC only works in a subset of the "competent network admin" scenario, but not in the cafe wifi scenario or the home lan scenario.
Can you prove this claim somehow?
Is there list of cafe wifi scenarios and home lan scenarios, you are referring to?
I can give you an address of a local Cafe here with a non-working DNSSEC. I am pretty sure where you live they have plenty of those cafes too.
Define please what is non-working DNSSEC. Does it use few internal names, which fail to validate? Does it refuse DNSSEC enabled query like resolved does? Is there any link describing their connection, which could you share? dig answers, traffic dumps or similar stuff?
Or German ICE trains public wifi doesn't allow DNSSEC.
What does that mean? Is there any bug/ticket related to it?
What does it reply to dig +dnssec? I have worked few times on Regiojet here in Czechia. Aside from few connections dropped, it worked just fine.
I think only login web dashboards are frequent reasons for dnssec failures. After they allow you internet access, it usually works fine (to me). Could this be also your case?
But I admit I connect most often just by smartphone, which does not have dnssec enabled. I have it just on my laptop and I never noticed such problem on it. I will check few myself.
With explanation how resolved fixes them if possible?
Our fix: we do not do DNSSEC by default.
That is incorrect. You do NOT ALLOW DNSSEC by default. Which is big difference. It even refuses to ask with DNSSEC when I make request for it. It refuses to pass me unmodified answer. I makes it the most broken DNS resolver I know about.
Have you tried asking them to fix it?
Note: DNS flag day 2019[1] made guessing due timeouts obsolete. According to DNS people, it was quite success, misbehaving servers are quite minimal. It should be enough to resend query with DNSSEC disabled just on FORMERR response.
Lennart
-- Lennart Poettering, Berlin
1. https://dnsflagday.net/2019/#experts
On Tue, Sep 29, 2020 at 4:51 pm, Petr Menšík pemensik@redhat.com wrote:
Anyway, we might forgive working dnssec validation. What we cannot forgive is lack of DNSSEC information passtrough in 2020.
I agree this should be fixed. See https://bugzilla.redhat.com/show_bug.cgi?id=1879028.
However, since this only matters for specialized server deployments, and will not matter for desktop usage or most server deployments, and since the workaround is very easy (just edit /etc/resolv.conf) it's really extreme to suggest it should be a release blocker when we have one week to go before final freeze. That timeframe is way too tight.
On 29 Sep 2020, at 22:04, Michael Catanzaro mcatanzaro@gnome.org wrote:
On Tue, Sep 29, 2020 at 4:51 pm, Petr Menšík pemensik@redhat.com wrote:
Anyway, we might forgive working dnssec validation. What we cannot forgive is lack of DNSSEC information passtrough in 2020.
I agree this should be fixed. See https://bugzilla.redhat.com/show_bug.cgi?id=1879028.
However, since this only matters for specialized server deployments, and will not matter for desktop usage or most server deployments, and since the workaround is very easy (just edit /etc/resolv.conf) it's really extreme to suggest it should be a release blocker when we have one week to go before final freeze. That timeframe is way too tight.
To step in here, regulatory compliance is a non optional requirement around the world.
Regulatory compliance applies to everybody in a jurisdiction, there is no such thing as a “specialized deployment” or environments where it “will not matter”. Compliance doesn’t care about an arbitrary freeze.
This is not a technical decision.
Regards, Graham —
On Tue, Sep 29, 2020 at 11:33 pm, Graham Leggett minfrin@sharp.fm wrote:
To step in here, regulatory compliance is a non optional requirement around the world.
Regulatory compliance applies to everybody in a jurisdiction, there is no such thing as a “specialized deployment” or environments where it “will not matter”. Compliance doesn’t care about an arbitrary freeze.
This is not a technical decision.
This is either a very strange misunderstanding, or trolling. I will assume positive intent. Internet RFCs are not regulatory requirements. If you're aware of some government regulation that requires us to forward RRSEC records, I would be very surprised, but please do let us know.
Michael
On Tue, Sep 29, 2020 at 5:44 PM Michael Catanzaro mcatanzaro@gnome.org wrote:
On Tue, Sep 29, 2020 at 11:33 pm, Graham Leggett minfrin@sharp.fm wrote:
To step in here, regulatory compliance is a non optional requirement around the world.
Regulatory compliance applies to everybody in a jurisdiction, there is no such thing as a “specialized deployment” or environments where it “will not matter”. Compliance doesn’t care about an arbitrary freeze.
This is not a technical decision.
This is either a very strange misunderstanding, or trolling. I will assume positive intent. Internet RFCs are not regulatory requirements. If you're aware of some government regulation that requires us to forward RRSEC records, I would be very surprised, but please do let us know.
Also, in case folks weren't aware, IETF RFCs are *intentionally* not described as standards. They are designed to be subject to revision at more or less any time. That's why they are titled as a "Request for Comments" (or RFC). The fact that they are used as standards documentation is sort of the nature of how things developed from an era where it was impossible to publish standards for free (organizations like the ISO require paying money for standards documentation).
But this also has the consequence that IETF RFCs do not have a requirement to be evaluated in the context of others and determined to be fully satisfiable along others. That is, it's possible to have mutually exclusive RFCs for a system that you have to implement.
So please keep that in mind when considering discussing DNS.
On Tue, 2020-09-29 at 17:50 -0400, Neal Gompa wrote:
On Tue, Sep 29, 2020 at 5:44 PM Michael Catanzaro mcatanzaro@gnome.org wrote:
On Tue, Sep 29, 2020 at 11:33 pm, Graham Leggett minfrin@sharp.fm wrote:
To step in here, regulatory compliance is a non optional requirement around the world.
Regulatory compliance applies to everybody in a jurisdiction, there is no such thing as a “specialized deployment” or environments where it “will not matter”. Compliance doesn’t care about an arbitrary freeze.
This is not a technical decision.
This is either a very strange misunderstanding, or trolling. I will assume positive intent. Internet RFCs are not regulatory requirements. If you're aware of some government regulation that requires us to forward RRSEC records, I would be very surprised, but please do let us know.
Also, in case folks weren't aware, IETF RFCs are *intentionally* not described as standards. They are designed to be subject to revision at more or less any time. That's why they are titled as a "Request for Comments" (or RFC). The fact that they are used as standards documentation is sort of the nature of how things developed from an era where it was impossible to publish standards for free (organizations like the ISO require paying money for standards documentation).
Sorry Neal, but this is not correct, in IETF there are several RFC types: - standard - informational - experimental - best current practice etc..
There are *definitely* officially defined by IETF "Standard" RFC, but not all RFC are standard for sure.
But this also has the consequence that IETF RFCs do not have a requirement to be evaluated in the context of others and determined to be fully satisfiable along others. That is, it's possible to have mutually exclusive RFCs for a system that you have to implement.
This is true, but much less so for standard RFCs.
So please keep that in mind when considering discussing DNS.
Always use a grain of salt with *any* standards, even ISO standards are often made by mandatory and optional parts, and sometimes they are conflicting too ... and then there are standards from different organizations that may be in partial conflict as well, welcome to the world :-)
On 29 Sep 2020, at 23:44, Michael Catanzaro mcatanzaro@gnome.org wrote:
This is either a very strange misunderstanding, or trolling. I will assume positive intent. Internet RFCs are not regulatory requirements. If you're aware of some government regulation that requires us to forward RRSEC records, I would be very surprised, but please do let us know.
Regulations like the GDPR exist, and ignorance of them is not a defence.
I am required by these regulations and many other regulations in multiple jurisdictions to make sure my users comply. If you have gone out of your way to break secure operation on Fedora, we will have to ban the use of Fedora by our users. I do not want to do that.
As I said, this is not a technical discussion. You need to defer this to compliance people, who I predict will simply tell you “comply”.
Regards, Graham —
On Wed, Sep 30, 2020 at 03:14:10PM +0200, Graham Leggett wrote:
I am required by these regulations and many other regulations in multiple jurisdictions to make sure my users comply. If you have gone out of your way to break secure operation on Fedora, we will have to ban the use of Fedora by our users. I do not want to do that.
Then don't ban them, and do your job instead?
The fact of the matter is that using out-of-the-box Fedora configurations *today* can leak "private" DNS queries, and if VPNs are in use, it is a virtual certainty.
To make Fedora "Compliant" using your definition, one already has to adjust the system configuration. This new approach, at worst, requires a slightly different configuration change to achieve the same results.
As I said, this is not a technical discussion. You need to defer this to compliance people, who I predict will simply tell you “comply”.
My $dayjob is headquartered in Europe and is in a _highly_ regulated, risk-adverse industry, with compliance officers coming out of the woodwork. Suffice it to say that what it means to "Comply" is highly context-sensitive.
But you are correct, this is not a problem that can be solved via technical means -- Many legitimate use cases have diametrically-opposed needs, and there is no way for Fedora to know out-of-the-box which use case should apply as the general default. Moreover, at the granularity of specific DNS lookups, the general default can easily be wrong.
- Solomon
On Wed, Sep 30, 2020 at 3:14 pm, Graham Leggett minfrin@sharp.fm wrote:
Regulations like the GDPR exist, and ignorance of them is not a defence.
I am required by these regulations and many other regulations in multiple jurisdictions to make sure my users comply. If you have gone out of your way to break secure operation on Fedora, we will have to ban the use of Fedora by our users. I do not want to do that.
As I said, this is not a technical discussion. You need to defer this to compliance people, who I predict will simply tell you “comply”.
Sorry, but I have not the faintest clue how your comment is relevant to this discussion regarding systemd-resolved.
Michael
Am 30.09.20 um 17:13 schrieb Michael Catanzaro:
On Wed, Sep 30, 2020 at 3:14 pm, Graham Leggett minfrin@sharp.fm wrote:
Regulations like the GDPR exist, and ignorance of them is not a defence.
I am required by these regulations and many other regulations in multiple jurisdictions to make sure my users comply. If you have gone out of your way to break secure operation on Fedora, we will have to ban the use of Fedora by our users. I do not want to do that.
As I said, this is not a technical discussion. You need to defer this to compliance people, who I predict will simply tell you “comply”.
Sorry, but I have not the faintest clue how your comment is relevant to this discussion regarding systemd-resolved.
I think, he meant the systemd-resolved fiallback to Cloudflare and Google. Is that in the fedora build? If so, i suggest to patch it out. That will fix the issue for me in perspective of the GDPR.
Best regards, Marius
On Thu, Oct 1, 2020 at 3:32 pm, Marius Schwarz fedoradev@cloud-foo.de wrote:
I think, he meant the systemd-resolved fiallback to Cloudflare and Google. Is that in the fedora build? If so, i suggest to patch it out. That will fix the issue for me in perspective of the GDPR.
Unless you explain this *very* clearly, I'm going to ignore it, because it seems farfetched. Fedora is not operating its own DNS server or collecting any sort of DNS-related data from you.
We are not going to patch out fallback to Cloudflare or Google because it is a non-issue. Fallback only happens when you have zero other DNS servers configured. When was the last time you connected to a network and there's no DHCP, no nothing? The number of users without some other working DNS is probably under 0.1%. Even then, I think you also have to disable NetworkManager for systemd-resolved to ever use its fallback DNS, because NetworkManager will configure a ~. DNS domain, causing systemd-resolved to never use its global DNS settings. (I think. That's my reading of the manpage. Testing welcome from anyone who wants to confirm that.)
So (if I'm right) we are talking about the exceeding rare combination of (a) no DNS set by DHCP, and also (b) user manually disabled NetworkManager. If you're really going to do (b) you will probably also disable systemd-resolved, right? Or make the one-line config file change to remove the fallback DNS? Or just manually set some DNS server? Seriously, this is a silly thing to worry about.
Finally, in the extremely unlikely event you do somehow wind up with Cloudflare and Google DNS, then you should *celebrate*, because they have extremely strong privacy policies for their DNS. Unless you think they are just lying about their data collection practices -- which they are not -- you have nothing to worry about from their DNS [1][2]. In contrast, your ISP is probably selling your DNS queries to advertisers. If you disagree, doesn't matter, because you're probably never going to see this fallback.
Michael
[1] https://developers.google.com/speed/public-dns/privacy [2] https://developers.cloudflare.com/1.1.1.1/privacy/public-dns-resolver/
On to, 01 loka 2020, Michael Catanzaro wrote:
On Thu, Oct 1, 2020 at 3:32 pm, Marius Schwarz fedoradev@cloud-foo.de wrote:
I think, he meant the systemd-resolved fiallback to Cloudflare and Google. Is that in the fedora build? If so, i suggest to patch it out. That will fix the issue for me in perspective of the GDPR.
Unless you explain this *very* clearly, I'm going to ignore it, because it seems farfetched. Fedora is not operating its own DNS server or collecting any sort of DNS-related data from you.
We are not going to patch out fallback to Cloudflare or Google because it is a non-issue. Fallback only happens when you have zero other DNS servers configured. When was the last time you connected to a network and there's no DHCP, no nothing? The number of users without some other working DNS is probably under 0.1%. Even then, I think you also have to disable NetworkManager for systemd-resolved to ever use its fallback DNS, because NetworkManager will configure a ~. DNS domain, causing systemd-resolved to never use its global DNS settings. (I think. That's my reading of the manpage. Testing welcome from anyone who wants to confirm that.)
We use the drop-in snippet configuration file in /etc/systemd/resolved.conf.d/zzz-ipa.conf to configure this behavior on IPA servers which deploy integrated DNS server. It works, so there is a confirmation.
So (if I'm right) we are talking about the exceeding rare combination of (a) no DNS set by DHCP, and also (b) user manually disabled NetworkManager. If you're really going to do (b) you will probably also disable systemd-resolved, right? Or make the one-line config file change to remove the fallback DNS? Or just manually set some DNS server? Seriously, this is a silly thing to worry about.
You can also drop a configuration snippet in /etc/systemd/resolved.conf.d/ to contain
FallbackDNS=<servers>
This will disable global DNS servers for any case.
Am 01.10.20 um 16:36 schrieb Alexander Bokovoy:
You can also drop a configuration snippet in /etc/systemd/resolved.conf.d/ to contain
FallbackDNS=<servers>
This will disable global DNS servers for any case.
if that would be the default, it would be ok.
Am 01.10.20 um 16:03 schrieb Michael Catanzaro:
We are not going to patch out fallback to Cloudflare or Google because it is a non-issue. Fallback only happens when you have zero other DNS servers configured. When was the last time you connected to a network and there's no DHCP, no nothing? The number of users without some other working DNS is probably under 0.1%.
BTW: thumbs up for the DOT proposal.
I will make it very clear and easy:
O== Situation for Germany
GDPR is in place as a EU LAW. The protection rules are only active for companies or organizations, not for private people.
2013 a german court (Kammergericht Berlin) ruled, that IP addresses are Personal Data. It has never been overruled.
Personal Data can only be send to none eu countries and corporations, if there is a data protection law in place that has the same or better level of protection as the eu law has ( or if it's necessary to buy stuff (a house, car, whatever ). The pact the EU did with the US was called Privacy Shield. It imploded (for the second time) a few months ago. From the moment the eu court rule was public, transfer of personal data into the us was illegal.
If you send a DNS REQUEST to a US DNS server from within a company network, and with ipv6 the internal ip is sent out i learned lately, you have sent personal data which is protected under the GDRP. It's not unlikely to use company pcs for private webvisits while having a meal break.
Therefor, a os that has google and cloudflare as a default, even if it's unlikely to happen as you point out, which sends out dns with personal data in it to a us dns server, brings the company in great trouble with the law. In the end, this means, you as a company/org need to pay a (possibly) shitload of money as a fine and therefor they can't use this os anymore. (someone else on the list pointed this out too.) The consequence is, Fedora is a juristic risk. [The fine is higher, if you as corp/org did not document this data transfer in your data protection memos] Of course a working dns setup will prevent this problem, but thats not the point. Activists in germany and other countries try to get more and more gov projects to OSS due to privacy issues with M$. It would be a shame if Fedora would also count as a potential problem.
Do we all really want this, for the benefit on 0.1%(as you say) have a dns lookup instead of a hint, that their systems are broken?
Pls remember: I'm just the messenger, Í didn't write the laws ;)
Funfact: last time I checked the northern germany police pc in my city, they used a fedora based desktop system. I like that fact :D But i'm pretty sure, they don't like a cloudflare fallback dns once they reach F33 ( if ever ).
best regards, Marius Schwarz
On Fri, Oct 2, 2020 at 12:34 am, Marius Schwarz fedoradev@cloud-foo.de wrote:
If you send a DNS REQUEST to a US DNS server from within a company network, and with ipv6 the internal ip is sent out i learned lately, you have sent personal data which is protected under the GDRP. It's not unlikely to use company pcs for private webvisits while having a meal break.
Hm, thanks for the explanation. I guess the DNS request would indeed be the *first* way you lose, because you have to do DNS before you do anything else. But you are going to lose immediately after anyway:
* Immediately after you connect to the network, Fedora connects to http://fedoraproject.org/static/hotspot.txt to see if you're behind a captive portal * Next, GNOME Software starts checking for updates in the background. You've leaked "personal data" to fedoraproject.org again, and also fwupd. * You open Firefox, it downloads Safe Browsing data from Google. (Admittedly this one is probably only behind a European CDN, but maybe Google is having a bad day, or maybe IP address logs are sent to the US.) Oh yeah, it also displays news from Pocket. Probably it does more connections to the US that I don't know about. * You switch to Financial Mode in Calculator, it downloads exchange rate data. * Anything crashes. A truncated stack trace gets sent to Fedora.
I'm sure my list is missing quite a lot. If your interpretation is correct, then I suppose German companies should immediately discontinue use of Fedora, and also most other computer operating systems....
Michael
On Fri, Oct 02, 2020 at 07:16:38AM -0500, Michael Catanzaro wrote:
On Fri, Oct 2, 2020 at 12:34 am, Marius Schwarz fedoradev@cloud-foo.de wrote:
If you send a DNS REQUEST to a US DNS server from within a company network, and with ipv6 the internal ip is sent out i learned lately, you have sent personal data which is protected under the GDRP. It's not unlikely to use company pcs for private webvisits while having a meal break.
Hm, thanks for the explanation. I guess the DNS request would indeed be the *first* way you lose, because you have to do DNS before you do anything else. But you are going to lose immediately after anyway:
- Immediately after you connect to the network, Fedora connects to
http://fedoraproject.org/static/hotspot.txt to see if you're behind a captive portal
- Next, GNOME Software starts checking for updates in the background. You've
leaked "personal data" to fedoraproject.org again, and also fwupd.
- You open Firefox, it downloads Safe Browsing data from Google. (Admittedly
this one is probably only behind a European CDN, but maybe Google is having a bad day, or maybe IP address logs are sent to the US.) Oh yeah, it also displays news from Pocket. Probably it does more connections to the US that I don't know about.
- You switch to Financial Mode in Calculator, it downloads exchange rate
data.
- Anything crashes. A truncated stack trace gets sent to Fedora.
I'm sure my list is missing quite a lot. If your interpretation is correct, then I suppose German companies should immediately discontinue use of Fedora, and also most other computer operating systems....
Only those that think that they are smarter that a user and ignore her/his privacy. The things in the list above definitely either shouldn't be done at all (like captive portla checks and checks for updates), or ask for explicit user consent (everything else).
On Fri, Oct 02, 2020 at 02:34:15PM +0200, Eugene Syromiatnikov wrote:
Only those that think that they are smarter that a user and ignore her/his privacy.
In other words, all of them?
FFS, if Fedora is "bad" for doing these things, how is MacOS, iOS, Android, or even Windows acceptible?
(out-of-the-box, that is. because that's what we're talking about here)
The things in the list above definitely either shouldn't be done at all (like captive portla checks and checks for updates), or ask for explicit user consent (everything else).
Great, more things for the user to click through *every single time*.
- Solomon
On Fri, Oct 02, 2020 at 09:23:12AM -0400, Solomon Peachy wrote:
On Fri, Oct 02, 2020 at 02:34:15PM +0200, Eugene Syromiatnikov wrote:
Only those that think that they are smarter that a user and ignore her/his privacy.
In other words, all of them?
FFS, if Fedora is "bad" for doing these things, how is MacOS, iOS, Android, or even Windows acceptible?
(out-of-the-box, that is. because that's what we're talking about here)
They are not, and that is why people use Linux.
The things in the list above definitely either shouldn't be done at all (like captive portla checks and checks for updates), or ask for explicit user consent (everything else).
Great, more things for the user to click through *every single time*.
The great UX invention of "do not ask this question again" saves the day again.
On Fri, Oct 02, 2020 at 03:37:21PM +0200, Eugene Syromiatnikov wrote:
FFS, if Fedora is "bad" for doing these things, how is MacOS, iOS, Android, or even Windows acceptible?
(out-of-the-box, that is. because that's what we're talking about here)
They are not, and that is why people use Linux.
In other words, this has nothing to do with legal (eg GPDR) compliance.
- Solomon
On 10/2/20 2:16 PM, Michael Catanzaro wrote:
On Fri, Oct 2, 2020 at 12:34 am, Marius Schwarz fedoradev@cloud-foo.de wrote:
If you send a DNS REQUEST to a US DNS server from within a company network, and with ipv6 the internal ip is sent out i learned lately, you have sent personal data which is protected under the GDRP. It's not unlikely to use company pcs for private webvisits while having a meal break.
Hm, thanks for the explanation. I guess the DNS request would indeed be the *first* way you lose, because you have to do DNS before you do anything else. But you are going to lose immediately after anyway:
- Immediately after you connect to the network, Fedora connects to
http://fedoraproject.org/static/hotspot.txt to see if you're behind a captive portal
Fedora is contacting fedora server, seems predictable.
- Next, GNOME Software starts checking for updates in the background.
You've leaked "personal data" to fedoraproject.org again, and also fwupd.
It checks also to Fedora servers, right?
- You open Firefox, it downloads Safe Browsing data from Google.
(Admittedly this one is probably only behind a European CDN, but maybe Google is having a bad day, or maybe IP address logs are sent to the US.) Oh yeah, it also displays news from Pocket. Probably it does more connections to the US that I don't know about.
- You switch to Financial Mode in Calculator, it downloads exchange rate
data.
Might ask question to user, whether that is okay. Can you please fill a RFE bug?
- Anything crashes. A truncated stack trace gets sent to Fedora.
I'm sure my list is missing quite a lot. If your interpretation is correct, then I suppose German companies should immediately discontinue use of Fedora, and also most other computer operating systems....
I think you are missing one important detail. When you choose to install Fedora, you are likely to accept it sends something to its servers. Anyway, they would usually have your IP somewhere, when you downloaded system image.
However, forwarding queries to every name you visit online to some party, which you never agreed to or maybe heard its name, is something different. It just provides your site history to company never mentioned even in configuration files. It is only mentioned by resolvectl, which normal user would never hear about.
It seems this should be easily configurable on installation (kickstart default or something), but by default should be empty.
Prepared commented out FallbackDNS=8.8.8.8,... would work well.
Am 05.10.20 um 11:12 schrieb Petr Menšík:
- Immediately after you connect to the network, Fedora connects to
http://fedoraproject.org/static/hotspot.txt to see if you're behind a captive portal
Fedora is contacting fedora server, seems predictable.
It's one thing to contact your repo or distro servers, and another if it's a known dataminer, that gets all domainnames you visit.
Best regards, Marius
On Tue, Oct 06, 2020 at 11:00:23PM +0200, Marius Schwarz wrote:
It's one thing to contact your repo or distro servers, and another if it's a known dataminer, that gets all domainnames you visit.
So.. given that both Google and Cloudfare have actual European business offices, aren't they bound by the GPDR too?
I get that it's fashionable these days to dump all over $Big_Tech, but we're better off basing our arguments on actual facts and logic?
Indeed, is there even a legal "Fedora" entity in Europe? Shouldn't we all be getting up in arms about Red Hat, especially now that it's wholly owned by, which is itself another massive data miner?
This seems to be a strange hill to fight over, especially for a last-ditch fallback that will only get used under an intentional [mis-]configuration of the network and/or Fedora system.
- Solomon
Am 06.10.20 um 23:21 schrieb Solomon Peachy:
It's one thing to contact your repo or distro servers, and another if it's a known dataminer, that gets all domainnames you visit.
So.. given that both Google and Cloudfare have actual European business offices, aren't they bound by the GPDR too?
I get that it's fashionable these days to dump all over $Big_Tech, but we're better off basing our arguments on actual facts and logic?
If you have a contract with them, which you don't have, and the data stays in europe, it would be another thing.
As a matter of fact, I have mailed our federal dp office and my statewise dp office for a dp statement.
As soon as a statement, no matter which opinion they have, arrives, i will post the result here on the list, so we all have a better understanding of the matter. Due to the complexity of this matter, I dont expect an answere soon ;)
Most likely i will receive a consultative call first and than it will takes months before something happens. Had it already for article 32 1a GDPR and the enforcement of TLS1_2 with SMTP in 2018. (btw.. i was right ;) )
best regards, Marius
On Fri, 2 Oct 2020, Michael Catanzaro wrote:
Hm, thanks for the explanation. I guess the DNS request would indeed be the *first* way you lose, because you have to do DNS before you do anything else. But you are going to lose immediately after anyway:
- Immediately after you connect to the network, Fedora connects to
http://fedoraproject.org/static/hotspot.txt to see if you're behind a captive portal
- Next, GNOME Software starts checking for updates in the background. You've
leaked "personal data" to fedoraproject.org again, and also fwupd.
If the locally configured DNS server supports Query Minimalization as per RFC 7816, at this point you would have only revealed "." or ".org"
If it further supports DNS-over-TLS, and more TLDs will start to support this, then nothing would be leaked. The world is steadilly moving towards this. Add encrypted SNI, and you see this improves even more. That is why governments are actually afraid of the opposite of GDPR right now. The fear of missing out of seeing DNS/SNI data.
- You open Firefox, it downloads Safe Browsing data from Google. (Admittedly
this one is probably only behind a European CDN, but maybe Google is having a bad day, or maybe IP address logs are sent to the US.)
This argument is that any browsing is a GDPR violation of every browser and OS. It is not a helpfull discussion, and if worth discussing, it should be discussed by laywers, not software engineers.
I'm sure my list is missing quite a lot. If your interpretation is correct, then I suppose German companies should immediately discontinue use of Fedora, and also most other computer operating systems....
The goal should always be to do the least amount of personal information gathering or leaking. Stating "but it leaks over there too" is not a very strong argument to leak data yourself.
Paul
OK, you convinced me: https://src.fedoraproject.org/rpms/systemd/pull-request/37. Let's see what others say.
Zbyszek
On Fri, Oct 02, 2020 at 12:34:32AM +0200, Marius Schwarz wrote:
Am 01.10.20 um 16:36 schrieb Alexander Bokovoy:
You can also drop a configuration snippet in /etc/systemd/resolved.conf.d/ to contain
FallbackDNS=<servers>
This will disable global DNS servers for any case.
if that would be the default, it would be ok.
Am 01.10.20 um 16:03 schrieb Michael Catanzaro:
We are not going to patch out fallback to Cloudflare or Google because it is a non-issue. Fallback only happens when you have zero other DNS servers configured. When was the last time you connected to a network and there's no DHCP, no nothing? The number of users without some other working DNS is probably under 0.1%.
BTW: thumbs up for the DOT proposal.
I will make it very clear and easy:
O== Situation for Germany
GDPR is in place as a EU LAW. The protection rules are only active for companies or organizations, not for private people.
2013 a german court (Kammergericht Berlin) ruled, that IP addresses are Personal Data. It has never been overruled.
Personal Data can only be send to none eu countries and corporations, if there is a data protection law in place that has the same or better level of protection as the eu law has ( or if it's necessary to buy stuff (a house, car, whatever ). The pact the EU did with the US was called Privacy Shield. It imploded (for the second time) a few months ago. From the moment the eu court rule was public, transfer of personal data into the us was illegal.
If you send a DNS REQUEST to a US DNS server from within a company network, and with ipv6 the internal ip is sent out i learned lately, you have sent personal data which is protected under the GDRP. It's not unlikely to use company pcs for private webvisits while having a meal break.
Therefor, a os that has google and cloudflare as a default, even if it's unlikely to happen as you point out, which sends out dns with personal data in it to a us dns server, brings the company in great trouble with the law. In the end, this means, you as a company/org need to pay a (possibly) shitload of money as a fine and therefor they can't use this os anymore. (someone else on the list pointed this out too.) The consequence is, Fedora is a juristic risk. [The fine is higher, if you as corp/org did not document this data transfer in your data protection memos] Of course a working dns setup will prevent this problem, but thats not the point. Activists in germany and other countries try to get more and more gov projects to OSS due to privacy issues with M$. It would be a shame if Fedora would also count as a potential problem.
Do we all really want this, for the benefit on 0.1%(as you say) have a dns lookup instead of a hint, that their systems are broken?
Pls remember: I'm just the messenger, Í didn't write the laws ;)
Funfact: last time I checked the northern germany police pc in my city, they used a fedora based desktop system. I like that fact :D But i'm pretty sure, they don't like a cloudflare fallback dns once they reach F33 ( if ever ).
best regards, Marius Schwarz _______________________________________________ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
On Thu, 1 Oct 2020, Michael Catanzaro wrote:
We are not going to patch out fallback to Cloudflare or Google because it is a non-issue. Fallback only happens when you have zero other DNS servers configured. When was the last time you connected to a network and there's no DHCP, no nothing? The number of users without some other working DNS is probably under 0.1%.
DNS discovery is currrently a hot topic at the IETF and there are various proposals circulating on how a client should behave to find its best DNS resolver.
Please see the ADD and DPRIVE working groups and their documents. I posted a few direct links in the last few days already. I think a mechanism that has been architectured by a wider group of engineers from a large number of different backgrounds and use cases would be a more appropriate venue to address this complex policy issue.
Personally, I prefer to prompt the user for permission before deciding to send their personal data to (mostly US based) entities.
And while the majorit of desktop users _might_ be okay with this implicit decision, it is always better to confirm that explicitely. You might think that UI is as bad as the COOKIE popups we now get, but lawyers disagree with us - whether we like or not that is a universe we live in.
Fruthermore it seems the servers running this will almost always never want this to happen, as most enterprises these days, especially in light of TLS 1.3 and encrypted SNI, are more and more reliant on using the DNS stream as an active firewall.
Paul
On Thu, 2020-10-01 at 09:03 -0500, Michael Catanzaro wrote:
On Thu, Oct 1, 2020 at 3:32 pm, Marius Schwarz fedoradev@cloud-foo.de wrote:
I think, he meant the systemd-resolved fiallback to Cloudflare and Google. Is that in the fedora build? If so, i suggest to patch it out. That will fix the issue for me in perspective of the GDPR.
Unless you explain this *very* clearly, I'm going to ignore it, because it seems farfetched. Fedora is not operating its own DNS server or collecting any sort of DNS-related data from you.
We are not going to patch out fallback to Cloudflare or Google because it is a non-issue. Fallback only happens when you have zero other DNS servers configured. When was the last time you connected to a network and there's no DHCP, no nothing? The number of users without some other working DNS is probably under 0.1%. Even then, I think you also have to disable NetworkManager for systemd-resolved to ever use its fallback DNS, because NetworkManager will configure a ~. DNS domain, causing systemd-resolved to never use its global DNS settings. (I think. That's my reading of the manpage. Testing welcome from anyone who wants to confirm that.)
So (if I'm right) we are talking about the exceeding rare combination of (a) no DNS set by DHCP, and also (b) user manually disabled NetworkManager. If you're really going to do (b) you will probably also disable systemd-resolved, right? Or make the one-line config file change to remove the fallback DNS? Or just manually set some DNS server? Seriously, this is a silly thing to worry about.
Finally, in the extremely unlikely event you do somehow wind up with Cloudflare and Google DNS, then you should *celebrate*, because they have extremely strong privacy policies for their DNS. Unless you think they are just lying about their data collection practices -- which they are not -- you have nothing to worry about from their DNS [1][2]. In contrast, your ISP is probably selling your DNS queries to advertisers. If you disagree, doesn't matter, because you're probably never going to see this fallback.
Michael, I think the issue here is not Google DNS vs ISP DNS, but internal DNS vs spilling it out to Google/Cloudflare.
That said, if it really is an internal DNS and there are strong policies around it I assume that the perimeter or the local machine firewall will be configured to block UDP packets to port 53 to any other external servers ...
This leaves out only some machines or some cases where a misconfiguration may cause this fallback to kick in. The occurrence is probably rare enough not to be a problem in practice at least from the pov of GDPR.
Simo.
Am 01.10.20 um 19:36 schrieb Simo Sorce:
That said, if it really is an internal DNS and there are strong policies around it I assume that the perimeter or the local machine firewall will be configured to block UDP packets to port 53 to any other external servers ...
This leaves out only some machines or some cases where a misconfiguration may cause this fallback to kick in. The occurrence is probably rare enough not to be a problem in practice at least from the pov of GDPR.
you know, that you contradict yourself here? :)
If the corp has blocked port 53 except for the internal dns server, how should the fallback packet get out?
I think, it's not important how often the default is used, it's the fact that it's hidden and therefor surprising for the corp itself, which makes it even more risky to run the os, than it's worth giving ( or in your example not to give ) the 0.1% a fallback answere.
IRL admins who know about it, as we all do now, we can avoid the problem. But for a company, which has to justify the surprising result of a DP audit, it will not be an easy talk with the dp buero. Just for the lols, I will ask our highest federal dp advocate tomorrow, what he thinks about this.
Best regards, Marius
On Fri, 2020-10-02 at 00:50 +0200, Marius Schwarz wrote:
Am 01.10.20 um 19:36 schrieb Simo Sorce:
That said, if it really is an internal DNS and there are strong policies around it I assume that the perimeter or the local machine firewall will be configured to block UDP packets to port 53 to any other external servers ...
This leaves out only some machines or some cases where a misconfiguration may cause this fallback to kick in. The occurrence is probably rare enough not to be a problem in practice at least from the pov of GDPR.
you know, that you contradict yourself here? :)
If the corp has blocked port 53 except for the internal dns server, how should the fallback packet get out?
It will not, that's the point, I do not see any contradiction.
I think, it's not important how often the default is used, it's the fact that it's hidden and therefor surprising for the corp itself, which makes it even more risky to run the os, than it's worth giving ( or in your example not to give ) the 0.1% a fallback answere.
I guess then corporations will immediately ban both Microsfot Windows and Apple MacOS because they connect to the respective motherships as well as dozen of other random IP addresses all the time ...
IRL admins who know about it, as we all do now, we can avoid the problem. But for a company, which has to justify the surprising result of a DP audit, it will not be an easy talk with the dp buero. Just for the lols, I will ask our highest federal dp advocate tomorrow, what he thinks about this.
IRL any admin knows that there is the potential for a networked machine to connect to random place, if they have a concern about that they take appropriate measures to prevent it, either via firewalling or by enforcing specific configurations.
I am not thrilled by this fallback either, but I think there are enough assurances that in any reasonably configured (and functional) network this fallback will not be triggered, and therefore there isn't a persistent privacy leak threat to be concerned about.
But both mine and yours are opinions, and I am pretty sure Fedora will do what's needed if there is some judicial determination that binds this specific kind of issue one way or another.
Simo.
On Mon, Sep 28, 2020 at 9:27 AM Andrew Lutomirski luto@mit.edu wrote:
On Mon, Sep 28, 2020 at 4:48 AM Zbigniew Jędrzejewski-Szmek < zbyszek@in.waw.pl> wrote:
On Mon, Sep 28, 2020 at 12:44:13AM -0400, Paul Wouters wrote:
Subject: Re: Fedora 33 System-Wide Change proposal: systemd-resolved
paul@thinkpad:~$ dig +dnssec vpn.nohats.ca @127.0.0.53
; <<>> DiG 9.11.22-RedHat-9.11.22-1.fc33 <<>> +dnssec vpn.nohats.ca @
127.0.0.53
;; global options: +cmd ;; Got answer: ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 51669 ;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1
;; OPT PSEUDOSECTION: ; EDNS: version: 0, flags: do; udp: 65494 ; OPT=5: 05 07 08 0a 0d 0e 0f (".......") ; OPT=6: 01 02 04 ("...") ; OPT=7: 01 (".") ;; QUESTION SECTION: ;vpn.nohats.ca. IN A
;; ANSWER SECTION: vpn.nohats.ca. 10 IN A 193.110.157.148
;; Query time: 143 msec ;; SERVER: 127.0.0.53#53(127.0.0.53) ;; WHEN: Mon Sep 28 00:18:32 EDT 2020 ;; MSG SIZE rcvd: 81
libreswan will see this result as an attack, and fail to resolve DNS
names
in its configuration. My postfix daemon will hold on to mail because it cannot get a DNSSEC proof of denial of existence of TLSA records if all DNSSEC records are filtered - even for domains that don't use DNSSEC because the denial of existence of DNSSEC for a specific domain requires the return of DNSSEC records that systemd now does not return.
I am sorry that I did not follow the fedora list carefully enough to notice this feature when it was proposed.
This change is harmful to network security, impacts existing
installations
depending on DNSSEC security, and leaks private queries for VPN/internal domains to the open internet, and prefers faster non-dnssec answers over dnssec validated answers. It fails various types of queries, misimplements part of the DNS protocol. Not only according to me, but to 20years+ developers of the bind software as well.
You're mixing a few different things here. We decided to not enable DNSSEC in resolved with this change, at least initially. For most users, DNSSEC is problematic because various intermediary DNS servers found in hotspots and routers don't support DNSSEC properly, leading to hard-to-debug validation failures. DNSSEC support in resolved can be enabled through resolved.conf. This may be a reasonable thing to do in an environment where the configured dns servers are known to support dnssec properly.
Paul may well have been mixing different things here, but I don't think you answered the one that seems like the most severe problem: systemd-resolved removed perfectly valid DNSSEC records that were supplied by the upstream server. One might reasonably debate whether Fedora's default DNS resolver configuration should validate DNSSEC, but I think it should honor the DO bit in client requests and return DNSSEC data.
Could the F33 default be changed to at least forward DNSSEC data to clients that ask for it?
(My personal view is that either this should happen or that systemd-resolved-as-default should be reverted for F33, but I'm not on FESCo.)
I should add:
After reading https://github.com/systemd/systemd/issues/8967, I really don't think that systemd-resolved's benefits outweigh its harms as a default resolver for Fedora. If someone wants to write a libfriendlydnsresolver and encourage/patch programs to use it instead of using glibc's resolver or reading resolv.conf, then that could be debated on its merits. But the actual contents of /etc/resolv.conf should follow the relevant standards, and systemd-resolved does not.
Perhaps systemd-resolved could change its mind and decide to comply with all relevant standards. But until then, it seems inappropriate to me for it to be the default in Fedora.
On Mon, Sep 28, 2020 at 09:44:13AM -0700, Andrew Lutomirski wrote:
After reading https://github.com/systemd/systemd/issues/8967, I really don't think that systemd-resolved's benefits outweigh its harms as a default resolver for Fedora. If someone wants to write a libfriendlydnsresolver and encourage/patch programs to use it instead of using glibc's resolver or reading resolv.conf, then that could be debated on its merits. But the actual contents of /etc/resolv.conf should follow the relevant standards, and systemd-resolved does not.
Perhaps systemd-resolved could change its mind and decide to comply with all relevant standards. But until then, it seems inappropriate to me for it to be the default in Fedora.
Pfff, now I'm confused. Here is a case where systemd-resolved implements the standard, and some people were unhappy because they were relying on sloppy implementations which don't follow the RFC. Nevertheless, we added an opt-in switch to make this work. (Since this feature mostly matters in "special" setups like k8s, where you need to do a lot of local setup anyway, requiring a one-line option seems to be reasonable).
Zbyszek
Zbigniew Jędrzejewski-Szmek zbyszek@in.waw.pl writes:
Pfff, now I'm confused. Here is a case where systemd-resolved implements the standard, and some people were unhappy because they were relying on sloppy implementations which don't follow the RFC.
Yes, welcome to software development!
Sometimes people care about the standard and sometimes the standard isn't helpful. You can bemoan it, or belittle the users as you are doing here, but *that's how it is*. The only way you can find out which approach to take is to *figure out what the people in the space expect*. In other words, it requires human contact and understanding of the problems - not just guessing in a vacuum.
Thanks, --Robbie
On Mon, Sep 28, 2020 at 01:14:14PM -0400, Robbie Harwood wrote:
Zbigniew Jędrzejewski-Szmek zbyszek@in.waw.pl writes:
Pfff, now I'm confused. Here is a case where systemd-resolved implements the standard, and some people were unhappy because they were relying on sloppy implementations which don't follow the RFC.
Yes, welcome to software development!
Sometimes people care about the standard and sometimes the standard isn't helpful. You can bemoan it, or belittle the users as you are doing here, but *that's how it is*.
I agree, but I was replying to a mail which said that systemd-resolved does not follow the standard. In *this* particular case, that complaint is off target.
The only way you can find out which approach to take is to *figure out what the people in the space expect*. In other words, it requires human contact and understanding of the problems - not just guessing in a vacuum.
There is more than one space... and different people have different expectations. What systemd-resolved was doing seems the appropriate thing for 90% of cases, but we also implemented an opt-in for the other behaviour. So I don't think you're being fair when you're saying "[I] bemoan it, or belittle the users", since I implemented the opt-in and I didn't even say anything about the users.
Zbyszek
On Mon, Sep 28, 2020 at 05:26:50PM +0000, Zbigniew Jędrzejewski-Szmek wrote:
On Mon, Sep 28, 2020 at 01:14:14PM -0400, Robbie Harwood wrote:
Zbigniew Jędrzejewski-Szmek zbyszek@in.waw.pl writes:
Pfff, now I'm confused. Here is a case where systemd-resolved implements the standard, and some people were unhappy because they were relying on sloppy implementations which don't follow the RFC.
Yes, welcome to software development!
Sometimes people care about the standard and sometimes the standard isn't helpful. You can bemoan it, or belittle the users as you are doing here, but *that's how it is*.
I agree, but I was replying to a mail which said that systemd-resolved does not follow the standard. In *this* particular case, that complaint is off target.
How is systemd-resolved compliant with this requirement of RFC 3397 section 4:
[2] Resolve a name that contains any dots by first trying it as an FQDN and if that fails, with the local domain name (or searchlist if specified) appended.
when it does not resolve "dk." via DNS?
Zbigniew Jędrzejewski-Szmek zbyszek@in.waw.pl writes:
On Mon, Sep 28, 2020 at 01:14:14PM -0400, Robbie Harwood wrote:
Zbigniew Jędrzejewski-Szmek zbyszek@in.waw.pl writes:
Pfff, now I'm confused. Here is a case where systemd-resolved implements the standard, and some people were unhappy because they were relying on sloppy implementations which don't follow the RFC.
Yes, welcome to software development!
Sometimes people care about the standard and sometimes the standard isn't helpful. You can bemoan it, or belittle the users as you are doing here, but *that's how it is*.
I agree, but I was replying to a mail which said that systemd-resolved does not follow the standard. In *this* particular case, that complaint is off target.
The only way you can find out which approach to take is to *figure out what the people in the space expect*. In other words, it requires human contact and understanding of the problems - not just guessing in a vacuum.
There is more than one space... and different people have different expectations. What systemd-resolved was doing seems the appropriate thing for 90% of cases, but we also implemented an opt-in for the other behaviour. So I don't think you're being fair when you're saying "[I] bemoan it, or belittle the users", since I implemented the opt-in and I didn't even say anything about the users.
Then I have misinterpreted your intent in the opening of your email - "Pfff" struck me as dismissive, and this was compounded by your statement that "some people were unhappy". I see this was not your intent and withdraw that criticism.
Thanks, --Robbie
On Mon, 28 Sep 2020 at 13:05, Zbigniew Jędrzejewski-Szmek zbyszek@in.waw.pl wrote:
On Mon, Sep 28, 2020 at 09:44:13AM -0700, Andrew Lutomirski wrote:
After reading https://github.com/systemd/systemd/issues/8967, I really don't think that systemd-resolved's benefits outweigh its harms as a default resolver for Fedora. If someone wants to write a libfriendlydnsresolver and encourage/patch programs to use it instead of using glibc's resolver or reading resolv.conf, then that could be debated on its merits. But the actual contents of /etc/resolv.conf should follow the relevant standards, and systemd-resolved does not.
Perhaps systemd-resolved could change its mind and decide to comply with all relevant standards. But until then, it seems inappropriate to me for it to be the default in Fedora.
Pfff, now I'm confused. Here is a case where systemd-resolved implements the standard, and some people were unhappy because they were relying on sloppy implementations which don't follow the RFC. Nevertheless, we added an opt-in switch to make this work. (Since this feature mostly matters in "special" setups like k8s, where you need to do a lot of local setup anyway, requiring a one-line option seems to be reasonable).
Hey for those of us in the peanuts gallery watching this play out.. could each of you point out which standards and RFC you are complying too. There are a lot of ones and funny enough.. they don't all agree with each other at times.
On Mon, Sep 28, 2020 at 01:15:36PM -0400, Stephen John Smoogen wrote:
On Mon, 28 Sep 2020 at 13:05, Zbigniew Jędrzejewski-Szmek zbyszek@in.waw.pl wrote:
On Mon, Sep 28, 2020 at 09:44:13AM -0700, Andrew Lutomirski wrote:
After reading https://github.com/systemd/systemd/issues/8967, I really don't think that systemd-resolved's benefits outweigh its harms as a default resolver for Fedora. If someone wants to write a libfriendlydnsresolver and encourage/patch programs to use it instead of using glibc's resolver or reading resolv.conf, then that could be debated on its merits. But the actual contents of /etc/resolv.conf should follow the relevant standards, and systemd-resolved does not.
Perhaps systemd-resolved could change its mind and decide to comply with all relevant standards. But until then, it seems inappropriate to me for it to be the default in Fedora.
Pfff, now I'm confused. Here is a case where systemd-resolved implements the standard, and some people were unhappy because they were relying on sloppy implementations which don't follow the RFC. Nevertheless, we added an opt-in switch to make this work. (Since this feature mostly matters in "special" setups like k8s, where you need to do a lot of local setup anyway, requiring a one-line option seems to be reasonable).
Hey for those of us in the peanuts gallery watching this play out.. could each of you point out which standards and RFC you are complying too. There are a lot of ones and funny enough.. they don't all agree with each other at times.
https://www.iab.org/documents/correspondence-reports-documents/2013-2/iab-st... in this particular case.
Zbyszek
Zbigniew Jędrzejewski-Szmek skrev:
On Mon, Sep 28, 2020 at 01:15:36PM -0400, Stephen John Smoogen wrote:
Hey for those of us in the peanuts gallery watching this play out.. could each of you point out which standards and RFC you are complying too. There are a lot of ones and funny enough.. they don't all agree with each other at times.
https://www.iab.org/documents/correspondence-reports-documents/2013-2/iab-st... in this particular case.
That IAB statement isn't itself a standard, but it references several standards. It even quotes RFC 3397 as saying:
[2] Resolve a name that contains any dots by first trying it as an FQDN and if that fails, with the local domain name (or searchlist if specified) appended.
[3] Resolve a name containing no dots by appending with the searchlist right away, but once again, no implicit searchlists should be used.
The name "dk." contains one dot, while "dk" contains no dots. According to the quoted rules those two shall be resolved differently. "dk." shall be treated as a fully qualified domain name. Now people are saying in this thread that systemd-resolved treats both as local names and doesn't even try to look them up in DNS. So does systemd-resolved comply with this standard or not?
Björn Persson
On Mo, 28.09.20 20:52, Björn Persson (Bjorn@rombobjörn.se) wrote:
Zbigniew Jędrzejewski-Szmek skrev:
On Mon, Sep 28, 2020 at 01:15:36PM -0400, Stephen John Smoogen wrote:
Hey for those of us in the peanuts gallery watching this play out.. could each of you point out which standards and RFC you are complying too. There are a lot of ones and funny enough.. they don't all agree with each other at times.
https://www.iab.org/documents/correspondence-reports-documents/2013-2/iab-st... in this particular case.
That IAB statement isn't itself a standard, but it references several standards. It even quotes RFC 3397 as saying:
[2] Resolve a name that contains any dots by first trying it as an FQDN and if that fails, with the local domain name (or searchlist if specified) appended.
[3] Resolve a name containing no dots by appending with the searchlist right away, but once again, no implicit searchlists should be used.
The name "dk." contains one dot, while "dk" contains no dots. According to the quoted rules those two shall be resolved differently. "dk." shall be treated as a fully qualified domain name. Now people are saying in this thread that systemd-resolved treats both as local names and doesn't even try to look them up in DNS. So does systemd-resolved comply with this standard or not?
https://github.com/systemd/systemd/pull/17194
Lennart
-- Lennart Poettering, Berlin
* Zbigniew Jędrzejewski-Szmek:
https://www.iab.org/documents/correspondence-reports-documents/2013-2/iab-st... in this particular case.
I looked at this extensively a couple of months ago. There is also an ICANN recommendation along similar lines, but focusing more on stub resolver configuration and search path processing, which is a better fit for systemd-resolved.
The feedback I received from subject matter experts is that complying with these ICANN recommendations (for search path processing) would break about 60% of all deployed Kubernetes clusters (and not just inside containers). I think some people have since started on updating Kubernetes practices and recommendations, but I expect that it will be a few more months until we see first effects.
I don't know if the systemd developers have conducted similar research before implementing their stub resolver behavior, and why they have reached opposite conclusions.
One problem with DNS is that you cannot take the standards and official recommendations and use them as a reference for a new DNS implementation. Many of the specifications are very old, some are quite poor, several of the sub-protocols are very badly designed (like using timeouts for protocol version negotiation; obviously that one never made it into an RFC, but was still widely deployed), and the entire space is extremely politicized. Not just by governments, but also by groups of individuals who for some reason cannot get along at all.
Thanks, Florian
On Tue, Sep 29, 2020 at 10:27:37AM +0200, Florian Weimer wrote:
- Zbigniew Jędrzejewski-Szmek:
https://www.iab.org/documents/correspondence-reports-documents/2013-2/iab-st... in this particular case.
I looked at this extensively a couple of months ago. There is also an ICANN recommendation along similar lines, but focusing more on stub resolver configuration and search path processing, which is a better fit for systemd-resolved.
The feedback I received from subject matter experts is that complying with these ICANN recommendations (for search path processing) would break about 60% of all deployed Kubernetes clusters (and not just inside containers). I think some people have since started on updating Kubernetes practices and recommendations, but I expect that it will be a few more months until we see first effects.
No, I don't think anyone did this kind of research. But Kubernetes was a (the?) primary motivation to optionally allow dotless lookups. (The assumption is that if you're running k8s, you are not just going to install latest Fedora there, but would do local configuration for the deployment anyway and can include the override.)
One problem with DNS is that you cannot take the standards and official recommendations and use them as a reference for a new DNS implementation. Many of the specifications are very old, some are quite poor, several of the sub-protocols are very badly designed (like using timeouts for protocol version negotiation; obviously that one never made it into an RFC, but was still widely deployed), and the entire space is extremely politicized. Not just by governments, but also by groups of individuals who for some reason cannot get along at all.
Yes, sadly.
Zbyszek
On Mon, Sep 28, 2020 at 10:05 AM Zbigniew Jędrzejewski-Szmek < zbyszek@in.waw.pl> wrote:
On Mon, Sep 28, 2020 at 09:44:13AM -0700, Andrew Lutomirski wrote:
After reading https://github.com/systemd/systemd/issues/8967, I really don't think that systemd-resolved's benefits outweigh its harms as a default resolver for Fedora. If someone wants to write a libfriendlydnsresolver and encourage/patch programs to use it instead of using glibc's resolver or reading resolv.conf, then that could be debated on its merits. But the actual contents of /etc/resolv.conf should follow the relevant standards, and systemd-resolved does not.
Perhaps systemd-resolved could change its mind and decide to comply with all relevant standards. But until then, it seems inappropriate to me for it to be the default in Fedora.
Pfff, now I'm confused. Here is a case where systemd-resolved implements the standard, and some people were unhappy because they were relying on sloppy implementations which don't follow the RFC. Nevertheless, we added an opt-in switch to make this work. (Since this feature mostly matters in "special" setups like k8s, where you need to do a lot of local setup anyway, requiring a one-line option seems to be reasonable).
I agree that the "tld." case is probably unusual, and I didn't personally know that anyone had an actual website set up like this until today, but as someone who has configured DNS zones, I did know that it was valid to attach records to a TLD. And, indeed, loading https://dk./ on Firefox on Fedora 32 works!
I'm reminded of https://fedoraproject.org/wiki/Changes/Default_Local_DNS_Resolver. That change, as I recall, didn't end up accepted because it ran into a bunch of corner cases and (at least at the time) didn't solve them all adequately. I don't recall its proponents ever arguing that the corner cases were *wrong*, though. In contrast, for some reason, the proponents of systemd-resolved are telling us that the corner cases aren't bugs, are arguing that the plainly written text in the standards are somehow not the standards, and are kindly offering to add non-default options to make systemd-resolved do what it should do by default.
Sorry, but it really seems that systemd-resolved is not ready for prime time.
To be clear, I personally do not like the model of having each client application manage its own communication with the network's resolver (i.e. what happens today when /etc/resolve.conf mentions whatever nameserver was provided by DHCP), but I think that a better solution needs to be very well thought through and to honor the expectation that whatever is in /etc/resolv.conf actually follows the standards. Programs that use the glibc name resolution APIs or that use UDP via their favorite library are not human beings who can be retrained to do things better -- they are programs that were written to various manpages, standards, etc, and when one of them asks getaddrinfo(3) to resolve "dk.", they don't mean "do whatever the systemd-resolved authors thought this should do"; they meant "resolve dk. as per POSIX and the RFCs. Similarly, if a program submits a perfectly valid EDNS query asking for DNSSEC data (note: I mean DO -- I'm not referring to the AD bit or to any sort of server-side validation), then, if the network supports DNSSEC, the result should include the requested data.
If Fedora 33 is going to violate these assumptions, along with whatever else might be broken that various people on this thread have alluded to, IMO it should have an extremely good reason to do so. I haven't seen one yet.
On Mon, Sep 28, 2020 at 12:44 am, Paul Wouters paul@nohats.ca wrote:
My fedora mail server uses DNSSEC based TLSA records to prevent MITM attacks on the STARTTLS layer, which is now completely broken. My IPsec VPN server uses dnssec validation using the specified nameserves in /etc/resolve.conf that now point to systemd-resolvd that does not return DNSSEC records and is completely broken:
If you're running mail servers or VPN servers, you can probably configure the DNS to your liking, right? Either enable DNSSEC support in systemd-resolved, or disable systemd-resolved. I'm not too concerned about this....
Honestly, I don't have a strong opinion on whether systemd-resolved is used by default on servers. There you normally have just one DNS server that you want to use, or at most a fallback or two that should return the same results, and old-style nss-dns name resolution should be fine. On workstations, though, where we really *cannot* enable DNSSEC, where VPN users often expect split DNS, and where we cannot expect users to configure anything manually, systemd-resolved is solving a real problem that nss-dns will never be able to handle.
Michael
On Mon, Sep 28, 2020 at 09:23:47AM -0500, Michael Catanzaro wrote:
*cannot* enable DNSSEC, where VPN users often expect split DNS, and where we cannot expect users to configure anything manually, systemd-resolved is solving a real problem that nss-dns will never be able to handle.
Can we get some release notes / release announcement wording around the end-user benefits here?
On Mon, Sep 28, 2020 at 10:29 am, Matthew Miller mattdm@fedoraproject.org wrote:
On Mon, Sep 28, 2020 at 09:23:47AM -0500, Michael Catanzaro wrote:
*cannot* enable DNSSEC, where VPN users often expect split DNS, and where we cannot expect users to configure anything manually, systemd-resolved is solving a real problem that nss-dns will never be able to handle.
Can we get some release notes / release announcement wording around the end-user benefits here?
"Fedora 33 uses systemd-resolved for name resolution. Most users will not notice any difference, but VPN users will benefit from safer defaults that ensure DNS requests are sent to the same network that would receive the corresponding traffic, avoiding unexpected DNS leaks or failure to resolve internal domains."
* Michael Catanzaro:
"Fedora 33 uses systemd-resolved for name resolution. Most users will not notice any difference, but VPN users will benefit from safer defaults that ensure DNS requests are sent to the same network that would receive the corresponding traffic, avoiding unexpected DNS leaks or failure to resolve internal domains."
I think this is rather misleading.
* The change disables protection mechanisms built into corporate VPNs that require them to observe all DNS traffic. Now this may sound rather weak as far as countermeasures go, but DNS-based mechanisms are the only thing you have got if you do not enforce a client-side approach (ugh, no thanks), or disable split tunneling (i.e., default route over the VPN; frequently not possible with current VPN usage levels and virtual company meetings over video link).
* There is no real protocol for sharing internal domains, so systemd-resolved cannot know all of them, and resolving some of them will fail or receive unexpected resolution results (probably observable for some jboss.org subdomains for Red Hatters, but I don't work in that area, so I don't have a good example at hand).
Thanks, Florian
I don't think my description is misleading....
On Mon, Sep 28, 2020 at 5:28 pm, Florian Weimer fweimer@redhat.com wrote:
- The change disables protection mechanisms built into corporate VPNs that require them to observe all DNS traffic. Now this may sound rather weak as far as countermeasures go, but DNS-based mechanisms
are the only thing you have got if you do not enforce a client-side approach (ugh, no thanks), or disable split tunneling (i.e., default route over the VPN; frequently not possible with current VPN usage levels and virtual company meetings over video link).
Correct. If you want to observe all DNS traffic, that is no longer possible by default unless you also handle routing all traffic. I'd argue that's a good change. From a system design perspective, that's just how DNS ought to work.
I don't think it would be smart for employees to voluntarily opt-in to sending all DNS to their employer anyway... there's little benefit to the employee, and a lot of downside. Importantly, if you're looking in your network settings and you see a checkbox that says "Use this connection only for resources on its network," a reasonable user *expects* that the connection will *really* only be used for resources on its network, not that it will be used for everything except DNS, which randomly goes to who knows where depending on what else you're connected to. Our design must try to avoid this failure case: "Sadly for Distrustful Denise, her employer discovers that she has been making some embarrassing DNS requests that she had expected to go through public-vpn.example.com instead."
Of course, it's still possible to get the old behavior if you really want to, but it will now require custom configuration not available via GUI, and nobody really wants to opt-in to that behavior, so it's not likely to be used except in managed corporate systems. If you're using a managed system, you're probably surveilled anyway, so better assume nothing is safe.
- There is no real protocol for sharing internal domains, so systemd-resolved cannot know all of them, and resolving some of them will fail or receive unexpected resolution results (probably observable for some jboss.org subdomains for Red Hatters, but I
don't work in that area, so I don't have a good example at hand).
Yes, that's true. And there's not currently any good solution to that without resorting to the command line.
Michael
On Mon, 28 Sep 2020, Michael Catanzaro wrote:
I don't think it would be smart for employees to voluntarily opt-in to sending all DNS to their employer anyway... there's little benefit to the employee, and a lot of downside.
Again, it is not up to systemd to limit valid use cases.
Perhaps Listen or read to Paul Vixie, father of many Bind software releases:
https://www.youtube.com/watch?v=ZxTdEEuyxHU
https://www.theregister.com/2018/10/23/paul_vixie_slaps_doh_as_dns_privacy_f...
There are use cases for and against routing all DNS over your VPN. If systemd wants to play system resolver, it needs to be able to be configured for either use case. You don't get to limit our use cases.
network settings and you see a checkbox that says "Use this connection only for resources on its network," a reasonable user *expects* that the connection will *really* only be used for resources on its network, not that it will be used for everything except DNS, which randomly goes to who knows where depending on what else you're connected to. Our design must try to avoid this failure case: "Sadly for Distrustful Denise, her employer discovers that she has been making some embarrassing DNS requests that she had expected to go through public-vpn.example.com instead."
See my previous email with respect to RFC 8598. There is a standard for this. We supported this in libreswan with unbound before we even forked from openswan, 10 years ago. I had also patched openvpn when Red Hat swithced VPN service type but it seems that patch got lost along the way.
Of course, it's still possible to get the old behavior if you really want to, but it will now require custom configuration not available via GUI
Again, this mentality of "power users can fend for themselves" and "only our own use cases matter".
, and nobody really wants to opt-in to that behavior
Some people like using a "DNS firewall", or have their VPN admins require it. Don't map use cases only on your own desired use cases. I can't really stress this enough, as it is constantly coming up within systemd projects.
- There is no real protocol for sharing internal domains, so systemd-resolved cannot know all of them, and resolving some of them will fail or receive unexpected resolution results (probably observable for some jboss.org subdomains for Red Hatters, but I
don't work in that area, so I don't have a good example at hand).
Yes, that's true. And there's not currently any good solution to that without resorting to the command line.
See above. libreswan IPsec VPNs has supported this for 10+ years. No commandline required.
Paul
On Mon, Sep 28, 2020 at 12:14 pm, Paul Wouters paul@nohats.ca wrote:
There are use cases for and against routing all DNS over your VPN. If systemd wants to play system resolver, it needs to be able to be configured for either use case. You don't get to limit our use cases.
It *can* be configured for either case. It will do whatever you tell it to.
Our GUI network settings don't expose the ability to send DNS to a different network, but systemd-resolved itself does. So if you want to see your employees' DNS, go ahead and configure it! It's easy to do using resolvectl. We're just not going to make it easy for people to shoot themselves in the foot by providing a GUI setting for this. Almost nobody has a personal requirement to send non-employer DNS to their employer. Nobody woke up today and thought "oh no, I sure hope my employer knows how much time I spent today watching Kim Kardashian." The network configuration is complex enough already. Many users will have no clue what "use this connection only for resources on its network" means. Now we have to have separate GUI buttons to allow using the connection for DNS, but not for routing? That's just too complex, so relegating it to command line is appropriate.
See my previous email with respect to RFC 8598. There is a standard for this. We supported this in libreswan with unbound before we even forked from openswan, 10 years ago. I had also patched openvpn when Red Hat swithced VPN service type but it seems that patch got lost along the way.
I've never heard of IKEv2 or this RFC, but reading just the abstract, I guess it's a layer *above* systemd-resolved. You would probably want NetworkManager to implement this if it doesn't already, and then push appropriate configuration to systemd-resolved. I add some ??? question marks because I only read the abstract ???
Michael
On Mo, 28.09.20 12:14, Paul Wouters (paul@nohats.ca) wrote:
On Mon, 28 Sep 2020, Michael Catanzaro wrote:
I don't think it would be smart for employees to voluntarily opt-in to sending all DNS to their employer anyway... there's little benefit to the employee, and a lot of downside.
Again, it is not up to systemd to limit valid use cases.
Perhaps Listen or read to Paul Vixie, father of many Bind software releases:
https://www.youtube.com/watch?v=ZxTdEEuyxHU
https://www.theregister.com/2018/10/23/paul_vixie_slaps_doh_as_dns_privacy_f...
There are use cases for and against routing all DNS over your VPN. If systemd wants to play system resolver, it needs to be able to be configured for either use case. You don't get to limit our use cases.
Configure "." as "routing domain" on a specific iface and the lookups wil go there preferably. If you put that on your VPN iface this means DNS traffic goes there preferably. If you put that ont he main iface this means DNS traffic goes there preferably.
Ideally you'd use more fine grained routing domains however.
Lennart
-- Lennart Poettering, Berlin
On Monday, September 28, 2020 12:42:32 PM MST Lennart Poettering wrote:
On Mo, 28.09.20 12:14, Paul Wouters (paul@nohats.ca) wrote:
On Mon, 28 Sep 2020, Michael Catanzaro wrote:
I don't think it would be smart for employees to voluntarily opt-in to sending all DNS to their employer anyway... there's little benefit to the employee, and a lot of downside.
Again, it is not up to systemd to limit valid use cases.
Perhaps Listen or read to Paul Vixie, father of many Bind software releases:
https://www.youtube.com/watch?v=ZxTdEEuyxHU
https://www.theregister.com/2018/10/23/paul_vixie_slaps_doh_as_dns_privacy _feature_becomes_a_standard/
There are use cases for and against routing all DNS over your VPN. If systemd wants to play system resolver, it needs to be able to be configured for either use case. You don't get to limit our use cases.
Configure "." as "routing domain" on a specific iface and the lookups wil go there preferably. If you put that on your VPN iface this means DNS traffic goes there preferably. If you put that ont he main iface this means DNS traffic goes there preferably.
Ideally you'd use more fine grained routing domains however.
Lennart
Lennart,
Is that a NetworkManager setting or a systemd-resolved setting? Is that going to be exposed in the GUI, or is it something that gets hidden away?
How does systemd-resolved figure out what domains "should" be sent to a given connection's DNS server without some arcane incantation from the systemd docs?
On Mo, 28.09.20 23:37, John M. Harris Jr (johnmh@splentity.com) wrote:
Configure "." as "routing domain" on a specific iface and the lookups wil go there preferably. If you put that on your VPN iface this means DNS traffic goes there preferably. If you put that ont he main iface this means DNS traffic goes there preferably.
Is that a NetworkManager setting or a systemd-resolved setting? Is that going to be exposed in the GUI, or is it something that gets hidden away?
I am not an NM guy, but I think they expose this these days. I can tell you definitely though that this is easily accessible via "resolvectl domain <iface>" from the command line and from .network networkd configuration files.
How does systemd-resolved figure out what domains "should" be sent to a given connection's DNS server without some arcane incantation from the systemd docs?
As mentioned elsewhere:
1) Search domains are implicitly routing domains: if an interface has "redhat.com" as search domain we also use that as routing domain, i.e. all *.redhat.com lookups will go to this interface and not to others.
2) If neither search domains nor routing domains are configured on any interface for a domain, lookups are routed to all interfaces in parallel, and the first positive and last negative answer is used.
i.e. focus is primarily on "let's make DNS work" and "let's make the best of the little information we traditionally have", and any further, more complex routing requires additional configuration in NM, networkd or directly with resolvectl commands.
Lennart
-- Lennart Poettering, Berlin
On Tuesday, September 29, 2020 1:01:23 AM MST Lennart Poettering wrote:
On Mo, 28.09.20 23:37, John M. Harris Jr (johnmh@splentity.com) wrote:
Configure "." as "routing domain" on a specific iface and the lookups wil go there preferably. If you put that on your VPN iface this means DNS traffic goes there preferably. If you put that ont he main iface this means DNS traffic goes there preferably.
Is that a NetworkManager setting or a systemd-resolved setting? Is that going to be exposed in the GUI, or is it something that gets hidden away?
I am not an NM guy, but I think they expose this these days. I can tell you definitely though that this is easily accessible via "resolvectl domain <iface>" from the command line and from .network networkd configuration files.
How does systemd-resolved figure out what domains "should" be sent to a given connection's DNS server without some arcane incantation from the systemd docs?
As mentioned elsewhere:
Search domains are implicitly routing domains: if an interface has "redhat.com" as search domain we also use that as routing domain, i.e. all *.redhat.com lookups will go to this interface and not to others.
If neither search domains nor routing domains are configured on any interface for a domain, lookups are routed to all interfaces in parallel, and the first positive and last negative answer is used.
i.e. focus is primarily on "let's make DNS work" and "let's make the best of the little information we traditionally have", and any further, more complex routing requires additional configuration in NM, networkd or directly with resolvectl commands.
Lennart
Lennart,
Search domains have absolutely nothing to do with routing. Search domains are specifically used for resolving non-FQDN to FQDN. This isn't a reliable way to see what domains are handled by a VPN, or by any DNS server.
The Red Hat VPN is a good example of this, as not every internal subdomain is in search domains. That's the case for many VPNs, corporate or personal.
On Di, 29.09.20 03:49, John M. Harris Jr (johnmh@splentity.com) wrote:
Search domains have absolutely nothing to do with routing. Search domains are specifically used for resolving non-FQDN to FQDN. This isn't a reliable way to see what domains are handled by a VPN, or by any DNS server.
The Red Hat VPN is a good example of this, as not every internal subdomain is in search domains. That's the case for many VPNs, corporate or personal.
Please read what I wrote: we have nothing better. And no it's not a perfectly complete solution, I am aware of that. Configure the routes explicitly if you want, it's easy, and add the extra domains to the per-interface route and all will be jolly. If you don't, then things will still work, but mean that queries that aren't listed in any search domains will be sent to both the VPN and the main iface DNS, thus the RH VPN will work perfectly fine — only drawback is that those internal domains not listed as search domains might be seen on the internet. But what would expect here happens? If you don't tell us the routing we cannot do fully perfect routing to your wishes, you need to give us something.
Search domains on VPNs are an indicator that these domains are handled by the VPN, that's why we use them also as routing domains. But this doesn't mean it's the *only* routing domains we use. We use the ones you configure, primarily. But since the concept didn't previously exist we make the best from what we have.
Lennart
-- Lennart Poettering, Berlin
On Tuesday, September 29, 2020 3:59:14 AM MST Lennart Poettering wrote:
On Di, 29.09.20 03:49, John M. Harris Jr (johnmh@splentity.com) wrote:
Search domains have absolutely nothing to do with routing. Search domains are specifically used for resolving non-FQDN to FQDN. This isn't a reliable way to see what domains are handled by a VPN, or by any DNS server.
The Red Hat VPN is a good example of this, as not every internal subdomain is in search domains. That's the case for many VPNs, corporate or personal.
Please read what I wrote: we have nothing better. And no it's not a perfectly complete solution, I am aware of that. Configure the routes explicitly if you want, it's easy, and add the extra domains to the per-interface route and all will be jolly. If you don't, then things will still work, but mean that queries that aren't listed in any search domains will be sent to both the VPN and the main iface DNS, thus the RH VPN will work perfectly fine — only drawback is that those internal domains not listed as search domains might be seen on the internet. But what would expect here happens? If you don't tell us the routing we cannot do fully perfect routing to your wishes, you need to give us something.
Search domains on VPNs are an indicator that these domains are handled by the VPN, that's why we use them also as routing domains. But this doesn't mean it's the *only* routing domains we use. We use the ones you configure, primarily. But since the concept didn't previously exist we make the best from what we have.
Lennart
If you really must send DNS queries to both (which defeats the purpose of 'Split DNS'), then it may be better to just use the DNS server of the VPN when connected to VPN, then only check the LAN interface when the response is NXDOMAIN.
On Di, 29.09.20 04:03, John M. Harris Jr (johnmh@splentity.com) wrote:
Search domains on VPNs are an indicator that these domains are handled by the VPN, that's why we use them also as routing domains. But this doesn't mean it's the *only* routing domains we use. We use the ones you configure, primarily. But since the concept didn't previously exist we make the best from what we have.
If you really must send DNS queries to both (which defeats the purpose of 'Split DNS'), then it may be better to just use the DNS server of the VPN when connected to VPN, then only check the LAN interface when the response is NXDOMAIN.
As mentioned in this thread already: this policy makes sense for some cases but not for others.
For example, if I have my laptop in my home wifi, connected to RH VPN, then there are some names resolvable only via the local DNS. Specifically: my router's, my printer's and my NAS' address. And there are other names only resolvable via RH VPN. systemd-resolved for the first time gives me a chance for this to just work: it will send requests to both the RH DNS servers and the local ones, and uses the first successful reply, or the last failed reply. And that's quite frankly awesome, because that *never* worked before.
So sending the requests to all available DNS servers in absence of better routing info is a great enabler: it makes DNS "just work" for many cases, including my own, and I doubt it's a particularly exotic one.
Key, take-away here:
1. Ideally we'd just route company DNS traffic to VPN, and everything else to local LAN DNS. But that requires explicit routing info to be configured, we cannot auto-detect this info (beyond some minor inference from the search domains)
2. In some cases hence routing all DNS traffic via VPN and nothing via local might be a workable option. In others this would be wrong.
3. In others routing all DNS traffic via local DNS and nothing via VPN might be workable option. In others this would be wrong.
4. In all cases where we can't do #1 because we lack the info, and don't know whether to do #2 or #3 because it depends on the type of VPN use, then sending to everything in parallel and taking the first positive reply and the last negative one is the best chance to make things "just work".
Anyway, I think I am repeating myself here.
Lennart
-- Lennart Poettering, Berlin
Hi Lennart,
more below...
On 9/29/20 3:41 PM, Lennart Poettering wrote:
On Di, 29.09.20 04:03, John M. Harris Jr (johnmh@splentity.com) wrote:
Search domains on VPNs are an indicator that these domains are handled by the VPN, that's why we use them also as routing domains. But this doesn't mean it's the *only* routing domains we use. We use the ones you configure, primarily. But since the concept didn't previously exist we make the best from what we have.
If you really must send DNS queries to both (which defeats the purpose of 'Split DNS'), then it may be better to just use the DNS server of the VPN when connected to VPN, then only check the LAN interface when the response is NXDOMAIN.
As mentioned in this thread already: this policy makes sense for some cases but not for others.
For example, if I have my laptop in my home wifi, connected to RH VPN, then there are some names resolvable only via the local DNS. Specifically: my router's, my printer's and my NAS' address. And there are other names only resolvable via RH VPN. systemd-resolved for the first time gives me a chance for this to just work: it will send requests to both the RH DNS servers and the local ones, and uses the first successful reply, or the last failed reply. And that's quite frankly awesome, because that *never* worked before.
Would you please try dnssec-trigger? It does exactly this thing. Unlike resolved, it can do that also with DNSSEC support on client side. But it is not system default.
So no, that is not as fresh as you say. Just have to specify subdomains, that should be sent to specific server. Not just trying anyone that works. Mind my privacy, not everyone has to know what am I resolving.
So sending the requests to all available DNS servers in absence of better routing info is a great enabler: it makes DNS "just work" for many cases, including my own, and I doubt it's a particularly exotic one.
It takes privacy from you and make it much worse. Just because it does 'just work'. Wouldn't it make sense to let corporate admins specify, what should be routed there? Do you know enterprise VPN, which is configured by BFU?
Key, take-away here:
- Ideally we'd just route company DNS traffic to VPN, and everything else to local LAN DNS. But that requires explicit routing info to be configured, we cannot auto-detect this info (beyond some minor inference from the search domains)
Let company configure it via VPN, allow local override for power users.
- In some cases hence routing all DNS traffic via VPN and nothing via local might be a workable option. In others this would be wrong.
Can you please be more specific? Example we can argue about? _gateway is just bare name, it would make sense to send it to LAN. Anything else?
- In others routing all DNS traffic via local DNS and nothing via VPN might be workable option. In others this would be wrong.
Never heard about this. Can you provide an example, where is this configuration desired? Is there public product or company, that configures it service like this? Do they offer DNS server in DHCP/autoconfiguration, when they do not want it to be used?
- In all cases where we can't do #1 because we lack the info, and don't know whether to do #2 or #3 because it depends on the type of VPN use, then sending to everything in parallel and taking the first positive reply and the last negative one is the best chance to make things "just work".
Then work on getting the info. If you don't know which variant, choose the same as without resolved. That usually would be #2, right? If that VPN specified server and it has higher priority, route there. Use any place default route is sent to.
Anyway, I think I am repeating myself here.
Sure, you repeat yourself. But turn the deaf ear to arguments of others. Please use smart defaults, only when they don't endanger user's privacy nor security.
Lennart
-- Lennart Poettering, Berlin
Petr, Brno
On Di, 29.09.20 16:03, Petr Menšík (pemensik@redhat.com) wrote:
For example, if I have my laptop in my home wifi, connected to RH VPN, then there are some names resolvable only via the local DNS. Specifically: my router's, my printer's and my NAS' address. And there are other names only resolvable via RH VPN. systemd-resolved for the first time gives me a chance for this to just work: it will send requests to both the RH DNS servers and the local ones, and uses the first successful reply, or the last failed reply. And that's quite frankly awesome, because that *never* worked before.
Would you please try dnssec-trigger? It does exactly this thing. Unlike resolved, it can do that also with DNSSEC support on client side. But it is not system default.
Hmm? resolved can do DNSSEC too, as mentioned. DNSSEC support is opt-in though.
So no, that is not as fresh as you say. Just have to specify subdomains, that should be sent to specific server. Not just trying anyone that works. Mind my privacy, not everyone has to know what am I resolving.
I move my laptop around, I want something that just works. And I am pretty sure that's the same for Fedora: a DNSSEC implementation that is opt-out but requires manual config is a no-go.
So sending the requests to all available DNS servers in absence of better routing info is a great enabler: it makes DNS "just work" for many cases, including my own, and I doubt it's a particularly exotic one.
It takes privacy from you and make it much worse. Just because it does 'just work'. Wouldn't it make sense to let corporate admins specify, what should be routed there? Do you know enterprise VPN, which is configured by BFU?
It provides the same privacy guarantees as your solution — as long as you tell it the routing domains. The only difference: if you don't tell it anything resolved's behaviour "just works", while your solution means things do not just work.
You know that meme, "It's not DNS. There's no way it's DNS. It was DNS"? I am pretty sure we shouldn't adopt defaults that break for the common cases out-of-the-box. Hence, in absence of more explicit routing info I am very sure "just works" is better than "just fails".
Yes, the logic does potentially allow more than you#d want onto some interfaces, but I don't think a (fake?) sense of "privacy" is a good excuse for "let's ship something that cannot work by default".
Yes, I too would prefer if my regular, non-RH DNS traffic never goes to RH servers while I am in the VPN, and I can easily configure things that way. But if I am pretty sure the majority of people probably put more emphasis "please please work" than on "i'd rather have not working DNS than have DNS queries end up on RH's DNS servers".
Key, take-away here:
- Ideally we'd just route company DNS traffic to VPN, and everything else to local LAN DNS. But that requires explicit routing info to be configured, we cannot auto-detect this info (beyond some minor inference from the search domains)
Let company configure it via VPN, allow local override for power users.
VPNs generally do not propagate domain routing info. They do provide search domain info, which — as mentioned — we infer routing info from. And yes, we of course allow local override.
So: check and check? (to the level this is possible)
- In some cases hence routing all DNS traffic via VPN and nothing via local might be a workable option. In others this would be wrong.
Can you please be more specific? Example we can argue about? _gateway is just bare name, it would make sense to send it to LAN. Anything else?
This is what the person I replied to asked for: if VPN is up send all DNS traffic to VPN, none to local DNS server.
- In others routing all DNS traffic via local DNS and nothing via VPN might be workable option. In others this would be wrong.
Never heard about this. Can you provide an example, where is this configuration desired? Is there public product or company, that configures it service like this? Do they offer DNS server in DHCP/autoconfiguration, when they do not want it to be used?
You appear to one of those who wants that, i.e. no have DNS traffic end up at RH that shouldn't go there if you are in the VPN.
What this boils down to: there are primarily two usecases for VPN I see:
1. employee talks to company VPN to access very specific servers. For cases like this ideally only company DNS traffic goes to VPN and nothing else. i.e. porn domain lookups only go to local LAN DNS.
2. user uses public VPN services such as mozilla VPN or Private Internet Access or a similar service to obstruct the view on what they are doing to the local network. In this case all DNS traffic should go to VPN and nothing to local LAN DNS.
Both usecases are equally valid and probably equally popular. We can't guess what is right.
In the first case the local LAN is more "trusted" if you so will than the company VPN: pornsite DNS traffic should go to LAN, but not to VPN. In the latter case the local LAN is less trusted than then used VPN: pornsite DNS traffic should go to VPN, but never to LAN.
- In all cases where we can't do #1 because we lack the info, and don't know whether to do #2 or #3 because it depends on the type of VPN use, then sending to everything in parallel and taking the first positive reply and the last negative one is the best chance to make things "just work".
Then work on getting the info. If you don't know which variant, choose the same as without resolved. That usually would be #2, right? If that VPN specified server and it has higher priority, route there. Use any place default route is sent to.
Well, except that the info is not provided via openvpn and similar solutions. I can't make it up from thin air.
But I really want to access servers inside the RH VPN and my local router/printer/NAS at the same time, so I am totally happy that with resolved this now *just works* even if RH's VPN doesn't tell us anything about routing domains.
Anyway, I think I am repeating myself here.
Sure, you repeat yourself. But turn the deaf ear to arguments of others. Please use smart defaults, only when they don't endanger user's privacy nor security.
I don't think what you are saying is congruent. On one hand you want that VPNs do not get all DNS traffic apparently, on the other you say it should have preference over the LAN DNS in all cases if it is configured? So what is it now?
Anyway, I still believe that making things just work is the best option, and buying vague "privacy" by "it's ok if it doesn't work" is a shitty deal.
Lennart
-- Lennart Poettering, Berlin
On 9/29/20 5:21 PM, Lennart Poettering wrote:
On Di, 29.09.20 16:03, Petr Menšík (pemensik@redhat.com) wrote:
For example, if I have my laptop in my home wifi, connected to RH VPN, then there are some names resolvable only via the local DNS. Specifically: my router's, my printer's and my NAS' address. And there are other names only resolvable via RH VPN. systemd-resolved for the first time gives me a chance for this to just work: it will send requests to both the RH DNS servers and the local ones, and uses the first successful reply, or the last failed reply. And that's quite frankly awesome, because that *never* worked before.
Would you please try dnssec-trigger? It does exactly this thing. Unlike resolved, it can do that also with DNSSEC support on client side. But it is not system default.
Hmm? resolved can do DNSSEC too, as mentioned. DNSSEC support is opt-in though.
Are you sure? Can it?
# systemctl restart systemd-resolved $ resolvectl | grep DNSSEC DNSSEC setting: allow-downgrade DNSSEC supported: yes
# delv @127.0.0.53 ;; validating ./NS: got insecure response; parent indicates it should be secure ;; insecurity proof failed resolving './NS/IN': 127.0.0.53#53 ;; resolution failed: insecurity proof failed
# delv ; fully validated . 503266 IN NS a.root-servers.net. . 503266 IN NS b.root-servers.net. . 503266 IN NS c.root-servers.net. . 503266 IN NS d.root-servers.net. . 503266 IN NS e.root-servers.net. . 503266 IN NS f.root-servers.net. . 503266 IN NS g.root-servers.net. . 503266 IN NS h.root-servers.net. . 503266 IN NS i.root-servers.net. . 503266 IN NS j.root-servers.net. . 503266 IN NS k.root-servers.net. . 503266 IN NS l.root-servers.net. . 503266 IN NS m.root-servers.net. . 503266 IN RRSIG NS 8 0 518400 20201012050000 20200929040000 46594 . EUNUgwVVvcgdX6JRCPfmhFPuIJW8DZf7ww1hQAgek0GPDT7kc75fER5Z NpASxPhrTQKentVt/C5ptwa0Z58i8O7XyN6Pu5ZIZrOpG5zV6y0FqLnI B7L01ugkmTdZIxfqTxbyiTh8hTWspskbAYQrnrSPiotX0+POYlw7ZIwX R6V7Y2mF48wFclaejrPRQy/ee03QKYyT9nYPahe7i0qnbmvk1JAfDiCw dR6Aa2hm/RSuW+7nJRqCbeDZZ4mU1lAZDiyUQ1ezAl/HcCCcpBud8iae DBYFXDX86EP0hXQexpzN5MLUQZuTCIq5Ihz9Vqk0orXmeaZ/k56t/2td /ak8sw==
# rpm -q systemd systemd-246.6-2.fc34.x86_64
How can I tell it works? Is servfail on dnssec-failed.org enough?
So no, that is not as fresh as you say. Just have to specify subdomains, that should be sent to specific server. Not just trying anyone that works. Mind my privacy, not everyone has to know what am I resolving.
I move my laptop around, I want something that just works. And I am pretty sure that's the same for Fedora: a DNSSEC implementation that is opt-out but requires manual config is a no-go.
So sending the requests to all available DNS servers in absence of better routing info is a great enabler: it makes DNS "just work" for many cases, including my own, and I doubt it's a particularly exotic one.
It takes privacy from you and make it much worse. Just because it does 'just work'. Wouldn't it make sense to let corporate admins specify, what should be routed there? Do you know enterprise VPN, which is configured by BFU?
It provides the same privacy guarantees as your solution — as long as you tell it the routing domains. The only difference: if you don't tell it anything resolved's behaviour "just works", while your solution means things do not just work.
You know that meme, "It's not DNS. There's no way it's DNS. It was DNS"? I am pretty sure we shouldn't adopt defaults that break for the common cases out-of-the-box. Hence, in absence of more explicit routing info I am very sure "just works" is better than "just fails".
Yes, the logic does potentially allow more than you#d want onto some interfaces, but I don't think a (fake?) sense of "privacy" is a good excuse for "let's ship something that cannot work by default".
Yes, I too would prefer if my regular, non-RH DNS traffic never goes to RH servers while I am in the VPN, and I can easily configure things that way. But if I am pretty sure the majority of people probably put more emphasis "please please work" than on "i'd rather have not working DNS than have DNS queries end up on RH's DNS servers".
Key, take-away here:
- Ideally we'd just route company DNS traffic to VPN, and everything else to local LAN DNS. But that requires explicit routing info to be configured, we cannot auto-detect this info (beyond some minor inference from the search domains)
Let company configure it via VPN, allow local override for power users.
VPNs generally do not propagate domain routing info. They do provide search domain info, which — as mentioned — we infer routing info from. And yes, we of course allow local override.
So: check and check? (to the level this is possible)
- In some cases hence routing all DNS traffic via VPN and nothing via local might be a workable option. In others this would be wrong.
Can you please be more specific? Example we can argue about? _gateway is just bare name, it would make sense to send it to LAN. Anything else?
This is what the person I replied to asked for: if VPN is up send all DNS traffic to VPN, none to local DNS server.
- In others routing all DNS traffic via local DNS and nothing via VPN might be workable option. In others this would be wrong.
Never heard about this. Can you provide an example, where is this configuration desired? Is there public product or company, that configures it service like this? Do they offer DNS server in DHCP/autoconfiguration, when they do not want it to be used?
You appear to one of those who wants that, i.e. no have DNS traffic end up at RH that shouldn't go there if you are in the VPN.
What this boils down to: there are primarily two usecases for VPN I see:
employee talks to company VPN to access very specific servers. For cases like this ideally only company DNS traffic goes to VPN and nothing else. i.e. porn domain lookups only go to local LAN DNS.
user uses public VPN services such as mozilla VPN or Private Internet Access or a similar service to obstruct the view on what they are doing to the local network. In this case all DNS traffic should go to VPN and nothing to local LAN DNS.
Both usecases are equally valid and probably equally popular. We can't guess what is right.
In the first case the local LAN is more "trusted" if you so will than the company VPN: pornsite DNS traffic should go to LAN, but not to VPN. In the latter case the local LAN is less trusted than then used VPN: pornsite DNS traffic should go to VPN, but never to LAN.
- In all cases where we can't do #1 because we lack the info, and don't know whether to do #2 or #3 because it depends on the type of VPN use, then sending to everything in parallel and taking the first positive reply and the last negative one is the best chance to make things "just work".
Then work on getting the info. If you don't know which variant, choose the same as without resolved. That usually would be #2, right? If that VPN specified server and it has higher priority, route there. Use any place default route is sent to.
Well, except that the info is not provided via openvpn and similar solutions. I can't make it up from thin air.
But I really want to access servers inside the RH VPN and my local router/printer/NAS at the same time, so I am totally happy that with resolved this now *just works* even if RH's VPN doesn't tell us anything about routing domains.
Anyway, I think I am repeating myself here.
Sure, you repeat yourself. But turn the deaf ear to arguments of others. Please use smart defaults, only when they don't endanger user's privacy nor security.
I don't think what you are saying is congruent. On one hand you want that VPNs do not get all DNS traffic apparently, on the other you say it should have preference over the LAN DNS in all cases if it is configured? So what is it now?
Anyway, I still believe that making things just work is the best option, and buying vague "privacy" by "it's ok if it doesn't work" is a shitty deal.
Lennart
-- Lennart Poettering, Berlin
On Tue, Sep 29, 2020 at 6:32 pm, Petr Menšík pemensik@redhat.com wrote:
Are you sure? Can it?
It cannot: https://bugzilla.redhat.com/show_bug.cgi?id=1879028
On 9/29/20 5:21 PM, Lennart Poettering wrote:
On Di, 29.09.20 16:03, Petr Menšík (pemensik@redhat.com) wrote:
For example, if I have my laptop in my home wifi, connected to RH VPN, then there are some names resolvable only via the local DNS. Specifically: my router's, my printer's and my NAS' address. And there are other names only resolvable via RH VPN. systemd-resolved for the first time gives me a chance for this to just work: it will send requests to both the RH DNS servers and the local ones, and uses the first successful reply, or the last failed reply. And that's quite frankly awesome, because that *never* worked before.
Would you please try dnssec-trigger? It does exactly this thing. Unlike resolved, it can do that also with DNSSEC support on client side. But it is not system default.
Hmm? resolved can do DNSSEC too, as mentioned. DNSSEC support is opt-in though.
So no, that is not as fresh as you say. Just have to specify subdomains, that should be sent to specific server. Not just trying anyone that works. Mind my privacy, not everyone has to know what am I resolving.
I move my laptop around, I want something that just works. And I am pretty sure that's the same for Fedora: a DNSSEC implementation that is opt-out but requires manual config is a no-go.
No-one demands 100% working validation. We just need 100% working DNSSEC passthru via resolved, so proper validators can ensure results. There are problems with DNSSEC. Most of them have known solutions. Just because your café and modem vendor haven't use any of them.
DNSSEC blocking resolver on fully DNSSEC aware network in year 2020 is NO-GO to me. It is simple, just don't stand in the way.
So sending the requests to all available DNS servers in absence of better routing info is a great enabler: it makes DNS "just work" for many cases, including my own, and I doubt it's a particularly exotic one.
It takes privacy from you and make it much worse. Just because it does 'just work'. Wouldn't it make sense to let corporate admins specify, what should be routed there? Do you know enterprise VPN, which is configured by BFU?
It provides the same privacy guarantees as your solution — as long as you tell it the routing domains. The only difference: if you don't tell it anything resolved's behaviour "just works", while your solution means things do not just work.
It "works", but provides the other party whole information about what I visit. Did you know this was the reason for invention of DNS over TLS and DNS over HTTPS? Even with such encrypted channel, you would leak it out. If that's the price for "I do not know how but it works", then I say no, thank you.
It it does not know, you can find a way to make it work. When it works and leaks, you just won't know.
You know that meme, "It's not DNS. There's no way it's DNS. It was DNS"?
"It's not systemd. There's no way it's systemd. It was systemd"? Ever heard this version?
I am pretty sure we shouldn't adopt defaults that break for the common cases out-of-the-box. Hence, in absence of more explicit routing info I am very sure "just works" is better than "just fails".
I have no problem with proper autoconfiguration. Throwing queries to every connection I have is plain wrong and always was. That's the reason why others let it fail in this case.
Yes, the logic does potentially allow more than you#d want onto some interfaces, but I don't think a (fake?) sense of "privacy" is a good excuse for "let's ship something that cannot work by default".> Yes, I too would prefer if my regular, non-RH DNS traffic never goes to RH servers while I am in the VPN, and I can easily configure things that way. But if I am pretty sure the majority of people probably put more emphasis "please please work" than on "i'd rather have not working DNS than have DNS queries end up on RH's DNS servers".But you omit easy configuration on RH VPN server, which would just fix
it. Way to autoconfigure the service without special configuration done by mere mortal.
Key, take-away here:
- Ideally we'd just route company DNS traffic to VPN, and everything else to local LAN DNS. But that requires explicit routing info to be configured, we cannot auto-detect this info (beyond some minor inference from the search domains)
Let company configure it via VPN, allow local override for power users.
VPNs generally do not propagate domain routing info. They do provide search domain info, which — as mentioned — we infer routing info from. And yes, we of course allow local override.
So: check and check? (to the level this is possible)
- In some cases hence routing all DNS traffic via VPN and nothing via local might be a workable option. In others this would be wrong.
Can you please be more specific? Example we can argue about? _gateway is just bare name, it would make sense to send it to LAN. Anything else?
This is what the person I replied to asked for: if VPN is up send all DNS traffic to VPN, none to local DNS server.
- In others routing all DNS traffic via local DNS and nothing via VPN might be workable option. In others this would be wrong.
Never heard about this. Can you provide an example, where is this configuration desired? Is there public product or company, that configures it service like this? Do they offer DNS server in DHCP/autoconfiguration, when they do not want it to be used?
You appear to one of those who wants that, i.e. no have DNS traffic end up at RH that shouldn't go there if you are in the VPN.
What this boils down to: there are primarily two usecases for VPN I see:
employee talks to company VPN to access very specific servers. For cases like this ideally only company DNS traffic goes to VPN and nothing else. i.e. porn domain lookups only go to local LAN DNS.
user uses public VPN services such as mozilla VPN or Private Internet Access or a similar service to obstruct the view on what they are doing to the local network. In this case all DNS traffic should go to VPN and nothing to local LAN DNS.
Both usecases are equally valid and probably equally popular. We can't guess what is right.
In the first case the local LAN is more "trusted" if you so will than the company VPN: pornsite DNS traffic should go to LAN, but not to VPN. In the latter case the local LAN is less trusted than then used VPN: pornsite DNS traffic should go to VPN, but never to LAN.
Sure, we have agreement at least. Because we miss way to know which is the kind, use configuration from VPN or local network to choose. I think case 2. is the legacy way. If we have no hint from service configuration, choose the most compatible. Make that choose predictable, secure and well documented.
- In all cases where we can't do #1 because we lack the info, and don't know whether to do #2 or #3 because it depends on the type of VPN use, then sending to everything in parallel and taking the first positive reply and the last negative one is the best chance to make things "just work".
Then work on getting the info. If you don't know which variant, choose the same as without resolved. That usually would be #2, right? If that VPN specified server and it has higher priority, route there. Use any place default route is sent to.
Well, except that the info is not provided via openvpn and similar solutions. I can't make it up from thin air.
It is provided by DHCP. IP4.DOMAIN[1]: redhat.com
But I really want to access servers inside the RH VPN and my local router/printer/NAS at the same time, so I am totally happy that with resolved this now *just works* even if RH's VPN doesn't tell us anything about routing domains.
I work for Red Hat too as you may have noticed. Because we made sure with dnssec-trigger it tells us to route redhat.com via VPN. So like you, I can access local resources in my home network and private redhat.com hosts at the same configuration. It works automagically because we requested it. We did not request jboss.com and that's missing. Surprise?
Wonders wonders, no resolved involved, just unbound+dnssec-trigger. Might not be under active development, but still works to some extent.
Anyway, I think I am repeating myself here.
Sure, you repeat yourself. But turn the deaf ear to arguments of others. Please use smart defaults, only when they don't endanger user's privacy nor security.
I don't think what you are saying is congruent. On one hand you want that VPNs do not get all DNS traffic apparently, on the other you say it should have preference over the LAN DNS in all cases if it is configured? So what is it now?
Make VPN administrator responsible. Do not invent behaviour, use what was most likely intended by network provider. Document how to create working configurations instead of trying to workaround bugs of others. DNS standards have a story about it and they learned it hard way. One of reasons DNS is complicated today. Learn from their mistakes.
Anyway, I still believe that making things just work is the best option, and buying vague "privacy" by "it's ok if it doesn't work" is a shitty deal.
Depends what do you trade for. My privacy expectation is simple: what I do is not seen by anyone I wouldn't allow. Meaning my browsing history does just one way with defined exceptions.
Lennart
-- Lennart Poettering, Berlin
On Tue, Sep 29, 2020 at 5:21 pm, Lennart Poettering mzerqung@0pointer.de wrote:
Yes, I too would prefer if my regular, non-RH DNS traffic never goes to RH servers while I am in the VPN, and I can easily configure things that way. But if I am pretty sure the majority of people probably put more emphasis "please please work" than on "i'd rather have not working DNS than have DNS queries end up on RH's DNS servers".
For clarity, if you check the box "Use this connection only for resources on its network," then NetworkManager will configure a search domain to ensure that non-RH DNS traffic will not go to RH servers. Lennart is describing systemd-resolved's default mode of operation on systems without NetworkManager, but NetworkManager is default is all Fedora editions.
On Tue, Sep 29, 2020 at 3:43 PM Lennart Poettering mzerqung@0pointer.de wrote:
On Di, 29.09.20 04:03, John M. Harris Jr (johnmh@splentity.com) wrote:
Search domains on VPNs are an indicator that these domains are handled by the VPN, that's why we use them also as routing domains. But this doesn't mean it's the *only* routing domains we use. We use the ones you configure, primarily. But since the concept didn't previously exist we make the best from what we have.
If you really must send DNS queries to both (which defeats the purpose of 'Split DNS'), then it may be better to just use the DNS server of the VPN when connected to VPN, then only check the LAN interface when the response is NXDOMAIN.
As mentioned in this thread already: this policy makes sense for some cases but not for others.
For example, if I have my laptop in my home wifi, connected to RH VPN, then there are some names resolvable only via the local DNS. Specifically: my router's, my printer's and my NAS' address. And there are other names only resolvable via RH VPN. systemd-resolved for the first time gives me a chance for this to just work: it will send requests to both the RH DNS servers and the local ones, and uses the first successful reply, or the last failed reply. And that's quite frankly awesome, because that *never* worked before.
So sending the requests to all available DNS servers in absence of better routing info is a great enabler: it makes DNS "just work" for many cases, including my own, and I doubt it's a particularly exotic one.
It is not an exotic one, but this behavior was in the past considered a vulnerability (information disclosure) [0]. Are we re-introducing it? I guess yes, and it can be that the benefits of it outweigh the vulnerability, but we should be explicit about it in our release notes.
[0]. CVE-2018-1000135 https://bugzilla.redhat.com/show_bug.cgi?id=1558238
Key, take-away here:
- Ideally we'd just route company DNS traffic to VPN, and everything else to local LAN DNS. But that requires explicit routing info to be configured, we cannot auto-detect this info (beyond some minor inference from the search domains)
Do we know which fedora shipped VPNs work well with split-dns and which will lead to leaking the web sites accessed?
regards, Nikos
On Tue, Sep 29, 2020 at 4:06 pm, Nikos Mavrogiannopoulos nmav@redhat.com wrote:
It is not an exotic one, but this behavior was in the past considered a vulnerability (information disclosure) [0]. Are we re-introducing it? I guess yes, and it can be that the benefits of it outweigh the vulnerability, but we should be explicit about it in our release notes.
[0]. CVE-2018-1000135 https://bugzilla.redhat.com/show_bug.cgi?id=1558238
If all I knew about this was what Lennart just wrote, I would be very concerned, because preventing DNS leaks is very important. But Lennart is not considering that NetworkManager is not going to configure systemd-resolved to operate like this. Lennart's described behavior only applies if you give systemd-resolved absolutely no information for how to route the DNS. But NetworkManager will not do that, it will do the right thing. E.g. if you have one "primary VPN" configured that accepts all traffic, your DNS goes to that VPN. It's not going to leak DNS queries to your router or to your ISP. If you have a VPN that only accepts traffic on its own network, it gets that DNS and not more. This is way better than the status quo prior to systemd-resolved where unexpected behavior was the norm.
In particular, if you have a VPN that does not select "use this connection only for resources on its network," then NetworkManager will configure a DNS domain ~. corresponding to the VPN's tun interface. All DNS goes there and only there unless it matches another search domain.
Michael
On Tuesday, September 29, 2020 6:41:12 AM MST Lennart Poettering wrote:
On Di, 29.09.20 04:03, John M. Harris Jr (johnmh@splentity.com) wrote:
Search domains on VPNs are an indicator that these domains are handled by the VPN, that's why we use them also as routing domains. But this doesn't mean it's the *only* routing domains we use. We use the ones you configure, primarily. But since the concept didn't previously exist we make the best from what we have.
If you really must send DNS queries to both (which defeats the purpose of 'Split DNS'), then it may be better to just use the DNS server of the VPN when connected to VPN, then only check the LAN interface when the response is NXDOMAIN.
As mentioned in this thread already: this policy makes sense for some cases but not for others.
For example, if I have my laptop in my home wifi, connected to RH VPN, then there are some names resolvable only via the local DNS. Specifically: my router's, my printer's and my NAS' address. And there are other names only resolvable via RH VPN. systemd-resolved for the first time gives me a chance for this to just work: it will send requests to both the RH DNS servers and the local ones, and uses the first successful reply, or the last failed reply. And that's quite frankly awesome, because that *never* worked before.
So sending the requests to all available DNS servers in absence of better routing info is a great enabler: it makes DNS "just work" for many cases, including my own, and I doubt it's a particularly exotic one.
There seems to be some confusion in this thread as to how exactly that works. It seems that the individual pushing this change is under the assumption that this implements 'Split DNS', that only requests for VPN resources are queried from the VPN's DNS server(s).
Key, take-away here:
- Ideally we'd just route company DNS traffic to VPN, and everything else to local LAN DNS. But that requires explicit routing info to be configured, we cannot auto-detect this info (beyond some minor inference from the search domains)
Well, we have the routing information, unless the VPN is one flat network. However, routing has absolutely nothing to do with DNS. Routing is a layer below, and is most likely required for your DNS queries to actually get to the DNS server(s) inside the VPN.
In some cases hence routing all DNS traffic via VPN and nothing via local might be a workable option. In others this would be wrong.
In others routing all DNS traffic via local DNS and nothing via VPN might be workable option. In others this would be wrong.
In all cases where we can't do #1 because we lack the info, and don't know whether to do #2 or #3 because it depends on the type of VPN use, then sending to everything in parallel and taking the first positive reply and the last negative one is the best chance to make things "just work".
This seems to be the part where the Change Owner is confused as to how this works, which is concerning at best. We need to be very careful when changing functionality that is so essential to the way Fedora systems use network resources, this is a pretty big change in functionality, and doesn't provide 'Split DNS', the issue the Change was originally created to solve.
Hi,
For example, if I have my laptop in my home wifi, connected to RH VPN, then there are some names resolvable only via the local DNS. Specifically: my router's, my printer's and my NAS' address. And there are other names only resolvable via RH VPN. systemd-resolved for the first time gives me a chance for this to just work: it will send requests to both the RH DNS servers and the local ones, and uses the first successful reply, or the last failed reply. And that's quite frankly awesome, because that *never* worked before.
Sorry, but that is not correct.
NetworkManager can handle split-dns just fine, by using dnsmasq and reconfiguring it via dbus when vpn connections come and go. I can easily add more servers + zones by dropping a config file snippet into the /etc/NetworkManager/dnsmasq.d/ directory, for example to resolve the hostnames for my kvm guests on the libvirt network.
That works for ages on my RHEL-7 workstation where systemd-resolved doesn't even exist ...
So sending the requests to all available DNS servers in absence of better routing info is a great enabler:
I fail to see why sending queries to all servers is a good plan. The redhat vpn dns servers surely can't resolve the hostnames for my local lan, and frankly they shouldn't even know which hosts I try to access. Likewise my ISP shouldn't know which non-public RH servers I try to access.
Key, take-away here:
- Ideally we'd just route company DNS traffic to VPN, and everything else to local LAN DNS. But that requires explicit routing info to be configured, we cannot auto-detect this info (beyond some minor inference from the search domains)
Well, if auto-detect doesn't work we should (a) fix the vpn protocols/implementations and (b) fallback to manually configuring things until this is done.
VPN use, then sending to everything in parallel and taking the first positive reply and the last negative one is the best chance to make things "just work".
At the expense of leaking information, which I don't consider a good trade-off.
take care, Gerd
Am 30.09.20 um 10:05 schrieb Gerd Hoffmann:
So sending the requests to all available DNS servers in absence of better routing info is a great enabler:
I fail to see why sending queries to all servers is a good plan. The redhat vpn dns servers surely can't resolve the hostnames for my local lan, and frankly they shouldn't even know which hosts I try to access. Likewise my ISP shouldn't know which non-public RH servers I try to access.
Your absolutly right, the described scenario would be a privacy and security nightmare.
Best regards, Marius Schwarz
On Wed, Sep 30, 2020 at 10:05 am, Gerd Hoffmann kraxel@redhat.com wrote:
Sorry, but that is not correct.
NetworkManager can handle split-dns just fine, by using dnsmasq and reconfiguring it via dbus when vpn connections come and go. I can easily add more servers + zones by dropping a config file snippet into the /etc/NetworkManager/dnsmasq.d/ directory, for example to resolve the hostnames for my kvm guests on the libvirt network.
That works for ages on my RHEL-7 workstation where systemd-resolved doesn't even exist ...
We actually considered dnsmasq, but NetworkManager developers recommended systemd-resolved. See: https://pagure.io/fedora-workstation/issue/123#comment-621603
I agree dnsmasq would have been a lot better than the status quo prior to F33. We would probably have used that if systemd-resolved didn't exist. If we could have a do-over, we should have started using it long ago.
So sending the requests to all available DNS servers in absence of better routing info is a great enabler:
I fail to see why sending queries to all servers is a good plan. The redhat vpn dns servers surely can't resolve the hostnames for my local lan, and frankly they shouldn't even know which hosts I try to access. Likewise my ISP shouldn't know which non-public RH servers I try to access.
I've tried to stop this line of discussion a bit earlier, since it's based on a misunderstanding of how NetworkManager uses systemd-resolved. I agree we should prioritize avoiding DNS leaks, and that's actually the primary motivating factor for the switch to systemd-resolved (you can see how much more attention I devote to this topic in the change proposal compared to the other benefits of systemd-resolved). NetworkManager will not configure systmed-resolved to send queries all over the place. We need a local resolver (systemd-resolved, dnsmasq, etc.) to ensure DNS queries go where users expect; it's not something we can do with traditional resolv.conf managed by NetworkManager.
Michael
Reading along, it's _at_best_ unclear what the eventual 'resolution' of this^ is.
What _is_ clear is that there's significant disagreement -- which, unfortunately, has at times here become nasty & personal -- about needed vs planned functionality, and, of late, regulatory compliance.
And, iiuc, though obviously very much up in the air, this is all relevant to F33 release, coming in weeks?
Can someone please clarify, ideally with some level of certainty:
If we've F32 systems in place, that do NOT use systemd-resolved &/or NetworkManager, but rather our own/preferred DNS client implementations with systemd-networkd,
will a system *upgrade*, from F32 to F33, force/require any changes to those configurations? or will systems be left as-is, and we can expect uninterrupted functionality?
which of these proposed systemd-resolved system-wide changes are NON-optional in _usage_? can they _all_ be turned-off/disabled?
bottom-line -- how much system breakage of existing infrastructure, if any, should we be planning for with a F32 -> F33 upgrade path?
On Wed, Sep 30, 2020 at 12:15 PM PGNet Dev pgnet.dev@gmail.com wrote:
Reading along, it's _at_best_ unclear what the eventual 'resolution' of this^ is.
What _is_ clear is that there's significant disagreement -- which, unfortunately, has at times here become nasty & personal -- about needed vs planned functionality, and, of late, regulatory compliance.
And, iiuc, though obviously very much up in the air, this is all relevant to F33 release, coming in weeks?
Can someone please clarify, ideally with some level of certainty:
If we've F32 systems in place, that do NOT use systemd-resolved &/or NetworkManager, but rather our own/preferred DNS client implementations with systemd-networkd,
will a system *upgrade*, from F32 to F33, force/require any changes to those configurations? or will systems be left as-is, and we can expect uninterrupted functionality?
which of these proposed systemd-resolved system-wide changes are NON-optional in _usage_? can they _all_ be turned-off/disabled?
bottom-line -- how much system breakage of existing infrastructure, if any, should we be planning for with a F32 -> F33 upgrade path?
If you're not using NetworkManager, this change has _zero_ impact.
On 9/30/20 9:16 AM, Neal Gompa wrote:
If you're not using NetworkManager, this change has _zero_ impact.
perfect.
clearly, i've missed or lost the obviousness of that incredibly useful tidbit in this novella :-/
thx!
On Wednesday, 30 September 2020 at 18:16, Neal Gompa wrote: [...]
If you're not using NetworkManager, this change has _zero_ impact.
What if I'm using NetworkManager and dnssec-trigger? This has been working very well for me for the last couple of releases and I'd hate to be forced to manually reconfigure things so that it starts working again.
Regards, Dominik
On Wed, Sep 30, 2020 at 6:43 pm, Dominik 'Rathann' Mierzejewski dominik@greysector.net wrote:
What if I'm using NetworkManager and dnssec-trigger? This has been working very well for me for the last couple of releases and I'd hate to be forced to manually reconfigure things so that it starts working again.
The upgrade process is designed to do the right thing for users who stick with our defaults. Manual intervention is required for unusual cases like this. You'll need to manually disable systemd-resolved after upgrade, restore /etc/resolv.conf from the backup file that will be created during upgrade, and restart NetworkManager. That would look something like:
# systemctl disable systemd-resolved.service # systmectl stop systemd-resolved.service # mv /etc/resolv.conf.orig-with-nm /etc/resolv.conf # systemctl restart NetworkManager.service
Michael
On 9/30/20 9:50 AM, Michael Catanzaro wrote:
You'll need to manually disable systemd-resolved after upgrade, restore /etc/resolv.conf from the backup file that will be created during upgrade
So the upgrade WILL ignore current F32 state -- systemd-resolved DISABLED + 'my' /etc/resolv.conf -- and enable + overwrite (respectively) each, regardless of whether we're _using_ NetworkManager (afaict it's impossible to completely remove all NM* cruft)?
On Wed, Sep 30, 2020 at 9:54 am, PGNet Dev pgnet.dev@gmail.com wrote:
So the upgrade WILL ignore current F32 state -- systemd-resolved DISABLED + 'my' /etc/resolv.conf -- and enable + overwrite (respectively) each, regardless of whether we're _using_ NetworkManager (afaict it's impossible to completely remove all NM* cruft)?
It only touches your /etc/resolv.conf if you are using NetworkManager, but I think it enables systemd-resolved regardless. You'll have to disable it if you don't want it.
On 9/30/20 7:11 PM, Michael Catanzaro wrote:
On Wed, Sep 30, 2020 at 9:54 am, PGNet Dev pgnet.dev@gmail.com wrote:
So the upgrade WILL ignore current F32 state -- systemd-resolved DISABLED + 'my' /etc/resolv.conf -- and enable + overwrite (respectively) each, regardless of whether we're _using_ NetworkManager (afaict it's impossible to completely remove all NM* cruft)?
It only touches your /etc/resolv.conf if you are using NetworkManager, but I think it enables systemd-resolved regardless. You'll have to disable it if you don't want it.
Shouldn't it change resolv.conf only in case NM is active AND resolv.conf is generated by Network Manager?
resolv.conf with dnssec-trigger is generated anyway, there is little need to backup it. It looks like:
# Generated by dnssec-trigger-script nameserver 127.0.0.1
I haven't tested upgrade to f33 yet. dnssec-trigger also tries protect resolv.conf for overwriting by chattr +i /etc/resolv.conf. I think it would break upgrade if change was attempted by systemd.
Also, it would battle for port 53 for anyone having already local resolver. It would just fail to start, if user enabled listening on any address. Which is now unfortunately default for dnsmasq. It might be a little random, which one would win on machine start. I guess systemd-resolved would win, because unlike named(bind) or unbound, it has before nss-lookup.target and network.target. Normal dns services start After=network.target.
Fortunately, it would not conflict with resolvers listening on localhost only, because it listens on different IP address. But it is before dns in nsswitch.conf, so every query would be done before "proper" dns resolvers. I don't this that is correct, when resolved has serious limitations in security.
I expect it would break any local resolver configured, unless it can detect existing resolver and avoid activation of systemd-resolved.
On Wed, Sep 30, 2020 at 9:58 pm, Petr Menšík pemensik@redhat.com wrote:
Shouldn't it change resolv.conf only in case NM is active AND resolv.conf is generated by Network Manager?
Correct, that's indeed what it does. (Since Zbigniew changed it yesterday. Previously, it did not check if NM is active.)
The systemd RPM scriplet will run sed, sed will see "Generated by dnssec-trigger-script" and say "that's not "Generated by NetworkManager" and it will leave your resolv.conf alone. So you're right, you don't need to worry about resolv.conf.
Even if it did try to remove the immutable /etc/resolv.conf, the upgrade script ignores failures, so you would see an error message in the middle of the transaction about rm failing but it wouldn't fail the whole upgrade.
It *will* still 'systemctl enable systemd-resolved.service', though, so you'll still need to disable that or you'll indeed have a battle for port 53. It will indeed probably break any local resolver you have configured.
On Wed, Sep 30, 2020, at 4:17 PM, Michael Catanzaro wrote:
On Wed, Sep 30, 2020 at 9:58 pm, Petr Menšík pemensik@redhat.com wrote:
Shouldn't it change resolv.conf only in case NM is active AND resolv.conf is generated by Network Manager?
Correct, that's indeed what it does. (Since Zbigniew changed it yesterday. Previously, it did not check if NM is active.)
I know this is already an epic thread but, just a FYI: This type of thing will completely not work on an rpm-ostree based system because the %post is run server side. It can't compute anything based on per-user data (and that's *also* true even when doing client side layering - the `/etc` that the `%post` script sees is the defaults, not anything a user has customized).
In general, any upgrade logic that needs to take into account user configuration needs to be a systemd unit, not a `%post`.
On Wed, Sep 30, 2020 at 7:34 pm, Colin Walters walters@verbum.org wrote:
I know this is already an epic thread but, just a FYI: This type of thing will completely not work on an rpm-ostree based system because the %post is run server side. It can't compute anything based on per-user data (and that's *also* true even when doing client side layering - the `/etc` that the `%post` script sees is the defaults, not anything a user has customized).
In general, any upgrade logic that needs to take into account user configuration needs to be a systemd unit, not a `%post`.
Hmmm, that probably means that users who upgrade Silverblue from F32 -> F33 will be broken.
Michael Catanzaro wrote:
On Wed, Sep 30, 2020 at 6:43 pm, Dominik 'Rathann' Mierzejewski dominik@greysector.net wrote:
What if I'm using NetworkManager and dnssec-trigger? This has been working very well for me for the last couple of releases and I'd hate to be forced to manually reconfigure things so that it starts working again.
The upgrade process is designed to do the right thing for users who stick with our defaults. Manual intervention is required for unusual cases like this. You'll need to manually disable systemd-resolved after upgrade, restore /etc/resolv.conf from the backup file that will be created during upgrade, and restart NetworkManager. That would look something like:
# systemctl disable systemd-resolved.service # systmectl stop systemd-resolved.service # mv /etc/resolv.conf.orig-with-nm /etc/resolv.conf # systemctl restart NetworkManager.service
So there's no need to revert any changes to /etc/nsswitch.conf? I've seen some discussion about that file in relation to systemd-resolved. It seemed far from easy to understand how to make it work correctly.
Björn Persson
On Wed, Sep 30, 2020 at 11:49 pm, Björn Persson Bjorn@xn--rombobjrn-67a.se wrote:
So there's no need to revert any changes to /etc/nsswitch.conf? I've seen some discussion about that file in relation to systemd-resolved. It seemed far from easy to understand how to make it work correctly.
You don't have to touch /etc/nsswitch.conf because it's designed to work with or without systemd-resolved running: resolve [!UNAVAIL=return]. If it's not running it will fall back to nss-myhostname and then nss-dns.
WARNING: Do not make manual changes to /etc/nsswitch.conf! Remember to edit /etc/authselect/user-sswitch.conf instead, then run 'sudo authselect apply-changes'
Dne 01. 10. 20 v 0:10 Michael Catanzaro napsal(a):
On Wed, Sep 30, 2020 at 11:49 pm, Björn Persson Bjorn@xn--rombobjrn-67a.se wrote:
So there's no need to revert any changes to /etc/nsswitch.conf? I've seen some discussion about that file in relation to systemd-resolved. It seemed far from easy to understand how to make it work correctly.
You don't have to touch /etc/nsswitch.conf because it's designed to work with or without systemd-resolved running: resolve [!UNAVAIL=return]. If it's not running it will fall back to nss-myhostname and then nss-dns.
WARNING: Do not make manual changes to /etc/nsswitch.conf! Remember to edit /etc/authselect/user-sswitch.conf instead, then run 'sudo authselect apply-changes'
I would appreciate if somebody took care about authselect tickets:
https://bugzilla.redhat.com/show_bug.cgi?id=1878752
The file you have referenced apparently lives in vacuum. You refer to it, but nobody owns it.
Vít
On Wed, Sep 30, 2020 at 11:50:00AM -0500, Michael Catanzaro wrote:
On Wed, Sep 30, 2020 at 6:43 pm, Dominik 'Rathann' Mierzejewski dominik@greysector.net wrote:
What if I'm using NetworkManager and dnssec-trigger? This has been working very well for me for the last couple of releases and I'd hate to be forced to manually reconfigure things so that it starts working again.
The upgrade process is designed to do the right thing for users who stick with our defaults. Manual intervention is required for unusual cases like this. You'll need to manually disable systemd-resolved after upgrade, restore /etc/resolv.conf from the backup file that will be created during upgrade, and restart NetworkManager. That would look something like:
# systemctl disable systemd-resolved.service # systmectl stop systemd-resolved.service # mv /etc/resolv.conf.orig-with-nm /etc/resolv.conf # systemctl restart NetworkManager.service
After the latest updates, this should not necessary (though it will still work). Instead, specify a preset (before the upgrade):
sudo bash -c 'mkdir -p /etc/systemd/system-preset && echo "disable systemd-resolved.service" >/etc/systemd/system-preset/20-systemd-resolved-disable.preset'
This is also described in https://fedoraproject.org/wiki/Changes/systemd-resolved#Fully_opting_out_of_...
Zbyszek
On Tue, 2020-09-29 at 12:59 +0200, Lennart Poettering wrote:
On Di, 29.09.20 03:49, John M. Harris Jr (johnmh@splentity.com) wrote:
Search domains have absolutely nothing to do with routing. Search domains are specifically used for resolving non-FQDN to FQDN. This isn't a reliable way to see what domains are handled by a VPN, or by any DNS server.
The Red Hat VPN is a good example of this, as not every internal subdomain is in search domains. That's the case for many VPNs, corporate or personal.
Please read what I wrote: we have nothing better. And no it's not a perfectly complete solution, I am aware of that. Configure the routes explicitly if you want, it's easy, and add the extra domains to the per-interface route and all will be jolly. If you don't, then things will still work, but mean that queries that aren't listed in any search domains will be sent to both the VPN and the main iface DNS, thus the RH VPN will work perfectly fine — only drawback is that those internal domains not listed as search domains might be seen on the internet. But what would expect here happens? If you don't tell us the routing we cannot do fully perfect routing to your wishes, you need to give us something.
Search domains on VPNs are an indicator that these domains are handled by the VPN, that's why we use them also as routing domains. But this doesn't mean it's the *only* routing domains we use. We use the ones you configure, primarily. But since the concept didn't previously exist we make the best from what we have.
I see conflicting information here from you and Michael Catanzaro.
You have mentioned quite a few times a fan out and leakage of name searches on all interfaces, while Michael said in response to me that if you do not select the magic option to do split DNS routing that all queries should go to the VPN only, which is it ?
And how do you configure domains that are not provided as search domain by the VPN so that they automatically are searched for via the VPN (and flushed at the time) when the VPN comes up?
In split DNS situation fan out is quite bad because you can get completely different answers, where generally the VPN has the correct answer for you but the local DNS server will probably win the latency game ...
Simo.
On Di, 29.09.20 13:56, Simo Sorce (simo@redhat.com) wrote:
On Tue, 2020-09-29 at 12:59 +0200, Lennart Poettering wrote:
On Di, 29.09.20 03:49, John M. Harris Jr (johnmh@splentity.com) wrote:
Search domains have absolutely nothing to do with routing. Search domains are specifically used for resolving non-FQDN to FQDN. This isn't a reliable way to see what domains are handled by a VPN, or by any DNS server.
The Red Hat VPN is a good example of this, as not every internal subdomain is in search domains. That's the case for many VPNs, corporate or personal.
Please read what I wrote: we have nothing better. And no it's not a perfectly complete solution, I am aware of that. Configure the routes explicitly if you want, it's easy, and add the extra domains to the per-interface route and all will be jolly. If you don't, then things will still work, but mean that queries that aren't listed in any search domains will be sent to both the VPN and the main iface DNS, thus the RH VPN will work perfectly fine — only drawback is that those internal domains not listed as search domains might be seen on the internet. But what would expect here happens? If you don't tell us the routing we cannot do fully perfect routing to your wishes, you need to give us something.
Search domains on VPNs are an indicator that these domains are handled by the VPN, that's why we use them also as routing domains. But this doesn't mean it's the *only* routing domains we use. We use the ones you configure, primarily. But since the concept didn't previously exist we make the best from what we have.
I see conflicting information here from you and Michael Catanzaro.
You have mentioned quite a few times a fan out and leakage of name searches on all interfaces, while Michael said in response to me that if you do not select the magic option to do split DNS routing that all queries should go to the VPN only, which is it ?
It might the latter. It's up to NM really: it depends what they tell resolved. If they default to associating the "." routing domain with a VPN this has the effect that all lookups will be preferably routed over that.
If they don't do that, and define no routing domains, then the interface it will be in the regular pool of ifaces where we send stuff for which no routing domain is defined anywhere.
So, I defer to Michael here: I didn't actually check what NM opted there. It might very well be that they default to configuring "." as routing domain for VPNs.
Lennart
-- Lennart Poettering, Berlin
On Tue, 2020-09-29 at 20:01 +0200, Lennart Poettering wrote:
On Di, 29.09.20 13:56, Simo Sorce (simo@redhat.com) wrote:
On Tue, 2020-09-29 at 12:59 +0200, Lennart Poettering wrote:
On Di, 29.09.20 03:49, John M. Harris Jr (johnmh@splentity.com) wrote:
Search domains have absolutely nothing to do with routing. Search domains are specifically used for resolving non-FQDN to FQDN. This isn't a reliable way to see what domains are handled by a VPN, or by any DNS server.
The Red Hat VPN is a good example of this, as not every internal subdomain is in search domains. That's the case for many VPNs, corporate or personal.
Please read what I wrote: we have nothing better. And no it's not a perfectly complete solution, I am aware of that. Configure the routes explicitly if you want, it's easy, and add the extra domains to the per-interface route and all will be jolly. If you don't, then things will still work, but mean that queries that aren't listed in any search domains will be sent to both the VPN and the main iface DNS, thus the RH VPN will work perfectly fine — only drawback is that those internal domains not listed as search domains might be seen on the internet. But what would expect here happens? If you don't tell us the routing we cannot do fully perfect routing to your wishes, you need to give us something.
Search domains on VPNs are an indicator that these domains are handled by the VPN, that's why we use them also as routing domains. But this doesn't mean it's the *only* routing domains we use. We use the ones you configure, primarily. But since the concept didn't previously exist we make the best from what we have.
I see conflicting information here from you and Michael Catanzaro.
You have mentioned quite a few times a fan out and leakage of name searches on all interfaces, while Michael said in response to me that if you do not select the magic option to do split DNS routing that all queries should go to the VPN only, which is it ?
It might the latter. It's up to NM really: it depends what they tell resolved. If they default to associating the "." routing domain with a VPN this has the effect that all lookups will be preferably routed over that.
If they don't do that, and define no routing domains, then the interface it will be in the regular pool of ifaces where we send stuff for which no routing domain is defined anywhere.
So, I defer to Michael here: I didn't actually check what NM opted there. It might very well be that they default to configuring "." as routing domain for VPNs.
Ah thanks, this is good to know, it is hard to figure out what is going on when there is conflicting information.
If NM does configure resolved this way it is a better option as a default.
Simo.
On Tue, Sep 29, 2020 at 8:01 pm, Lennart Poettering mzerqung@0pointer.de wrote:
So, I defer to Michael here: I didn't actually check what NM opted there. It might very well be that they default to configuring "." as routing domain for VPNs.
Yes, this is what happens.
Qualification: it's what should happen, sans bugs. We do have one known bug, https://bugzilla.redhat.com/show_bug.cgi?id=1863041, but we only know of one affected user, and the NetworkManager developers have figured out know how to fix it. This bug probably doesn't affect most users (at least it works properly for me).
On Tue, Sep 29, 2020 at 4:00 AM Lennart Poettering mzerqung@0pointer.de wrote:
On Di, 29.09.20 03:49, John M. Harris Jr (johnmh@splentity.com) wrote:
Search domains have absolutely nothing to do with routing. Search
domains are
specifically used for resolving non-FQDN to FQDN. This isn't a reliable
way to
see what domains are handled by a VPN, or by any DNS server.
The Red Hat VPN is a good example of this, as not every internal
subdomain is
in search domains. That's the case for many VPNs, corporate or personal.
Please read what I wrote: we have nothing better. And no it's not a perfectly complete solution, I am aware of that. Configure the routes explicitly if you want, it's easy, and add the extra domains to the per-interface route and all will be jolly. If you don't, then things will still work, but mean that queries that aren't listed in any search domains will be sent to both the VPN and the main iface DNS, thus the RH VPN will work perfectly fine — only drawback is that those internal domains not listed as search domains might be seen on the internet. But what would expect here happens? If you don't tell us the routing we cannot do fully perfect routing to your wishes, you need to give us something.
Search domains on VPNs are an indicator that these domains are handled by the VPN, that's why we use them also as routing domains. But this doesn't mean it's the *only* routing domains we use. We use the ones you configure, primarily. But since the concept didn't previously exist we make the best from what we have.
These heuristics seem fairly problematic, but this is solvable. Fedora has a considerable amount of influence on GNOME and NetworkManager. How about adjusting the UI to actually cover these cases? The idea that the VPN configuration would go off into the weeds if a new checkbox showed up seems silly — setting up a VPN is fundamentally a power user operation.
This could all be first class parts of VPN config. There could be a set of options: use this VPN to look up all DNS domains or use this VPN to look up the following domains. Each domain in the list could have an optional indication that the user *also* wants it to be a “search domain” to get the behavior that a query with no trailing dot will try that domain as a suffix. And the behavior of broadcasting queries in parallel to the non-VPN network should be configurable as well. As someone who has configured corporate and personal VPNs, I would have made use of these options, and my various VPNs would all be configured differently.
Right now we have a situation where the underlying system is quite configurable, but (in networking and elsewhere) GNOME likes to hide detailed configuration in gsettings or otherwise make it very hard to discover. For things like touchpad config, I respect GNOME’s goal of keeping it simple even if I disagree. For networking, I think that the genuinely simple cases (connect to WiFi, use that WiFI) should be approachable to non-technical users, but setting up something like a VPN is inherently complex, and trying to hide that complexity makes everything harder.
On 9/29/20 10:01 AM, Lennart Poettering wrote:
On Mo, 28.09.20 23:37, John M. Harris Jr (johnmh@splentity.com) wrote:
Configure "." as "routing domain" on a specific iface and the lookups wil go there preferably. If you put that on your VPN iface this means DNS traffic goes there preferably. If you put that ont he main iface this means DNS traffic goes there preferably.
Is that a NetworkManager setting or a systemd-resolved setting? Is that going to be exposed in the GUI, or is it something that gets hidden away?
I am not an NM guy, but I think they expose this these days. I can tell you definitely though that this is easily accessible via "resolvectl domain <iface>" from the command line and from .network networkd configuration files.
How does systemd-resolved figure out what domains "should" be sent to a given connection's DNS server without some arcane incantation from the systemd docs?
As mentioned elsewhere:
- Search domains are implicitly routing domains: if an interface has "redhat.com" as search domain we also use that as routing domain, i.e. all *.redhat.com lookups will go to this interface and not to others.
This is the same algorithm dnssec-trigger uses for this purpose. Because nothing better is there. dnssec-trigger ignores search domain in user configuration, using it exclusively to "route" domain requests to correct servers.
Anyway, there exist at least three implementations able to do split DNS scenario, systemd-resolved, dnsmasq and dnssec-trigger+unbound. For some reason, there is no common layer of configuring it inside Network Manager. I think routed domains should be configured there and any split DNS provider should get it from there. No VPN should put it to specific implementation, be it systemd or unbound.
- If neither search domains nor routing domains are configured on any interface for a domain, lookups are routed to all interfaces in parallel, and the first positive and last negative answer is used.
This is very tragic way of doing it for privacy. You just shout out every request you have to all parties without any priority. This is not how it was done without systemd-resolved. This is much worse. Please reconsider it. If no split view configuration known, all names should be handled to VPN. If more of them, use priority of NM connection or manual configuration. Allow local override.
As was said, search is done used for something else by default.
Please don't force me share my porn sites with both employer AND ISP. Just one of them is enough. Wait for timeout from the first one before trying next one.
i.e. focus is primarily on "let's make DNS work" and "let's make the best of the little information we traditionally have", and any further, more complex routing requires additional configuration in NM, networkd or directly with resolvectl commands.
Use any information provided by network administrators, but allow local overrides. But stay secure at the same time. Put complexity to network configuration, resolved should fetch it from there.
Lennart
-- Lennart Poettering, Berlin _______________________________________________ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Petr
On Mon, 2020-09-28 at 23:37 -0700, John M. Harris Jr wrote:
On Monday, September 28, 2020 12:42:32 PM MST Lennart Poettering wrote:
On Mo, 28.09.20 12:14, Paul Wouters (paul@nohats.ca) wrote:
On Mon, 28 Sep 2020, Michael Catanzaro wrote:
I don't think it would be smart for employees to voluntarily opt-in to sending all DNS to their employer anyway... there's little benefit to the employee, and a lot of downside.
Again, it is not up to systemd to limit valid use cases.
Perhaps Listen or read to Paul Vixie, father of many Bind software releases:
https://www.youtube.com/watch?v=ZxTdEEuyxHU
https://www.theregister.com/2018/10/23/paul_vixie_slaps_doh_as_dns_privacy _feature_becomes_a_standard/
There are use cases for and against routing all DNS over your VPN. If systemd wants to play system resolver, it needs to be able to be configured for either use case. You don't get to limit our use cases.
Configure "." as "routing domain" on a specific iface and the lookups wil go there preferably. If you put that on your VPN iface this means DNS traffic goes there preferably. If you put that ont he main iface this means DNS traffic goes there preferably.
Ideally you'd use more fine grained routing domains however.
Lennart
Lennart,
Is that a NetworkManager setting or a systemd-resolved setting? Is that going to be exposed in the GUI, or is it something that gets hidden away?
NM gets "routing domains" for a given connection (eg, network interface) from a couple different sources:
1) DHCP 2) SLAAC RDNSS 3) VPN 4) manually configured in the connection info, eg:
nmcli con mod rh-openvpn ipv4.dns-search "foobar.com"
It passes this information on to resolved or its own local caching DNS configuration which uses dnsmasq, which both use it for directing lookups for those domains to the DNS servers detected/configured for that interface.
Dan
How does systemd-resolved figure out what domains "should" be sent to a given connection's DNS server without some arcane incantation from the systemd docs?
-- John M. Harris, Jr.
devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
* Michael Catanzaro:
I don't think it would be smart for employees to voluntarily opt-in to sending all DNS to their employer anyway... there's little benefit to the employee, and a lot of downside. Importantly, if you're looking in your network settings and you see a checkbox that says "Use this connection only for resources on its network," a reasonable user *expects* that the connection will *really* only be used for resources on its network, not that it will be used for everything except DNS, which randomly goes to who knows where depending on what else you're connected to. Our design must try to avoid this failure case: "Sadly for Distrustful Denise, her employer discovers that she has been making some embarrassing DNS requests that she had expected to go through public-vpn.example.com instead."
Eh, for a corporate laptop (which is not physically connected to the corporate network due to present circumstances), I do expect that DNS is handled by the corporate DNS servers. I would also like to route all network traffic over the corporate VPN, but as I tried to explain, that is just not feasible at the moment. I also don't see how this is a trust issue for a *corporate* laptop used for work purposes.
Thanks, Florian
You can do this, but again, you need to use the command line. E.g. 'resolvectl dns tun0 8.8.8.8'
We're actually no longer debating how systemd-resolved works; rather, we're now debating how NetworkManager chooses to configure systemd-resolved. systemd-resolved just does what it's told to do. It's actually NetworkManager that decides to split DNS according to routing by default as a matter of policy. It could do otherwise if it wanted to, but I think this is a good default. Nothing stops you from changing it though. :)
On Monday, September 28, 2020 9:39:17 AM MST Michael Catanzaro wrote:
You can do this, but again, you need to use the command line. E.g. 'resolvectl dns tun0 8.8.8.8'
We're actually no longer debating how systemd-resolved works; rather, we're now debating how NetworkManager chooses to configure systemd-resolved. systemd-resolved just does what it's told to do. It's actually NetworkManager that decides to split DNS according to routing by default as a matter of policy. It could do otherwise if it wanted to, but I think this is a good default. Nothing stops you from changing it though. :)
Michael, By what mechanism does NetworkManager "split DNS according to routing"? If it hasn't already made a request from both your cleartext and your VPN connection's DNS servers, it has no way of knowing what network should be used to get the right results. Routing and DNS are unrelated.
On Mon, Sep 28, 2020 at 11:41:12PM -0700, John M. Harris Jr wrote:
On Monday, September 28, 2020 9:39:17 AM MST Michael Catanzaro wrote:
You can do this, but again, you need to use the command line. E.g. 'resolvectl dns tun0 8.8.8.8'
We're actually no longer debating how systemd-resolved works; rather, we're now debating how NetworkManager chooses to configure systemd-resolved. systemd-resolved just does what it's told to do. It's actually NetworkManager that decides to split DNS according to routing by default as a matter of policy. It could do otherwise if it wanted to, but I think this is a good default. Nothing stops you from changing it though. :)
Michael, By what mechanism does NetworkManager "split DNS according to routing"? If it hasn't already made a request from both your cleartext and your VPN connection's DNS servers, it has no way of knowing what network should be used to get the right results. Routing and DNS are unrelated.
NetworkManager pushes DNS server configuration (and associated bits like domain search and routing domains) over dbus to resolved. That way it "[tells resolved how to] split DNS according to routing". Of course, after the name has been resolved to an IP address, the packets to that IP address are routed too. So there is "routing" in the sense of deciding which interface is appropriate for a given DNS name and "routing" in the sense of deciding which interface is appropriate for a given IP address.
Zbyszek
Hi,
NetworkManager pushes DNS server configuration (and associated bits like domain search and routing domains) over dbus to resolved. That way it "[tells resolved how to] split DNS according to routing". Of course, after the name has been resolved to an IP address, the packets to that IP address are routed too. So there is "routing" in the sense of deciding which interface is appropriate for a given DNS name and "routing" in the sense of deciding which interface is appropriate for a given IP address.
This explain how this issue came in the first place. https://gnuguix-drive.mycozy.cloud/public?sharecode=YvERPGX14g5S
Cheers, Cuckoo's Calling.
On Tuesday, September 29, 2020 5:13:48 AM MST Zbigniew Jędrzejewski-Szmek wrote:
On Mon, Sep 28, 2020 at 11:41:12PM -0700, John M. Harris Jr wrote:
On Monday, September 28, 2020 9:39:17 AM MST Michael Catanzaro wrote:
You can do this, but again, you need to use the command line. E.g. 'resolvectl dns tun0 8.8.8.8'
We're actually no longer debating how systemd-resolved works; rather, we're now debating how NetworkManager chooses to configure systemd-resolved. systemd-resolved just does what it's told to do. It's
actually NetworkManager that decides to split DNS according to routing by default as a matter of policy. It could do otherwise if it wanted to, but I think this is a good default. Nothing stops you from changing
it though. :)
Michael, By what mechanism does NetworkManager "split DNS according to routing"? If it hasn't already made a request from both your cleartext and your VPN connection's DNS servers, it has no way of knowing what network should be used to get the right results. Routing and DNS are unrelated.
NetworkManager pushes DNS server configuration (and associated bits like domain search and routing domains) over dbus to resolved. That way it "[tells resolved how to] split DNS according to routing". Of course, after the name has been resolved to an IP address, the packets to that IP address are routed too. So there is "routing" in the sense of deciding which interface is appropriate for a given DNS name and "routing" in the sense of deciding which interface is appropriate for a given IP address.
It seems that the terminology is fairly confusing, considering it's right alongside actual routing configuration.. Okay, so "routing" means something wildly different than you'd think with systemd-resolved, got it.
In most cases, in order to get to a DNS server inside a VPN, your packets have to have a route which can reach the IP of that server for that interface, which is configured using NetworkManager (or a VPN config file, imported into NM). Anyone that understands basic networking will likely be confused by this terminology.
That aside, where in NetworkManager do these "routing domains" get specified?
On Tue, 2020-09-29 at 09:18 -0700, John M. Harris Jr wrote:
On Tuesday, September 29, 2020 5:13:48 AM MST Zbigniew Jędrzejewski- Szmek wrote:
On Mon, Sep 28, 2020 at 11:41:12PM -0700, John M. Harris Jr wrote:
On Monday, September 28, 2020 9:39:17 AM MST Michael Catanzaro wrote:
You can do this, but again, you need to use the command line. E.g. 'resolvectl dns tun0 8.8.8.8'
We're actually no longer debating how systemd-resolved works; rather, we're now debating how NetworkManager chooses to configure systemd-resolved. systemd-resolved just does what it's told to do. It's
actually NetworkManager that decides to split DNS according to routing by default as a matter of policy. It could do otherwise if it wanted to, but I think this is a good default. Nothing stops you from changing
it though. :)
Michael, By what mechanism does NetworkManager "split DNS according to routing"? If it hasn't already made a request from both your cleartext and your VPN connection's DNS servers, it has no way of knowing what network should be used to get the right results. Routing and DNS are unrelated.
NetworkManager pushes DNS server configuration (and associated bits like domain search and routing domains) over dbus to resolved. That way it "[tells resolved how to] split DNS according to routing". Of course, after the name has been resolved to an IP address, the packets to that IP address are routed too. So there is "routing" in the sense of deciding which interface is appropriate for a given DNS name and "routing" in the sense of deciding which interface is appropriate for a given IP address.
It seems that the terminology is fairly confusing, considering it's right alongside actual routing configuration.. Okay, so "routing" means something wildly different than you'd think with systemd-resolved, got it.
In most cases, in order to get to a DNS server inside a VPN, your packets have to have a route which can reach the IP of that server for that interface, which is configured using NetworkManager (or a VPN config file, imported into NM). Anyone that understands basic networking will likely be confused by this terminology.
That aside, where in NetworkManager do these "routing domains" get specified?
In the connection itself (GUI or CLI), or they come from DHCP or SLAAC or the VPN.
nmcli con mod rh-openvpn ipv4.dns-search "foobar.com" nmcli con mod rh-openvpn ipv4.never-default true
combined with having a local caching DNS server (or resolved) enabled will route queries for those search domains only to the VPN-provided DNS servers.
There are corresponding GUI boxes for these in nm-connection-editor, GNOME network settings, and KDE.
Dan
-- John M. Harris, Jr.
devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
On Tuesday, September 29, 2020 9:36:38 AM MST Dan Williams wrote:
On Tue, 2020-09-29 at 09:18 -0700, John M. Harris Jr wrote:
On Tuesday, September 29, 2020 5:13:48 AM MST Zbigniew Jędrzejewski- Szmek wrote:
On Mon, Sep 28, 2020 at 11:41:12PM -0700, John M. Harris Jr wrote:
On Monday, September 28, 2020 9:39:17 AM MST Michael Catanzaro wrote:
You can do this, but again, you need to use the command line. E.g. 'resolvectl dns tun0 8.8.8.8'
We're actually no longer debating how systemd-resolved works; rather, we're now debating how NetworkManager chooses to configure systemd-resolved. systemd-resolved just does what it's told to do. It's
actually NetworkManager that decides to split DNS according to routing by default as a matter of policy. It could do otherwise if it wanted to, but I think this is a good default. Nothing stops you from changing
it though. :)
Michael, By what mechanism does NetworkManager "split DNS according to routing"? If it hasn't already made a request from both your cleartext and your VPN connection's DNS servers, it has no way of knowing what network should be used to get the right results. Routing and DNS are unrelated.
NetworkManager pushes DNS server configuration (and associated bits like domain search and routing domains) over dbus to resolved. That way it "[tells resolved how to] split DNS according to routing". Of course, after the name has been resolved to an IP address, the packets to that IP address are routed too. So there is "routing" in the sense of deciding which interface is appropriate for a given DNS name and "routing" in the sense of deciding which interface is appropriate for a given IP address.
It seems that the terminology is fairly confusing, considering it's right alongside actual routing configuration.. Okay, so "routing" means something wildly different than you'd think with systemd-resolved, got it.
In most cases, in order to get to a DNS server inside a VPN, your packets have to have a route which can reach the IP of that server for that interface, which is configured using NetworkManager (or a VPN config file, imported into NM). Anyone that understands basic networking will likely be confused by this terminology.
That aside, where in NetworkManager do these "routing domains" get specified?
In the connection itself (GUI or CLI), or they come from DHCP or SLAAC or the VPN.
nmcli con mod rh-openvpn ipv4.dns-search "foobar.com" nmcli con mod rh-openvpn ipv4.never-default true
combined with having a local caching DNS server (or resolved) enabled will route queries for those search domains only to the VPN-provided DNS servers.
There are corresponding GUI boxes for these in nm-connection-editor, GNOME network settings, and KDE.
Dan,
This would require a list of search domains a mile long, and for the end user to know what needs to go over the VPN anyway. Additionally, this may well break scripts that expect a given short name complication, but end up getting it from a different domain, since they're all in search domains now.
On Mon, 2020-09-28 at 10:51 -0500, Michael Catanzaro wrote:
I don't think my description is misleading....
On Mon, Sep 28, 2020 at 5:28 pm, Florian Weimer fweimer@redhat.com wrote:
- The change disables protection mechanisms built into corporate VPNs that require them to observe all DNS traffic. Now this may sound rather weak as far as countermeasures go, but DNS-based mechanisms
are the only thing you have got if you do not enforce a client-side approach (ugh, no thanks), or disable split tunneling (i.e., default route over the VPN; frequently not possible with current VPN usage levels and virtual company meetings over video link).
Correct. If you want to observe all DNS traffic, that is no longer possible by default unless you also handle routing all traffic. I'd argue that's a good change. From a system design perspective, that's just how DNS ought to work.
No, this is wrong, DNS and traffic routing are absolutely disjoint hitngs, and you cannot assume that DNS ought to work as traffic routing, because it never did.
Traditionally you had a locally defined DNS server that would answer your queries, and for traffic you had routers that would decide where the traffic go, and there is absolutely no correlation between the two.
Actually from the DNS pov, traditionally, what is called "split DNS" is almost a capital sin. In fact the default *should* be to use a VPN DNS *unless* the user explicitly tells you it doesn't fully trust the VPN and wants to try to split out the traffic.
Multiple VPNs definitely pose a problem, so a proper interface may allow the user to define a "primary" VPN so that would be the one where you send the bulk of DNS traffic.
I don't think it would be smart for employees to voluntarily opt-in to sending all DNS to their employer anyway...
This statement really needs to be justified at length, in normal copropate environment the equipment is owned by the company and all DNS traffic *is* expected to go through the VPN so as not to leak your queries to the internet directly.
there's little benefit to the employee, and a lot of downside.
Sorry, but this is quite debatable, and entirely depends on the relationship between employer and employee among other things. Nothing you can assert or assume.
Importantly, if you're looking in your network settings and you see a checkbox that says "Use this connection only for resources on its network," a reasonable user *expects* that the connection will *really* only be used for resources on its network, not that it will be used for everything except DNS,
This is in fact a good indication of the intention a user have. I use in some context and not others.
which randomly goes to who knows where depending on what else you're connected to. Our design must try to avoid this failure case: "Sadly for Distrustful Denise, her employer discovers that she has been making some embarrassing DNS requests that she had expected to go through public-vpn.example.com instead."
This is where you should have a "This is my primary VPN" option, which means all DNS traffic goes there when it is active.
Of course, it's still possible to get the old behavior if you really want to, but it will now require custom configuration not available via GUI,
And this is a big problem, you are trying to enhance privacy in one direction, but failing to do so in the other. This should be considered a bug, given all the other reasons you gave for the change.
and nobody really wants to opt-in to that behavior, so it's not likely to be used except in managed corporate systems.
I do for my corporate laptop, so I think you have incorrectly assessed the situation. I would also want to opt-into sending everytihng on the VPN if it were a privacy-VPN, there MUST be a button that says "use *only* _this_ VPN for *all* DNS traffic" if you care about user's privacy.
If you're using a managed system, you're probably surveilled anyway, so better assume nothing is safe.
- There is no real protocol for sharing internal domains, so systemd-resolved cannot know all of them, and resolving some of them will fail or receive unexpected resolution results (probably observable for some jboss.org subdomains for Red Hatters, but I
don't work in that area, so I don't have a good example at hand).
Yes, that's true. And there's not currently any good solution to that without resorting to the command line.
And this is a bug.
my 2c, Simo.
On Mon, Sep 28, 2020 at 2:44 pm, Simo Sorce simo@redhat.com wrote:
No, this is wrong, DNS and traffic routing are absolutely disjoint hitngs, and you cannot assume that DNS ought to work as traffic routing, because it never did.
Hi Simo,
Apologies for a long reply, but I wanted to try to address at least most of your concerns.
I would say that rather than *assuming* how things work, we're *defining* it. We have to pick some default behavior, and I think splitting DNS is least-likely to subvert user expectations, therefore least-likely to lead to privacy mistakes.
Now, we only have one GUI option here: "use this connection only for resources on its network." If we want to offer the user the choice of whether to split DNS along with routing, well, then we need more than one option, and we're really getting into the weeds and probably exceeding a reasonable complexity limit for an already-complex network configuration dialog. This is a niche use-case IMO -- with the possible exception of managed corporate deployments where our defaults do not matter, more on that below -- and it's much easier to handle this case by letting the user manually specify the DNS server to use for the connection, which will now actually work in F33 with systemd-resolved. Try this in F32 and all the servers you've configured from various network connections all get jumbled together in /etc/resolv.conf, in uncertain order, with only the one that's lucky enough to be first actually used. Behavior changes based on the order that you connect or disconnect your VPNs and DNS winds up going to unexpected places. But in F33, it just works, so you can easily configure your DNS exactly as you please. So if you want your VPN to get all your DNS but only traffic for its own network, you can still do it.
Traditionally you had a locally defined DNS server that would answer your queries, and for traffic you had routers that would decide where the traffic go, and there is absolutely no correlation between the two.
Actually from the DNS pov, traditionally, what is called "split DNS" is almost a capital sin. In fact the default *should* be to use a VPN DNS *unless* the user explicitly tells you it doesn't fully trust the VPN and wants to try to split out the traffic.
That's still the case. All this discussion about split DNS is only relevant to the case where the user checks the box "use this connection only for resources on its network" (or imports a VPN profile that selects that automatically). By default, it's not checked, nothing is split, all DNS and routing goes to the VPN.
Multiple VPNs definitely pose a problem, so a proper interface may allow the user to define a "primary" VPN so that would be the one where you send the bulk of DNS traffic.
Yeah, I suspect we have room for improvement there. But regardless, you'll get a sensible result as long as you have no more than one VPN for which "use this connection only for resources on its network" is not checked. As long as there's only one with that box unchecked, it's your primary VPN, and the rest are secondary.
I don't think it would be smart for employees to voluntarily opt-in to sending all DNS to their employer anyway...
This statement really needs to be justified at length, in normal copropate environment the equipment is owned by the company and all DNS traffic *is* expected to go through the VPN so as not to leak your queries to the internet directly.
In a normal corporate environment, your machines are managed by the corporation and may (or may not) be subject to Big Brother surveillance, possibly to a much higher degree than just watching your DNS. However closely you're being watched, one thing is certain: the corporation will configure your DNS for you, however it pleases. Our defaults just don't matter, because the corporation will change them. It will decide where your DNS goes and where your traffic goes. If it wants to get all your DNS but not all your traffic, the configuration with systemd-resolved is a little different than it was traditionally, but it's not hard to do.
The case where our defaults really matter is for unmanaged systems, where the corporation is not configuring your defaults, the computer is owned by the end user, and the end user is configuring the VPN but is not an expert in DNS or routing or any network-related topics. I'm firmly convinced that almost nobody in this case would intentionally choose for the corporation to receive DNS but not corresponding traffic, because that can only benefit the corporation. There's really very few conceivable situations where that benefits the user. I can think of only one: you want to resolve an internal domain using the corporate DNS server, but don't have a search domain configured for it. (In that case, the solution is to manually add a search domain.) The much more common case is the corporation finds something it doesn't like in employee's DNS (for a G-rated example, let's say too much time on Facebook) and employee gets in trouble for that.
Please note that we still default to sending *everything* to the corporation, all DNS and all traffic. Only if you check the "use this connection only for resources on its network" box do we then go into split mode, on the assumption that split DNS is likely what the user wants. So the change here is that if you ask for the split, now you actually get a proper split with no DNS leaks. If you don't ask for the split, of course there's not going to be a split.
(In F32, the user would *think* everything is split if selecting this option, but DNS would not be split -- only traffic -- so the corporation would still see your embarrassing DNS queries. Or, depending on connection order, the opposite might occur, the corporation might receive traffic but no DNS at all, breaking internal name resolution.)
I hope that addresses most of your next points, so I'll skip down a few:
Of course, it's still possible to get the old behavior if you really want to, but it will now require custom configuration not available via GUI,
And this is a big problem, you are trying to enhance privacy in one direction, but failing to do so in the other. This should be considered a bug, given all the other reasons you gave for the change.
I'm confused here. My goal is to avoid subverting user expectations by defaulting to sending DNS wherever the traffic goes if the user has configured a split. The biggest risk to privacy is if we fail to adhere to user expectations, right?
I do for my corporate laptop, so I think you have incorrectly assessed the situation. I would also want to opt-into sending everytihng on the VPN if it were a privacy-VPN, there MUST be a button that says "use *only* _this_ VPN for *all* DNS traffic" if you care about user's privacy.
So focusing only on the case where you have configured a split (because if there's no split, none of this is relevant): we don't have an easy button to let you use one VPN for DNS corresponding to traffic that it doesn't receive. Of course, it's possible to configure that by manually setting the other VPNs to use custom DNS, but it's a very strange case. If you're not routing all traffic through the VPN, then when is it important for privacy that the DNS also goes to that VPN? That's probably not what you really want? Normally, if you are using a VPN for privacy, you want to ensure all DNS goes through it, *except* DNS for internal resources associated with your non-primary VPNs.
If you're using a managed system, you're probably surveilled anyway, so better assume nothing is safe.
- There is no real protocol for sharing internal domains, so systemd-resolved cannot know all of them, and resolving some of
them
will fail or receive unexpected resolution results (probably observable for some jboss.org subdomains for Red Hatters, but I don't work in that area, so I don't have a good example at hand).
Yes, that's true. And there's not currently any good solution to that without resorting to the command line.
And this is a bug.
Yeah, this is a fair complaint.
Of course, this problem is avoidable by unchecking "use this connection only for resources on its network" if you use only one VPN. And failing that: at least the situation is not worse than it was before.
Michael
On Mon, Sep 28, 2020 at 03:51:51PM -0500, Michael Catanzaro wrote:
That's still the case. All this discussion about split DNS is only relevant to the case where the user checks the box "use this connection only for resources on its network" (or imports a VPN profile that selects that automatically). By default, it's not checked, nothing is split, all DNS and routing goes to the VPN.
I think the VPN plugin and VPN server has some input, no? All the VPN servers I've used send routes to the VPN client to determine which traffic the client should send via the VPN. How does that interact with "use this connection only for resources on its network"? Does the user preference take precendence over the VPN server-provided routes? What if the VPN server doesn't send any route other than 0.0.0.0/0?
On Mon, Sep 28, 2020 at 5:18 pm, Chuck Anderson cra@alum.wpi.edu wrote:
I think the VPN plugin and VPN server has some input, no? All the VPN servers I've used send routes to the VPN client to determine which traffic the client should send via the VPN. How does that interact with "use this connection only for resources on its network"? Does the user preference take precendence over the VPN server-provided routes? What if the VPN server doesn't send any route other than 0.0.0.0/0?
Good question! So good that I don't know the answer. Yes, the VPN plugin indeed gets to make decision based on configuration pushed by the VPN server. The NetworkManager developers are experts in how these settings interact. I *think* the routes provided by the VPN take precedence over the checkbox (but only for routing, not for DNS)? But this would certainly be good to document and explore more fully.
This is actually at issue in https://bugzilla.redhat.com/show_bug.cgi?id=1863041 where we currently wind up doing the wrong thing by default. See e.g. comment #81 where the VPN plugin is constructing routing information to pass to NetworkManager.
On Mon, 2020-09-28 at 16:40 -0500, Michael Catanzaro wrote:
On Mon, Sep 28, 2020 at 5:18 pm, Chuck Anderson cra@alum.wpi.edu wrote:
I think the VPN plugin and VPN server has some input, no? All the VPN servers I've used send routes to the VPN client to determine which traffic the client should send via the VPN. How does that interact with "use this connection only for resources on its network"? Does the user preference take precendence over the VPN server-provided routes? What if the VPN server doesn't send any route other than 0.0.0.0/0?
Good question! So good that I don't know the answer. Yes, the VPN plugin indeed gets to make decision based on configuration pushed by the VPN server. The NetworkManager developers are experts in how these settings interact. I *think* the routes provided by the VPN take precedence over the checkbox (but only for routing, not for DNS)? But this would certainly be good to document and explore more fully.
If you check "Use this connection..." then NetworkManager will:
(a) never set the default route through the VPN (b) enable split DNS using the VPN-provided (or manually configured) DNS search domains
If you do not check that box, then the VPN will become the default route and all your non-local traffic will be sent to it.
Unfortunately you cannot rely on VPNs to "do the right thing" and always pass back 0.0.0.0 when it wants all the traffic. Plus the user may not want to pass all traffic to the VPN, regardless of what the VPN wants. If you have a corporate laptop and the company wants all your data to go through the VPN, then that laptop is presumably well-managed and the IT admin will enforce that "Use this connection..." is unchecked.
Dan
This is actually at issue in https://bugzilla.redhat.com/show_bug.cgi?id=1863041 where we currently wind up doing the wrong thing by default. See e.g. comment #81 where the VPN plugin is constructing routing information to pass to NetworkManager.
devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
On Tue, 2020-09-29 at 11:20 -0500, Dan Williams wrote:
On Mon, 2020-09-28 at 16:40 -0500, Michael Catanzaro wrote:
On Mon, Sep 28, 2020 at 5:18 pm, Chuck Anderson cra@alum.wpi.edu wrote:
I think the VPN plugin and VPN server has some input, no? All the VPN servers I've used send routes to the VPN client to determine which traffic the client should send via the VPN. How does that interact with "use this connection only for resources on its network"? Does the user preference take precendence over the VPN server-provided routes? What if the VPN server doesn't send any route other than 0.0.0.0/0?
Good question! So good that I don't know the answer. Yes, the VPN plugin indeed gets to make decision based on configuration pushed by the VPN server. The NetworkManager developers are experts in how these settings interact. I *think* the routes provided by the VPN take precedence over the checkbox (but only for routing, not for DNS)? But this would certainly be good to document and explore more fully.
If you check "Use this connection..." then NetworkManager will:
(a) never set the default route through the VPN (b) enable split DNS using the VPN-provided (or manually configured) DNS search domains
If you do not check that box, then the VPN will become the default route and all your non-local traffic will be sent to it.
Unfortunately you cannot rely on VPNs to "do the right thing" and always pass back 0.0.0.0 when it wants all the traffic. Plus the user may not want to pass all traffic to the VPN, regardless of what the VPN wants. If you have a corporate laptop and the company wants all your data to go through the VPN, then that laptop is presumably well-managed and the IT admin will enforce that "Use this connection..." is unchecked.
Dan, I think that unfortunately we cannot conflate "Use this connection..." to both decide on packet routing and well as DNS routing.
There are definitely VPNs where routing allows only to reach internal networks and does not allow passthrough, and at the same time VPN expects that all DNS resolution happens through the VPN DNS server as they selectively override names so some traffic is routed over VPN when connected but over the regular internet when not (via DNS views).
I am not saying this is good or bad, it just is, and if we conflate this setting we cannot express that setup, which is common (for example this is the recommended/required configuration for our RH VPN IIRC).
I am also *not* saying we should have two flags that read the same but just add "for DNS" in one and "for packets" in the other, as most users would probably be confused.
In general I would say that for the common case the default should be to send queries to the VPN even if there is packet routing split, especially if we are thinking about the "café case" I would definitely trust more the DNS server via the VPN than the one spoofed by the café broken wifi.
Maybe the way to do this is to provide a different switch that say something like "I trust this connection to protect my privacy". Then if you do not want to trust the DNS provided by the VPN for everything, you can toggle that one off (default would be on), and that will cause split DNS as well based on configured domains.
WDYT ?
Simo.
On Tue, 2020-09-29 at 15:14 -0400, Simo Sorce wrote:
On Tue, 2020-09-29 at 11:20 -0500, Dan Williams wrote:
On Mon, 2020-09-28 at 16:40 -0500, Michael Catanzaro wrote:
On Mon, Sep 28, 2020 at 5:18 pm, Chuck Anderson <cra@alum.wpi.edu
wrote:
I think the VPN plugin and VPN server has some input, no? All the VPN servers I've used send routes to the VPN client to determine which traffic the client should send via the VPN. How does that interact with "use this connection only for resources on its network"? Does the user preference take precendence over the VPN server- provided routes? What if the VPN server doesn't send any route other than 0.0.0.0/0?
Good question! So good that I don't know the answer. Yes, the VPN plugin indeed gets to make decision based on configuration pushed by the VPN server. The NetworkManager developers are experts in how these settings interact. I *think* the routes provided by the VPN take precedence over the checkbox (but only for routing, not for DNS)? But this would certainly be good to document and explore more fully.
If you check "Use this connection..." then NetworkManager will:
(a) never set the default route through the VPN (b) enable split DNS using the VPN-provided (or manually configured) DNS search domains
If you do not check that box, then the VPN will become the default route and all your non-local traffic will be sent to it.
Unfortunately you cannot rely on VPNs to "do the right thing" and always pass back 0.0.0.0 when it wants all the traffic. Plus the user may not want to pass all traffic to the VPN, regardless of what the VPN wants. If you have a corporate laptop and the company wants all your data to go through the VPN, then that laptop is presumably well- managed and the IT admin will enforce that "Use this connection..." is unchecked.
Dan, I think that unfortunately we cannot conflate "Use this connection..." to both decide on packet routing and well as DNS routing.
I'm not disagreeing that in a perfect world we'd actually differentiate two different things.
But it's been this way for 12+ years with NM and (even though I've been out of NetworkManager for a long time) I've never seen this much discussion about the topic. Clearly something has worked for the majority of users for a long, long time.
But perhaps it's time to evaluate improvements.
There are definitely VPNs where routing allows only to reach internal networks and does not allow passthrough, and at the same time VPN expects that all DNS resolution happens through the VPN DNS server as they selectively override names so some traffic is routed over VPN when connected but over the regular internet when not (via DNS views).
I am not saying this is good or bad, it just is, and if we conflate this setting we cannot express that setup, which is common (for example this is the recommended/required configuration for our RH VPN IIRC).
Sure, those setups are not possible with the binary option we have to day with NM.
I am also *not* saying we should have two flags that read the same but just add "for DNS" in one and "for packets" in the other, as most users would probably be confused.
In general I would say that for the common case the default should be to send queries to the VPN even if there is packet routing split, especially if we are thinking about the "café case" I would definitely trust more the DNS server via the VPN than the one spoofed by the café broken wifi.
That is the current default, if you leave "Use this connection..." unchecked. Of course that also sends your traffic to the VPN, which may not actually work depending on your VPN config.
Maybe the way to do this is to provide a different switch that say something like "I trust this connection to protect my privacy". Then if you do not want to trust the DNS provided by the VPN for everything, you can toggle that one off (default would be on), and that will cause split DNS as well based on configured domains.
Well... I trust this connection to protect my privacy in some ways, but not others. I don't think it's that clear-cut.
I honestly don't expect most users to understand or care about the difference bewteen split IP routing and split DNS routing. But for those that do, perhaps there should be additional configuration possible. That's an RFE for the NM team.
Dan
WDYT ?
Simo.
-- Simo Sorce RHEL Crypto Team Red Hat, Inc
devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
* Michael Catanzaro:
Of course, this problem is avoidable by unchecking "use this connection only for resources on its network" if you use only one VPN. And failing that: at least the situation is not worse than it was before.
Have you actually tried this with a corporate VPN recently?
It requires that the VPN is actually able to route general Internet traffic for you. At present, I don't think this is a reasonable assumption.
I think the workaround you are suggesting is not available to most users.
Thanks, Florian
On Mon, 28 Sep 2020, Michael Catanzaro wrote:
If you're running mail servers or VPN servers, you can probably configure the DNS to your liking, right? Either enable DNSSEC support in systemd-resolved, or disable systemd-resolved. I'm not too concerned about this....
You should be concerned about this.
The bar should never be "breaking the system is fine for advanced users"
The signal you are sending me (and others) right now is the wrong signal. systemd should works for me, the user.
Paul
* Michael Catanzaro:
If you're running mail servers or VPN servers, you can probably configure the DNS to your liking, right? Either enable DNSSEC support in systemd-resolved, or disable systemd-resolved. I'm not too concerned about this....
What about end users who just enable a VPN client?
My understanding is that the DNS request routing in systemd-resolved effectively disables any security mechanisms on the VPN side, and instructs most current browsers to route DNS requests to centralized DNS servers for all requests (i.e., overriding what came from both the VPN and DHCP).
Thanks, Florian
On Mon, Sep 28, 2020 at 4:39 pm, Florian Weimer fweimer@redhat.com wrote:
My understanding is that the DNS request routing in systemd-resolved effectively disables any security mechanisms on the VPN side, and instructs most current browsers to route DNS requests to centralized DNS servers for all requests (i.e., overriding what came from both the VPN and DHCP).
No... certainly not. Previously, VPNs only worked properly if you have exactly one VPN, and it's configured to receive all traffic. Using a VPN that receives traffic only for resources on its network, or using multiple VPNs at once, would result in DNS leaks. In fact, making VPNs work properly is the *only* reason I'm involved in this. I was frustrated to see that Fedora sometimes sent my requests for internal Red Hat resources to my public VPN's DNS server instead of Red Hat's DNS servers. See [1] for a comparison between previous and new behavior.
Now, we do currently have the one bug where NetworkManager doesn't configure systemd-resolved properly [2], but we only know of one affected user, and that's going to be fixed. Your VPNs will probably work properly in F33 with no configuration changes.
[1] https://fedoraproject.org/wiki/Changes/systemd-resolved#Split_DNS [2] https://bugzilla.redhat.com/show_bug.cgi?id=1863041
* Michael Catanzaro:
On Mon, Sep 28, 2020 at 4:39 pm, Florian Weimer fweimer@redhat.com wrote:
My understanding is that the DNS request routing in systemd-resolved effectively disables any security mechanisms on the VPN side, and instructs most current browsers to route DNS requests to centralized DNS servers for all requests (i.e., overriding what came from both the VPN and DHCP).
No... certainly not. Previously, VPNs only worked properly if you have exactly one VPN, and it's configured to receive all traffic. Using a VPN that receives traffic only for resources on its network, or using multiple VPNs at once, would result in DNS leaks. In fact, making VPNs work properly is the *only* reason I'm involved in this. I was frustrated to see that Fedora sometimes sent my requests for internal Red Hat resources to my public VPN's DNS server instead of Red Hat's DNS servers. See [1] for a comparison between previous and new behavior.
But the DNS view provided by the Red Hat VPN is what disables the centralized DNS resolvers in browsers in these configurations. The magic browser probe no longer fails with the change in DNS routing (which the proposal confusingly names “Split DNS”) because it goes out over the public Internet, where it is not filtered, unlike the Red Hat VPN.
Thanks, Florian
On Mon, Sep 28, 2020 at 5:18 pm, Florian Weimer fweimer@redhat.com wrote:
But the DNS view provided by the Red Hat VPN is what disables the centralized DNS resolvers in browsers in these configurations. The magic browser probe no longer fails with the change in DNS routing (which the proposal confusingly names “Split DNS”) because it goes out over the public Internet, where it is not filtered, unlike the Red Hat VPN.
Hm, I'm pretty sure this is a Firefox-specific issue, right? Fedora's Firefox is patched to use system DNS, so it shouldn't matter for us. I'm not aware of any other browser that ignores system DNS; at least, I'm fairly certain Chrome and Epiphany will both never do this.
Applications will always be able to ignore system DNS should they choose to do so....
* Michael Catanzaro:
On Mon, Sep 28, 2020 at 5:18 pm, Florian Weimer fweimer@redhat.com wrote:
But the DNS view provided by the Red Hat VPN is what disables the centralized DNS resolvers in browsers in these configurations. The magic browser probe no longer fails with the change in DNS routing (which the proposal confusingly names “Split DNS”) because it goes out over the public Internet, where it is not filtered, unlike the Red Hat VPN.
Hm, I'm pretty sure this is a Firefox-specific issue, right? Fedora's Firefox is patched to use system DNS, so it shouldn't matter for us. I'm not aware of any other browser that ignores system DNS; at least, I'm fairly certain Chrome and Epiphany will both never do this.
It seems that you are right about Chromium:
| We have no plans to support this approach. We believe that our | deployment model is significantly different from Mozilla's, and as a | result canary domains won't be needed.
https://www.chromium.org/developers/dns-over-https
However, you wrote earlier that “split DNS” is not available over nss_dns, so I think Chromium is still impacted because it uses the same interfaces that nss_dns would use in this mode (i.e., not nss_resolve).
Thanks, Florian
nss-dns is allright. All you need to have is dns server with domain configurable servers.
Those are: - unbound (with dnssec-trigger autoconfigured) - dnsmasq - systemd-resolved - probably knot-resolver - bind (not more difficult to reconfigure runtime)
Maybe more. It is not about nss, because /etc/resolv.conf does not support any domain:server-ip tuples. It would work better with local cache. resolved is not the only possibility. Just use /etc/resolv.conf set to localhost and configure forwarders in your server from NM (or networkd).
On 9/28/20 5:43 PM, Florian Weimer wrote:
- Michael Catanzaro:
On Mon, Sep 28, 2020 at 5:18 pm, Florian Weimer fweimer@redhat.com wrote:
But the DNS view provided by the Red Hat VPN is what disables the centralized DNS resolvers in browsers in these configurations. The magic browser probe no longer fails with the change in DNS routing (which the proposal confusingly names “Split DNS”) because it goes out over the public Internet, where it is not filtered, unlike the Red Hat VPN.
Hm, I'm pretty sure this is a Firefox-specific issue, right? Fedora's Firefox is patched to use system DNS, so it shouldn't matter for us. I'm not aware of any other browser that ignores system DNS; at least, I'm fairly certain Chrome and Epiphany will both never do this.
It seems that you are right about Chromium:
| We have no plans to support this approach. We believe that our | deployment model is significantly different from Mozilla's, and as a | result canary domains won't be needed.
https://www.chromium.org/developers/dns-over-https
However, you wrote earlier that “split DNS” is not available over nss_dns, so I think Chromium is still impacted because it uses the same interfaces that nss_dns would use in this mode (i.e., not nss_resolve).
Thanks, Florian
On Tue, Sep 29, 2020 at 4:28 pm, Petr Menšík pemensik@redhat.com wrote:
nss-dns is allright. All you need to have is dns server with domain configurable servers.
Those are:
- unbound (with dnssec-trigger autoconfigured)
- dnsmasq
- systemd-resolved
- probably knot-resolver
- bind (not more difficult to reconfigure runtime)
Maybe more. It is not about nss, because /etc/resolv.conf does not support any domain:server-ip tuples. It would work better with local cache. resolved is not the only possibility. Just use /etc/resolv.conf set to localhost and confi
Great, that will work wonderfully for those of us who run our own DNS server and configure it to split DNS as we prefer, and who never use VPNs, and who own zero laptops. For the rest of the world, nss-dns is not alright.
On 9/29/20 10:05 PM, Michael Catanzaro wrote:
On Tue, Sep 29, 2020 at 4:28 pm, Petr Menšík pemensik@redhat.com wrote:
nss-dns is allright. All you need to have is dns server with domain configurable servers.
Those are:
- unbound (with dnssec-trigger autoconfigured)
- dnsmasq
- systemd-resolved
- probably knot-resolver
- bind (not more difficult to reconfigure runtime)
Maybe more. It is not about nss, because /etc/resolv.conf does not support any domain:server-ip tuples. It would work better with local cache. resolved is not the only possibility. Just use /etc/resolv.conf set to localhost and confi
Great, that will work wonderfully for those of us who run our own DNS server and configure it to split DNS as we prefer, and who never use VPNs, and who own zero laptops. For the rest of the world, nss-dns is not alright.
Isn't the whole issue just to have that server configured correctly? Just omit manual configuration. VPNs are not solved only by resolved. dnssec-trigger solves it the same way. It needs only integration with NM.
systemd-resolved is also just dns server with few more options. Bundled into single package with more features, that might have been separate. I own a laptop, connect VPN everyday and it works just fine. Did you know dnsmasq can be configured in very similar way?
I think systemd-resolved mixed too many bits together.
--
Petr Menšík Software Engineer Red Hat, http://www.redhat.com/ email: pemensik@redhat.com PGP: DFCF908DB7C87E8E529925BC4931CA5B6C9FC5CB
On Mon, Sep 28, 2020 at 10:34:07AM -0500, Michael Catanzaro wrote:
Hm, I'm pretty sure this is a Firefox-specific issue, right? Fedora's Firefox is patched to use system DNS, so it shouldn't matter for us. I'm not aware of any other browser that ignores
Is this actually the case? I can't find such a patch in https://src.fedoraproject.org/rpms/firefox/tree/master but maybe I'm just not looking in the right place.
On Mon, Sep 28, 2020 at 12:04:27PM -0400, Matthew Miller wrote:
Hm, I'm pretty sure this is a Firefox-specific issue, right? Fedora's Firefox is patched to use system DNS, so it shouldn't matter for us. I'm not aware of any other browser that ignores
Is this actually the case? I can't find such a patch in https://src.fedoraproject.org/rpms/firefox/tree/master but maybe I'm just not looking in the right place.
Ah, got it:
$ grep trr * firefox-redhat-default-prefs.js:pref("network.trr.mode", 5);
On Mon, Sep 28, 2020 at 10:05:09AM -0500, Michael Catanzaro wrote:
[1] https://fedoraproject.org/wiki/Changes/systemd-resolved#Split_DNS
This link second time… there's a lot of text, but no example of configuration file for split dns. Is it because end user cannot easily configure split dns permanently?
On Mon, Sep 28, 2020 at 6:56 pm, Tomasz Torcz tomek@pipebreaker.pl wrote:
This link second time… there's a lot of text, but no example of configuration file for split dns. Is it because end user cannot easily configure split dns permanently?
You can configure custom DNS servers per-network in NetworkManager settings. This will actually work as expected now, unlike in F32. There's no single file to configure this, since it's a per-connection setting and NetworkManager stores configuration in separate files for each connection.
If you don't want split DNS, just don't check "use this connection only for resources on its network" when creating the VPN. (By default, VPNs still get all traffic, so split DNS is actually only used if you check that box, or configure custom DNS servers.)
<snip>
This entire discussion is generating enough emails per hour to be an IRC discussion. Could we please move this discussion to #fedora-devel or someplace more appropriate?
On Mon, Sep 28, 2020 at 10:38:27AM -0700, Erich Eickmeyer wrote:
<snip>
This entire discussion is generating enough emails per hour to be an IRC discussion. Could we please move this discussion to #fedora-devel or someplace more appropriate?
Well, not everyone is on IRC, and email is sometimes a better medium for longer discussions.
Sometimes this list just has a really high volume of emails. ;(
kevin
On Tue, Sep 29, 2020 at 07:27:37AM -0700, Kevin Fenzi wrote:
On Mon, Sep 28, 2020 at 10:38:27AM -0700, Erich Eickmeyer wrote:
<snip>
This entire discussion is generating enough emails per hour to be an IRC discussion. Could we please move this discussion to #fedora-devel or someplace more appropriate?
Well, not everyone is on IRC, and email is sometimes a better medium for longer discussions.
Sometimes this list just has a really high volume of emails. ;(
Yeah. This is a complicated subject. IRC is not good for multi paragraph questions and answers and clarifications.
Zbyszek
On Mo, 28.09.20 16:39, Florian Weimer (fweimer@redhat.com) wrote:
- Michael Catanzaro:
If you're running mail servers or VPN servers, you can probably configure the DNS to your liking, right? Either enable DNSSEC support in systemd-resolved, or disable systemd-resolved. I'm not too concerned about this....
What about end users who just enable a VPN client?
My understanding is that the DNS request routing in systemd-resolved effectively disables any security mechanisms on the VPN side, and instructs most current browsers to route DNS requests to centralized DNS servers for all requests (i.e., overriding what came from both the VPN and DHCP).
That's not precisely true. resolved maintains DNS server info per-interface, i.e. your vpn will have one set of servers attached to them, and your main interface another. We then try to route lookups to these servers following some logic that tries to make the best of what is known. Specifically, you can configure "routing" domains for each iface, which will bind traffic within some domain onto such interface. If none is configured then this is implicitly populated by the search domains configured along with the DNS server info, if that exists. If for some lookup no such routing domain is known then we'll send traffic to the DNS servers of all interfaces in parallel, using the first positive/last negative reply.
This emphasizes that DNS lookups should just work, and provides — unlike nss-dns/resolv.conf — a way how in VPN setups you can route your lookups explicitly to avoid they leak to the wrong networks.
You can also specify "." as routing domain on some iface btw, which has the affect of routing all traffic preferable to that iface taking it away from all others (except those which also have routing domains configured for the relavant domains).
So, yes, you have tight control where things go, and can configure this per domain. For example you can tell resolved to route redhat.com queries to the RH VPN iface, and everything else to internet.
Previously, in the status pre-resolved all you could do is all-or-nothing. Either everything goes to VPN or all goes to main iface. (You can get this behaviour by resolved too, via the "." routing domain if you like).
But it's a bit unfair to claim things where a step back while they are actually a step forward, since we have the routing infra now.
I have the suspicion the main issue you are having is that we default to "all in parallel" if in doubt about lookups, while you want "vpn always wins" if in doubt about lookups. I am think our approach is more robust which is why we took it.
Lennart
-- Lennart Poettering, Berlin
On Sunday, September 27, 2020 9:44:13 PM MST Paul Wouters wrote:
Subject: Re: Fedora 33 System-Wide Change proposal: systemd-resolved
I was just hit by the first bug in systemd-resolved 4 days after I upgraded to fedora33. I will file a bug report for that, but I wanted to discuss something more fundamental.
systemd-resolved has a number of architectural flaws. When these were pointed out, bugs are not accepted and closed or ignored. Worse, I was told that systemd-resolved would not become the system DNS resolver, so I could just choose to not use it.
Unfortunately, with my upgrade to fedora 33 I was unwittingly upgraded to systemd-resolved. I want to remove it from my system, but I cannot because it is not even a sub-package of systemd, it is part of the core systemd package. This goes against promises made in the past.
Not only that, this change apparently "obsoletes" /etc/resolv.conf, which is just not acceptable.
It is my opinion as a long time DNS RFC author and package maintainer that systemd-resolved is not a suitable DNS resolver. Downgrading DNSSEC allowing local DNS to bypass DNSSEC is bad enough, but I read on:
https://fedoraproject.org/wiki/Changes/systemd-resolved
that DNSSEC is now completely disabled. Again, this is completely unacceptable. DNSSEC is part of the core of DNS protocols. My fedora mail server uses DNSSEC based TLSA records to prevent MITM attacks on the STARTTLS layer, which is now completely broken. My IPsec VPN server uses dnssec validation using the specified nameserves in /etc/resolve.conf that now point to systemd-resolvd that does not return DNSSEC records and is completely broken:
paul@thinkpad:~$ dig +dnssec vpn.nohats.ca @127.0.0.53
; <<>> DiG 9.11.22-RedHat-9.11.22-1.fc33 <<>> +dnssec vpn.nohats.ca @127.0.0.53
;; global options: +cmd
;; Got answer: ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 51669 ;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1
;; OPT PSEUDOSECTION: ; EDNS: version: 0, flags: do; udp: 65494 ; OPT=5: 05 07 08 0a 0d 0e 0f (".......") ; OPT=6: 01 02 04 ("...") ; OPT=7: 01 (".") ;; QUESTION SECTION: ;vpn.nohats.ca. IN A
;; ANSWER SECTION: vpn.nohats.ca. 10 IN A 193.110.157.148
;; Query time: 143 msec ;; SERVER: 127.0.0.53#53(127.0.0.53) ;; WHEN: Mon Sep 28 00:18:32 EDT 2020 ;; MSG SIZE rcvd: 81
libreswan will see this result as an attack, and fail to resolve DNS names in its configuration. My postfix daemon will hold on to mail because it cannot get a DNSSEC proof of denial of existence of TLSA records if all DNSSEC records are filtered - even for domains that don't use DNSSEC because the denial of existence of DNSSEC for a specific domain requires the return of DNSSEC records that systemd now does not return.
I am sorry that I did not follow the fedora list carefully enough to notice this feature when it was proposed.
This change is harmful to network security, impacts existing installations depending on DNSSEC security, and leaks private queries for VPN/internal domains to the open internet, and prefers faster non-dnssec answers over dnssec validated answers. It fails various types of queries, misimplements part of the DNS protocol. Not only according to me, but to 20years+ developers of the bind software as well.
The first mandatory step is to not disable DNSSEC. If I put on my security hat, I would actually have to look into filing a CVE for Fedora 33.
A further discussion should happen on the future of systemd-resolved as the default Fedora (and later RHEL/CentOS) system resolver.
Paul
Paul,
This is only a handful of issues with it, and there are many. I tried to bring up that replacing /etc/resolv.conf would cause needless trouble, and that looking for the comment NetworkManager puts in it wasn't sufficient, but my messages were ignored.
Not only will this needlessly break existing configurations, but it will leak all of your private queries to Google if you haven't specified a DNS server. They hard-coded 8.8.8.8 and 8.8.4.4 into their resolver, and that still isn't patched out for Fedora.
On 9/28/20 6:44 AM, Paul Wouters wrote:
Subject: Re: Fedora 33 System-Wide Change proposal: systemd-resolved
I was just hit by the first bug in systemd-resolved 4 days after I upgraded to fedora33. I will file a bug report for that, but I wanted to discuss something more fundamental.
systemd-resolved has a number of architectural flaws. When these were pointed out, bugs are not accepted and closed or ignored. Worse, I was told that systemd-resolved would not become the system DNS resolver, so I could just choose to not use it.
Unfortunately, with my upgrade to fedora 33 I was unwittingly upgraded to systemd-resolved. I want to remove it from my system, but I cannot because it is not even a sub-package of systemd, it is part of the core systemd package. This goes against promises made in the past.
First time? It's standard operating procedure for systemd from the inception of that project.
Den mån 31 aug. 2020 kl 22:40 skrev Michael Catanzaro <mcatanzaro@gnome.org
:
On Mon, Aug 31, 2020 at 10:19 pm, Andreas Tunek andreas.tunek@gmail.com wrote:
I can't get that command to do anything useful in either F32 or F33.
You should see something like:
$ resolvectl query example.com example.com: 2606:2800:220:1:248:1893:25c8:1946 -- link: tun0 93.184.216.34 -- link: tun0
-- Information acquired via protocol DNS in 86.4ms. -- Data is authenticated: no
And that should be working out-of-the-box in F33.
However, if I write "http://nas-name.local in F32 in either Firefox or Gnome Web I can connect to my NAS, but if I write the same in F33 in the same programs I get an error.
Ah, I bet this your NAS is mDNS, which is broken right now: https://bugzilla.redhat.com/show_bug.cgi?id=1867830
Thanks, I added the little info I have to the bug report.
/Andreas
Until the root cause is diagnosed, here is a workaround you can try. As root, open up /etc/authselect/user-nsswitch.conf and look at the hosts line. In a freshly-installed F33 system, it should look like this:
hosts: resolve [!UNAVAIL=return] myhostname files mdns4_minimal [NOTFOUND=return] dns
This says: "if resolved is running, and it doesn't return a hostname, then STOP and don't try anything else." Everything else is listed only for the case where resolved is not running. But since resolved is currently not resolving mDNS as expected, let's change it to check with avahi first, then check with resolved second, like this:
hosts: mdns4_minimal [NOTFOUND=return] resolve [!UNAVAIL=return] myhostname files dns
Then, as root, run:
# authselect apply-changes
and then restart your browser. That's not the configuration we want to use in F33, but hopefully it will "fix" your problem. Please let me know if it works!
Michael
devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
On Sunday, August 30, 2020 3:30:22 AM MST Andreas Tunek wrote:
On my F33 test system (new install yesterday) I can see my NAS but I can't connect to it. Could that be due to this change?
If you used your own DNS servers, yes. You'll need to fix this file:
Remove the systemd thing: `rm /etc/resolv.conf`
Add your file back
cat << EOF search your.domain nameserver 1.2.3.4 nameserver 1.2.3.5 EOF > /etc/resolv.conf
Sub out 'your.domain' for your domain, and the IP addresses for your IP address(es). Remove the 2nd entry if you only have one.
You may wish to do that with an editor, such as Emacs, Vim or ed. I did it with shell expansion to make it easier to format into an email.
On 2020-08-30 18:30, Andreas Tunek wrote:
On my F33 test system (new install yesterday) I can see my NAS but I can't connect to it. Could that be due to this change?
Could you me a bit more specific?
I just installed from Fedora-Workstation-Live-x86_64-33-20200829.n.0.iso to a VM and everything seems working just fine for me.
[egreshko@f33g ~]$ resolvectl Global LLMNR setting: resolve MulticastDNS setting: resolve DNSOverTLS setting: no DNSSEC setting: no DNSSEC supported: no Current DNS Server: 192.168.122.1 DNS Servers: 192.168.122.1 Fallback DNS Servers: 1.1.1.1 8.8.8.8 1.0.0.1 8.8.4.4 2606:4700:4700::1111 2001:4860:4860::8888 2606:4700:4700::1001 2001:4860:4860::8844 DNS Domain: greshko.com
Link 2 (enp1s0) Current Scopes: DNS LLMNR/IPv4 LLMNR/IPv6
[egreshko@f33g ~]$ ll /etc/resolv.conf -rw-r--r--. 1 root root 74 Aug 31 10:33 /etc/resolv.conf
[egreshko@f33g ~]$ cat /etc/resolv.conf # Generated by NetworkManager search greshko.com nameserver 192.168.122.1
[egreshko@f33g ~]$ df -T Filesystem Type 1K-blocks Used Available Use% Mounted on /dev/vda2 btrfs 32504832 6213600 26109168 20% / /dev/vda2 btrfs 32504832 6213600 26109168 20% /home /dev/vda1 ext4 999320 184228 746280 20% /boot nas:/volume1/aux nfs4 5621463168 1920182016 3701281152 35% /aux
On Mon, Aug 31, 2020 at 1:09 pm, Vitaly Zaitsev via devel devel@lists.fedoraproject.org wrote:
You system is not using systemd-resolved.
It mostly is, though, via nss-resolve. Only stuff that uses resolv.conf directly is broken and not using resolved, which is https://bugzilla.redhat.com/show_bug.cgi?id=1873856. Anything using glibc should be fine.
Michael
On 2020-08-31 21:18, Michael Catanzaro wrote:
On Mon, Aug 31, 2020 at 1:09 pm, Vitaly Zaitsev via devel devel@lists.fedoraproject.org wrote:
You system is not using systemd-resolved.
It mostly is, though, via nss-resolve. Only stuff that uses resolv.conf directly is broken and not using resolved, which is https://bugzilla.redhat.com/show_bug.cgi?id=1873856. Anything using glibc should be fine.
OK, I went to using a rawhide VM. That system does have the symlink to /run/systemd/resolve/stub-resolv.conf.
But with that symlink it seems there is no integration with NetworkManager. So, if one has configured (which is the default) to acquire addresses automatically then DNS server info supplied by DHCP is ignored.
Is my take incorrect?
On Mon, Aug 31, 2020 at 9:36 pm, Ed Greshko ed.greshko@greshko.com wrote:
But with that symlink it seems there is no integration with NetworkManager. So, if one has configured (which is the default) to acquire addresses automatically then DNS server info supplied by DHCP is ignored.
Is my take incorrect?
Hopefully. NetworkManager should be pushing DNS configuration from DHCP to systemd-resolved. Sounds like you might be encountering some strange bug....
On 2020-09-01 01:18, Michael Catanzaro wrote:
On Mon, Aug 31, 2020 at 9:36 pm, Ed Greshko ed.greshko@greshko.com wrote:
But with that symlink it seems there is no integration with NetworkManager. So, if one has configured (which is the default) to acquire addresses automatically then DNS server info supplied by DHCP is ignored.
Is my take incorrect?
Hopefully. NetworkManager should be pushing DNS configuration from DHCP to systemd-resolved. Sounds like you might be encountering some strange bug....
OK, I looked a bit closer at the issue.
Indeed, systemd-resolved is getting the DNS server information. The actual issue is a lack of a DNS Domain. (a.k.a. "search" setting)
I don't think the DHCP server for the QEMU VMs is supplying a domain. However, Network Manager will add a "search" option to resolv.conf when "hostname" returns a FQDN.
This is using F33 Workstation.
[egreshko@f33g ~]$ hostname f33g.greshko.com
[egreshko@f33g ~]$ host nas Host nas not found: 2(SERVFAIL)
[egreshko@f33g ~]$ host nas.greshko.com nas.greshko.com has address 192.168.1.142 nas.greshko.com has IPv6 address 2001:b030:112f::19
[egreshko@f33g ~]$ resolvectl Global LLMNR setting: resolve MulticastDNS setting: resolve DNSOverTLS setting: no DNSSEC setting: no DNSSEC supported: no Fallback DNS Servers: 1.1.1.1 8.8.8.8 1.0.0.1 8.8.4.4 2606:4700:4700::1111 2001:4860:4860::8888 2606:4700:4700::1001 2001:4860:4860::8844
Link 2 (enp1s0) Current Scopes: DNS LLMNR/IPv4 LLMNR/IPv6 DefaultRoute setting: yes LLMNR setting: yes MulticastDNS setting: no DNSOverTLS setting: no DNSSEC setting: no DNSSEC supported: no Current DNS Server: 192.168.122.1 DNS Servers: 192.168.122.1 DNS Domain: ~.
On Tue, Sep 01, 2020 at 07:50:46AM +0800, Ed Greshko wrote:
I don't think the DHCP server for the QEMU VMs is supplying a domain. However, Network Manager will add a "search" option to resolv.conf when "hostname" returns a FQDN.
Ed opened https://bugzilla.redhat.com/show_bug.cgi?id=1874419. To follow up on this from a slightly different perspective:
NM *does* support pushing "search domains" into systemd-resolved, and in general this is expected to work. For example, I now have a company VPN configured using NM, and the "redhat.com" search domain is active:
$ resolvectl status tun0 Link 35 (tun0) Current Scopes: DNS LLMNR/IPv4 LLMNR/IPv6 ... Current DNS Server: 10.38.5.26 DNS Servers: 10.45.248.15 10.38.5.26 DNS Domain: redhat.com
$ resolvectl query www www.redhat.com www: 23.211.151.51 -- link: hub0 2a02:26f0:1300:186::d44 -- link: hub0 2a02:26f0:1300:190::d44 -- link: hub0 (e3396.dscx.akamaiedge.net)
-- Information acquired via protocol DNS in 263.0ms. -- Data is authenticated: no www.redhat.com: 2a02:26f0:1300:190::d44 -- link: hub0 2a02:26f0:1300:186::d44 -- link: hub0 23.211.151.51 -- link: hub0 (e3396.dscx.akamaiedge.net)
-- Information acquired via protocol DNS in 2.4ms. -- Data is authenticated: no
(It says "hub0" because the address is public, so it would not be routed through tun0. The relevant part is that "www" gets treated as "www.redhat.com" by resolved.)
What does *not* happen, is the domainname part of a hostname received via DHCP being installed as a search domain. In the case of a lease received via DHCP by NetworkManager, it's NetworkManager that decides what config to push to resolved. As discussed in https://bugzilla.redhat.com/show_bug.cgi?id=1874419, automatically installing the domain name as search domain this is not expected and probably not a good idea as a default.
Zbyszek
On Monday, August 31, 2020 1:22:58 AM MST Ed Greshko wrote:
On 2020-08-30 18:30, Andreas Tunek wrote:
On my F33 test system (new install yesterday) I can see my NAS but I can't connect to it. Could that be due to this change?
Could you me a bit more specific?
I just installed from Fedora-Workstation-Live-x86_64-33-20200829.n.0.iso to a VM and everything seems working just fine for me.
[egreshko@f33g ~]$ resolvectl Global LLMNR setting: resolve MulticastDNS setting: resolve DNSOverTLS setting: no DNSSEC setting: no DNSSEC supported: no Current DNS Server: 192.168.122.1 DNS Servers: 192.168.122.1 Fallback DNS Servers: 1.1.1.1 8.8.8.8 1.0.0.1 8.8.4.4 2606:4700:4700::1111 2001:4860:4860::8888 2606:4700:4700::1001 2001:4860:4860::8844 DNS Domain: greshko.com
Link 2 (enp1s0) Current Scopes: DNS LLMNR/IPv4 LLMNR/IPv6
[egreshko@f33g ~]$ ll /etc/resolv.conf -rw-r--r--. 1 root root 74 Aug 31 10:33 /etc/resolv.conf
[egreshko@f33g ~]$ cat /etc/resolv.conf # Generated by NetworkManager search greshko.com nameserver 192.168.122.1
[egreshko@f33g ~]$ df -T Filesystem Type 1K-blocks Used Available Use% Mounted on /dev/vda2 btrfs 32504832 6213600 26109168 20% / /dev/vda2 btrfs 32504832 6213600 26109168 20% /home /dev/vda1 ext4 999320 184228 746280 20% /boot nas:/volume1/aux nfs4 5621463168 1920182016 3701281152 35% /aux
Ed,
Where did you set these fallback servers? This is something that you specifically chose to do, and not systemd, right?
On 2020-08-31 21:40, John M. Harris Jr wrote:
On Monday, August 31, 2020 1:22:58 AM MST Ed Greshko wrote:
On 2020-08-30 18:30, Andreas Tunek wrote:
On my F33 test system (new install yesterday) I can see my NAS but I can't connect to it. Could that be due to this change?
Could you me a bit more specific?
I just installed from Fedora-Workstation-Live-x86_64-33-20200829.n.0.iso to a VM and everything seems working just fine for me.
[egreshko@f33g ~]$ resolvectl Global LLMNR setting: resolve MulticastDNS setting: resolve DNSOverTLS setting: no DNSSEC setting: no DNSSEC supported: no Current DNS Server: 192.168.122.1 DNS Servers: 192.168.122.1 Fallback DNS Servers: 1.1.1.1 8.8.8.8 1.0.0.1 8.8.4.4 2606:4700:4700::1111 2001:4860:4860::8888 2606:4700:4700::1001 2001:4860:4860::8844 DNS Domain: greshko.com
Link 2 (enp1s0) Current Scopes: DNS LLMNR/IPv4 LLMNR/IPv6
[egreshko@f33g ~]$ ll /etc/resolv.conf -rw-r--r--. 1 root root 74 Aug 31 10:33 /etc/resolv.conf
[egreshko@f33g ~]$ cat /etc/resolv.conf # Generated by NetworkManager search greshko.com nameserver 192.168.122.1
[egreshko@f33g ~]$ df -T Filesystem Type 1K-blocks Used Available Use% Mounted on /dev/vda2 btrfs 32504832 6213600 26109168 20% / /dev/vda2 btrfs 32504832 6213600 26109168 20% /home /dev/vda1 ext4 999320 184228 746280 20% /boot nas:/volume1/aux nfs4 5621463168 1920182016 3701281152 35% /aux
Ed,
Where did you set these fallback servers? This is something that you specifically chose to do, and not systemd, right?
My understanding is the Fallback Servers are hard coded. From the resolved.conf man page
FallbackDNS= A space-separated list of IPv4 and IPv6 addresses to use as the fallback DNS servers. Please see DNS= for acceptable format of adddresses. Any per-link DNS servers obtained from systemd- networkd.service(8) take precedence over this setting, as do any servers set via DNS= above or /etc/resolv.conf. This setting is hence only used if no other DNS server information is known. If this option is not given, a compiled-in list of DNS servers is used instead.
On Monday, August 31, 2020 6:43:37 AM MST Ed Greshko wrote:
On 2020-08-31 21:40, John M. Harris Jr wrote:
On Monday, August 31, 2020 1:22:58 AM MST Ed Greshko wrote:
On 2020-08-30 18:30, Andreas Tunek wrote:
On my F33 test system (new install yesterday) I can see my NAS but I can't connect to it. Could that be due to this change?
Could you me a bit more specific?
I just installed from Fedora-Workstation-Live-x86_64-33-20200829.n.0.iso to
a VM and everything seems working just fine for me.
[egreshko@f33g ~]$ resolvectl Global
LLMNR setting: resolveMulticastDNS setting: resolve
DNSOverTLS setting: no
DNSSEC setting: no DNSSEC supported: noCurrent DNS Server: 192.168.122.1
DNS Servers: 192.168.122.1Fallback DNS Servers: 1.1.1.1
8.8.8.8 1.0.0.1 8.8.4.4 2606:4700:4700::1111 2001:4860:4860::8888 2606:4700:4700::1001 2001:4860:4860::8844 DNS Domain: greshko.comLink 2 (enp1s0)
Current Scopes: DNS LLMNR/IPv4 LLMNR/IPv6[egreshko@f33g ~]$ ll /etc/resolv.conf -rw-r--r--. 1 root root 74 Aug 31 10:33 /etc/resolv.conf
[egreshko@f33g ~]$ cat /etc/resolv.conf # Generated by NetworkManager search greshko.com nameserver 192.168.122.1
[egreshko@f33g ~]$ df -T Filesystem Type 1K-blocks Used Available Use% Mounted on /dev/vda2 btrfs 32504832 6213600 26109168 20% / /dev/vda2 btrfs 32504832 6213600 26109168 20% /home /dev/vda1 ext4 999320 184228 746280 20% /boot nas:/volume1/aux nfs4 5621463168 1920182016 3701281152 35% /aux
Ed,
Where did you set these fallback servers? This is something that you specifically chose to do, and not systemd, right?
My understanding is the Fallback Servers are hard coded. From the resolved.conf man page
FallbackDNS= A space-separated list of IPv4 and IPv6 addresses to use as the fallback DNS servers. Please see DNS= for acceptable format of adddresses. Any per-link DNS servers obtained from systemd- networkd.service(8) take precedence over this setting, as do anyservers set via DNS= above or /etc/resolv.conf. This setting is hence only used if no other DNS server information is known. If this option is not given, a compiled-in list of DNS servers is used instead.
I thought we'd already addressed that months ago. Does everyone here realize we're about to ship a release with "fallback" to Google and Cloudflare DNS?
On Monday, August 31, 2020 8:32:49 AM MST Vitaly Zaitsev via devel wrote:
On 31.08.2020 17:07, John M. Harris Jr wrote:
ship a release with "fallback" to Google and Cloudflare DNS?
Big Brother will be happy. :-)
Sure, those two companies will be thrilled, I'm sure. This is a huge disservice to our users. Why in the world does systemd try to force DNS servers when none are configured? If no DNS servers are configured, there should be no DNS servers in use.
On 2020-09-01 08:52, John M. Harris Jr wrote:
On Monday, August 31, 2020 8:32:49 AM MST Vitaly Zaitsev via devel wrote:
On 31.08.2020 17:07, John M. Harris Jr wrote:
ship a release with "fallback" to Google and Cloudflare DNS?
Big Brother will be happy. :-)
Sure, those two companies will be thrilled, I'm sure. This is a huge disservice to our users. Why in the world does systemd try to force DNS servers when none are configured? If no DNS servers are configured, there should be no DNS servers in use.
Standard DNS has a hierarchical structure with roots and delegation. The idea of asking somebody to do DNS resolution for you comes from the widespread tendency to centralize everything (i.e. inability to understand how the Internet was originally designed).
Insisting on using a DNS server for name resolution is like insisting on using a proxy for HTTP access.
The only sane DNS server we should have is the one on localhost (doing proper caching according to TTLs). I do not know what this new systemd thing will provide (and hardcoded defaults would be a wrong beginning); in my case it has been bind on localhost for years; it lets me have local zones (e.g. plugged in when a VPN goes up) and I can also make it authoritative for external things I want to override (i.e. playing like a super-powered /etc/hosts).
Regards.
On Tue, Sep 1, 2020 at 3:34 AM Roberto Ragusa mail@robertoragusa.it wrote:
On 2020-09-01 08:52, John M. Harris Jr wrote:
On Monday, August 31, 2020 8:32:49 AM MST Vitaly Zaitsev via devel wrote:
On 31.08.2020 17:07, John M. Harris Jr wrote:
ship a release with "fallback" to Google and Cloudflare DNS?
Big Brother will be happy. :-)
Sure, those two companies will be thrilled, I'm sure. This is a huge disservice to our users. Why in the world does systemd try to force DNS servers when none are configured? If no DNS servers are configured, there should be no DNS servers in use.
Standard DNS has a hierarchical structure with roots and delegation. The idea of asking somebody to do DNS resolution for you comes from the widespread tendency to centralize everything (i.e. inability to understand how the Internet was originally designed).
Hiding it inside yet another systemd structure without following the existing standards is, sadly, typical of systemd. It also puts at risk restricted environments where providing no DNS is deliberately used to restrict outbound network use, such as virtual machines or chroot cages without an enabled /etc/resolv.conf. That includes the "mock" build environment where "pip install" is kept network disabled by the lack of DNS. It will also completely screw up VPN setups where out-of-band DNS servers break internal versus external service access management.
Insisting on using a DNS server for name resolution is like insisting on using a proxy for HTTP access.
It's secretly cooking the fries in bacon grease. It not only offends people such as vegetarians, Muslims, and Jews but it creates an unnecessary health risk for people with the "Alpha-Gal" meat allergy.
On Tue, Sep 1, 2020 at 8:17 am, Nico Kadel-Garcia nkadel@gmail.com wrote:
Hiding it inside yet another systemd structure without following the existing standards is, sadly, typical of systemd. It also puts at risk restricted environments where providing no DNS is deliberately used to restrict outbound network use, such as virtual machines or chroot cages without an enabled /etc/resolv.conf. That includes the "mock" build environment where "pip install" is kept network disabled by the lack of DNS.
So open up /etc/systemd/resolved.conf and set FallbackDNS= (set it to nothing). That will override fallback to Cloudflare or Google. Then you're done.
Realistically, this fallback is unlikely to ever be used anyway, so it doesn't matter very much. And if you're operating a restricted environment and you don't know how to configure DNS, you likely have bigger problems than systemd....
It will also completely screw up VPN setups where out-of-band DNS servers break internal versus external service access management.
No it won't. systemd is not going to use a fallback DNS server if your VPN provides its own DNS. It's not stupid. This is very easily verified simply by typing 'resolvectl' and seeing what DNS servers it has configured for a particular tun interface.
Michael
On Tuesday, September 1, 2020 7:22:49 AM MST Michael Catanzaro wrote:
On Tue, Sep 1, 2020 at 8:17 am, Nico Kadel-Garcia nkadel@gmail.com wrote:
Hiding it inside yet another systemd structure without following the existing standards is, sadly, typical of systemd. It also puts at risk restricted environments where providing no DNS is deliberately used to restrict outbound network use, such as virtual machines or chroot cages without an enabled /etc/resolv.conf. That includes the "mock" build environment where "pip install" is kept network disabled by the lack of DNS.
So open up /etc/systemd/resolved.conf and set FallbackDNS= (set it to nothing). That will override fallback to Cloudflare or Google. Then you're done.
This is not something that any user should ever have to do. If there are no configured DNS servers, either by DHCP or manual configuration, the user doesn't want DNS.
Realistically, this fallback is unlikely to ever be used anyway, so it doesn't matter very much. And if you're operating a restricted environment and you don't know how to configure DNS, you likely have bigger problems than systemd....
If this is unlikely to be used, can we get this set to empty by default in Fedora?
On Tue, Sep 01, 2020 at 08:17:49AM -0400, Nico Kadel-Garcia wrote:
Big Brother will be happy. :-)
Sure, those two companies will be thrilled, I'm sure. This is a huge disservice to our users. Why in the world does systemd try to force DNS servers when none are configured? If no DNS servers are configured, there should be no DNS servers in use.
Standard DNS has a hierarchical structure with roots and delegation. The idea of asking somebody to do DNS resolution for you comes from the widespread tendency to centralize everything (i.e. inability to understand how the Internet was originally designed).
Hiding it inside yet another systemd structure without following the existing standards is, sadly, typical of systemd. It also puts at risk restricted environments where providing no DNS is deliberately used to restrict outbound network use, such as virtual machines or chroot cages without an enabled /etc/resolv.conf. That includes the "mock" build environment where "pip install" is kept network disabled by the lack of DNS.
Other sentences in this paragraph have already been disambiguated by others, so I'll reply only to the part about mock: That's not how mock works (for the last few years). $ mock --shell # ip route default via 127.0.0.1 dev lo proto static # dig fedoraproject.org @8.8.8.8 (hangs for the duration of timeout)
Relying on lack of resolv.conf configuration in the buildroot to prevent network use would be very brittle and easily circumvented, even by accident if some tool included IP addresses internally.
Zbyszek
* Roberto Ragusa:
Standard DNS has a hierarchical structure with roots and delegation. The idea of asking somebody to do DNS resolution for you comes from the widespread tendency to centralize everything (i.e. inability to understand how the Internet was originally designed).
DNS originally did not have a clear hierarchical structure. You just asked one server you liked, and it would point you towards a server somewhat closer to the data source if it did not have the answer itself.
The hierarchy was always there in the background, to ensure eventually successful lookups, but it gained prominence in implementations only once it became apparent that you can only use data from servers that are actually authoritative for the subtree in which the data resides. Otherwise, you end up with rather trivial spoofing attacks.
Insisting on using a DNS server for name resolution is like insisting on using a proxy for HTTP access.
I'm not sure if that's the appropriate analogy. Most of us don't run BGP on their laptops, and DNS is closer to that layer than to HTTP.
But it definitely doesn't make sense to create a deep hierarchy of resolvers, somehow mirroring the hierarchy of delegation.
The only sane DNS server we should have is the one on localhost (doing proper caching according to TTLs).
Many networks block outgoing UDP traffic, so you cannot run DNS locally at all. There are also concerns that the DNS infrastructure cannot handle the load unless there is one level of shared caching betweeen the endpoints and the authoritative servers. Those DNS caches certainly suppress some of the problematic client behavior (but they also add their own share of broken queries, of course).
Thanks, Florian
On Tue, Sep 01, 2020 at 02:28:57PM +0200, Florian Weimer wrote:
There are also concerns that the DNS infrastructure cannot handle the load unless there is one level of shared caching betweeen the endpoints and the authoritative servers. Those DNS caches certainly suppress some of the problematic client behavior (but they also add their own share of broken queries, of course).
Well, all that distributed caching hierarchy is undermined by some auxiliary non-DNS Google project anyway: https://arstechnica.com/gadgets/2020/08/a-chrome-feature-is-creating-enormou...
* John M. Harris, Jr.:
Sure, those two companies will be thrilled, I'm sure. This is a huge disservice to our users. Why in the world does systemd try to force DNS servers when none are configured? If no DNS servers are configured, there should be no DNS servers in use.
Acutally, the historic default is to use localhost (127.0.0.1). This is what an empty or missing /etc/resolv.conf file has always meant.
(Okay, there was apparently a time when localhost could also be reached at 0.0.0.0, and that was the default before 127.0.0.1. But that likely predates the Linux networking stack.)
Thanks, Florian
On Tuesday, September 1, 2020 5:19:15 AM MST Florian Weimer wrote:
- John M. Harris, Jr.:
Sure, those two companies will be thrilled, I'm sure. This is a huge disservice to our users. Why in the world does systemd try to force DNS servers when none are configured? If no DNS servers are configured, there
should be no DNS servers in use.
Acutally, the historic default is to use localhost (127.0.0.1). This is what an empty or missing /etc/resolv.conf file has always meant.
(Okay, there was apparently a time when localhost could also be reached at 0.0.0.0, and that was the default before 127.0.0.1. But that likely predates the Linux networking stack.)
Well, that was only reading host files, right? Or do you mean the system would actually perform lookup against itself?
* John M. Harris, Jr.:
On Tuesday, September 1, 2020 5:19:15 AM MST Florian Weimer wrote:
- John M. Harris, Jr.:
Sure, those two companies will be thrilled, I'm sure. This is a huge disservice to our users. Why in the world does systemd try to force DNS servers when none are configured? If no DNS servers are configured, there
should be no DNS servers in use.
Acutally, the historic default is to use localhost (127.0.0.1). This is what an empty or missing /etc/resolv.conf file has always meant.
(Okay, there was apparently a time when localhost could also be reached at 0.0.0.0, and that was the default before 127.0.0.1. But that likely predates the Linux networking stack.)
Well, that was only reading host files, right? Or do you mean the system would actually perform lookup against itself?
It actually sends UDP packets to 127.0.0.1, port 53.
Thanks, Florian
On Sat, Aug 29, 2020 at 3:12 pm, Chris Murphy lists@colorremedies.com wrote:
Are these the expected behavior?
4. is unexpected. The file should not be generated by NetworkManager. NetworkManager should notice that the file is a symlink to /run/systemd/resolve/stub-resolv.conf and leave it alone. (In point 3. the file should be a symlink.)
On Sun, Aug 30, 2020 at 9:04 am, Michael Catanzaro mcatanzaro@gnome.org wrote:
- is unexpected. The file should not be generated by NetworkManager.
NetworkManager should notice that the file is a symlink to /run/systemd/resolve/stub-resolv.conf and leave it alone. (In point 3. the file should be a symlink.)
I don't know what to do about this. Ideally we would figure out what's wrong and sneak a freeze exception into the beta release. If the file in 3. is not a symlink, then that would be what's wrong, but it ought to be a symlink.
On Sun, Aug 30, 2020 at 9:06 am, Michael Catanzaro mcatanzaro@gnome.org wrote:
I don't know what to do about this. Ideally we would figure out what's wrong and sneak a freeze exception into the beta release. If the file in 3. is not a symlink, then that would be what's wrong, but it ought to be a symlink.
I just did a test install. From the live environment, post-installation, I see:
$ pwd /mnt/sysroot/etc $ ls -l | grep resolv -rw-r--r--. 1 root root 729 Aug 30 11:19 resolv.conf
So the file is indeed not a symlink. That tells NetworkManager that the file is to be managed by NetworkManager, not by systemd.
So basically you discovered our change is totally broken. Thanks for testing. I've reported https://bugzilla.redhat.com/show_bug.cgi?id=1873856.
Michael
On Sunday, August 30, 2020 8:36:46 AM MST Michael Catanzaro wrote:
On Sun, Aug 30, 2020 at 9:06 am, Michael Catanzaro mcatanzaro@gnome.org wrote:
I don't know what to do about this. Ideally we would figure out what's wrong and sneak a freeze exception into the beta release. If the file in 3. is not a symlink, then that would be what's wrong, but it ought to be a symlink.
I just did a test install. From the live environment, post-installation, I see:
$ pwd /mnt/sysroot/etc $ ls -l | grep resolv -rw-r--r--. 1 root root 729 Aug 30 11:19 resolv.conf
So the file is indeed not a symlink. That tells NetworkManager that the file is to be managed by NetworkManager, not by systemd.
So basically you discovered our change is totally broken. Thanks for testing. I've reported https://bugzilla.redhat.com/show_bug.cgi?id=1873856.
This is one of the reasons I've been warning you about this logic from the start. This heavy-handed change is going to be a nightmare if executed this way. A simple fix would be to leave existing systems alone, and only create that symlink on new installs. That would simplify the logic, and prevent breaking peoples' systems needlessly. All users would benefit this way.
On 30.08.2020 16:04, Michael Catanzaro wrote:
The file should not be generated by NetworkManager. NetworkManager should notice that the file is a symlink to /run/systemd/resolve/stub-resolv.conf and leave it alone. (In point 3. the file should be a symlink.)
Network Manager can be configured to use an external resolv.conf generator for example by adding a new config file /etc/NetworkManager/conf.d/99-resolved.conf:
[main] dns=systemd-resolved
After adding this option you don't need to make /etc/resolv.conf a symbolic link to /run/NetworkManager/resolv.conf.
On 7/26/20 6:15 PM, John M. Harris Jr wrote:
Please do not disable reading from /etc/resolv.conf.
Where did you get the impression that the change would do that?
Relevant quotes:
In short, '''applications that read /etc/resolv.conf will continue to work as before.'''
/etc/resolv.conf will no longer be read when performing name resolution using glibc; however, it is still provided for compatibility with applications that manually read this file to perform name resolution.
On Sunday, July 26, 2020 7:47:08 PM MST Gordon Messmer wrote:
On 7/26/20 6:15 PM, John M. Harris Jr wrote:
Please do not disable reading from /etc/resolv.conf.
Where did you get the impression that the change would do that?
Relevant quotes:
In short, '''applications that read /etc/resolv.conf will continue to work as before.'''
/etc/resolv.conf will no longer be read when performing name resolution using glibc; however, it is still provided for compatibility with applications that manually read this file to perform name resolution.
Relevant quotes: After upgrade, /etc/resolv.conf will be managed by systemd and symlinked to / run/systemd/resolve/stub-resolv.conf.
Writing to /etc/resolv.conf will no longer work as expected.
-
Adam Williamson -
Alexander Bokovoy -
Andreas Tunek -
Andrew Lutomirski -
Ben Cotton -
Björn Persson -
Chris Adams -
Chris Murphy -
Christopher -
Chuck Anderson -
Colin Walters -
Cuckoo's Calling -
Dan Williams -
Daniel Walsh -
David Sommerseth -
Denys Vlasenko -
Dominik 'Rathann' Mierzejewski -
drago01 -
Ed Greshko -
Erich Eickmeyer -
Eugene Syromiatnikov -
Florian Weimer -
Gary Buhrmaster -
Gerd Hoffmann -
Gordon Messmer -
Graham Leggett -
Ian Pilcher -
James Cassell -
John M. Harris Jr -
Kevin Fenzi -
Lennart Poettering -
Marius Schwarz -
Markus Larsson -
Matthew Miller -
Michael Catanzaro -
Michael Cronenworth -
Neal Gompa -
Nico Kadel-Garcia -
Nicolas Mailhot -
Nikos Mavrogiannopoulos -
Paul Wouters -
Pavel Raiskup -
Petr Menšík -
pgfed -
PGNet Dev
-
Robbie Harwood -
Roberto Ragusa -
Simo Sorce -
Solomon Peachy -
stan -
Stephen John Smoogen -
Tom Hughes -
Tomas Mraz -
Tomasz Torcz -
Vitaly Zaitsev -
Vít Ondruch -
Zbigniew Jędrzejewski-Szmek -
Zdenek Dohnal