systemd-resolved breakage

[Sam Varshavchik]

I was surprised that a

dig www.unicode.org a

was coming back with a SERVFAIL. I have bind running on the same server, so  
I thought it just had a hiccup. Just to be sure I haven't fat-fingered  
anything and accidentally set dns lookups to go to another dns server on the  
network, I did an explicit "dig @127.0.0.1", and that worked. Must be it,  
right? I looked at /etc/resolv.conf and found out that the default server is  
127.0.0.53, and then remembered in F33 it was decided to have systemd stick  
its nose into another place it didn't belong in the first place.

So, systemd's resolver was broken, for some reason, with no indication why  
or how. Nothing was logged anywhere, as I far as I could see. Typical. As I  
was researching for the proper way to get rid of yet another pile of systemd  
junk, I got to the point of finding out about "resolvectl dns" which proudly  
reported the server's IP address as the only DNS server it knows about, then  
resolution via 127.0.0.53 started working again. And again, without any  
explanation.

Reading the amusing list of buzzwords in "man resolvectl", like "server  
feature probing logic" it is unclear why any of that crap is needed. If the  
whole purpose of this whole house of cards is to have a proxy that silently  
redirect DNS lookups to whatever DNS server is configured via the currently  
active connection, I can see the value in that but I do not see what's to be  
gained by a fragile "server feature probing logic" that, apparently,  
sometimes decides that the DNS server on the same host doesn't really exist,  
and decides to reject all requests, on its own initiative, with SERVFAILs.  
And provide no useful logging, whatsoever. 'journalctl -u systemd-resolved'  
has no explanation, of any kind, whatsoever.

DNS resolution is a critical function. systemd-resolved failed it,  
miserably. And without having even the courage to provide any kind of a  
useful logging that can be used to file a bug report.

[Tom Horsley]

(Relatively) simple fix:

systemctl stop systemd-resolvd
systemctl disable systemd-resolvd

Edit /etc/NetworkManager/NetworkManager.conf And in the [main]
section stick this:

[main]
dns=none

Now rm -f /etc/resolv.conf

Now create your own /etc/resolv.conf file from scratch with
the nameserver and search directives you actually want.

Systemd won't stick its nose in, NetworkManager won't clobber
your resolve.conf file, your network will actually function :-).

[Petr Menšík]

Sad thing is, I want Network Manager to write my resolv.conf as it did
before. I just want systemd-resolved disabled and keep simple text file
in /etc/resolv.conf.

I haven't found automatic way to recover my system, after I do:
systemctl disable --now systemd-resolved

It keep broken /etc/resolv.conf, which I have to fix manually.

systemctl restart NetworkManager does not help.

Am I missing something?

would NetworkManager.conf:
dns=default

Write resolv.conf again? Can I make it rewrite after disabling
systemd-resolved? Why doesn't it restore /etc/resolv.conf on
systemd-resolved shutdown?

On 11/13/20 2:22 PM, Tom Horsley wrote:
> (Relatively) simple fix:
>
> systemctl stop systemd-resolvd
> systemctl disable systemd-resolvd
>
> Edit /etc/NetworkManager/NetworkManager.conf And in the [main]
> section stick this:
>
> [main]
> dns=none
>
> Now rm -f /etc/resolv.conf
>
> Now create your own /etc/resolv.conf file from scratch with
> the nameserver and search directives you actually want.
>
> Systemd won't stick its nose in, NetworkManager won't clobber
> your resolve.conf file, your network will actually function :-).
> _______________________________________________
> users mailing list -- [e-mail]
> To unsubscribe send an email to [e-mail]
> Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives: https://lists.fedoraproject.org/archives/list/[e-mail]

[Tom H]

On Fri, Nov 13, 2020 at 3:17 PM Petr Menšík 
wrote:
> Sad thing is, I want Network Manager to write my resolv.conf as it
> did before. I just want systemd-resolved disabled and keep simple
> text file in /etc/resolv.conf.
>
> I haven't found automatic way to recover my system, after I do:
> systemctl disable --now systemd-resolved
>
> It keep broken /etc/resolv.conf, which I have to fix manually.
>
> systemctl restart NetworkManager does not help.
>
> Am I missing something?
>
> would NetworkManager.conf:
> dns=default
>
> Write resolv.conf again? Can I make it rewrite after disabling
> systemd-resolved? Why doesn't it restore /etc/resolv.conf on
> systemd-resolved shutdown?

It should work if you replace "rc-manager=symlink" in the settings of
my previous email with "rc-manager=file" and set "/etc/resolv.conf" as
a file.

[Sam Varshavchik]

Petr Menšík writes:
> Am I missing something?

Yes, system is being shoved down your gullet, whether you like it or not.
> would NetworkManager.conf:
> dns=default
>
> Write resolv.conf again? Can I make it rewrite after disabling
> systemd-resolved? Why doesn't it restore /etc/resolv.conf on
> systemd-resolved shutdown?

Yes, /etc/resolv.conf is a symlink. Apparently NetworkManager has been  
changed to write /run/NetworkManager/resolv.conf instead, so you can set  
/etc/resolv.conf symlink to point to it.

This whole house of cards is an epic fail. There is already a perfectly  
usable architecture in place to handle this: alternatives. Both  
NetworkManager and systemd should use alternatives to claim dibs on  
/etc/resolv.conf, and alternatives will manage the symlink itself.

This is just plain dumb. /etc/resolv.conf should be an alternatives-managed  
link. If it were, you'd immediately know how to obtain the list of all the  
alternative configurations, and have a simple mechanism for selecting how  
you want DNS to be configured, and complete freedom to use or not use  
systemd-resolved. I refuse to believe that this obvious solution never  
occured to anyone, instead it was decided to intentionally cram systemd- 
resolved down everyone's throat and make people jump through hoops to get  
rid of it.

[Tom H]

On Sat, Nov 14, 2020 at 12:42 AM Sam Varshavchik
 wrote:
> Petr Menšík writes:
> > Am I missing something?
>
> Yes, system is being shoved down your gullet, whether you like it
> or not.

Whichever distribution you use, you're at the mercy of its developers.
In this particular case, the developer responsible for the change
didn't care about breaking some users' setups by not forcing this
change for upgrades. That's life.

[Tom Horsley]

On Sat, 14 Nov 2020 11:56:01 +0100
Tom H wrote:
> > Yes, system is being shoved down your gullet, whether you like it
> > or not.
>
> Whichever distribution you use, you're at the mercy of its developers.
> In this particular case, the developer responsible for the change
> didn't care about breaking some users' setups by not forcing this
> change for upgrades. That's life.

There are computer viruses, then there's systemd - the world's
only computer fungus - it keeps growing over more and more
of the system :-).

[Anthony F McInerney]

On Fri, 13 Nov 2020 at 23:42, Sam Varshavchik  wrote:
> Petr Menšík writes:
> > Am I missing something?
>
> Yes, system is being shoved down your gullet, whether you like it or not.
> > would NetworkManager.conf:
> > dns=default
> >
> > Write resolv.conf again? Can I make it rewrite after disabling
> > systemd-resolved? Why doesn't it restore /etc/resolv.conf on
> > systemd-resolved shutdown?
>
> Yes, /etc/resolv.conf is a symlink. Apparently NetworkManager has been
> changed to write /run/NetworkManager/resolv.conf instead, so you can set
> /etc/resolv.conf symlink to point to it.
>
> This whole house of cards is an epic fail. There is already a perfectly
> usable architecture in place to handle this: alternatives. Both
> NetworkManager and systemd should use alternatives to claim dibs on
> /etc/resolv.conf, and alternatives will manage the symlink itself.
>
> This is just plain dumb. /etc/resolv.conf should be an
> alternatives-managed
> link. If it were, you'd immediately know how to obtain the list of all
> the
> alternative configurations, and have a simple mechanism for selecting how
> you want DNS to be configured, and complete freedom to use or not use
> systemd-resolved. I refuse to believe that this obvious solution never
> occured to anyone, instead it was decided to intentionally cram systemd-
> resolved down everyone's throat and make people jump through hoops to get
> rid of it.

Sounds reasonable considering the options for resolved itself seems to all
rely on the default set by the distro, with no option to change it for the
enduser. Unless they know how to faff about symlinking resolv.conf to
another 'option'.....

So considering the above the below options (man page) is there a simple way
in fedora to flip these options without manual symlinking?

/ETC/RESOLV.CONF
       Four modes of handling /etc/resolv.conf (see resolv.conf(5)) are
supported:

·   systemd-resolved maintains the
/run/systemd/resolve/stub-resolv.conf file for compatibility with
traditional Linux programs. This file may be symlinked from
           /etc/resolv.conf. This file lists the 127.0.0.53 DNS stub (see
above) as the only DNS server. It also contains a list of search domains
that are in use by
           systemd-resolved. The list of search domains is always kept
up-to-date. Note that /run/systemd/resolve/stub-resolv.conf should not be
used directly by
           applications, but only through a symlink from /etc/resolv.conf.
This file may be symlinked from /etc/resolv.conf in order to connect all
local clients that bypass
           local DNS APIs to systemd-resolved with correct search domains
settings. This mode of operation is recommended.

·   A static file /usr/lib/systemd/resolv.conf is provided that
lists the 127.0.0.53 DNS stub (see above) as only DNS server. This file may
be symlinked from
           /etc/resolv.conf in order to connect all local clients that
bypass local DNS APIs to systemd-resolved. This file does not contain any
search domains.

·   systemd-resolved maintains the /run/systemd/resolve/resolv.conf
file for compatibility with traditional Linux programs. This file may be
symlinked from
           /etc/resolv.conf and is always kept up-to-date, containing
information about all known DNS servers. Note the file format's
limitations: it does not know a concept
           of per-interface DNS servers and hence only contains system-wide
DNS server definitions. Note that /run/systemd/resolve/resolv.conf should
not be used directly by
           applications, but only through a symlink from /etc/resolv.conf.
If this mode of operation is used local clients that bypass any local DNS
API will also bypass
           systemd-resolved and will talk directly to the known DNS servers.

·   Alternatively, /etc/resolv.conf may be managed by other
packages, in which case systemd-resolved will read it for DNS configuration
data. In this mode of
           operation systemd-resolved is consumer rather than provider of
this configuration file.

Note that the selected mode of operation for this file is detected
fully automatically, depending on whether /etc/resolv.conf is a symlink to
       /run/systemd/resolve/resolv.conf or lists 127.0.0.53 as DNS server.

[Sam Varshavchik]

Anthony F McInerney writes:
> Note that the selected mode of operation for this file is detected  
> fully automatically, depending on whether /etc/resolv.conf is a symlink to
>        /run/systemd/resolve/resolv.conf or lists 127.0.0.53 as DNS server.

Again: why does anything have to be "detected fully automatically"?

We've had a mechanism for controlling symlinks for an eternity:  
alternatives.

I cannot see any valid technical argument against having /etc/resolv.conf  
managed by alternatives. If systemd-resolved wishes to have the highest  
alternatives priority, and thus ends up being the resolver by default,  
that's fine.

Does anyone have an argument /against/ having /etc/resolv.conf managed via  
alternatives? What can't you do with alternatives that you can with the  
current ham-fisted approach, where someone has to go out of their way to  
figure out how to switch to network manager-provided resolv.conf?

[Anthony F McInerney]

On Sat, 14 Nov 2020 at 19:35, Sam Varshavchik  wrote:
> Anthony F McInerney writes:
> >    Note that the selected mode of operation for this file is
>
> detected
> > fully automatically, depending on whether /etc/resolv.conf is a symlink
>
> to
> >    /run/systemd/resolve/resolv.conf or lists 127.0.0.53 as DNS
>
> server.
>
> Again: why does anything have to be "detected fully automatically"?
>
> We've had a mechanism for controlling symlinks for an eternity:
> alternatives.
>
> I cannot see any valid technical argument against having /etc/resolv.conf
> managed by alternatives. If systemd-resolved wishes to have the highest
> alternatives priority, and thus ends up being the resolver by default,
> that's fine.
>
> Does anyone have an argument /against/ having /etc/resolv.conf managed
> via
> alternatives? What can't you do with alternatives that you can with the
> current ham-fisted approach, where someone has to go out of their way to
> figure out how to switch to network manager-provided resolv.conf?

Can someone explain why systemd-resolved needs to symlink /etc/resolv.conf
to 4(or more) different places, instead of just having those 'detected
things' as options in /etc/systemd/resolved.conf ?

Also spotted this on the Arch Wiki

*"systemd-resolved* will work out of the box with a network manager
https://wiki.archlinux.org/index.php/Network_manager using
/etc/resolv.conf. No particular configuration is required since
*systemd-resolved* will be detected by following the /etc/resolv.conf symlink.
This is going to be the case with systemd-networkd
https://wiki.archlinux.org/index.php/Systemd-networkd or NetworkManager
https://wiki.archlinux.org/index.php/NetworkManager."

[Tom H]

On Sun, Nov 15, 2020 at 10:15 AM Anthony F McInerney
 wrote:
> Can someone explain why systemd-resolved needs to symlink
> /etc/resolv.conf to 4(or more) different places, instead of just
> having those 'detected things' as options in
> /etc/systemd/resolved.conf ?

The different symlinks/files provide different options.

/etc/resolv.conf is a symlink to /run/systemd/resolve/stub-resolv.conf
so
/etc/resolv.conf has "127.0.0.53" and "search"
resolved forwards queries to the upstream server

/etc/resolv.conf is a symlink to /usr/lib/systemd/resolv.conf
so
/etc/resolv.conf has "127.0.0.53"
resolved forwards queries to the upstream server

/etc/resolv.conf is a symlink to /run/systemd/resolve/resolv.conf
so
/etc/resolv.conf has "upstream server" and "search"
resolved only creates this file as if it were "resolvconf"

/etc/resolv.conf is a file
so
resolved uses the configured "server" and "search" as its "upstream
server" and "search"

I doubt that anyone would want resolved to create/modify the symlinks
depending on "/etc/systemd/resolved.conf".

Sam's idea of using the "alternatives" infrastrucure would be better.
But that would create other problems.

Assume that you're using networkd and resolved. If you install
NetworkManager, the preferred, higher-priority alternative'll be
"/run/NetworkManager/resolv.conf" whether you want it or not.
> Also spotted this on the Arch Wiki
>
> "systemd-resolved will work out of the box with a network manager
> using /etc/resolv.conf. No particular configuration is required
> since systemd-resolved will be detected by following the
> /etc/resolv.conf symlink. This is going to be the case with
> systemd-networkd or NetworkManager."

Sure, but this isn't the problem here. When you upgrade from F32 to
F33, the systemd post-install script enables resolved and creates a
resolved symlink.

[Anthony F McInerney]

On Sun, 15 Nov 2020 at 20:22, Tom H  wrote:
> On Sun, Nov 15, 2020 at 10:15 AM Anthony F McInerney
>  wrote:
> > Can someone explain why systemd-resolved needs to symlink
> > /etc/resolv.conf to 4(or more) different places, instead of just
> > having those 'detected things' as options in
> > /etc/systemd/resolved.conf ?
>
> The different symlinks/files provide different options.

Yes I was well aware of that, most people who had to read the man page to
find out about the magic symlink and which one had broken their system also
know this now too.

I doubt that anyone would want resolved to create/modify the symlinks
> depending on "/etc/systemd/resolved.conf".

Which indeed, was my point, why would "anyone want resolved to
create/modify the symlinks" or even BE one, instead of just a static file,
that some other random app doesn't claim ownership of with a symlink, that
does some 'magic' with enforce options, you'll know about once you read the
man page.

[Tom H]

On Fri, Nov 13, 2020 at 2:22 PM Tom Horsley  wrote:
> systemctl stop systemd-resolvd
> systemctl disable systemd-resolvd

resolved
> Edit /etc/NetworkManager/NetworkManager.conf And in the [main]
> section stick this:
>
> [main]
> dns=none

Since "/etc/NetworkManager/NetworkManager.conf" is
"%config(noreplace)", it'd be better to create
"/etc/NetworkManager/conf.d/dns.conf" with the above.

Or you can leverage NM:

- set "dns=default" and "rc-manager=symlink" in "[main]" in
"/etc/NetworkManager/conf.d/dns.conf"

- set "dns=127.0.0.1" in
"/etc/NetworkManager/system-connections/<SamV>.nmconnection"

- set "/etc/resolv.conf" as a symlink to "../run/NetworkManager/resolv.conf"

There is another thread re systemd-resolved and adguard.

Ed Greshko had a solution to use /etc/resolve.conf instead of the stub file systemd-resolved uses.

"To revert to the previous behavior you need to remove the symbolic link

lrwxrwxrwx. 1 root root 39 Sep  1 17:52 resolv.conf ->
../run/systemd/resolve/stub-resolv.conf

disable or mask systemd-resolved.service, create an empty /etc/resolv.conf and reboot."

Would that work in this case?