Please stop hijacking /etc/resolv.conf

[Sam Varshavchik]

I updated from F39 to F40. I used to have:

/etc/resolv.conf -> /run/NetworkManager/no-stub-resolv.conf

Everything got messed up because the update hijacked this symlink again:

lrwxrwxrwx. 1 root root 39 May 29 09:44 /etc/resolv.conf -> ../run/systemd/resolve/stub-resolv.conf

I was confident that F39 did not have systemd-resolved installed, it must've  
been pulled into the update.

I repeated the experiment on two other F39 systems today. I confirmed that  
neither one of them had systemd-resolved installed. F40 pulled it in, and  
the package's scriptlet clobbered /etc/resolv.conf

It would be real nice if someone finally STOPed this repeated hijacking of  
/etc/resolv.conf, and breaking network connectivity, is this too much to ask? 
Although this was fairly simple to fix, this kind of behavior does not  
improve systemd's existing reputation.

[Jeffrey Walton]

On Wed, May 29, 2024 at 6:55 PM Sam Varshavchik  wrote:
> I updated from F39 to F40. I used to have:
>
> /etc/resolv.conf -> /run/NetworkManager/no-stub-resolv.conf
>
> Everything got messed up because the update hijacked this symlink again:
>
> lrwxrwxrwx. 1 root root 39 May 29 09:44 /etc/resolv.conf -> ../run/systemd/resolve/stub-resolv.conf
>
> I was confident that F39 did not have systemd-resolved installed, it must've
> been pulled into the update.
>
> I repeated the experiment on two other F39 systems today. I confirmed that
> neither one of them had systemd-resolved installed. F40 pulled it in, and
> the package's scriptlet clobbered /etc/resolv.conf
>
> It would be real nice if someone finally STOPed this repeated hijacking of
> /etc/resolv.conf, and breaking network connectivity, is this too much to ask?
> Although this was fairly simple to fix, this kind of behavior does not
> improve systemd's existing reputation.

It sounds like something for https://bugzilla.redhat.com/.

Jeff

[Sam Varshavchik]

Jeffrey Walton writes:
> On Wed, May 29, 2024 at 6:55 PM Sam Varshavchik   
> wrote:
> > I updated from F39 to F40. I used to have:
> >
> > /etc/resolv.conf -> /run/NetworkManager/no-stub-resolv.conf
> >
> > Everything got messed up because the update hijacked this symlink again:
> >
> > lrwxrwxrwx. 1 root root 39 May 29 09:44 /etc/resolv.conf - 
> > ../run/systemd/resolve/stub-resolv.conf
> >
> > I was confident that F39 did not have systemd-resolved installed, it
>
> must've
> > been pulled into the update.
> >
> > I repeated the experiment on two other F39 systems today. I confirmed that
> > neither one of them had systemd-resolved installed. F40 pulled it in, and
> > the package's scriptlet clobbered /etc/resolv.conf
> >
> > It would be real nice if someone finally STOPed this repeated hijacking of
> > /etc/resolv.conf, and breaking network connectivity, is this too much to
>
> ask?
> > Although this was fairly simple to fix, this kind of behavior does not
> > improve systemd's existing reputation.
>
> It sounds like something for https://bugzilla.redhat.com/.

Yes, where it will gather dust for about a year, until it gets closed when  
F40 goes EOL.

[Barry Scott]

> On 29 May 2024, at 23:49, Sam Varshavchik  wrote:
>
> I updated from F39 to F40. I used to have:
>
> /etc/resolv.conf -> /run/NetworkManager/no-stub-resolv.conf
>
> Everything got messed up because the update hijacked this symlink again:
>
> lrwxrwxrwx. 1 root root 39 May 29 09:44 /etc/resolv.conf -> ../run/systemd/resolve/stub-resolv.conf
>
> I was confident that F39 did not have systemd-resolved installed, it must've been pulled into the update.
>
> I repeated the experiment on two other F39 systems today. I confirmed that neither one of them had systemd-resolved installed. F40 pulled it in, and the package's scriptlet clobbered /etc/resolv.conf
>
> It would be real nice if someone finally STOPed this repeated hijacking of /etc/resolv.conf, and breaking network connectivity, is this too much to ask?Although this was fairly simple to fix, this kind of behavior does not improve systemd's existing reputation.

My understanding is that you can configure either systemd-resolved or NetworkManger to setup what you need
to be in resolv.conf and then can leave how the symlink is managed as is.

Have you thought to do that setup?

Barry
> --
> _______________________________________________
> users mailing list -- [e-mail]
> To unsubscribe send an email to [e-mail]
> Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives: https://lists.fedoraproject.org/archives/list/[e-mail]
> Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue

[Sam Varshavchik]

Barry Scott writes:
> > On 29 May 2024, at 23:49, Sam Varshavchik  wrote:
> >
> > I updated from F39 to F40. I used to have:
> >
> > /etc/resolv.conf -> /run/NetworkManager/no-stub-resolv.conf
> >
> > Everything got messed up because the update hijacked this symlink again:
> >
> > lrwxrwxrwx. 1 root root 39 May 29 09:44 /etc/resolv.conf - 
> > ../run/systemd/resolve/stub-resolv.conf
> >
> > I was confident that F39 did not have systemd-resolved installed, it
>
> must've been pulled into the update.
> > I repeated the experiment on two other F39 systems today. I confirmed that
>
> neither one of them had systemd-resolved installed. F40 pulled it in, and  
> the package's scriptlet clobbered /etc/resolv.conf
> > It would be real nice if someone finally STOPed this repeated hijacking of
>
> /etc/resolv.conf, and breaking network connectivity, is this too much to ask? 
> Although this was fairly simple to fix, this kind of behavior does not  
> improve systemd's existing reputation.
>
> My understanding is that you can configure either systemd-resolved or  
> NetworkManger to setup what you need
> to be in resolv.conf and then can leave how the symlink is managed as is.
>
> Have you thought to do that setup?

That's exactly what I've done. I'm using NetworkManager. The symlink  
normally points to NetworkManager-written resolv.conf.

dnf system-upgrade download --releasever=40 followed by dnf system-upgrade  
reboot ended up installing systemd-resolved, and then replacing the symlink.  
That's precisely the problem.

[Kevin Fenzi]

For whatever it's worth, the logic to replace or not replace
/etc/resolv.conf has gone through a bunch of adjustments, but currently,
as far as I can see it's:

if systemctl -q is-enabled systemd-resolved.service &>/dev/null &&
   ! systemd-analyze cat-config systemd/resolved.conf 2>/dev/null |
        grep -iqE '^DNSStubListener\s*=\s*(no?|false|0|off)\s*$'; then

if ! test -e /etc/resolv.conf && ! test -L /etc/resolv.conf; then
    ln -sv ../run/systemd/resolve/stub-resolv.conf /etc/resolv.conf || :
  elif test -d /run/systemd/system/ &&
     ! mountpoint /etc/resolv.conf &>/dev/null; then
    ln -fsv ../run/systemd/resolve/stub-resolv.conf /etc/resolv.conf || :
  fi
fi

So, it's:

* Is systemd-resolved enabled? and is there a resolved.conf file that
does not have DNSStubListener option set to off/false/no.

* If thats the case, then is there not already a /etc/resolv.conf file and
it's not a link? ie, it doesn't exist, or it's a regular file.
Then make the link.

* Else if thats not true, then see if systemd is running at all mounting
/etc/resolv.conf from somewhere else.

So, I think if you:

disable systemd-resolved 
or
make /etc/resolv.conf a real file, not a link. 
or
set 'DNSStubListener=no' in /etc/systemd/resolved.conf

It will not be replaced anymore.

kevin

[Todd Zullinger]

Kevin Fenzi wrote:
[...snip loads of useful information...]
> So, I think if you:
>
> disable systemd-resolved

To that end, the change proposal¹ when systemd-resolved was
enabled by default (F33) contains an example of how to
ensure the service remains disabled -- which would avoid the
situation Sam had where it got installed due to a dependency
chain and then started because the preset enables it.

sudo bash -c 'mkdir -p /etc/systemd/system-preset && echo "disable systemd-resolved.service" >/etc/systemd/system-preset/20-systemd-resolved-disable.preset'

I used that at the time and have not had systemd-resolved
start up on any updates or upgrades (so far and IIRC).

¹ https://fedoraproject.org/wiki/Changes/systemd-resolved#Fully_opting_out_of_...

[Sam Varshavchik]

Todd Zullinger writes:
> Kevin Fenzi wrote:
> [...snip loads of useful information...]
> > So, I think if you:
> >
> > disable systemd-resolved
>
> To that end, the change proposal¹ when systemd-resolved was
> enabled by default (F33) contains an example of how to
> ensure the service remains disabled -- which would avoid the
> situation Sam had where it got installed due to a dependency
> chain and then started because the preset enables it.

Well, if it was installed due to a dependency, then it would be reasonably  
expected that whatever declared it as a dependency required a working  
systemd-resolved. After all, why declare it as a dependency, otherwise?
> sudo bash -c 'mkdir -p /etc/systemd/system-preset && echo "disable systemd- 
> resolved.service" >/etc/systemd/system-preset/20-systemd-resolved- 
> disable.preset'
>
> I used that at the time and have not had systemd-resolved
> start up on any updates or upgrades (so far and IIRC).

And doing that would presumably break whatever pulled in systemd-resolved,  
due to a bone-fide dependency on a working systemd-resolved.

So, either something needed systemd-resolved, or not. If it did not really  
need it, it should not've been declared as a dependency. If it did, then a  
preset that disables systemd-resolved would break something.

This whole preset mechanism seems like a workaround for a functional or a  
design gap.

[Jeffrey Walton]

On Thu, May 30, 2024 at 4:33 PM Sam Varshavchik  wrote:
> Todd Zullinger writes:
> > Kevin Fenzi wrote:
> > [...snip loads of useful information...]
> > > So, I think if you:
> > >
> > > disable systemd-resolved
> >
> > To that end, the change proposal¹ when systemd-resolved was
> > enabled by default (F33) contains an example of how to
> > ensure the service remains disabled -- which would avoid the
> > situation Sam had where it got installed due to a dependency
> > chain and then started because the preset enables it.
>
> Well, if it was installed due to a dependency, then it would be reasonably
> expected that whatever declared it as a dependency required a working
> systemd-resolved. After all, why declare it as a dependency, otherwise?

I could see this behavior as a default. Folks who need
systemd-resolved (but appear to be missing it) have it automatically
installed to help keep their system running. That is, the system is
trying to help folks who are less savvy than you.

I guess what the logic or script is missing (the one Kevin detailed)
is, what to do if NetworkManager is installed and running. That seems
like where the problem occured in your case. In your case (and others
like you), it seems like the condition 'NetworkManager is installed
and running' should foreclose on installing and running
systemd-resolved.

Jeff

[Samuel Sieb]

On 5/30/24 2:12 PM, Jeffrey Walton wrote:
> I guess what the logic or script is missing (the one Kevin detailed)
> is, what to do if NetworkManager is installed and running. That seems
> like where the problem occured in your case. In your case (and others
> like you), it seems like the condition 'NetworkManager is installed
> and running' should foreclose on installing and running
> systemd-resolved.

They are not exclusive.  Most people have both running unless they've 
explicitly changed things.

[Jeffrey Walton]

On Thu, May 30, 2024 at 6:06 PM Samuel Sieb  wrote:
> On 5/30/24 2:12 PM, Jeffrey Walton wrote:
> > I guess what the logic or script is missing (the one Kevin detailed)
> > is, what to do if NetworkManager is installed and running. That seems
> > like where the problem occured in your case. In your case (and others
> > like you), it seems like the condition 'NetworkManager is installed
> > and running' should foreclose on installing and running
> > systemd-resolved.
>
> They are not exclusive.  Most people have both running unless they've
> explicitly changed things.

This seems like an unusual configuration (to me). I'm not arguing with
you; it is just my observation. You cannot serve two masters at the
same time. One has to go.

Jeff

[Samuel Sieb]

On 5/30/24 3:44 PM, Jeffrey Walton wrote:
> On Thu, May 30, 2024 at 6:06 PM Samuel Sieb  wrote:
> > On 5/30/24 2:12 PM, Jeffrey Walton wrote:
> > > I guess what the logic or script is missing (the one Kevin detailed)
> > > is, what to do if NetworkManager is installed and running. That seems
> > > like where the problem occured in your case. In your case (and others
> > > like you), it seems like the condition 'NetworkManager is installed
> > > and running' should foreclose on installing and running
> > > systemd-resolved.
> >
> > They are not exclusive.  Most people have both running unless they've
> > explicitly changed things.
>
> This seems like an unusual configuration (to me). I'm not arguing with
> you; it is just my observation. You cannot serve two masters at the
> same time. One has to go.

NetworkManager doesn't do DNS, so what do you think would be 
conflicting?  This is the default Fedora configuration.

[Sam Varshavchik]

Samuel Sieb writes:
> On 5/30/24 3:44 PM, Jeffrey Walton wrote:
> > On Thu, May 30, 2024 at 6:06 PM Samuel Sieb  wrote:
> > > On 5/30/24 2:12 PM, Jeffrey Walton wrote:
> > > > I guess what the logic or script is missing (the one Kevin detailed)
> > > > is, what to do if NetworkManager is installed and running. That seems
> > > > like where the problem occured in your case. In your case (and others
> > > > like you), it seems like the condition 'NetworkManager is installed
> > > > and running' should foreclose on installing and running
> > > > systemd-resolved.
> > >
> > > They are not exclusive.  Most people have both running unless they've
> > > explicitly changed things.
> >
> > This seems like an unusual configuration (to me). I'm not arguing with
> > you; it is just my observation. You cannot serve two masters at the
> > same time. One has to go.
>
> NetworkManager doesn't do DNS, so what do you think would be conflicting?   
> This is the default Fedora configuration.

NetworkManager installs resolv.conf, based on active network connections.  
NetworkManager itself does not DNS but it knows very well who does, and it  
updates resolv.conf accordingly.

[Mike Wright]

On 5/30/24 13:18, Todd Zullinger wrote:
> Kevin Fenzi wrote:
> [...snip loads of useful information...]
> > So, I think if you:
> >
> > disable systemd-resolved
>
> sudo bash -c 'mkdir -p /etc/systemd/system-preset && echo "disable systemd-resolved.service" >/etc/systemd/system-preset/20-systemd-resolved-disable.preset'

Here's one that doesn't eat yet another inode ( and saves some typing )

for f in disable stop mask; do sudo systemctl $f systemd-resolved; done

[Sam Varshavchik]

Kevin Fenzi writes:
> So, I think if you:
>
> disable systemd-resolved
> or
> make /etc/resolv.conf a real file, not a link.
> or
> set 'DNSStubListener=no' in /etc/systemd/resolved.conf
>
> It will not be replaced anymore.

I did not even /have/ systemd-resolved installed.

dnf system-upgrade installed it on its own, enabled it, and /etc/resolv.conf  
was replaced as a result.

[Kevin Fenzi]

On Thu, May 30, 2024 at 04:25:44PM GMT, Sam Varshavchik wrote:
> Kevin Fenzi writes:
> > So, I think if you:
> >
> > disable systemd-resolved
> > or
> > make /etc/resolv.conf a real file, not a link.
> > or
> > set 'DNSStubListener=no' in /etc/systemd/resolved.conf
> >
> > It will not be replaced anymore.
>
> I did not even /have/ systemd-resolved installed.
>
> dnf system-upgrade installed it on its own, enabled it, and /etc/resolv.conf
> was replaced as a result.

Interesting. So, the only thing I see that requires systemd-resolved is
anaconda-core. However, systemd itself has a 'recommends' for it, so
perhaps thats what pulled it back in?

So, possibly keeping it installed, but masked is a better way to make
sure it never gets enabled?

kevin

[Sam Varshavchik]

Kevin Fenzi writes:
> On Thu, May 30, 2024 at 04:25:44PM GMT, Sam Varshavchik wrote:
> > Kevin Fenzi writes:
> > > So, I think if you:
> > >
> > > disable systemd-resolved
> > > or
> > > make /etc/resolv.conf a real file, not a link.
> > > or
> > > set 'DNSStubListener=no' in /etc/systemd/resolved.conf
> > >
> > > It will not be replaced anymore.
> >
> > I did not even /have/ systemd-resolved installed.
> >
> > dnf system-upgrade installed it on its own, enabled it, and
>
> /etc/resolv.conf
> > was replaced as a result.
>
> Interesting. So, the only thing I see that requires systemd-resolved is
> anaconda-core. However, systemd itself has a 'recommends' for it, so
> perhaps thats what pulled it back in?

That's probably it. Sifting through /var/log/dnf.log: this was removed  
together with systemd-resolved, when I removed it.

But this is a red herring. It was anaconda itself that was also pulled in as  
a dependency.

Installing dependencies:
 anaconda                                      x86_64  40.22.3-1.fc40                      fedora                   102 k
 anaconda-core                                 x86_64  40.22.3-1.fc40                      fedora                   2.8 M
 anaconda-gui                                  x86_64  40.22.3-1.fc40                      fedora                   562 k
 anaconda-tui                                  x86_64  40.22.3-1.fc40                      fedora                   228 k
> So, possibly keeping it installed, but masked is a better way to make
> sure it never gets enabled?

Masking is just solving the X of an XY problem. Either the dependencies  
being masked are not needed, or if they are needed masking would break  
something.

[Felix Miata]

Sam Varshavchik composed on 2024-05-31 12:13 (UTC-0400):
> Masking is just solving the X of an XY problem. Either the dependencies  
> being masked are not needed, or if they are needed masking would break  
> something.

I like KISS:

# inxi -Sz
System:
  Kernel: 6.8.11-300.fc40.x86_64 arch: x86_64 bits: 64
  Console: pty pts/0 Distro: Fedora Linux 40 (Forty)
# rpm -qa | egrep 'solv|netw|nage|wicd' | sort
libsemanage-3.6-3.fc40.x86_64
libsolv-0.7.29-1.fc40.x86_64
systemd-networkd-255.7-1.fc40.x86_64
# systemctl list-unit-files | egrep 'net|solv|nage'
display-manager.service                      alias           -
systemd-network-generator.service            disabled        enabled
systemd-networkd-wait-online.service         disabled        disabled
systemd-networkd-wait-online@.service        disabled        disabled
systemd-networkd.service                     disabled        disabled
systemd-networkd.socket                      enabled         disabled
network-online.target                        static          -
network-pre.target                           static          -
network.target                               static          -
# [ -f  /etc/resolv.conf ] && lsattr /etc/resolv.conf
----i---------e------- /etc/resolv.conf
# ip a show eth0 | grep inet
    inet 192.168.<redact>/24 brd 192.168.<redact> scope global eth0
#

[Tim]

On Fri, 2024-05-31 at 08:53 -0700, Kevin Fenzi wrote:
> Interesting. So, the only thing I see that requires systemd-resolved
> is anaconda-core. However, systemd itself has a 'recommends' for it,
> so perhaps thats what pulled it back in?
>
> So, possibly keeping it installed, but masked is a better way to make
> sure it never gets enabled?

Is Anaconda more than just the OS installer, now?  Is it needed post-
install?

[Kevin Fenzi]

On Sat, Jun 01, 2024 at 06:07:24AM GMT, Tim via users wrote:
> On Fri, 2024-05-31 at 08:53 -0700, Kevin Fenzi wrote:
> > Interesting. So, the only thing I see that requires systemd-resolved
> > is anaconda-core. However, systemd itself has a 'recommends' for it,
> > so perhaps thats what pulled it back in?
> >
> > So, possibly keeping it installed, but masked is a better way to make
> > sure it never gets enabled?
>
> Is Anaconda more than just the OS installer, now?  Is it needed post-
> install?

No, it's not needed. It's somewhat of a historical artifact of the way
some installs work that it's there. There was some talk about it
removing itself at the end, but I am not sure where that ended up.

kevin

[Tim]

Tim:
> > Is Anaconda more than just the OS installer, now?  Is it needed post-
> > install?

Kevin Fenzi:
> No, it's not needed. It's somewhat of a historical artifact of the way
> some installs work that it's there. There was some talk about it
> removing itself at the end, but I am not sure where that ended up.

I used to remove it, wasn't sure whether I could still do that.

I see the value in *some* people keeping it, if *they* were working on
improving it.  But maybe that should be a testing repo thing.

[Sam Varshavchik]

Tim via users writes:
> Tim:
> > > Is Anaconda more than just the OS installer, now?  Is it needed post-
> > > install?
>
> Kevin Fenzi:
> > No, it's not needed. It's somewhat of a historical artifact of the way
> > some installs work that it's there. There was some talk about it
> > removing itself at the end, but I am not sure where that ended up.
>
> I used to remove it, wasn't sure whether I could still do that.

dnf remove systemd-resolved also removed it, without any issues.

[Jeffrey Walton]

On Mon, Jun 3, 2024 at 6:42 AM Sam Varshavchik  wrote:
> Tim via users writes:
> > Tim:
> > > > Is Anaconda more than just the OS installer, now?  Is it needed post-
> > > > install?
> >
> > Kevin Fenzi:
> > > No, it's not needed. It's somewhat of a historical artifact of the way
> > > some installs work that it's there. There was some talk about it
> > > removing itself at the end, but I am not sure where that ended up.
> >
> > I used to remove it, wasn't sure whether I could still do that.
>
> dnf remove systemd-resolved also removed it, without any issues.

Didn't a simple removal cause the problems you are encountering?

[Sam Varshavchik]

Jeffrey Walton writes:
> On Mon, Jun 3, 2024 at 6:42 AM Sam Varshavchik  wrote:
> > Tim via users writes:
> > > Tim:
> > > > > Is Anaconda more than just the OS installer, now?  Is it needed post-
> > > > > install?
> > >
> > > Kevin Fenzi:
> > > > No, it's not needed. It's somewhat of a historical artifact of the way
> > > > some installs work that it's there. There was some talk about it
> > > > removing itself at the end, but I am not sure where that ended up.
> > >
> > > I used to remove it, wasn't sure whether I could still do that.
> >
> > dnf remove systemd-resolved also removed it, without any issues.
>
> Didn't a simple removal cause the problems you are encountering?

Nope, quite the opposite: it was a simple addition. A dnf system-upgrade  
pulled in anaconda. Which pulled in system-resolved. Which overwrote  
/etc/resolv.conf

Uninstalling the whole mess actually mostly restored the symlink back to  
NetworkManager. Except that it was no-stub-resolv.conf originally. Which has  
the wrong selinux context, in F40. So, we're not out of the woods, just yet…