Manual intervention required: broken /etc/nsswitch.conf and /etc/resolv.conf for F33 early adopters

[Michael Catanzaro]

Hi,

I've you've installed or upgraded to Fedora 33 (or to rawhide) prior to 
today, your /etc/nsswitch and /etc/resolv.conf are probably in a broken 
state that requires manual intervention to resolve. This has caused 
breakage for mDNS and VPN users [1][2][3]. Apologies for this breakage. 
Anyone installing with current nightly images or upgrading as of today 
should be OK, so users installing F33 beta or upgrading from F32 to F33 
beta will be unaffected.

To fix /etc/nsswitch.conf, edit the hosts line in 
/etc/authselect/user-nsswitch.conf to look like this:

hosts:      files mdns4_minimal [NOTFOUND=return] resolve 
[!UNAVAIL=return] myhostname dns

Then run:

# authselect apply-changes

Then, fix /etc/resolv.conf:

# rm /etc/resolv.conf
# ln -s ../run/systemd/resolve/stub-resolv.conf

And restart NetworkManager:

# systemctl restart NetworkManager.service

Michael

[1] https://bugzilla.redhat.com/show_bug.cgi?id=1873856
[2] https://bugzilla.redhat.com/show_bug.cgi?id=1867830
[3] https://bugzilla.redhat.com/show_bug.cgi?id=1863041

[Michael Catanzaro]

On Wed, Sep 9, 2020 at 2:08 pm, Michael Catanzaro 
<mcatanzaro(a)gnome.org&gt; wrote:
>  Then, fix /etc/resolv.conf:
>  
>  # rm /etc/resolv.conf
>  # ln -s ../run/systemd/resolve/stub-resolv.conf 
Sigh, sorry. Fixed instructions:

# rm /etc/resolv.conf
# ln -s ../run/systemd/resolve/stub-resolv.conf /etc/resolv.conf

It should look like this:

# ls -l /etc | grep resolv
lrwxrwxrwx. 1 root root 39 Sep 9 14:13 resolv.conf -> 
../run/systemd/resolve/stub-resolv.conf

(Unless you intentionally want to configure DNS differently and are OK 
with having a non-default setup.)

Michael

[Mikhail Gavrilov]

# authselect apply-changes
[error] [/etc/authselect/nsswitch.conf] has unexpected content!
[error] Unexpected changes to the configuration were detected.
[error] Refusing to activate profile unless those changes are removed or overwrite is
requested.
Some unexpected changes to the configuration were detected. Use 'select' command
instead.

# cat /etc/resolv.conf
# This file is managed by man:systemd-resolved(8). Do not edit.
#
# This is a dynamic resolv.conf file for connecting local clients to the
# internal DNS stub resolver of systemd-resolved. This file lists all
# configured search domains.
#
# Run "resolvectl status" to see details about the uplink DNS servers
# currently in use.
#
# Third party programs should typically not access this file directly, but only
# through the symlink at /etc/resolv.conf. To manage man:resolv.conf(5) in a
# different way, replace this symlink by a static file or a different symlink.
#
# See man:systemd-resolved.service(8) for details about the supported modes of
# operation for /etc/resolv.conf.

nameserver 127.0.0.53
options edns0 trust-ad

# ls -l /etc | grep resolv
lrwxrwxrwx.  1 root root        39 Sep 10 10:56 resolv.conf ->
../run/systemd/resolve/stub-resolv.conf
-rw-r--r--.  1 root root        76 Aug 11 06:15 resolv.conf.orig-with-nm

And after creating a fresh VPN connection via NetworkManager name
resolving not working again.

# ping git.sbis.ru
ping: git.sbis.ru: Name or service not known

$ resolvectl domain
Global:
Link 2 (enp5s0): ~.
Link 3 (wlp4s0):
Link 4 (virbr0):
Link 5 (virbr0-nic):
Link 7 (vpn0):

[Kamil Paral]

On Thu, Sep 10, 2020 at 8:54 AM Mikhail Gavrilov <
mikhail.v.gavrilov(a)gmail.com&gt; wrote:

>  # authselect apply-changes
>  [error] [/etc/authselect/nsswitch.conf] has unexpected content!
>  [error] Unexpected changes to the configuration were detected.
>  [error] Refusing to activate profile unless those changes are removed or
>  overwrite is requested.
>  Some unexpected changes to the configuration were detected. Use 'select'
>  command instead.
>  
I see the same error.

[Michael Catanzaro]

On Thu, Sep 10, 2020 at 9:36 am, Kamil Paral <kparal(a)redhat.com&gt; wrote:
>  On Thu, Sep 10, 2020 at 8:54 AM Mikhail Gavrilov 
>  <mikhail.v.gavrilov(a)gmail.com&gt; wrote:
> > # authselect apply-changes
> >  [error] [/etc/authselect/nsswitch.conf] has unexpected content!
> >  [error] Unexpected changes to the configuration were detected.
> >  [error] Refusing to activate profile unless those changes are 
> > removed or overwrite is requested.
> >  Some unexpected changes to the configuration were detected. Use 
> > 'select' command instead.
>  
>  I see the same error. 
I'm a bit concerned that two different people are seeing this. I don't 
think we have any scriptlets tha writes to /etc/nsswitch.conf or 
/etc/authselect/nsswitch.conf on its own. But maybe, for non-live 
installs, that could happen if the systemd RPM gets installed before 
the authselect RPM? Then systemd would think /etc/nsswitch.conf is not 
managed by authselect. Hm....

Anyway, if you can find a way to get to this state from a clean 
install, without touching those files manually, then please do report a 
bug. That shouldn't be happening.

[Zdenek Kabelac]

Dne 10. 09. 20 v 17:53 Michael Catanzaro napsal(a):
>  On Thu, Sep 10, 2020 at 9:36 am, Kamil Paral
> <kparal(a)redhat.com&gt; wrote:
> > On Thu, Sep 10, 2020 at 8:54 AM Mikhail Gavrilov 
> > <mikhail.v.gavrilov(a)gmail.com&gt; wrote:
> >> # authselect apply-changes
> >>  [error] [/etc/authselect/nsswitch.conf] has unexpected content!
> >>  [error] Unexpected changes to the configuration were detected.
> >>  [error] Refusing to activate profile unless those changes are removed or 
> >> overwrite is requested.
> >>  Some unexpected changes to the configuration were detected. Use 'select'
>
> >> command instead.
> >
> > I see the same error.
>  
>  I'm a bit concerned that two different people are seeing this. I don't think 
Well count me as 3rd. - I've seen this to happen on upgrade from fc32 rawhide
to fc34 rawhide.

It's been basically horrible experience reported here:

https://bugzilla.redhat.com/show_bug.cgi?id=1875000

so I've originally accounted zero length nsswitch.conf issue to this problem,
but it most like relates to this thread (and yeah - it took me a while
to figure out how to make networking DNS running properly again,
but nothing compered with restoration of crashed DNF transaction...

Regards

Zdenek

[Christopher Engelhard]

On 10.09.20 17:53, Michael Catanzaro wrote:
>  I'm a bit concerned that two different people are seeing this. I
> don't
>  think we have any scriptlets tha writes to /etc/nsswitch.conf or
>  /etc/authselect/nsswitch.conf on its own. But maybe, for non-live
>  installs, that could happen if the systemd RPM gets installed before the
>  authselect RPM? Then systemd would think /etc/nsswitch.conf is not
>  managed by authselect. Hm....
>  
>  Anyway, if you can find a way to get to this state from a clean install,
>  without touching those files manually, then please do report a bug. That
>  shouldn't be happening. 
I have this same issue on a system that was installed as F33 rawhide &
is now F34 rawhide. I never touched those files, nor did I do anything
that should have caused anything else to change them.

I'll file a bug - would this be against authselect or  systemd?

Christopher

[Michael Catanzaro]

On Thu, Sep 10, 2020 at 8:02 pm, Christopher Engelhard <ce(a)lcts.de&gt; 
wrote:
>  I'll file a bug - would this be against authselect or  systemd?
I would start with authselect, even if it might turn out to be a 
systemd RPM issue.

That said, it would be really helpful if someone could find a way to 
reproduce this, since otherwise we won't know how to fix whatever has 
gone wrong.

Michael

[Kamil Paral]

On Thu, Sep 10, 2020 at 5:54 PM Michael Catanzaro <mcatanzaro(a)gnome.org&gt;
wrote:

>  On Thu, Sep 10, 2020 at 9:36 am, Kamil Paral
> <kparal(a)redhat.com&gt; wrote:
>  > On Thu, Sep 10, 2020 at 8:54 AM Mikhail Gavrilov
>  > <mikhail.v.gavrilov(a)gmail.com&gt; wrote:
>  >> # authselect apply-changes
>  >>  [error] [/etc/authselect/nsswitch.conf] has unexpected content!
>  >>  [error] Unexpected changes to the configuration were detected.
>  >>  [error] Refusing to activate profile unless those changes are
>  >> removed or overwrite is requested.
>  >>  Some unexpected changes to the configuration were detected. Use
>  >> 'select' command instead.
>  >
>  > I see the same error.
>
>  I'm a bit concerned that two different people are seeing this. I don't
>  think we have any scriptlets tha writes to /etc/nsswitch.conf or
>  /etc/authselect/nsswitch.conf on its own. But maybe, for non-live
>  installs, that could happen if the systemd RPM gets installed before
>  the authselect RPM? Then systemd would think /etc/nsswitch.conf is not
>  managed by authselect. Hm....
>
>  Anyway, if you can find a way to get to this state from a clean
>  install, without touching those files manually, then please do report a
>  bug. That shouldn't be happening.
>  
I did edit /etc/nsswitch.conf manually, because obviously I needed a
working system :) The confusing part here is that the error message claims
that **/etc/authselect/nsswitch.conf** has unexpected content. So this
doesn't seem to be related to /etc/nsswitch.conf having been edited by
hand. Is it? If it is, what is the proper way to revert back to upstream
defaults in this case? Should I append "--force" to the command? (I don't
want to destroy my system, that's why I'm not simply trying it. All of this
seems awfully complex and fragile).

[Michael Catanzaro]

On Fri, Sep 11, 2020 at 10:21 am, Kamil Paral <kparal(a)redhat.com&gt; wrote:
>  I did edit /etc/nsswitch.conf manually, because obviously I needed a
>
>  working system :) The confusing part here is that the error message 
>  claims that **/etc/authselect/nsswitch.conf** has unexpected content. 
>  So this doesn't seem to be related to /etc/nsswitch.conf having been 
>  edited by hand. Is it? 
I think it's probably related. It's unexpected for it to not match your 
/etc/nsswitch.conf.

>  If it is, what is the proper way to revert back to upstream defaults
>
>  in this case? Should I append "--force" to the command? (I don't want 
>  to destroy my system, that's why I'm not simply trying it. All of 
>  this seems awfully complex and fragile). 
Yes, use --force. That tells authselect that it is OK to overwrite your 
manual configuration. Otherwise, authselect is careful to not overwrite 
files that have been edited manually.

[Kamil Paral]

On Fri, Sep 11, 2020 at 7:03 PM Michael Catanzaro <mcatanzaro(a)gnome.org&gt;
wrote:

>  > If it is, what is the proper way to revert back to upstream defaults
>  > in this case? Should I append "--force" to the command? (I don't want
>  > to destroy my system, that's why I'm not simply trying it. All of
>  > this seems awfully complex and fragile).
>
>  Yes, use --force. That tells authselect that it is OK to overwrite your
>  manual configuration. Otherwise, authselect is careful to not overwrite
>  files that have been edited manually.
>  
Sigh, authselect is even more complicated than I thought. I can't simply do
"sudo authselect apply-changes --force". I need to do "sudo authselect
select $something --force" first, according to the man page. And I have no
idea what $something is. Can you please provide an exact set of commands to
run? Thanks.

[Pavel Březina]

On 9/14/20 12:11 PM, Kamil Paral wrote:
>  On Fri, Sep 11, 2020 at 7:03 PM Michael Catanzaro
> <mcatanzaro(a)gnome.org 
>  <mailto:[e-mail]>> wrote:
>  
>  
>       > If it is, what is the proper way to revert back to upstream defaults
>       > in this case? Should I append "--force" to the command? (I don't
>      want
>       > to destroy my system, that's why I'm not simply trying it. All of
>       > this seems awfully complex and fragile).
>  
>      Yes, use --force. That tells authselect that it is OK to overwrite your
>      manual configuration. Otherwise, authselect is careful to not overwrite
>      files that have been edited manually.
>  
>  
>  Sigh, authselect is even more complicated than I thought. I can't simply 
>  do "sudo authselect apply-changes --force". I need to do "sudo 
>  authselect select $something --force" first, according to the man page. 
>  And I have no idea what $something is. Can you please provide an exact 
>  set of commands to run? Thanks. 
apply-changes only works if you have valid authselect configuration (no 
manual changes) and you edit /etc/authselect/user-nsswitch.conf or one 
of the selected profile template.

If you need to restore previous configuration you can either use 
backup-restore (if there is any backup, see backup-list) or you can use 
'authselect current --raw' which will print you what authselect command 
line was used.

authselect select `authselect current --raw` --force

The default in Fedora is 'authselect select sssd' with optional 
with-fingerprint and with-mkhomedir.

>  
>  
>  _______________________________________________
>  devel mailing list -- devel(a)lists.fedoraproject.org
>  To unsubscribe send an email to devel-leave(a)lists.fedoraproject.org
>  Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
>  List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
>  List Archives:
> https://lists.fedoraproject.org/archives/list/[e-mail]
>

[Kamil Paral]

On Mon, Sep 14, 2020 at 12:37 PM Pavel Březina <pbrezina(a)redhat.com&gt; wrote:

>  apply-changes only works if you have valid authselect configuration
> (no
>  manual changes) and you edit /etc/authselect/user-nsswitch.conf or one
>  of the selected profile template.
>
>  If you need to restore previous configuration you can either use
>  backup-restore (if there is any backup, see backup-list) or you can use
>  'authselect current --raw' which will print you what authselect command
>  line was used.
>
>  authselect select `authselect current --raw` --force
>  
Thanks, that seems to have worked.

[Thomas Haller]

On Thu, 2020-09-10 at 06:53 +0000, Mikhail Gavrilov wrote:
>  And after creating a fresh VPN connection via NetworkManager name
>  resolving not working again.
>  
>  # ping git.sbis.ru
>  ping: git.sbis.ru: Name or service not known
>  
>  $ resolvectl domain
>  Global:
>  Link 2 (enp5s0): ~.
>  Link 3 (wlp4s0):
>  Link 4 (virbr0):
>  Link 5 (virbr0-nic):
>  Link 7 (vpn0): 

Hi,


there isn't much information provided here, so I can only guess what's
wrong.


1) the VPN profile does not get the default route (has `ipv4.never-
default=yes).

2) the VPN profile does not specify any search domains, and neither
does the VPN server push any search domains.


If the VPN profile would get a default route (contrary to 1), then
NetworkManager would automatically add search domain "~.". But as that
doesn't happen, the VPN is not considered for resolving any domains.

Fix your configuration to either:

- let your VPN server announce the domains that it should use.

- explicilty configure the desired search domains on the VPN. In the
simpliest case just

 $ nmcli connection modify "$VPN_PROFILE" +ipv4.dns-search '~.'

"~." is a routeing-only search domain for everything.


That's what Beniamino already said at 
https://bugzilla.redhat.com/show_bug.cgi?id=1863041#c33



best,
Thomas

[Mikhail Gavrilov]

From here https://bugzilla.redhat.com/show_bug.cgi?id=1863041#c46 I have expected what
newly-created connection would work properly without manually changing ipv4.dns-search to
~. on the specific VPN connection.

[Thomas Haller]

On Thu, 2020-09-10 at 10:28 +0000, Mikhail Gavrilov wrote:
>  From here https://bugzilla.redhat.com/show_bug.cgi?id=1863041#c46 I
>  have expected what newly-created connection would work properly
>  without manually changing ipv4.dns-search to ~. on the specific VPN
>  connection. 
Hi,


I think you need

   nmcli connection modify "$VPN_PROFILE" +ipv4.dns-search "~."


It doesn't matter whether you newly create a profile. What only matters
are the settings (the content) of the connection profile, as you see it
with `nmcli connection show "$PROFILE"`. Now how you created it.


You have a wrong configuration ("wrong" least with respect to how
NetworkManager currently behaves):

  - with split DNS enabled
  - the VPN has DNS servers configured (either manually or pushed by 
    server).
  - a VPN profile that has no search domains (neither manually nor 
    pushed by server)
  - the VPN is not configured to route all traffic.

Consequently, that DNS server isn't gonna get used.


There is a possibility that NetworkManager could improve to
automatically add the search domain "~." in such cases. But until that
is happens, you have to adjust your connection profile (or your VPN
server to announce the proper search domain).
https://bugzilla.redhat.com/show_bug.cgi?id=1863041#c49


Without systemd-resolved (without split DNS support), NetworkManager
behaves differently because it configures all DNS servers in
/etc/resolv.conf -- regardless of the search domains. That's why
switching to systemd-resolved breaks your previously working setup. In
the end, the behavior is different whether split-DNS or not is enabled,
so this might just be expected, albeit it's very unfortunate to break
previously working setups.



best,
Thomas

[Vít Ondruch]

Hi Michael,

Could you please provide more details? This is content of my nsswitch.conf:


~~~

$ grep mdns4_minimal /etc/authselect/user-nsswitch.conf
hosts:      files mdns4_minimal [NOTFOUND=return] dns myhostname
~~~


How that happened? From what version of what package it happened? Why
should I do some changes manually and they are not handled automatically?


Thx


Vít


Dne 09. 09. 20 v 21:08 Michael Catanzaro napsal(a):
>  Hi,
>
>  I've you've installed or upgraded to Fedora 33 (or to rawhide) prior
>  to today, your /etc/nsswitch and /etc/resolv.conf are probably in a
>  broken state that requires manual intervention to resolve. This has
>  caused breakage for mDNS and VPN users [1][2][3]. Apologies for this
>  breakage. Anyone installing with current nightly images or upgrading
>  as of today should be OK, so users installing F33 beta or upgrading
>  from F32 to F33 beta will be unaffected.
>
>  To fix /etc/nsswitch.conf, edit the hosts line in
>  /etc/authselect/user-nsswitch.conf to look like this:
>
>  hosts:      files mdns4_minimal [NOTFOUND=return] resolve
>  [!UNAVAIL=return] myhostname dns
>
>  Then run:
>
>  # authselect apply-changes
>
>  Then, fix /etc/resolv.conf:
>
>  # rm /etc/resolv.conf
>  # ln -s ../run/systemd/resolve/stub-resolv.conf
>
>  And restart NetworkManager:
>
>  # systemctl restart NetworkManager.service
>
>  Michael
>
>  [1] https://bugzilla.redhat.com/show_bug.cgi?id=1873856
>  [2] https://bugzilla.redhat.com/show_bug.cgi?id=1867830
>  [3] https://bugzilla.redhat.com/show_bug.cgi?id=1863041
>
>  _______________________________________________
>  devel mailing list -- devel(a)lists.fedoraproject.org
>  To unsubscribe send an email to devel-leave(a)lists.fedoraproject.org
>  Fedora Code of Conduct:
>  https://docs.fedoraproject.org/en-US/project/code-of-conduct/
>  List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
>  List Archives:
>  https://lists.fedoraproject.org/archives/list/[e-mail]

[Michael Catanzaro]

On Thu, Sep 10, 2020 at 9:24 am, Vít Ondruch <vondruch(a)redhat.com&gt; 
wrote:
>  Hi Michael,
>  
>  Could you please provide more details? This is content of my 
>  nsswitch.conf:
>  
>  
>  ~~~
>  
>  $ grep mdns4_minimal /etc/authselect/user-nsswitch.conf
>  hosts:      files mdns4_minimal [NOTFOUND=return] dns myhostname
>  ~~~
>  
>  
>  How that happened? From what version of what package it happened? Why
>  should I do some changes manually and they are not handled 
>  automatically? 
You're completely missing resolved from your hosts line, so you'll get 
legacy name resolution behavior via nss-dns (which is what Ubuntu does).

There were multiple different bugs here, so it's a bit hard to keep 
track of them all. systemd package was being too cautious about 
deleting /etc/resolv.conf, so some users wound up with /etc/resolv.conf 
managed by NetworkManager even though systemd is running, which was not 
what I had intended. Plus we originally planned for resolved to handle 
mDNS resolution instead of avahi, but we discovered that it doesn't 
work as expected. We have hacky scripts in the systemd and nss-mdns 
packages to manage the hosts line, plus it's managed by authselect 
itself. They all have to agree on expected configuration and it's all a 
bit fragile. The systemd package script is careful to only touch this 
line if the order matches the previous Fedora default configuration, so 
we have to choose between automatically reordering the nss modules to 
fix this for users of F33 pre-beta, vs. clobbering intentional 
configuration changes for users who actually wanted the hosts line to 
be different. If this had reached stable releases, then we'd really 
have to do that clobbering, but it didn't, so seems best to be more 
conservative here. These scripts would not be needed if we could add 
systemd nss modules and nss-mdns to the nsswitch.conf provided by the 
glibc package.

Michael

[Vít Ondruch]

Dne 10. 09. 20 v 17:32 Michael Catanzaro napsal(a):
>  On Thu, Sep 10, 2020 at 9:24 am, Vít Ondruch <vondruch(a)redhat.com&gt; wrote:
> > Hi Michael,
> >
> > Could you please provide more details? This is content of my
> > nsswitch.conf:
> >
> >
> > ~~~
> >
> > $ grep mdns4_minimal /etc/authselect/user-nsswitch.conf
> > hosts:      files mdns4_minimal [NOTFOUND=return] dns myhostname
> > ~~~
> >
> >
> > How that happened? From what version of what package it happened? Why
> > should I do some changes manually and they are not handled
> > automatically?
>
>  You're completely missing resolved from your hosts line, so you'll get
>  legacy name resolution behavior via nss-dns (which is what Ubuntu does).
>
>  There were multiple different bugs here, so it's a bit hard to keep
>  track of them all. systemd package was being too cautious about
>  deleting /etc/resolv.conf, so some users wound up with
>  /etc/resolv.conf managed by NetworkManager even though systemd is
>  running, which was not what I had intended. Plus we originally planned
>  for resolved to handle mDNS resolution instead of avahi, but we
>  discovered that it doesn't work as expected. We have hacky scripts in
>  the systemd and nss-mdns packages to manage the hosts line, plus it's
>  managed by authselect itself. They all have to agree on expected
>  configuration and it's all a bit fragile. The systemd package script
>  is careful to only touch this line if the order matches the previous
>  Fedora default configuration, so we have to choose between
>  automatically reordering the nss modules to fix this for users of F33
>  pre-beta, vs. clobbering intentional configuration changes for users
>  who actually wanted the hosts line to be different. If this had
>  reached stable releases, then we'd really have to do that clobbering,
>  but it didn't, so seems best to be more conservative here. These
>  scripts would not be needed if we could add systemd nss modules and
>  nss-mdns to the nsswitch.conf provided by the glibc package. 

Thank you for the details.

And now I wonder, is there a way to restore the pristine state of these
files without touching them directly (with exception of deleting them)?
E.g.:

~~~

$ rm /etc/authselect/user-nsswitch.conf

$ sudo dnf reinstall authselect

~~~

Also I don't understand why I have files on my system which are not
owned by anything:

~~~

$ rpm -qf /etc/authselect/user-nsswitch.conf
file /etc/authselect/user-nsswitch.conf is not owned by any package

~~~

How are these files created for fresh system?

And also, is there long term vision to prevent issues, that nobody
really knows who edited which configuration file? I understand that
there is some legacy, multiple projects involved, etc. but I don't want
to care about random configuration files in /etc.


Vít