The latest systemd update broke ssh. Specifically, "ssh hostname", where hostname was not a FQDN no longer worked, and I had to use "ssh hostname.domain", for my LAN.
Of course, /etc/resolv.conf was pointing to systemd's resolver, and features:
search .
this was despite my DHCP server offering the LAN's domain name properly.
I don't remember if I fixed this laptop previously, or it was using systemd's resolver for a while. I just symlinked /etc/resolv.conf to /run/NetworkManager/no-stub-resolv.conf, which had the correct DNS search string, as always.
This seemed to have fixed DNS lookups, per se:
$ host jack jack.email-scan.com has address 192.168.0.6
But ssh was still broken, and failing.
An strace of ssh revealed that it still insists on talking to systemd.
connect(3, {sa_family=AF_UNIX, sun_path="/run/systemd/resolve/io.systemd.Resolve"}, 42) = 0 sendto(3, "{"method":"io.systemd.Resolve.ResolveHostname","parameters": {"name":"jack","flags":0}}\0", 87, MSG_DONTWAIT|MSG_NOSIGNAL, NULL, 0) = 87 mmap(NULL, 135168, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f103ff5c000
I don't see where it's picking this up. The only thing I can think of is nsswitch.conf, but the "hosts" entry there is identical to what's on another, un-upgraded system.
I don't see where it's picking this up. The only thing I can think of is nsswitch.conf, but the "hosts" entry there is identical to what's on another, un-upgraded system.
Does anyone know how to fumigate systemd out of ssh?
On 4/17/22 12:26, Sam Varshavchik wrote:
I don't see where it's picking this up. The only thing I can think of is nsswitch.conf, but the "hosts" entry there is identical to what's on another, un-upgraded system.
Maybe remove the "resolve" entry from the hosts line:
$ rpm -qf /lib64/libnss_resolve* systemd-libs-250.3-8.fc36.x86_64
Gordon Messmer writes:
On 4/17/22 12:26, Sam Varshavchik wrote:
I don't see where it's picking this up. The only thing I can think of is nsswitch.conf, but the "hosts" entry there is identical to what's on another, un-upgraded system.
Maybe remove the "resolve" entry from the hosts line:
$ rpm -qf /lib64/libnss_resolve* systemd-libs-250.3-8.fc36.x86_64
yeah. systemd's injection has two prongs: via nsswitch and via resolv.conf. Apparently it's not enough to let glibc's default resolver read whatever's in /etc/resolv.conf and have systemd come in that door, it's also necessary to inject it via nsswitch.conf's back door.
Apparently it's still been doing DNS resolution for host lookups, but it was mostly working, so stayed under the radar, until the recent update broke it. Additionally, the systemd-resolved rpm is actively hijacking /etc/resolv.conf. From the package's scriptlet:
if systemctl -q is-enabled systemd-resolved.service &>/dev/null && ! systemd-analyze cat-config systemd/resolved.conf 2>/dev/null | grep -iqE '^DNSStubListener\s*=\s*(no?|false|0|off)\s*$'; then
if ! test -e /etc/resolv.conf && ! test -L /etc/resolv.conf; then ln -sv ../run/systemd/resolve/stub-resolv.conf /etc/resolv.conf || : elif test -d /run/systemd/system/ && ! mountpoint /etc/resolv.conf &>/dev/null; then ln -fsv ../run/systemd/resolve/stub-resolv.conf /etc/resolv.conf || : fi fi
The preceding comment explicitly states that the purpose of this is to reset the symlink back to systemd when systemd-resolved gets updated, if it's installed. Awesome.
On Sun, Apr 17, 2022 at 06:34:48PM -0400, Sam Varshavchik wrote:
but it was mostly working, so stayed under the radar, until the recent update broke it. Additionally, the systemd-resolved rpm is actively hijacking /etc/resolv.conf. From the package's scriptlet:
Sam, can you _please_ not use such colorfully-negative lagnguage here?
None of this is "infestation", "hijacking", or something to "fumigate". This is all from an approved Change for Fedora Linux 33, which you can read here: https://fedoraproject.org/wiki/Changes/systemd-resolved, and which was documented in the release notes at https://docs.fedoraproject.org/en-US/fedora/f33/release-notes/sysadmin/Netwo....
You're welcome to disagree with the technical decision (although _elaborate_ disagreement isn't very helpful without an alternative plan and someone to implement it!). But, as you can see from the "Benefit to Fedora" section of the proposal, this was done by people actively working to make Fedora Linux better for users, and the Fedora Engineering Steering Committee weighed that benefit along with considerable feedback in making the decision.
So, all of this language which implies pestilence or some kind of illicit undertaking is wildly out of line.
On Mon, 18 Apr 2022 15:29:01 -0400 Matthew Miller mattdm@fedoraproject.org wrote:
On Sun, Apr 17, 2022 at 06:34:48PM -0400, Sam Varshavchik wrote:
but it was mostly working, so stayed under the radar, until the recent update broke it. Additionally, the systemd-resolved rpm is actively hijacking /etc/resolv.conf. From the package's scriptlet:
Sam, can you _please_ not use such colorfully-negative lagnguage here?
None of this is "infestation", "hijacking", or something to "fumigate". This is all from an approved Change for Fedora Linux 33, which you can read here: https://fedoraproject.org/wiki/Changes/systemd-resolved, and which was documented in the release notes at https://docs.fedoraproject.org/en-US/fedora/f33/release-notes/sysadmin/Netwo....
You're welcome to disagree with the technical decision (although _elaborate_ disagreement isn't very helpful without an alternative plan and someone to implement it!). But, as you can see from the "Benefit to Fedora" section of the proposal, this was done by people actively working to make Fedora Linux better for users, and the Fedora Engineering Steering Committee weighed that benefit along with considerable feedback in making the decision.
So, all of this language which implies pestilence or some kind of illicit undertaking is wildly out of line.
Better for users? Are you kidding? All this systemd is reinventing the wheel. Go back to Windows, when you grow up.
Without BR, Bob
Tom Horsley wrote:
On Mon, 18 Apr 2022 23:35:35 +0200 Bob Marcan wrote:
Better for users? Are you kidding? All this systemd is reinventing the wheel. Go back to Windows, when you grow up.
Nah! It is the first computer fungus, slowly growing over all the rest of Linux.
With a very grudging list admin hat on...
Before anyone feels the need to pile onto this thread, please carefully read the Fedora Code of Conduct and list guidelines:
Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
You are welcome to express yourself, but it is not at all unreasonable that you do so within the very loose rules outlined in the CoC and other list guidelines.
On Mon, 18 Apr 2022, Todd Zullinger wrote:
Tom Horsley wrote:
Nah! It is the first computer fungus, slowly growing over all the rest of Linux.
With a very grudging list admin hat on...
Before anyone feels the need to pile onto this thread, please carefully read the Fedora Code of Conduct and list guidelines:
TH was making fun of code, not the person who is just trying to figure out how to live with it.
Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
Does making fun of *code* violate either?
Said code reminds me a bit of evercookies and of a problem some people have with their phones: they have to redo the settings after every "update". The updates occur wothout warning or notice and the settings include privacy settings.
You are welcome to express yourself, but it is not at all unreasonable that you do so within the very loose rules outlined in the CoC and other list guidelines.
The term fungus might have been chosen just for humor, but it might also have been an especially accurate choice: Fungal infections are notoriously difficult to get rid of. Antibiotics do not work on them. There is no vaccine for a fungus.
Since the given code was purportedly the result of a well thought out decision, making fun of it might be the only way to get rid of it. An ordinary bug report would simply be an invitation to a closed NOTABUG.
On Fri, Apr 22, 2022 at 10:24 AM Michael Hennebry hennebry@web.cs.ndsu.nodak.edu wrote:
Does making fun of *code* violate either?
Maybe, maybe not. But it doesn't do any good either. Are you aware of a single instance of somebody making fun of code, and the author of said code then saying, "Oh, now that you have made fun of my code, I see that I didn't think things through?" It doesn't happen. Making fun of a person's creations makes that person defensive, and less tractable than they might have been at first. It is counterproductive.
We primates have a distinct tendency to descend into warring tribes of poop-slinging chimpanzees. Many organizations have codified standards of professional behavior for precisely this reason. Once the poop-slinging starts, forward progress comes to a rapid halt.
Those who are unhappy with a technical product have a choice. Either (1) whine endlessly about it, which will change nothing; or (2) figure out how to persuade somebody to change things. The latter requires making a technical argument. List the disadvantages of the current approach. Outline an alternate approach that does not have those disadvantages. Be frank about the disadvantages of the alternate approach. Approach the person or people with decision-making power and persuade them that your alternate approach deserves a hard look. Offer to help implement the alternate approach.
Persuading people takes work. Mocking not only doesn't persuade, it does the opposite.
On Fri, 22 Apr 2022, Jerry James wrote:
On Fri, Apr 22, 2022 at 10:24 AM Michael Hennebry hennebry@web.cs.ndsu.nodak.edu wrote:
Does making fun of *code* violate either?
Maybe, maybe not. But it doesn't do any good either. Are you aware of a single instance of somebody making fun of code, and the author of said code then saying, "Oh, now that you have made fun of my code, I see that I didn't think things through?" It doesn't happen. Making fun of a person's creations makes that person defensive, and less
tractable than they might have been at first. It is counterproductive.
I also know of no technical argument on the fedora forum that persuaded fedora's authors. Perhaps you do. Whether the decision under discussion was made for technical reasons depends on how broadly one defines technical reasons. In the current thread, an implicit assumption seems to be that its authors are immune to rational argument, technical or otherwise.
A more likely benefit might occur when the time comes for another such decision. Make it easy to customize or routinely destroy a user's attempt at customization? In the case at hand, destruction was chosen.
We primates have a distinct tendency to descend into warring tribes of poop-slinging chimpanzees. Many organizations have codified standards of professional behavior for precisely this reason. Once the poop-slinging starts, forward progress comes to a rapid halt.
Those who are unhappy with a technical product have a choice. Either (1) whine endlessly about it, which will change nothing; or (2) figure out how to persuade somebody to change things. The latter requires making a technical argument. List the disadvantages of the current approach. Outline an alternate approach that does not have those disadvantages. Be frank about the disadvantages of the alternate approach. Approach the person or people with decision-making power and persuade them that your alternate approach deserves a hard look. Offer to help implement the alternate approach.
Presenting the alternative has already been done: Delete the offending code.
Persuading people takes work. Mocking not only doesn't persuade, it does the opposite.
Persuading people is sometimes impossible. Sometimes the best one can do is try to keep it from getting worse. One way to get rid of smoking would be for non-smokers to make a collective decision to make fun of smokers. My guess is that it would greatly reduce the number of children who take up the habit. I expect it would also have the effect of sacrificing rather a lot of adult smokers, not to mention children who are already smoking.
BTW as a vocabulary choice, I'd put whining in the same category as infested.
Michael Hennebry writes:
I also know of no technical argument on the fedora forum that persuaded fedora's authors.
As far as I know there isn't really a technical argument for systemd or any particular systemd.* on Fedora workstations. The various traditional inits and daemons work fine in that environment.[1]
I have seen technical arguments for systemd and its modules made in the context of servers, though (search the lists for Lennart Poettering (sp?), you'll find some pretty quick, and there are plenty that he didn't make, too, I just don't have an easy search key for them). Since it's the server industry that pays geld for RHEL, and often contributes dollars and dev time to Fedora as well, they're going to be listened to for server editions. And then maintainers are going to think "we have to keep the paying customers satisfied, and we'd really prefer to maintain only one subsystem for each purpose". Pretty clearly that one subsystem is going to be the one the large customers/donors prefer for their business systems.
If you're talking about whatever bit of code is annoying the OP, well, if you put maintenance of your system's software in the hands of third parties, you're either going to have to put up with such annoyance sometimes because maintainers are neither omniscient to foresee all problems, nor omnipotent to program away all the problems they do foresee. Or you're going to have to take an active role in the maintenance of the software systems you depend on.
One way to get rid of smoking would be for non-smokers to make a collective decision to make fun of smokers.
That might work for smoking, although more likely because the smokers will leave when they see you coming, rather than because they stop smoking. But "leaving when they see you coming" is not exactly what you're hoping for from maintainers.
Footnotes: [1] However, maintenance on non-systemd software is going to decrease over time, and eventually it's all going to be "legacy" and riddled with technical debt -- and that will be the technical argument.
On Sun, 24 Apr 2022, Stephen J. Turnbull wrote:
Michael Hennebry writes:
One way to get rid of smoking would be for non-smokers to make a collective decision to make fun of smokers.
That might work for smoking, although more likely because the smokers will leave when they see you coming, rather than because they stop smoking. But "leaving when they see you coming" is not exactly what you're hoping for from maintainers.
The point seems to have been missed. The primary salutary effect of a *collective* decision would be on prospective smokers, e.g. children. Current adult smokers 'twould be more likely to kill than cure, hence the sacrifice mentioned in a non-quoted portion of my post.
The author of the complained-of code roughly corresponds to the adult smoker. The possible author of the next such annoyance roughly corresponds to the prospective smoker.
Michael Hennebry writes:
The point seems to have been missed.
The same could be said here, but it would be valid.
The primary salutary effect of a *collective* decision would be on prospective smokers, e.g. children.
You make me curious what experience in open source maintenance you have.
In mine, again, that's true for smokers, not so for volunteer maintainers. Over the years in several projects, one fairly toxic, others more polite, I've seen potential maintainers (and list moderators) balk at taking responsibility because of visible harrassment from users. There just isn't a whole generation of "young" users waiting to grow into "adult smoker" maintainers the way a half-century ago there was a whole generation of children waiting to grow into adult smokers *because* that was a mark of adulthood. Few users *want* to be maintainers, so any disincentive to accepting responsibility that isn't itself productive work is a bad idea when dealing with volunteers.
The other thing I've seen happen more than once is that the people doing the work set up alternative channels, or even go to private channels. The rules there are not necessarily "safe space", either, but pure harassment about WONTFIX issues gets one ejected. You'll notice that the people defending the decision here aren't the people who made it -- *they* escaped to some other channel long ago.
Harassing volunteer maintainers in open source is just a toxic lose- lose proposition. Even if they're so bad you just want them gone, forking the project is almost always cheaper, faster, and more effective. You don't get that trifecta often in software, take it while the taking's good. :-D
Steve
On Mon, 2022-04-25 at 14:50 +0900, Stephen J. Turnbull wrote:
You make me curious what experience in open source maintenance you have.
I got involved in the mailing lists for non-open source programs on the Amiga, that'd be in the 1990s, where some program authors were promoting user-input. We found bugs, sometimes even found the solutions. We caught cases where the programs didn't follow the specs, and pointed out what the program should be doing. And helped users figure out how to do what they wanted. Not that much different than this list.
On the YAM list (yet another mailer, a freeware email program) things went rather well nearly all the time. I did get told off for calling some a dickhead once, the person telling me off said they weren't saying he wasn't, but don't call him that. Others said there was a surprising lack of swearing, considering I'm Australian. That was mostly over off-topic discussions, anyway. I don't recall there being any of "this program does dumb things" flamefests.
On the Miami list (a TCP/IP stack), the programmer was more headstrong, but about the only headbutting I remember was, from time to time, some people suggesting the config files should be plain text so they could be user-edited, and the programmer saying no the program should write all its own configuration riles (so you can't set up mutually-exclusive options, for instance - it just would let you do so).
The IBrowse list (web browser) suffered from repeatedly AWOL programmer, with some go-betweens that could occasionally contact them, and it being near abandonware for its whole existence. There's still people working on it, despite it being over a decade out of touch with the modern web (no Java, basic JavaScripting, no CSS). It's slightly more graphical than Lynx, still in the realms of the very early graphical web browsers.
There was probably a tiny bit of argy bargy from people expecting free software (though it wasn't expensive).
But overall, one thing we rarely had to contend with was programmers foisting unwanted features on us, nor removing wanted features. And it was rather rare that users made dopey feature requests.
Michael Hennebry writes:
TH was making fun of code, not the person who is just trying to figure out how to live with it.
No, the allegation was that disparaging comments directed at the code are perceived as impuning the character of the code's source, and not the character who's using the code.
This is not the first time I've came across such a thing, and this defensive reflex is as old as Linux is.
Since the given code was purportedly the result of a well thought out decision, making fun of it might be the only way to get rid of it. An ordinary bug report would simply be an invitation to a closed NOTABUG.
No, that's not what will happen. What will happen is: absolutely nothing. Crickets will chirp for about a year, or a little bit more, and the bug will get closed after the release EOLs.
Matthew Miller writes:
On Sun, Apr 17, 2022 at 06:34:48PM -0400, Sam Varshavchik wrote:
but it was mostly working, so stayed under the radar, until the recent update broke it. Additionally, the systemd-resolved rpm is actively hijacking /etc/resolv.conf. From the package's scriptlet:
Sam, can you _please_ not use such colorfully-negative lagnguage here?
None of this is "infestation", "hijacking", or something to "fumigate". This is all from an approved Change for Fedora Linux 33, which you can read here: https://fedoraproject.org/wiki/Changes/systemd-resolved, and which was documented in the release notes at https://docs.fedoraproject.org/en-US/fedora/f33/release- notes/sysadmin/Networking/.
This looks like an appeal to authority, and not an argument on its own merits.
But let's go back and revisit all of that, if you insist.
# Fedora will continue its history of enabling new systemd-provided services # whenever it makes sense to do so.
Note that this statement assumes that it "makes sense to do so". There's no preceding discussion, or overview, that ever defined an objective standard that established any metric for the "makes sense to do so", or not, part. What does "makes sense to do so" mean?Just a simple statement that "it makes sense to do so", and that was the end of the discussion.
# Standardizing on upstream systemd services is beneficial to the broader # Linux ecosystem in addition to Fedora, since standardizing reduces # behavior differences between different Linux distributions.
Really? The majority of Linux distributions already adopted systemd- resolved, when this was proposed?
# Sadly, Fedora is no longer leading in this area. Ubuntu has # enabled systemd-resolved by default since Ubuntu 16.10, so by the time # Fedora 33 is released, we will be three years behind Ubuntu here.
This is the part that actually prompted me to do decide to reply – I was going to let it go, at first.
But I just happened to have have Ubuntu 20 right here, recently installed, without any customizations. Let's take a look.
mrsam@ripper:~$ uname -a Linux ripper 5.13.0-39-generic #44~20.04.1-Ubuntu SMP Thu Mar 24 16:43:35 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux mrsam@ripper:~$ grep hosts /etc/nsswitch.conf hosts: files mdns4_minimal [NOTFOUND=return] dns
Nope, no systemd here. I have no recollection if editing this manually. This is what I ended up with, after installing Focal.
/etc/resolv.conf is pointing to systemd's resolver, but there's no redirection in nsswitch.conf.
So, grand total: the status of systemd-resolved in Ubuntu was apparently used to propose doing something that systemd-resolved was /not/ configured for, in Ubuntu.
Or maybe systemd-resolved was also hooked in nsswitch.conf originally, in Ubuntu 16, and Ubuntu decided to back away from that?
On Mon, Apr 18, 2022 at 07:47:28PM -0400, Sam Varshavchik wrote:
This looks like an appeal to authority, and not an argument on its own merits.
But let's go back and revisit all of that, if you insist.
This is quite missing my point. I'm not interested in _arguing_ at all. The point is: your hyperbole about "hijacking" and etc. is not appropriate. This is an intentional, discussed, and approved change that went through the proper processes.
It's fine for you to discuss the technical aspects — and even the "merits", as you said. But if that's what you want to do, do that. Several years ago, all of the vitriol and trolling on this list got so bad we had to shut down pretty much every systemd discussion. Let's not go back to that.
On Mon, 2022-04-18 at 20:57 -0400, Matthew Miller wrote:
This is quite missing my point. I'm not interested in _arguing_ at all. The point is: your hyperbole about "hijacking" and etc. is not appropriate. This is an intentional, discussed, and approved change that went through the proper processes.
When one service interferes with another, hijacking seems like exactly the right descriptive term to use.
It's fine for you to discuss the technical aspects — and even the "merits", as you said. But if that's what you want to do, do that. Several years ago, all of the vitriol and trolling on this list got so bad we had to shut down pretty much every systemd discussion. Let's not go back to that.
Sounds like several cases of: Can't take criticism. I am never wrong. Won't back down.
Tim via users writes:
On Mon, 2022-04-18 at 20:57 -0400, Matthew Miller wrote:
This is quite missing my point. I'm not interested in _arguing_ at all. The point is: your hyperbole about "hijacking" and etc. is not appropriate. This is an intentional, discussed, and approved change that went through the proper processes.
When one service interferes with another, hijacking seems like exactly the right descriptive term to use.
Help me out here: wasn't there a point of order made, way back when: hey, if you want to disable systemd-resolved, just manually replace the /etc/resolv.conf symlink?
I have a vivid recollection of that.
But the latest systemd-resolved update force-updates the /etc/resolv.conf symlink, even if it was manually overridden.
It's fine for you to discuss the technical aspects — and even the "merits", as you said. But if that's what you want to do, do that. Several years ago, all of the vitriol and trolling on this list got so bad we had to shut down pretty much every systemd discussion. Let's not go back to that.
Sounds like several cases of: Can't take criticism. I am never wrong. Won't back down.
And I did make an attempt to have a discussion on the merits: specifically how Ubuntu's systemd usage was offered as a supporting argument. But, looking at an actual Ubuntu server next to me, I discovered this is not what ended up happening in Fedora.
Now, after a brief retrospective, perhaps my choice of words was a little bit too blunt, so I'll try to moderate it. I just keep making the same mistake, over and over again. I, personally, never get bothered when someone flames the crap out of me, or my warez. As they say: sticks and stones. I just don't know why I should, and my mistake is assuming that noone else gives a crap what I mouth off, either. So, why should I care much about what comes out of my mouth? Who cares? Well, apparently, people do. So be it.
So, there must be a good reason why the systemd-resolved now forcefully resets the /etc/resolv.conf symlink back to itself. The cited Bugzilla entry says something about the /etc/resolv.conf being missing after a fresh install, or something like that. I forget the exact situation where that happened. But as far as I can tell, the package's scriptlet also explicitly checks if the symlink exists, but points elsewhere, and "corrects" that too. I emphasize: the scriptlet checks if the symlink is missing, if so it gets created. That's right on target. But then, if the sysmlink exists, then another check is made, and it still gets reset. I couldn't immediately figure out why it did that.
And then it's still bothers me how systemd-resolved hooks /etc/nsswitch.conf, how that's not publicized very well, either; and how exciting this becomes when, after, a systemd-resolved update, it breaks for whatever reason.
On Mon, Apr 18, 2022 at 11:43 PM Sam Varshavchik mrsam@courier-mta.com wrote:
Help me out here: wasn't there a point of order made, way back when: hey, if you want to disable systemd-resolved, just manually replace the /etc/resolv.conf symlink?
*and* disable the systemd unit, I believe. At least that is what I have done, and I haven't noticed any unwanted reversions.
On Mon, 18 Apr 2022 23:43:36 -0400 Sam Varshavchik wrote:
Help me out here: wasn't there a point of order made, way back when: hey, if you want to disable systemd-resolved, just manually replace the /etc/resolv.conf symlink?
You also need to systemctl disable systemd-resolved (and probably systemctl mask systemd-resolved). That way it doesn't update anything at all. That's how I'm currently set and everyone is happily using the dnsmasq I have installed to serve my local lan.
Systemd didn't invent replacing resolv.conf, various dhcp client shofware have done that for years (and it was just as irritating when they did it).
systemd-resolved can also be just uninstalled. Provides at least very basic symlink removal.
On 4/19/22 14:27, Tom Horsley wrote:
On Mon, 18 Apr 2022 23:43:36 -0400 Sam Varshavchik wrote:
Help me out here: wasn't there a point of order made, way back when: hey, if you want to disable systemd-resolved, just manually replace the /etc/resolv.conf symlink?
You also need to systemctl disable systemd-resolved (and probably systemctl mask systemd-resolved). That way it doesn't update anything at all. That's how I'm currently set and everyone is happily using the dnsmasq I have installed to serve my local lan.
I think the major problem is with upgrades from previous versions of Fedora, where nothing similar were required. systemd-resolved just grabbed /etc/resolv.conf and made many other dhcp clients unable to get it back. They also don't offer simple way to disable resolved and keep DNS working.
Systemd didn't invent replacing resolv.conf, various dhcp client shofware have done that for years (and it was just as irritating when they did it).
systemd invented replacing /etc/resolv.conf with symlink leading to a private /run directory. No other software did that automatically AFAIK. If you disable systemd-resolved later, it would stay broken. Other clients just overwritten the contents of /etc/resolv.conf. A common trick is using chattr +i /etc/resolv.conf when you want to prevent random rewrites.
When it should reconfigure system name resolution, it has to use some way. Current resolvconf from systemd-resolved package is useless without systemd-resolved enabled. I don't know about any better generic interface to configure system nameservers.
On Apr 18, 2022, at 20:58, Matthew Miller mattdm@fedoraproject.org wrote:
It's fine for you to discuss the technical aspects — and even the "merits", as you said. But if that's what you want to do, do that. Several years ago, all of the vitriol and trolling on this list got so bad we had to shut down pretty much every systemd discussion. Let's not go back to that.
Sadly, these kinds of tired flame wars just kill any interest in being part of the community. People just love to argue, but they scare off people new to the forum, and make existing people disinterested in continuing to contribute.
-- Jonathan Billings
Could repeated flames around systemd mean something is wrong with the way systemd introduces new features?
If people complain often, maybe those changes should have been made in opt-in mode. Especially on upgrades from previous releases. The mentioned change were invasive and has broken multiple scenarios.
Could perhaps systemd team invest more time to propagating their features and persuading people to start using (and wanting) new features, instead of just throwing them at people?
Just my 2 cents.
Regards,
Petr
On 4/19/22 02:57, Matthew Miller wrote:
On Mon, Apr 18, 2022 at 07:47:28PM -0400, Sam Varshavchik wrote:
This looks like an appeal to authority, and not an argument on its own merits.
But let's go back and revisit all of that, if you insist.
This is quite missing my point. I'm not interested in _arguing_ at all. The point is: your hyperbole about "hijacking" and etc. is not appropriate. This is an intentional, discussed, and approved change that went through the proper processes.
It's fine for you to discuss the technical aspects — and even the "merits", as you said. But if that's what you want to do, do that. Several years ago, all of the vitriol and trolling on this list got so bad we had to shut down pretty much every systemd discussion. Let's not go back to that.
On Wed, 20 Apr 2022 17:04:39 +0200 Petr Menšík pemensik@redhat.com wrote:
Could repeated flames around systemd mean something is wrong with the way systemd introduces new features?
If people complain often, maybe those changes should have been made in opt-in mode. Especially on upgrades from previous releases. The mentioned change were invasive and has broken multiple scenarios.
Could perhaps systemd team invest more time to propagating their features and persuading people to start using (and wanting) new features, instead of just throwing them at people?
Just my 2 cents.
Regards,
Petr
Not just 2 cents. A huge amount. :-)
Bob
On 4/20/22 08:04, Petr Menšík wrote:
Could repeated flames around systemd mean something is wrong with the way systemd introduces new features?
If people complain often, maybe those changes should have been made in opt-in mode. Especially on upgrades from previous releases. The mentioned change were invasive and has broken multiple scenarios.
Could perhaps systemd team invest more time to propagating their features and persuading people to start using (and wanting) new features, instead of just throwing them at people?
The benefits have been well explained. The problem is that some people really don't like change even if it's for the better. And sometimes things do break when changed and instead of finding out why it breaks and how to fix, they just say how terrible the new software is and that it should never have been used.
The switch to systemd was a huge improvement and all the major distros and probably most of the rest have switched to it.
systemd-resolved is working fine for most people. I had to check if I'm using it because I hadn't even noticed any change. It will likely be extra useful for VPNs once I get around to setting up split DNS for that.
On Sat, 23 Apr 2022, Samuel Sieb wrote:
The benefits have been well explained. The problem is that some people really don't like change even if it's for the better. And sometimes things do break when changed and instead of finding out why it breaks and how to fix, they just say how terrible the new software is and that it should never have been used.
Nyet. OP was not complaining about change. OP was complaining about his inability to change his system. OP did discover the reason. OP discovered the reason was systemd code apparently designed to frustrate just that change. OP's complaint was not about systemd generally, 'twas about a single rather awful policy decision.
What would a systemd evangelist suggest as a minimal workaround?
Michael Hennebry writes:
On Sat, 23 Apr 2022, Samuel Sieb wrote:
The benefits have been well explained. The problem is that some people really don't like change even if it's for the better. And sometimes things do break when changed and instead of finding out why it breaks and how to fix, they just say how terrible the new software is and that it should never have been used.
Nyet. OP was not complaining about change. OP was complaining about his inability to change his system.
Not quite. The initial complaint was that a systemd update broke DNS lookups. This was surprising, because on that system systemd-resolved was purportedly disabled by redirecting the /etc/resolv.conf link, which was originally presented as the way to disable it. Subsequently, it was established that:
1) The updated systemd-resolved package now force-replaces the symlink
2) Even if the symlink is manually adjusted again, DNS resolution is still non-functional because systemd-resolved is also hooked via nsswitch.conf.
3) The bug that was referenced as the reason for changing the scriptlet appears to be because the /etc/resolv.conf was missing in some edge cases, and the scriptlet did check to see if it doesn't exist and creates it. Except that it also checks if it does exist, and then it gets force-replaced to point to systemd-resolved. I couldn't find an explanation for that.
I also don't recall the reason for nsswitch.conf hook. Playing devil's advocate: I can think of one technical reason, but I see no point in speculating. Instead of explaining this, the merits of the original Fedora change proposal were referenced, which – I don't recall to what extent the actual technical merits were listed, I only remember its main argument: well, it's enabled in Ubuntu.
Inspecting a fresh Ubuntu 20 install that I had here I discovered that, yes, it's running systemd-resolved, but it's not hooked in nsswitch.conf.
What also rubs a little bit of the wrong way is that the nsswitch module is called "resolve". It wasn't apparent that it came from systemd. "resolve" is very generic, after all, who can suspect an innocent module named "resolve", as the culprit for non-functional DNS resolution? It should've been called "systemd".
OP did discover the reason. OP discovered the reason was systemd code apparently designed to frustrate just that change. OP's complaint was not about systemd generally, 'twas about a single rather awful policy decision.
What would a systemd evangelist suggest as a minimal workaround?
Well, the suggestion seems to be: remove systemd-resolved if it doesn't work for you.
Which is an entirely fair proposition right now.
I'll just make a friendly wager: 100 quatloos that either systemd or some other core component will depend on systemd-resolved in Fedora 40, and there will be no way to remove it; and by Fedora 45 it'll probably be unmaskable.
On Sun, Apr 24, 2022 at 12:00:50PM -0500, Michael Hennebry wrote:
On Sat, 23 Apr 2022, Samuel Sieb wrote:
The benefits have been well explained. The problem is that some people really don't like change even if it's for the better. And sometimes things do break when changed and instead of finding out why it breaks and how to fix, they just say how terrible the new software is and that it should never have been used.
Nyet. OP was not complaining about change. OP was complaining about his inability to change his system. OP did discover the reason. OP discovered the reason was systemd code apparently designed to frustrate just that change. OP's complaint was not about systemd generally, 'twas about a single rather awful policy decision.
What would a systemd evangelist suggest as a minimal workaround?
I suggest the OP misunderstood the setup or intended setup.
The scriptlets will set /etc/resolv.conf to point to the systemd-resolved resolver if:
* The /etc/resolv.conf file doesn't exist yet AND * systemd is being used to boot (so, it's not a container, etc) AND * systemd-resolved service is enabled AND * DNSStubListener is not set to no/false/off in systemd/resolved.conf
So, if you wish to have systemd-resolved not manage your dns, you can: * make a /etc/resolv.conf file and put whatever you want in it. * disable the systemd-resolved service * Set DNSStubListener to no/false/off in systemd/resolved.conf
There's no reason you can't disable this if you like (barring bugs which there have been some of definitely).
That said, there's lots of advantages to systemd-resolved... it allows you to split dns based on interface (ie, vpn requests can go to a server on the vpn instead of to all nameservers in the public interfaces), you can override lots of other things on a per interface basis, you can manage the dns cache easily, you can enable/disable/set dnssec prefs, etc.
kevin
Kevin Fenzi writes:
I suggest the OP misunderstood the setup or intended setup.
The scriptlets will set /etc/resolv.conf to point to the systemd-resolved resolver if:
- The /etc/resolv.conf file doesn't exist yet
AND
- systemd is being used to boot (so, it's not a container, etc)
AND
- systemd-resolved service is enabled
AND
- DNSStubListener is not set to no/false/off in systemd/resolved.conf
That's the first ln -s call in the scriptlet.
Now, look at the second one, the one with the -f flag.
On Sun, Apr 24, 2022 at 02:12:50PM -0400, Sam Varshavchik wrote:
Kevin Fenzi writes:
I suggest the OP misunderstood the setup or intended setup.
The scriptlets will set /etc/resolv.conf to point to the systemd-resolved resolver if:
- The /etc/resolv.conf file doesn't exist yet
AND
- systemd is being used to boot (so, it's not a container, etc)
AND
- systemd-resolved service is enabled
AND
- DNSStubListener is not set to no/false/off in systemd/resolved.conf
That's the first ln -s call in the scriptlet.
Now, look at the second one, the one with the -f flag.
Which systemd version are you looking at?
bug #2074122 was just recently fixed in https://bodhi.fedoraproject.org/updates/FEDORA-2022-1bf06aa328 systemd-249.11-2.fc35
perhaps thats what you hit?
kevin
Kevin Fenzi writes:
That's the first ln -s call in the scriptlet.
Now, look at the second one, the one with the -f flag.
Which systemd version are you looking at?
systemd-249.11-1.fc35.src.rpm
bug #2074122 was just recently fixed in https://bodhi.fedoraproject.org/updates/FEDORA-2022-1bf06aa328 systemd-249.11-2.fc35
perhaps thats what you hit?
No. As it's been pointed out, even the stock /etc/resolv.conf's comments read that the only thing that needs to be done is to replace this symlink.
However, as of systemd-249.11-1.fc35.src.rpm, if you have installed a manual /etc/resolv.conf symlink that update will reinstall the symlink back to systemd.
I haven't looked at systemd-249.11-2.fc35, but based on its description it's a different bug. The scriptlet reinstalls it, but then the symlink's target is broken. From the bug:
Running scriptlet: systemd-resolved-249.11-1.fc35.x86_64 69/69 '/etc/resolv.conf' → '../run/systemd/resolve/stub-resolv.conf'
That's the symlink getting replaced. I haven't looked at the -2's scriptlet, but I venture that the fix for that is unlikely to prevent an existing symlink from getting replaced here.
Sam Varshavchik wrote:
Kevin Fenzi writes:
That's the first ln -s call in the scriptlet.
Now, look at the second one, the one with the -f flag.
Which systemd version are you looking at?
systemd-249.11-1.fc35.src.rpm
bug #2074122 was just recently fixed in https://bodhi.fedoraproject.org/updates/FEDORA-2022-1bf06aa328 systemd-249.11-2.fc35
perhaps thats what you hit?
No. As it's been pointed out, even the stock /etc/resolv.conf's comments read that the only thing that needs to be done is to replace this symlink.
However, as of systemd-249.11-1.fc35.src.rpm, if you have installed a manual /etc/resolv.conf symlink that update will reinstall the symlink back to systemd.
I haven't looked at systemd-249.11-2.fc35, but based on its description it's a different bug. The scriptlet reinstalls it, but then the symlink's target is broken. From the bug:
Running scriptlet: systemd-resolved-249.11-1.fc35.x86_64 69/69 '/etc/resolv.conf' → '../run/systemd/resolve/stub-resolv.conf'
That's the symlink getting replaced. I haven't looked at the -2's scriptlet, but I venture that the fix for that is unlikely to prevent an existing symlink from getting replaced here.
The relevant parts of the diff from -1 to -2 are:
diff --git a/systemd.spec b/systemd.spec index 91e9b6d..f30f835 100644 --- a/systemd.spec +++ b/systemd.spec [...] @@ -927,7 +927,8 @@ if [ $1 -eq 0 ] ; then fi
%post resolved -[ $1 -gt 1 ] && exit 0 +[ $1 -eq 1 ] || exit 0 +# Initial installation
# Related to https://bugzilla.redhat.com/show_bug.cgi?id=1943263 if ls /usr/lib/systemd/libsystemd-shared-24[0-8].so &>/dev/null; then @@ -938,6 +939,9 @@ fi %systemd_post systemd-resolved.service
%posttrans resolved +[ $1 -eq 1 ] || exit 0 +# Initial installation + # Create /etc/resolv.conf symlink. # We would also create it using tmpfiles, but let's do this here # too before NetworkManager gets a chance. (systemd-tmpfiles invocation above @@ -957,11 +961,11 @@ if systemctl -q is-enabled systemd-resolved.service &>/dev/null && ! systemd-analyze cat-config systemd/resolved.conf 2>/dev/null | grep -iqE '^DNSStubListener\s*=\s*(no?|false|0|off)\s*$'; then
- if ! test -e /etc/resolv.conf && ! test -L /etc/resolv.conf; then - ln -sv ../run/systemd/resolve/stub-resolv.conf /etc/resolv.conf || : + if ! test -e /etc/resolv.conf; then + ln -sv ../run/systemd/resolve/stub-resolv.conf /etc/resolv.conf elif test -d /run/systemd/system/ && ! mountpoint /etc/resolv.conf &>/dev/null; then - ln -fsv ../run/systemd/resolve/stub-resolv.conf /etc/resolv.conf || : + ln -fsv ../run/systemd/resolve/stub-resolv.conf /etc/resolv.conf fi fi
That's from two commits:
0c79c0a (Do not touch /etc/resolv.conf on upgrades, 2022-04-12) 5f6666c (Revert "Avoid trying to create the symlink if there's a dangling symlink already", 2022-04-12)
It skips the code which touches /etc/resolv.conf except on initial install of systemd-resolved, at least, which would have prevented this issue for most folks in this thread, I believe.
Looking at the f36 and rawhide branches, it looks as though the fixes were synced to rawhide but not f36. Particularly, the `[ $1 -eq 1 ] || exit 0` in %posttrans is missing from f36.
I don't know if that's intentional or not. If not, it seems like parts of this bug could be waiting to bite folks on upgrade to f36.
It would be ideal if someone who cares could follow-up with the systemd package maintainers and ensure that's not an omission.
Todd Zullinger writes:
The relevant parts of the diff from -1 to -2 are:
%posttrans resolved +[ $1 -eq 1 ] || exit 0 +# Initial installation +
And that's why an update to 11-1.fc35 ran the rest of the script.
- if ! test -e /etc/resolv.conf && ! test -L /etc/resolv.conf; then - ln -sv ../run/systemd/resolve/stub-resolv.conf /etc/resolv.conf || : + if ! test -e /etc/resolv.conf; then + ln -sv ../run/systemd/resolve/stub-resolv.conf /etc/resolv.conf elif test -d /run/systemd/system/ && ! mountpoint /etc/resolv.conf &>/dev/null; then - ln -fsv ../run/systemd/resolve/stub-resolv.conf /etc/resolv.conf
|| : + ln -fsv ../run/systemd/resolve/stub-resolv.conf /etc/resolv.conf fi
Note the elif part. If the symlink exists, it's going to get reset if systemd is installed.
So, if you don't have systemd-resolved currently installed, and then install it, the existing symlink is going to get reset, one way or another. Which might be fine, you're taking an explicit, manual action.
It skips the code which touches /etc/resolv.conf except on initial install of systemd-resolved, at least, which would have prevented this issue for most folks in this thread, I believe.
Now it does.
On Sun, 2022-04-24 at 10:48 -0700, Kevin Fenzi wrote:
The scriptlets will set /etc/resolv.conf to point to the systemd-resolved resolver if:
- The /etc/resolv.conf file doesn't exist yet
AND
- systemd is being used to boot (so, it's not a container, etc)
AND
- systemd-resolved service is enabled
AND
- DNSStubListener is not set to no/false/off in systemd/resolved.conf
So, if you wish to have systemd-resolved not manage your dns, you can:
- make a /etc/resolv.conf file and put whatever you want in it.
- disable the systemd-resolved service
- Set DNSStubListener to no/false/off in systemd/resolved.conf
It seems to me that original complaints were that it didn't follow that set of rules. They had a resolv.conf file that it kept interfering with.
I'd well imagine that initial boot up, you don't have one. At your first boot it may be about to be configure by DHCP, does it wait until after that before taking over? You may be going to manually configure your network, do your after-the-fact configurations get acknowledged as the settings that will be kept and not interfered with?
On 4/24/22 10:00, Michael Hennebry wrote:
What would a systemd evangelist suggest as a minimal workaround?
systemctl disable systemd-resolved && systemctl stop systemd-resolved rm /etc/resolv.conf systemctl restart NetworkManager
As documented in the original change proposal: https://fedoraproject.org/wiki/Changes/systemd-resolved
Once upon a time, Gordon Messmer gordon.messmer@gmail.com said:
systemctl disable systemd-resolved && systemctl stop systemd-resolved
BTW for quite some time, this can just be "systemctl disable --now <service>".
rm /etc/resolv.conf systemctl restart NetworkManager
As documented in the original change proposal: https://fedoraproject.org/wiki/Changes/systemd-resolved
If that's what required, then an update to systemd-resolved is in order, since the generated resolv.conf says:
# through the symlink at /etc/resolv.conf. To manage man:resolv.conf(5) in a # different way, replace this symlink by a static file or a different symlink.
Nothing about disabling/stopping/restarting any services.