Is this proof that systemd is completely broken?

[Tom H]

On Sat, Jul 12, 2014 at 3:16 PM, Sam Varshavchik <mrsam at courier-mta.com> wrote:
> Tom H writes:


>> "NetworkManager-wait-online.service" has:
>>
>> <begin>
>> After=NetworkManager.service
>> Wants=network.target
>> Before=network.target network-online.target
>> </end>
>>
>> Perhaps you should replicate that in "wait-for-network.service" for it
>> to behave as you intend it.
>>
>> I assume that you can replicate "After=NetworkManager.service" with
>> "After=network.service" even though your network's being brought up by
>> "/etc/rc.d/init.d/network".
>
> Ok, I tried the following variation:
>
> [Unit]
> Description=Wait for network ports to be initialized
> Before=network.target network-online.target
> After=network.service
> Wants=network.target
>
> [Service]
> Type=oneshot
> ExecStart=/root/bin/wait-for-network
>
> [Install]
> WantedBy=multi-user.target
>
> With this, the first reboot initialized all servers properly, and the
> console logging during the boot went back to what it was about a month or
> two ago, before everything went off the rails.

Good; although Corinna's "/etc/systemd/system/network-waiter.service"
is a cleaner solution.

You could also override the Fedora-provided service with
"/etc/systemd/system/named-chroot.service":
[Unit]
Description=Berkeley Internet Name Domain (DNS)
Wants=nss-lookup.target
Before=nss-lookup.target
After=network-online.target ## or network.target network-online.target

But, if I understand correctly, you want to figure out why your unit
isn't ordered the way that it ought to be, so you're not just
interested in a solution that works.


>> What do you have in "/root/bin/wait-for-network"?
>
> Nothing particularly exciting. I'll attach it below. But whatever it does or
> does not have, it should not affect the order in which systemd starts
> processes. It can't possibly affect the order in which systemd starts
> processes. It has no means of controlling the order;

I wasn't questioning whether your script was affecting the startup order.


> yet, with the previous
> service file, with having just a "Before=network.target", systemd was
> clearly launching a different service that stated "After=network.target",
> before this one finished. Which was broken.
>
> On the last boot, the following script logged that all interfaces were
> already up when it started. Previously, it showed that only lo0 was up, and
> the rest came up after it waited ten seconds. So the difference now is
> clearly because of "After=network.service", but that alone doesn't explain
> it. Even without it, this service doesn't officially start, according to the
> documented specification of a one-shot service, until network.service
> finished bringing up all the interfaces.
>
> So that leaves the addition of "Wants=network.target". It apparently has an
> effect on the ordering of services that get started. Now, you read its
> description, in systemd's documentation and tell me if it says anything
> about controlling the order of services. The plain reading of the
> documentation only suggests that this controls what gets started, and not
> which order anything gets started with.

>From systemd.special:

network-online.target
Units that strictly require a configured network connection should pull in
network-online.target (via a Wants= type dependency) and order themselves after
it. This target unit is intended to pull in a service that delays further
execution until the network is sufficiently set up. What precisely this requires
is left to the implementation of the network managing service. Note the
distinction between this unit and network.target. This unit is an active unit
(i.e. pulled in by the consumer rather than the provider of this functionality)
and pulls in a service which possibly adds substantial delays to further
execution. In contrast, network.target is a passive unit (i.e. pulled in by the
provider of the functionality, rather than the consumer) that usually does not
delay execution much. Usually, network.target is part of the boot of most
systems, while network-online.target is not, except when at least one unit
requires it. Also see Running Services After the Network is up[1] for more
information. All mount units for remote network file systems automatically pull
in this unit, and order themselves after it. Note that networking daemons that
simply provide functionality to other hosts generally do not need to pull this
in.

("[1]" is the freedesktop url that Kevin posted.)


> After grepping around, it became even more obvious how brain-dead systemd
> is. The new service that I added is the only service that specifies
> "Wants=network.target". None of the stock Fedora services – including the
> ones that fail to start properly until the network connections are up –
> specify "Wants=network.target". No Fedora server package, that I have
> installed, specifyis "Wants=network-online.target" either; but I have some
> non-Fedora packages that specify "Wants=network-online.target". And until I
> installed this custom service, nothing else wanted network.target either.

"NetworkManager-wait-online.service" has "Wants=network.target", which
is why I suggested the same for "wait-for-network.service".


> So… Looks like the bug is as follows. The system comes up with systemd
> configured to bring up multi-user.target. But unless something explicitly
> specifies Wants=network.target, systemd is going to completely ignore this
> target, and completely ignore all Before or After network.target
> specification, from any service that it is starting, and ignore any obvious
> dependencies that result from Before or After network.target. All without
> giving any hint as to what it's doing.
>
> As I said: this used to work, at least for me, reliably, until about a month
> or two ago, and I see that there have been a couple of systemd updates
> since. I can almost hear someone already bleating "this is how systemd is
> designed to work, and the problem is with incorrect service files".
>
> Presuming that my guess as to what the bug is, is correct: this is a bunk
> answer. Either this was a major change in systemd's behavior, recently, or
> not. If not: 1) systemd documentation is crap, and this should've been
> documented under in the "Wants" documentation a long time ago, 2) systemd
> just broke everything, without bothering to announce this major change in
> behavior. And this is also crap.
>
> But this should not surprise anyone. Recall, earlier this year, Linus
> calling out systemd's maintainer for pulling the exact same kind of a stunt:
> breaking something, in this case a kernel boot with the "debug" option, and
> then acting as if it's the kernel's problem to solve. Same exact snobbery
> and arrogance.
>
> Well, anyway, here's my script, FWIW, which is really a moot point; since by
> using After=network.service it's not really necessary, any more.

Thanks. I was just curious what was creating the output that you
showed in your earlier email.