when startup delays become bugs

[Adam Williamson]

On Tue, 2013-05-14 at 21:28 -0600, Chris Murphy wrote:

> > As we've established, sendmail's one minute delay when the hostname is
> > not fully qualified looks ugly, but does not actually delay perceived
> > startup at all, because neither local nor remote login require
> > sendmail.service to be up.
> 
> With sendmail enabled, time to ssh success is ~40 seconds.
> 
> With 'systemctl mask sendmail', time to ssh success is ~18 seconds. So
> the sendmail delay does actually delay startup as it relates to remote
> login.

Ah, sorry, I thought you'd said otherwise, must have misread.

> The fully qualified domain name isn't something I'm compelled to use
> on other platforms so I don't understand the advantage of it being
> needed on Fedora. 

Do those 'other platforms' use sendmail? I don't think many other
distros ship it by default any more. Debian installs exim by default,
for instance.

> And if I use it, I end up with worse problems in that more services
> have long delays rather than just sendmail and sm-client, and I can no
> longer do ssh chris at f19q.local but rather I have to type
> chris at f19qlocaldomain.

I know you said that, but as I wrote in the bug, I can't reproduce that
at all. I use foo.localdomain hostnames, I don't have startup delays,
and avahi foo.local still works.

> The only two things I change from the stock installation is set the
> hostname, and 'systemctl enable sshd' and 'systemctl start sshd'.
> That's it.

Are you setting the hostname during installation or post-install?

> > 
> > Once you have the list of services with significant start times which
> > delay interaction for you, the next step is to investigate why they take
> > a long time to start up. It might be inevitable, or the result of a
> > local configuration issue that you can change, or it might be a bug. You
> > really just need to take a look at the service file, see what it does,
> > see if you can figure out specifically what part(s) of that service's
> > startup process are slow, and if there's anything that can be done to
> > improve it.
> 
> Is there a way to get more verbosity, selectively per service, to find
> what what it's doing or waiting on when it has a significant start
> time. 

You can isolate messages relating to specific services with journalctl:

journalctl _SYSTEMD_UNIT=foobar.service

I don't think there's a universal 'verbosity' setting for systemd units,
but I might be wrong, I don't know everything.

> And what's a significant start time? 2 seconds? 20 seconds?

Well, like I said, that's an impossible question to give a single
definitive answer to. It's your decision: you're the one doing
performance optimization for your workload. This is a question you ask
yourself, not someone else. What is a significant start time for you? Is
it worth an uncertain amount of your investigation time to save yourself
200 seconds on each boot? 20 seconds? 2 seconds? 0.2 seconds? It's
entirely up to you.

> Well it is relatively easy to just mask sendmail and disable sm-client,

If you don't care about local mail delivery, why not just remove
sendmail? Nothing requires it, it's just in the default package set. You
can 'yum remove' it safely.

>  especially seeing as I don't need an MTA running. Even if there are
> good reasons for one being installed by default, I really doubt the
> majority of Fedora users benefit from it being enabled by default.

We have that argument every six months or so; it's probably about time.
Some people still consider local mail delivery an essential function of
a *nix system, and a mechanism that applications should be able to
expect to be in place. I had a quick look earlier today and apparently
still no-one's written a simple local delivery only MTA for this
purpose, so if we want local mail delivery, we have to install either
sendmail, postfix or exim by default. They're all big horking
carrier-grade MTAs. Personally I'd find it more sensible to go with
postfix than sendmail, but it's a marginal call; if you just want
something that does basic local delivery with zero configuration it
really doesn't matter much which one you pick. The differences between
them affect more complex use cases, and if you're going to actually go
and configure an MTA to do something more complex than default local
mail delivery, then you can easily switch from sendmail to postfix or
exim if you prefer them.

I wouldn't hate it if we just said 'it's not 1976 any more, we're not
going to provide local mail delivery by default, sorry', but both sides
of the argument are reasonable positions. And I'd love it if someone
wrote a basic local-only MTA for distros to include instead of a full-on
MTA like sendmail or postfix, but I'm not volunteering to write it,
so...
-- 
Adam Williamson
Fedora QA Community Monkey
IRC: adamw | Twitter: AdamW_Fedora | identi.ca: adamwfedora
http://www.happyassassin.net