Living with Systemd

[Arthur Dent]

Hello All,

I am gradually getting used to systemd. I can now just about force my
fingers to type "systemctl restart httpd.service" even though my brain
is itching to write "service httpd restart" and I find this cheatsheet
to be very useful:
http://fedoraproject.org/wiki/SysVinit_to_Systemd_Cheatsheet

However, there is one thing I find *incredibly* frustrating and that is
the paucity of information when things go wrong.

When I am tinkering with some app or other and I mess something up (as I
often do), being told that the app has "entered a failed state", but not
*WHY* it failed is very unhelpful. At least with SystemV you would be
told that it's because a file is missing, permission problem, config
error... etc.

Looking into /var/log/messages is no help (it just repeats the same
message) and very often the program's own logs are of no use because the
app has not started logging.

I have looked at man systemctl but can see no "verbose" (or similar)
switch.

I find that the only way I can troubleshoot a failing process is to
start the program directly from its executable, or by manually running
the init.d script. Then I get some useful information as to why the
program failed, and fix it.

Jul 22 00:01:11 mydomain systemd[1]: httpd.service: control process exited, code=exited status=1
Jul 22 00:01:11 mydomain systemd[1]: Unit httpd.service entered failed state.

The above error was caused by one rule in the thousands of mod-security
rules that I had just updated.

Starting httpd directly from the init script I found this:
Syntax error on line 91 of /etc/httpd/modsecurity.d/base_rules/modsecurity_crs_20_protocol_violations.conf:
Error creating rule: Unknown variable: REQBODY_ERROR

and I could see exactly what I needed to fix.

Am I missing something? Is there a way to get helpful error messages
with systemd that I don't know about - or are we stuck with "code=exited
status=1"?

Mark

[Mark Eggers]

On Fri, 22 Jul 2011 11:00:02 +0100, Arthur Dent wrote:
> Hello All,
>
> I am gradually getting used to systemd. I can now just about force my
> fingers to type "systemctl restart httpd.service" even though my brain
> is itching to write "service httpd restart" and I find this cheatsheet
> to be very useful:
> http://fedoraproject.org/wiki/SysVinit_to_Systemd_Cheatsheet
>
> However, there is one thing I find *incredibly* frustrating and that is
> the paucity of information when things go wrong.
>
> When I am tinkering with some app or other and I mess something up (as I
> often do), being told that the app has "entered a failed state", but not
> *WHY* it failed is very unhelpful. At least with SystemV you would be
> told that it's because a file is missing, permission problem, config
> error... etc.
>
> Looking into /var/log/messages is no help (it just repeats the same
> message) and very often the program's own logs are of no use because the
> app has not started logging.
>
> I have looked at man systemctl but can see no "verbose" (or similar)
> switch.
>
> I find that the only way I can troubleshoot a failing process is to
> start the program directly from its executable, or by manually running
> the init.d script. Then I get some useful information as to why the
> program failed, and fix it.
>
> Jul 22 00:01:11 mydomain systemd[1]: httpd.service: control process
> exited, code=exited status=1 Jul 22 00:01:11 mydomain systemd[1]: Unit
> httpd.service entered failed state.
>
> The above error was caused by one rule in the thousands of mod-security
> rules that I had just updated.

This really doesn't address the lack of information that's available when 
a service fails to start using systemd. I agree that more information 
would be great.

However, for Apache HTTPD, you have another tool to see what's wrong with 
your configuration.
> From the command line, run:

/usr/sbin/apachectl configtest

This will return "Syntax OK" if the configuration file is OK. If there is 
a problem, there should be detailed information.

/usr/sbin/apachectl -t

also works.

. . . . just my two cents.

/mde/

[Arthur Dent]

On Fri, 2011-07-22 at 15:46 +0000, Mark Eggers wrote:
> This really doesn't address the lack of information that's available when 
> a service fails to start using systemd. I agree that more information 
> would be great.
>
> However, for Apache HTTPD, you have another tool to see what's wrong with 
> your configuration.
> > From the command line, run:
>
> /usr/sbin/apachectl configtest
>
> This will return "Syntax OK" if the configuration file is OK. If there is 
> a problem, there should be detailed information.
>
> /usr/sbin/apachectl -t
>
> also works.
>
> . . . . just my two cents.

Hi Mark,

Thanks for that. Very useful.

However, HTTPD was just an example - the most recent one for me. I have
also had to resort to running init scripts (or the binary executable)
manually for clamd, fail2ban and ddclient in the last couple of weeks
alone.

I was really hoping that someone would reply saying something like "You
idiot! - Don't you know, all you need to do is to use the
--tellmewhatswrong switch with systemctl and all will be revealed!" But
alas, no-one has said anything...

What happens with F16 when there will be no more init scripts?

Mark

[Mark Eggers]

On Fri, 22 Jul 2011 22:26:24 +0100, Arthur Dent wrote:
> I was really hoping that someone would reply saying something like "You
> idiot! - Don't you know, all you need to do is to use the
> --tellmewhatswrong switch with systemctl and all will be revealed!" But
> alas, no-one has said anything...
>
> What happens with F16 when there will be no more init scripts?

I agree.

That's an interesting question. I did a little poking around on:

http://www.freedesktop.org/wiki/Software/systemd

and didn't see anything that jumped out. Maybe there's a way to get more 
information via /etc/systemd/system.conf and /etc/rsyslog.conf. I'll have 
to research that.

Other than that, a visit to the IRC or mailing list might be in order.

. . . . just my two cents.
/mde/

[T.C. Hollingsworth]

On 7/22/11, Arthur Dent  wrote:
> I was really hoping that someone would reply saying something like "You
> idiot! - Don't you know, all you need to do is to use the
> --tellmewhatswrong switch with systemctl and all will be revealed!" But
> alas, no-one has said anything...

Not quite as easy as that, but here's what you can do:

Edit /etc/systemd/system.conf and set "DefaultStandardOutput" to one
of "syslog", "kmsg", "kmsg+console", or "syslog+console".  That will
redirect stdout from it's default destination of /dev/null to the
place you set.  You may set "DefaultStandardError" too if you wish,
but by default it goes to the same place as DefaultStandardOutput.
(Note that you can also do this per-service by editing it's unit file
[see "man systemd.exec" for details] or at boot-time via the kernel
argument "systemd.default_standard_output".)

Then, just run systemctl daemon-reexec (in the case of editing
system.conf) to make your change take effect, start the service, and
check the output where you told it to send it along to.

-T.C.

[Hiisi]

On Wed, 2011-08-03 at 02:46 -0700, T.C. Hollingsworth wrote:
> Edit /etc/systemd/system.conf and set "DefaultStandardOutput" to one
> of "syslog", "kmsg", "kmsg+console", or "syslog+console".  That will
> redirect stdout from it's default destination of /dev/null to the
> place you set.  You may set "DefaultStandardError" too if you wish,
> but by default it goes to the same place as DefaultStandardOutput.
> (Note that you can also do this per-service by editing it's unit file
> [see "man systemd.exec" for details] or at boot-time via the kernel
> argument "systemd.default_standard_output".)

Tip of the day!
Definitely DefaultStandardError should point to syslog+console by
default.
--
Hiisi

[Arthur Dent]

On Wed, 2011-08-03 at 02:46 -0700, T.C. Hollingsworth wrote:
> On 7/22/11, Arthur Dent  wrote:
> > I was really hoping that someone would reply saying something like "You
> > idiot! - Don't you know, all you need to do is to use the
> > --tellmewhatswrong switch with systemctl and all will be revealed!" But
> > alas, no-one has said anything...
>
> Not quite as easy as that, but here's what you can do:
>
> Edit /etc/systemd/system.conf and set "DefaultStandardOutput" to one
> of "syslog", "kmsg", "kmsg+console", or "syslog+console".  That will
> redirect stdout from it's default destination of /dev/null to the
> place you set.  You may set "DefaultStandardError" too if you wish,
> but by default it goes to the same place as DefaultStandardOutput.
> (Note that you can also do this per-service by editing it's unit file
> [see "man systemd.exec" for details] or at boot-time via the kernel
> argument "systemd.default_standard_output".)
>
> Then, just run systemctl daemon-reexec (in the case of editing
> system.conf) to make your change take effect, start the service, and
> check the output where you told it to send it along to.

Wow! really useful advice! Thank You. 
As Hiisi says - Tip of the Day and I also agree that syslog+console
should be the default.

However, I just tried this by causing a deliberate error in a
Mod_Security configuration and attempting to restart HTTPD using
systemctl restart httpd.service.

I still only got:
"Job failed. See system logs and 'systemctl status' for details."
at the command line. However, at least now in /var/log/messages I do get
this:

Aug  3 14:06:53 mydomain httpd[1977]: /etc/rc.d/init.d/functions: line 58: /dev/stderr: No such device or address
Aug  3 14:06:57 mydomain httpd[1977]: Stopping httpd: #033[60G[#033[0;32m  OK  #033[0;39m]
Aug  3 14:06:57 mydomain httpd[1988]: /etc/rc.d/init.d/functions: line 58: /dev/stderr: No such device or address
Aug  3 14:06:59 mydomain httpd[1988]: Starting httpd: #033[60G[#033[0;32m  OK  #033[0;39m]
Aug  3 14:10:37 mydomain httpd[2035]: /etc/rc.d/init.d/functions: line 58: /dev/stderr: No such device or address
Aug  3 14:10:38 mydomain httpd[2035]: Stopping httpd: #033[60G[#033[0;32m  OK  #033[0;39m]
Aug  3 14:10:38 mydomain httpd[2043]: /etc/rc.d/init.d/functions: line 58: /dev/stderr: No such device or address
Aug  3 14:10:38 mydomain httpd[2043]: Starting httpd: Syntax error on line 91 of /etc/httpd/modsecurity.d/base_rules/modsecurity_crs_20_protocol_violations.conf:
Aug  3 14:10:38 mydomain httpd[2043]: Error creating rule: Unknown variable: REQBODY_ERROR
Aug  3 14:10:38 mydomain httpd[2043]: #033[60G[#033[0;31mFAILED#033[0;39m]
Aug  3 14:10:38 mydomain systemd[1]: httpd.service: control process exited, code=exited status=1
Aug  3 14:10:38 mydomain systemd[1]: Unit httpd.service entered failed state.

Which tells me exactly what I want to know.
Thank You.

I presume that the reason I got no output to the console was because of
the problem with /etc/rc.d/init.d/functions. Lines 56-62 say:

if [ -z "${CONSOLETYPE:-}" ]; then
  if [ -r "/dev/stderr" ]; then
    CONSOLETYPE="$(/sbin/consoletype < /dev/stderr)"
  else
    CONSOLETYPE="$(/sbin/consoletype)"
  fi
fi

How can I fix this?

(Although I am a much happier bunny already!)

Thanks again

Mark

[T.C. Hollingsworth]

On 8/3/11, Arthur Dent  wrote:
> Wow! really useful advice! Thank You.
> As Hiisi says - Tip of the Day and I also agree that syslog+console
> should be the default.

Unfortunately, systemd's parallelization of service startup make
output on the console extremely ugly during boot, because several
different commands would output to the screen at the same time and be
all mixed together.  There was some talk on systemd-devel of a
curses-based boot screen that neatly organized service startup status
and output but that seems rather pie-in-the-sky at the moment.

That argument is much weaker for syslog though, since it's easily filtered.
> However, I just tried this by causing a deliberate error in a
> Mod_Security configuration and attempting to restart HTTPD using
> systemctl restart httpd.service.
>
> I still only got:
> "Job failed. See system logs and 'systemctl status' for details."
> at the command line. However, at least now in /var/log/messages I do get
> this:
>
> Aug  3 14:06:53 mydomain httpd[1977]: /etc/rc.d/init.d/functions: line 58:
> /dev/stderr: No such device or address
> Aug  3 14:06:57 mydomain httpd[1977]: Stopping httpd: #033[60G[#033[0;32m
> OK  #033[0;39m]
> Aug  3 14:06:57 mydomain httpd[1988]: /etc/rc.d/init.d/functions: line 58:
> /dev/stderr: No such device or address
> Aug  3 14:06:59 mydomain httpd[1988]: Starting httpd: #033[60G[#033[0;32m
> OK  #033[0;39m]
> Aug  3 14:10:37 mydomain httpd[2035]: /etc/rc.d/init.d/functions: line 58:
> /dev/stderr: No such device or address
> Aug  3 14:10:38 mydomain httpd[2035]: Stopping httpd: #033[60G[#033[0;32m
> OK  #033[0;39m]
> Aug  3 14:10:38 mydomain httpd[2043]: /etc/rc.d/init.d/functions: line 58:
> /dev/stderr: No such device or address
> Aug  3 14:10:38 mydomain httpd[2043]: Starting httpd: Syntax error on line
> 91 of
> /etc/httpd/modsecurity.d/base_rules/modsecurity_crs_20_protocol_violations.conf:
> Aug  3 14:10:38 mydomain httpd[2043]: Error creating rule: Unknown variable:
> REQBODY_ERROR
> Aug  3 14:10:38 mydomain httpd[2043]: #033[60G[#033[0;31mFAILED#033[0;39m]
> Aug  3 14:10:38 mydomain systemd[1]: httpd.service: control process exited,
> code=exited status=1
> Aug  3 14:10:38 mydomain systemd[1]: Unit httpd.service entered failed
> state.
>
> Which tells me exactly what I want to know.
> Thank You.
>
> I presume that the reason I got no output to the console was because of
> the problem with /etc/rc.d/init.d/functions. Lines 56-62 say:
>
> if [ -z "${CONSOLETYPE:-}" ]; then
>   if [ -r "/dev/stderr" ]; then
>     CONSOLETYPE="$(/sbin/consoletype < /dev/stderr)"
>   else
>     CONSOLETYPE="$(/sbin/consoletype)"
>   fi
> fi
>
> How can I fix this?

I get the same error, but it appears to be harmless; console output
WORKSFORME.  Perhaps you're expecting output in xterm?  Console ==
/dev/console (which usually == tty1), not the tty you happen to be
running systemctl on.  You can redirect output to an arbitrary tty but
only on a per-unit basis.  (In fact, this discussion has put a crazy
notion in my head to use that functionality in a script that makes
debugging services easier.)
> (Although I am a much happier bunny already!)
>
> Thanks again
>
> Mark

[Arthur Dent]

On Wed, 2011-08-03 at 10:32 -0700, T.C. Hollingsworth wrote:
> I get the same error, but it appears to be harmless; console output
> WORKSFORME.  Perhaps you're expecting output in xterm?  Console ==
> /dev/console (which usually == tty1), not the tty you happen to be
> running systemctl on.

Ahh OK - I didn't understand that.
Unfortunately however I cannot test this at the moment. You see I run
that (F15) box headless and administer it by SSH'ing into it. I could
get physical access to the machine later, but then I'd also have to get
a monitor, mouse and keyboard for it (not impossible - but a hassle
nevertheless).
> You can redirect output to an arbitrary tty but
> only on a per-unit basis.  (In fact, this discussion has put a crazy
> notion in my head to use that functionality in a script that makes
> debugging services easier.)

That sounds interesting. Please keep me posted.

At least I am one step closer with syslog working, but it could
certainly be easier...

Thanks very much for your help so far. Much appreciated.

Best regards

Mark

[T.C. Hollingsworth]

On 8/3/11, Arthur Dent  wrote:
> On Wed, 2011-08-03 at 10:32 -0700, T.C. Hollingsworth wrote:
> > I get the same error, but it appears to be harmless; console output
> > WORKSFORME.  Perhaps you're expecting output in xterm?  Console ==
> > /dev/console (which usually == tty1), not the tty you happen to be
> > running systemctl on.
>
> Ahh OK - I didn't understand that.
> Unfortunately however I cannot test this at the moment. You see I run
> that (F15) box headless and administer it by SSH'ing into it. I could
> get physical access to the machine later, but then I'd also have to get
> a monitor, mouse and keyboard for it (not impossible - but a hassle
> nevertheless).

Googling around it would appear that it's possible to redirect
/dev/console to a pty, so you might try that.

systemd provides the "tty" option to DefaultStandardError but doesn't
seem to offer the configuration to actually set which tty it goes to.
I'll ping systemd-devel to see if that is a documentation or design
oversight, because that would help tremendously.
> > You can redirect output to an arbitrary tty but
> > only on a per-unit basis.  (In fact, this discussion has put a crazy
> > notion in my head to use that functionality in a script that makes
> > debugging services easier.)
>
> That sounds interesting. Please keep me posted.
>
> At least I am one step closer with syslog working, but it could
> certainly be easier...
>
> Thanks very much for your help so far. Much appreciated.
>
> Best regards
>
> Mark

[Will Yonker]

> On 8/3/11, Arthur Dent  wrote:
> > Wow! really useful advice! Thank You.
> > As Hiisi

says - Tip of the Day and I also agree that syslog+console
> >

should be the default.
> Unfortunately, systemd's

parallelization of service startup make
> output on the console

extremely ugly during boot, because several
> different commands

would output to the screen at the same time and be
> all mixed

together.

Isn't it possible for you to tell if you were
started from a console and output to it if so?  I seem to remember
writing a script one time that needed to do that (i.e. output information
if called from the command line but be silent when called from cron).