Verbose output from Daemons on startup

[T.C. Hollingsworth]

On Sun, Oct 30, 2011 at 8:43 PM, Nathan D'elboux <nafe27 at hotmail.com> wrote:
> Hi All,
> I have built a new Linux box recently, have been using Fedora for years so
> decided to stick with it and put Fedora 15 on it
> I am familiar with strace and redirecting STOUT and STDERR for debugging
> apps on the CLI but what i am unfamiliar with is on bootup
> When the list of daemons begin to start up and eventually gets the [OK]
> message i have 1 daemon that will hang
> So my question is, is there some way whether it be by Console port or rescue
> disk can i get a verbose output of what the daemon is doing
> when attempting to boot up so i can verify what part exactly it is failing
> at?
> I want to possibly pipe all STOUT to another box via console or just on
> local screen but i want a verbose output so i can see more detail of what
> the kernel is doing on bootup behind the scenes

Usually its easier to debug daemon problems by just running them on
the command line.  Look for the ExecStart line in
/lib/systemd/system/<name>.service or if it uses classic initscripts
check the corresponding file in /etc/init.d for the command that is
run.  That way you can easily run it through strace if you need to.

That being said, on Fedora 16, systemd redirects stdout and stderr to
syslog, so you can view the output from the daemons in
/var/log/messages.  Unfortunately, IIRCC F15 systemd does not do this,
so edit /etc/systemd/system.conf and add a line that says
"DefaultStandardOutput=syslog".  The next time you boot, all daemons'
stdout and stderr will be stored in syslog so you can view it
afterward, either in /var/log/messages or with a viewer like
KSystemLog.  Messages from the kernel are also stored there, so you
can see everything in one place.

You can also set DefaultStandardOutput to "syslog+console" to have it
output to the console as well, but that can be problematic because
systemd starts many services in parallel.  (If you decide to do that,
make sure to remove "rhgb quiet" from your kernel command line as well
so Plymouth gets out of the way and kernel messages are printed to the
console also.)

It's also rather easy to run systemd native services through strace,
just run copy the .service file from /lib/systemd/system to
/etc/systemd/system and append strace the command on the ExecStart
line.  When you're done, just rm the file in /etc and the one in /lib
will take back over.  Classic initscripts are a little harder:  you
have to find the appropriate line in the script and change it, and
then change it back when you're done.

> I tried explaining it as well as i could so i apologize if some people get
> confused. I can post any other details needed if need be like Specs etc
> Thanks guys
> Cheers,
> Nathan

-T.C.