how to correctly log without journald (and its blobs)?

[Franta Hanzlík]

In previous Fedora distros was (IMHO) right way to log only to syslog,
without journald and its (for me) unwanted annoying binary logs, by using
configuration as:

*) Set systemd log target to syslog:
systemd.log_target=syslog (syslog-or-kmsg)  ---- on kernel cmdline
systemd.default_standard_output=syslog      --/
   and
LogTarget=syslog                            ---- in systemd.conf
DefaultStandardOutput=syslog                --/

*) configuring rsyslog to listen on /dev/log unix socket:
$SystemLogSocketName /dev/log                     ------ in rsyslog.conf
$ModLoad imuxsock                                 ----/
$OmitLocalLogging off                             ---/
$AddUnixListenSocket /run/systemd/journal/syslog  --/(legacy directives)
$AddUnixListenSocket /run/systemd/journal/socket  -/
$AddUnixListenSocket /run/systemd/journal/stdout  /
(not sure when all last three directives needed)

*) prevent to run journald:
systemctl mask systemd-journald.service
(and maybe mask systemd-journal-flush.service)

And it works fine.
But in actual Fedora 24 in systemd man page values syslog-or-kmsg and
syslog are missing in LogTarget option. As systemd/journald man pages
say hardly anything about exact mean of appropriate configuration for
this purpose, then please when someone more knowledgeable can advise:

- where to direct the systemd output? Maybe to kmsg an then read it in
rsyslog via imklog?
(rsyslog should have also imkmsg module, but is not in rsyslog-8.12.0-3.fc24)

- what about /run/log/journal/.../system.journal ?
On my test F24 system it have open rsyslogd, abrt-dump-journal-oops and
abrt-dump-journal-xorg (no need for these last two) processes. Uses it
also something else?

- what about /run/systemd/journal/{dev-log,socket,stdout} unix sockets?
Should rsyslogd listen on them?

- /dev/log seems be now symlink to /run/systemd/journal/dev-log socket
(as defined by systemd-journald-dev-log.socket) Should it be left
(because now something sends to /run/systemd/journal/dev-log), or can
be /run/systemd/journal/dev-log removed?

Thanks in advance for some clarification about this.
Franta Hanzlik

[Franta Hanzlík]

Nobody knows how to get rid of those unnecessary journald binary logs?

On Sun, 18 Sep 2016 13:01:49 +0200
Franta Hanzlík  wrote:
> In previous Fedora distros was (IMHO) right way to log only to syslog,
> without journald and its (for me) unwanted annoying binary logs, by using
> configuration as:
>
> *) Set systemd log target to syslog:
> systemd.log_target=syslog (syslog-or-kmsg)  ---- on kernel cmdline
> systemd.default_standard_output=syslog      --/
>    and
> LogTarget=syslog                            ---- in systemd.conf
> DefaultStandardOutput=syslog                --/
>
> *) configuring rsyslog to listen on /dev/log unix socket:
> $SystemLogSocketName /dev/log                     ------ in rsyslog.conf
> $ModLoad imuxsock                                 ----/
> $OmitLocalLogging off                             ---/
> $AddUnixListenSocket /run/systemd/journal/syslog  --/(legacy directives)
> $AddUnixListenSocket /run/systemd/journal/socket  -/
> $AddUnixListenSocket /run/systemd/journal/stdout  /
> (not sure when all last three directives needed)
>
> *) prevent to run journald:
> systemctl mask systemd-journald.service
> (and maybe mask systemd-journal-flush.service)
>
> And it works fine.
> But in actual Fedora 24 in systemd man page values syslog-or-kmsg and
> syslog are missing in LogTarget option. As systemd/journald man pages
> say hardly anything about exact mean of appropriate configuration for
> this purpose, then please when someone more knowledgeable can advise:
> where to direct the systemd output? Maybe to kmsg an then read it in
>
> rsyslog via imklog?
> (rsyslog should have also imkmsg module, but is not in rsyslog-8.12.0-3.fc24)
> what about /run/log/journal/.../system.journal ?
>
> On my test F24 system it have open rsyslogd, abrt-dump-journal-oops and
> abrt-dump-journal-xorg (no need for these last two) processes. Uses it
> also something else?
> what about /run/systemd/journal/{dev-log,socket,stdout} unix sockets?
>
> Should rsyslogd listen on them?
> /dev/log seems be now symlink to /run/systemd/journal/dev-log socket
>
> (as defined by systemd-journald-dev-log.socket) Should it be left
> (because now something sends to /run/systemd/journal/dev-log), or can
> be /run/systemd/journal/dev-log removed?
>
> Thanks in advance for some clarification about this.
> Franta Hanzlik

[Franta Hanzlík]

On Thu, 17 Nov 2016 15:50:24 +0100
Franta Hanzlík  wrote:
> Nobody knows how to get rid of those unnecessary journald binary logs?
>
> On Sun, 18 Sep 2016 13:01:49 +0200
> Franta Hanzlík  wrote:
> > In previous Fedora distros was (IMHO) right way to log only to syslog,
> > without journald and its (for me) unwanted annoying binary logs, by using
> > configuration as:
> >
> > *) Set systemd log target to syslog:
> > systemd.log_target=syslog (syslog-or-kmsg)  ---- on kernel cmdline
> > systemd.default_standard_output=syslog      --/
> >    and
> > LogTarget=syslog                            ---- in systemd.conf
> > DefaultStandardOutput=syslog                --/
> >
> > *) configuring rsyslog to listen on /dev/log unix socket:
> > $SystemLogSocketName /dev/log                     ------ in rsyslog.conf
> > $ModLoad imuxsock                                 ----/
> > $OmitLocalLogging off                             ---/
> > $AddUnixListenSocket /run/systemd/journal/syslog  --/(legacy directives)
> > $AddUnixListenSocket /run/systemd/journal/socket  -/
> > $AddUnixListenSocket /run/systemd/journal/stdout  /
> > (not sure when all last three directives needed)
> >
> > *) prevent to run journald:
> > systemctl mask systemd-journald.service
> > (and maybe mask systemd-journal-flush.service)
> >
> > And it works fine.
> > But in actual Fedora 24 in systemd man page values syslog-or-kmsg and
> > syslog are missing in LogTarget option. As systemd/journald man pages
> > say hardly anything about exact mean of appropriate configuration for
> > this purpose, then please when someone more knowledgeable can advise:
> > where to direct the systemd output? Maybe to kmsg an then read it in
> >
> > rsyslog via imklog?
> > (rsyslog should have also imkmsg module, but is not in rsyslog-8.12.0-3.fc24)
> > what about /run/log/journal/.../system.journal ?
> >
> > On my test F24 system it have open rsyslogd, abrt-dump-journal-oops and
> > abrt-dump-journal-xorg (no need for these last two) processes. Uses it
> > also something else?
> > what about /run/systemd/journal/{dev-log,socket,stdout} unix sockets?
> >
> > Should rsyslogd listen on them?
> > /dev/log seems be now symlink to /run/systemd/journal/dev-log socket
> >
> > (as defined by systemd-journald-dev-log.socket) Should it be left
> > (because now something sends to /run/systemd/journal/dev-log), or can
> > be /run/systemd/journal/dev-log removed?
> >
> > Thanks in advance for some clarification about this.
> > Franta Hanzlik

Eh, excuse for bad formulation. Binary log I can inhibit with specifiing
'Storage=none' in [Journal] section of /etc/systemd/journald.conf.

What I want is completely eliminate 'journald' program, of course (if I can
not get rid of the systemd :(

Thanks, Franta

[Tom H]

On Thu, Nov 17, 2016 at 5:03 PM, Franta Hanzlík  wrote:
> Eh, excuse for bad formulation. Binary log I can inhibit with
> specifiing 'Storage=none' in [Journal] section of
> /etc/systemd/journald.conf.
>
> What I want is completely eliminate 'journald' program,

That's not possible. You need journald as an rsyslog forwarder.

[Franta Hanzlík]

On Thu, 17 Nov 2016 19:43:11 +0200
Tom H  wrote:
> On Thu, Nov 17, 2016 at 5:03 PM, Franta Hanzlík  wrote:
> > Eh, excuse for bad formulation. Binary log I can inhibit with
> > specifiing 'Storage=none' in [Journal] section of
> > /etc/systemd/journald.conf.
> >
> > What I want is completely eliminate 'journald' program,
>
> That's not possible. You need journald as an rsyslog forwarder.

Tom thanks for reply. It seems to me, as there in conjunction with systemd
somehow often occurs "That's not possible".
But is not possible use for this (somehow) systemd option
"--log-target=kmsg" ?

And earlier, I think even in Fedora 21, it was possible to use the option
"--log-target=syslog" - I think, this option was served just for this
purpose. Where has she gone? Why now has been dropped?

Sorry for this questions, systemd documentation poorly (rather not at
all) describe this.

[Tom H]

On Thu, Nov 17, 2016, Franta Hanzlík  wrote:
> On Thu, 17 Nov 2016, Tom H  wrote:
> > On Thu, Nov 17, 2016, Franta Hanzlík  wrote:
> > > Eh, excuse for bad formulation. Binary log I can inhibit with
> > > specifiing 'Storage=none' in [Journal] section of
> > > /etc/systemd/journald.conf.
> > >
> > > What I want is completely eliminate 'journald' program,
> >
> > That's not possible. You need journald as an rsyslog forwarder.
>
> Tom thanks for reply.

You're welcome.
> It seems to me, as there in conjunction with systemd
> somehow often occurs "That's not possible".
> But is not possible use for this (somehow) systemd option
> "--log-target=kmsg" ?

I've just tried "systemd.log_target=kmsg" on the kernel cmdline with
rsyslog set up and it logged to "/var/log/" but "journalctl -k" has
some log lines that aren't there with the default "journal-or-kmsg"
setting. Perhaps "systemd.log_target=" sets the location that systemd
logs to _at_boot_ rather than a runtime target, especially since
journald has its own kernel cmdline settings that look like
"systemd.journald.forward_to_...=".
> And earlier, I think even in Fedora 21, it was possible to use the option
> "--log-target=syslog" - I think, this option was served just for this
> purpose. Where has she gone? Why now has been dropped?

The last version to have "--log-target=syslog" was 215. I suspect
that, like "--log-target=kmsg", it didn't mean that journald was
bypassed.
> Sorry for this questions, systemd documentation poorly (rather not at
> all) describe this.

systemd is, in general, well-documented but in this instance
"--log-target=" isn't explained clearly.

[Matthew Miller]

On Thu, Nov 17, 2016 at 04:03:52PM +0100, Franta Hanzlík wrote:
> What I want is completely eliminate 'journald' program, of course (if I can
> not get rid of the systemd :(

Are you trying to solve a particular problem other than distaste for
journald? If that's what it is, sure, go for it, but please do be aware
that you're straying quite far from what we actually work on testing.

The journal provides several very useful features. I find
'systemctl status sshd' (or service of your choice) to be really
useful. The filtered results are also really nice — much easier than
grepping because grep isn't field-aware. (Try journalctl -p err.)

So that said, if you really want a journald-free system based on
Fedora, you might try uselessd, a "minimized" fork of system.
http://uselessd.darknedgy.net/. We don't include it, but you might be
able to create a Fedora Remix which does. That, however, didn't last
very look due to lack of actual interest in development — so it's not
been updated in two years.

If that's not up your alley, you might try something like Linux From
Scratch to build your own system that works just like you like it — or
try Slackware. Or you might decide that it's not so horrible after all.

[Matthew Miller]

On Sun, Nov 20, 2016 at 07:12:29PM -0500, Matthew Miller wrote:
> The journal provides several very useful features. I find
> 'systemctl status sshd' (or service of your choice) to be really
> useful. The filtered results are also really nice — much easier than
> grepping because grep isn't field-aware. (Try journalctl -p err.)

Oh! And `journalctl -b` to get logs from this boot, and `journalctl
-b-1` to get logs from the previous, and so on.

[Tom H]

On Sun, Nov 20, 2016 at 7:12 PM, Matthew Miller
 wrote:
> So that said, if you really want a journald-free system based on
> Fedora, you might try uselessd, a "minimized" fork of system.
> http://uselessd.darknedgy.net/. We don't include it, but you might be
> able to create a Fedora Remix which does. That, however, didn't last
> very look due to lack of actual interest in development — so it's not
> been updated in two years.

Given that the first para ends with "uselessd is dead," avoiding it is
probably a better course of action.

nosh (I've never used it) is probably a better alternative albeit one
that'll require some getting used to judging from the documentation.
It has a systemctl shim, so at least that'll be familiar.

[Tom Horsley]

On Thu, 17 Nov 2016 15:50:24 +0100
Franta Hanzlík wrote:
> Nobody knows how to get rid of those unnecessary journald binary logs?

I can't get rid of journald, but I can reduce it to nothing more than
a proxy, relaying data to rsyslog, which then writes logs in readable text
as God intended them:

1. Make sure you install rsyslog if it isn't already.

2. Edit /etc/systemd/journald.conf and set these parameters:

Storage=none
ForwardToSyslog=yes

3. Edit /etc/rsyslog.conf and make sure
$ModLoad imuxsock
is *not* commented out.

4. Generate the /etc/rsyslog.d/listen.conf file with
$SystemLogSocketName /run/systemd/journal/syslog

5. Generate the /etc/rsyslog.d/sd-socket.conf file with
$AddUnixListenSocket /run/systemd/journal/syslog

Might as well reboot at this point. You get no more binary
logs, and everything gets forwarded to syslog.

Aside from the added bonus of readable logs, you no longer
get infinite boot loops caused by corrupt binary logs :-).

[Tom H]

On Thu, Nov 17, 2016 at 5:04 PM, Tom Horsley  wrote:
> Make sure you install rsyslog if it isn't already.
>
> Edit /etc/systemd/journald.conf and set these parameters:
>
> Storage=none
> ForwardToSyslog=yes
> Edit /etc/rsyslog.conf and make sure
>
> $ModLoad imuxsock
> is *not* commented out.
> Generate the /etc/rsyslog.d/listen.conf file with
>
> $SystemLogSocketName /run/systemd/journal/syslog
> Generate the /etc/rsyslog.d/sd-socket.conf file with
>
> $AddUnixListenSocket /run/systemd/journal/syslog

You don't need (5).

PS: Using non-legacy directives,

module(load="imuxsock" SysSock.Name="/run/systemd/journal/syslog")

is equivalent to

$ModLoad imuxsock
$SystemLogSocketName /run/systemd/journal/syslog

[Tom Horsley]

On Thu, 17 Nov 2016 19:53:04 +0200
Tom H wrote:
> PS: Using non-legacy directives,
>
> module(load="imuxsock" SysSock.Name="/run/systemd/journal/syslog")

You must be a rsyslog developer, because the last time I tried,
I couldn't find the slightest particle of human readable
documentation explaining how to use the wonderful new and
improved config format :-).

[Tom H]

On Thu, Nov 17, 2016 at 8:27 PM, Tom Horsley  wrote:
> On Thu, 17 Nov 2016 19:53:04 +0200 Tom H wrote:
> > PS: Using non-legacy directives,
> >
> > module(load="imuxsock" SysSock.Name="/run/systemd/journal/syslog")
>
> You must be a rsyslog developer

Definitely not, LOL
> because the last time I tried, I couldn't find the slightest particle
> of human readable documentation explaining how to use the wonderful
> new and improved config format :-).

It took me a while to figure it out but since there's a request within
the doc pages to use the new syntax, I'm trying...

[Christian Kujau]

I was able to disable journald similar to this:

systemctl stop systemd-journald-dev-log.socket systemd-journald.socket systemd-journald.service
systemctl mask systemd-journald-dev-log.socket systemd-journald.socket systemd-journald.service

(I tried to "disable" those services first, but somehow journald got started anyway on the next reboot).

Configure rsyslog with:

$AddUnixListenSocket                /var/run/systemd/journal/syslog
$SystemLogSocketName            /var/run/systemd/journal/syslog

Or with the new configuration format[0]:

module(load="imuxsock" SysSock.Name="/run/systemd/journal/syslog" Socket="/var/run/systemd/journal/syslog")

I also had to adjust the /dev/log symlink to point to /var/run/systemd/journal/syslog, so that programs like logger(1) will still work. Added the following to /etc/rc.local (or wherevery custom startup routines were configured):

ln -svf /var/run/systemd/journal/syslog /dev/log | logger -t "$0"

With all that journald is no longer running and rsyslog still works! :-)

[0] http://www.rsyslog.com/doc/v8-stable/configuration/modules/imuxsock.html

[Tom H]

On Sun, Apr 8, 2018 at 9:35 PM, Christian Kujau  wrote:
> I was able to disable journald similar to this:
>
> systemctl stop systemd-journald-dev-log.socket systemd-journald.socket systemd-journald.service
> systemctl mask systemd-journald-dev-log.socket systemd-journald.socket systemd-journald.service
>
> (I tried to "disable" those services first, but somehow journald got started anyway on the next reboot).
>
> Configure rsyslog with:
>
> $AddUnixListenSocket /var/run/systemd/journal/syslog
> $SystemLogSocketName /var/run/systemd/journal/syslog
>
> Or with the new configuration format[0]:
>
> module(load="imuxsock" SysSock.Name="/run/systemd/journal/syslog" Socket="/var/run/systemd/journal/syslog")
>
> I also had to adjust the /dev/log symlink to point to /var/run/systemd/journal/syslog, so that programs like logger(1) will still work. Added the following to /etc/rc.local (or wherevery custom startup routines were configured):
>
> ln -svf /var/run/systemd/journal/syslog /dev/log | logger -t "$0"

If you disable journald, this is a minimal configuration file that'll
use the default "/dev/log" and "proc/kmsg":

th@localhost ~ $ cat /etc/rsyslog.conf
module(load="imuxsock")
module(load="imklog")
global(workDirectory="/var/lib/rsyslog")
*.* action(type="omfile" file="/var/log/messages")
th@localhost ~ $

[ If you don't disable journald in dracut, you'll still have logs (for
the initramfs) output via "journalctl" reading "/run/log/journal/*" ]